It seems I can access the admin GUI from my WAN address by default. I would rather not be able to do this. How do I disable this access?
You are probably talking about having it reachable from the LAN through the WAN IP. Try accessing from the WAN port: it does not work. Not even ICMP ping replies by default from there.
Outside access is disabled by default. I had to create a specific rule to allow access to it from the outside world.
If you really want to you can try to create a block rule on the LAN interface to the WAN IP address. I am not sure what would happen to traffic at that point, however, as it is a very odd configuration.