slow dhcp on wan -> broken NAT

[synfinatic]

This was happening on 26.1.x and continues on 26.7.1.   I'm on AT&T fiber and using an AZORES WAS-110 fiber module/ONT to bypass the AT&T provided gateway.  For this to work, my `ixl0` interface uses DHCP/DHCPv6 to get IP addresses and I create a virtual IP (192.168.11.10/24) on `ixl0` in order to manage the WAS-110 module.

The problem is that for whatever reason, getting an IP address from AT&T is _slow_.  After rebooting the firewall, I often have to login and manually run `dhclient ixl0` to force it to try again and that works.  Once an IPv4 address is obtained, DHCPv6 seems to happen automatically.

The problem is that the system seems to not really register that an IPv4 address has been obtained and so NAT is broken- running `tcpdump -ni ixl0` shows traffic egressing with a source IP of 192.168.11.10- not the public IP that AT&T has given via DHCP.  I've figured out that I can fix this by going into the interface config setting and re-applying the existing settings, but this sucks.

Seems like the root cause of the issue is AT&T being slow to give out an IP address over DHCP- any way to make things more robust?

[synfinatic]

Meant to mention that for outbound NAT I'm using "Manual outbound NAT rule generation" and selecting NAT Address: "interface address"

[drosophila]

You are perfectly right: NAT uses the first IP address it finds, which will usually be the first the interface gets assigned. So if you assign the 192.168.11.10 as "virtual" but the primary assignment fails on first try, then, even if DHCP eventually succeeds, 192.168.11.10 will be first in the list and NAT will use that (check ifconfig output for the order, the GUI overview may lie about it). It can't know whether you want to NAT inter-LAN or towards the internet, after all, and there is no way to tell it.
I've had a similar issue and could fix it only by running a cron job that reorders the addresses every minute (the smallest interval it allows to use). Sucks big time.
In your case, it should be possible to hook into the "newwanip" trigger, since it gets a new WAN IP, which this trigger is all about. You'd still need to have a script for it and manually configure the trigger from the command line. Either way, it'll break on a fresh install because none of this gets stored in the config. :(

[synfinatic]

So looking again at the NAT settings, seems like there are 3 options in the drop down for `ixl0`:

* interface address
* ONTWAN address -- ONTWAN is the name for ixl0
* 192.168.11.10 ONT Management -- the virtual IP

Seems like using the 2nd option should provide the correct results?   Or are the 1st and 2nd effectively the same option with different names?

[lmoore]

Another thought you could try.

- Assuming you have configured a static IP address on your ONT, add a static route on your ONT to reach your management network (LAN) via 192.168.11.10 (optional)

- Add a Bridge interface and include WAN as a member

- Assign the bridge interface and enter ONT in the description field

- Refresh the web page

- Select [ONT] interface and select Enable Interface

- Set IPv4 configuration to Static IPv4

- Set IPv4 address to 192.168.11.10/24

- Save the settings

- Apply the changes

- Select your [WAN] interface

- Check Block private networks

- Save the settings

- Apply the changes

- Review your firewall rules allowing connections to the 192.168.11.0/24 network

- On ONT, if no static route to your management network (LAN) - see above, create a Source NAT rule to the ONT network using the bridge address

[lmoore]

Firewall and Source NAT rules attached - note the use of tags.

[meyergru]

@lmoore: That will not cut it. The problem is not that the modem cannot be reached.

The problem is that the normal WAN NAT rule does not work, because the translated NAT IP is the virtual IP and not the real WAN IP.

This is, because when the DHCP client on WAN times out on the first try, the virtual IP gets to take the "primary" address slot on the WAN interface. When DHCP later succeeds, this will not change, such that the virtual IP will still be used for normal unbound NAT (not for the NAT rule that you need to reach the modem).

This problem does not manifest when the WAN is on another interface than the modem, like with PPPoE or with a VLAN or when DHCP is fast enough to succeed before the virtual IP gets initialized.

A fix for this is probably non-trivial, because you would have to delete and re-create the virtual IP when the DHCP finally succeeds and that could break other things.

The easy way out would be to skip modem access completely under these conditions. Another way would be if the modem could be configured such that the GUI can be configured on another VLAN than the internet connection. If the ISP wants a VLAN, you should tag it on OpnSense if you can and not leave that to the modem. There seems to be a community firmware for that XGS-PON:

https://pon.wiki/guides/masquerade-as-the-orange-sa-livebox-7-with-the-was-110/#from-the-web-ui

And you probably used the "fix-vlans" setting from here:

https://pon.wiki/guides/masquerade-as-the-att-inc-bgw320-500-505-with-the-was-110/#configure-ont-settings

I think (aka "IDK") that AT&T needs a VLAN. As I said, when you let OpnSense do the tagging, it could work, because you would use different interfaces for ONT and WAN, then.

[Patrick M. Hausen]

If the modem has two ethernet ports you could use two interfaces on OPNsense (or VLANs distributed via a switch) to connect to

- the modem network
- the proper uplink

Just a thought. I do this. I do not piggyback the modem network on top of the PPPoE interface. I use two completely separated ones. I think this could be even more important for DHCP instead of PPPoE.

[lmoore]

That will not cut it. The problem is not that the modem cannot be reached.

It's worked for me for well over 10 years, originally on OpenBSD and using a vether interface for the DSL modem network. The vether interface was included as another member in the bridge.

Way back, I experimented with using a virtual IP to access the modem. I also set up DHClient to add the additional static IP to access the modem. The problem with this was that when the DHCP lease expired, due to a connectivity issue with the ISP, the modem was no longer reachable. This is when I reconsidered how to do it and the change in my configuration to use a bridge, with re0 being the WAN port.

I only have an IPv4 environment so I can't speak to IPv6 behaviour with this configuration.

In my environment I've disabled automatic NAT rules in OPNsense, electing to set up NAT rules when necessary. When my ISP has encounters a problem and my DHCP lease expires, I've never encountered an issue with NAT once the WAN interface eventually receives its DHCP lease, some times hours later.

My DSL modem has a static IP address and is configured with a static route to the management network, hence I do not use NAT to access the DSL modem.

If we consider for a moment what I posted, two interfaces are being used.

The first is the WAN interface. Using the interface of the OP, i.e. ixl0 in place of re0 (in my examples), DHCP should operate merrily on this interface.

The second interface which has the IPv4 address of 192.168.11.10 assigned to it is the bridge interface - named ONT in my example.

Based upon the information provided, I don't expect the address assigned to the bridge to be picked up and used for NAT on ixl0 in the OP's environment, just as it doesn't in my environment.

My production firewall is still on 26.1 and won't be upgraded until I've tested the upgrade using my test machine. However, I'm not expecting any problems with NAT to the Internet nor accessing my DSL modem after the upgrade.