This was happening on 26.1.x and continues on 26.7.1. I'm on AT&T fiber and using an AZORES WAS-110 fiber module/ONT to bypass the AT&T provided gateway. For this to work, my `ixl0` interface uses DHCP/DHCPv6 to get IP addresses and I create a virtual IP (192.168.11.10/24) on `ixl0` in order to manage the WAS-110 module.
The problem is that for whatever reason, getting an IP address from AT&T is _slow_. After rebooting the firewall, I often have to login and manually run `dhclient ixl0` to force it to try again and that works. Once an IPv4 address is obtained, DHCPv6 seems to happen automatically.
The problem is that the system seems to not really register that an IPv4 address has been obtained and so NAT is broken- running `tcpdump -ni ixl0` shows traffic egressing with a source IP of 192.168.11.10- not the public IP that AT&T has given via DHCP. I've figured out that I can fix this by going into the interface config setting and re-applying the existing settings, but this sucks.
Seems like the root cause of the issue is AT&T being slow to give out an IP address over DHCP- any way to make things more robust?
Meant to mention that for outbound NAT I'm using "Manual outbound NAT rule generation" and selecting NAT Address: "interface address"
You are perfectly right: NAT uses the first IP address it finds, which will usually be the first the interface gets assigned. So if you assign the 192.168.11.10 as "virtual" but the primary assignment fails on first try, then, even if DHCP eventually succeeds, 192.168.11.10 will be first in the list and NAT will use that (check ifconfig output for the order, the GUI overview may lie about it). It can't know whether you want to NAT inter-LAN or towards the internet, after all, and there is no way to tell it.
I've had a similar issue and could fix it only by running a cron job that reorders the addresses every minute (the smallest interval it allows to use). Sucks big time.
In your case, it should be possible to hook into the "newwanip" trigger, since it gets a new WAN IP, which this trigger is all about. You'd still need to have a script for it and manually configure the trigger from the command line. Either way, it'll break on a fresh install because none of this gets stored in the config. :(
So looking again at the NAT settings, seems like there are 3 options in the drop down for `ixl0`:
* interface address
* ONTWAN address -- ONTWAN is the name for ixl0
* 192.168.11.10 ONT Management -- the virtual IP
Seems like using the 2nd option should provide the correct results? Or are the 1st and 2nd effectively the same option with different names?
Another thought you could try.
- Assuming you have configured a static IP address on your ONT, add a static route on your ONT to reach your management network (LAN) via 192.168.11.10 (optional)
- Add a Bridge interface and include WAN as a member
- Assign the bridge interface and enter ONT in the description field
- Refresh the web page
- Select [ONT] interface and select Enable Interface
- Set IPv4 configuration to Static IPv4
- Set IPv4 address to 192.168.11.10/24
- Save the settings
- Apply the changes
- Select your [WAN] interface
- Check Block private networks
- Save the settings
- Apply the changes
- Review your firewall rules allowing connections to the 192.168.11.0/24 network
- On ONT, if no static route to your management network (LAN) - see above, create a Source NAT rule to the ONT network using the bridge address
Firewall and Source NAT rules attached - note the use of tags.
@lmoore: That will not cut it. The problem is not that the modem cannot be reached.
The problem is that the normal WAN NAT rule does not work, because the translated NAT IP is the virtual IP and not the real WAN IP.
This is, because when the DHCP client on WAN times out on the first try, the virtual IP gets to take the "primary" address slot on the WAN interface. When DHCP later succeeds, this will not change, such that the virtual IP will still be used for normal unbound NAT (not for the NAT rule that you need to reach the modem).
This problem does not manifest when the WAN is on another interface than the modem, like with PPPoE or with a VLAN or when DHCP is fast enough to succeed before the virtual IP gets initialized.
A fix for this is probably non-trivial, because you would have to delete and re-create the virtual IP when the DHCP finally succeeds and that could break other things.
The easy way out would be to skip modem access completely under these conditions. Another way would be if the modem could be configured such that the GUI can be configured on another VLAN than the internet connection. If the ISP wants a VLAN, you should tag it on OpnSense if you can and not leave that to the modem. There seems to be a community firmware for that XGS-PON:
https://pon.wiki/guides/masquerade-as-the-orange-sa-livebox-7-with-the-was-110/#from-the-web-ui
And you probably used the "fix-vlans" setting from here:
https://pon.wiki/guides/masquerade-as-the-att-inc-bgw320-500-505-with-the-was-110/#configure-ont-settings
I think (aka "IDK") that AT&T needs a VLAN. As I said, when you let OpnSense do the tagging, it could work, because you would use different interfaces for ONT and WAN, then.
If the modem has two ethernet ports you could use two interfaces on OPNsense (or VLANs distributed via a switch) to connect to
- the modem network
- the proper uplink
Just a thought. I do this. I do not piggyback the modem network on top of the PPPoE interface. I use two completely separated ones. I think this could be even more important for DHCP instead of PPPoE.
Quote from: meyergru on July 26, 2026, 09:20:10 PMThat will not cut it. The problem is not that the modem cannot be reached.
It's worked for me for well over 10 years, originally on OpenBSD and using a vether interface for the DSL modem network. The vether interface was included as another member in the bridge.
Way back, I experimented with using a virtual IP to access the modem. I also set up DHClient to add the additional static IP to access the modem. The problem with this was that when the DHCP lease expired, due to a connectivity issue with the ISP, the modem was no longer reachable. This is when I reconsidered how to do it and the change in my configuration to use a bridge, with re0 being the WAN port.
I only have an IPv4 environment so I can't speak to IPv6 behaviour with this configuration.
In my environment I've disabled automatic NAT rules in OPNsense, electing to set up NAT rules when necessary. When my ISP has encounters a problem and my DHCP lease expires, I've never encountered an issue with NAT once the WAN interface eventually receives its DHCP lease, some times hours later.
My DSL modem has a static IP address and is configured with a static route to the management network, hence I do not use NAT to access the DSL modem.
If we consider for a moment what I posted, two interfaces are being used.
The first is the WAN interface. Using the interface of the OP, i.e. ixl0 in place of re0 (in my examples), DHCP should operate merrily on this interface.
The second interface which has the IPv4 address of 192.168.11.10 assigned to it is the bridge interface - named ONT in my example.
Based upon the information provided, I don't expect the address assigned to the bridge to be picked up and used for NAT on ixl0 in the OP's environment, just as it doesn't in my environment.
My production firewall is still on 26.1 and won't be upgraded until I've tested the upgrade using my test machine. However, I'm not expecting any problems with NAT to the Internet nor accessing my DSL modem after the upgrade.
Quote from: Patrick M. Hausen on July 26, 2026, 09:27:13 PMIf the modem has two ethernet ports you could use two interfaces on OPNsense (or VLANs distributed via a switch) to connect to
- the modem network
- the proper uplink
Just a thought. I do this. I do not piggyback the modem network on top of the PPPoE interface. I use two completely separated ones. I think this could be even more important for DHCP instead of PPPoE.
This is what I found for the Azores WAS-110 (https://www.azoresnetworks.com/product/pon-cpe-65.html).
A modem with multiple Ethernet ports is ideal, in reality, many I have used only have a single port.
In my case, my modem is on a PCIe card and is installed in my firewall.
*deleted*
@lmoore: Now I understand. Interesting that you can use a bridge on an otherwise configure port... the way you do it you create two interfaces, which does the trick. I stand corrected, well done.
Actually, although this is kind of complicated, it should be added to the thread where ONT/modem access for the non-VLAN DHCP case is discussed, just because the "usual" VIP way is risky in case an initial DHCP takes too long. I cannot seem to find it just right now.
@meyergru It looks to me similar to creating a lagg with only one member to "abstract away" different interface names in a HA setup with different hardware.
@lmoore Clever one!
@meyergru & @patrick - Thank you!
I would think of the bridge more as a simple switch interface in this set up.
It would be nice if the vether-kmod could be fixed to work with FreeBSD 15 & 16 as it would make for a cleaner configuration.
FWIW: I tried this approach instead of my usual (working) virtual IP setup.
However, it did not work first - I could not even ping the ONT from OpnSense itself. Only after I disabled my null route for 192.168.0.0/16 did it work - and then, to my complete disbelief, it still worked when I re-enabled the null route!
There is something fishy going on. I have read several reports of incidents where only a reboot made certain things work after settings were changed with 26.7.x.
Try net.link.bridge.inherit_mac=1
Guess what? The ping stopped working again. It could well be something off with the MAC, because I have to set the MAC on the WAN interface manually and the bridge MAC differs.
Setting net.link.bridge.inherit_mac=1 does not change the bridge MAC immediately, BTW.
P.S.: Now it starts working again out of thin air, still with a different MAC - strange.
net.link.bridge.inherit_mac=1 makes the bridge inherit the MAC address of the first member interface the moment it is added.
Nope. Not when the MAC on the WAN interface is manually set. Yet I fear that when I manually set the same MAC on the ONT interface, it might break my internet connection.
I hope I have found it: net.link.bridge.pfil_member=0 could do the trick.
I have not set any System Tunables on the test machine..
Using the configurations I provided earlier and only changing the IPv4 address on the ONT interface, I have connected an old DSL modem to the WAN port.
Next, I connected my laptop to the LAN and manually configured the network settings for the 10.200.128.0/24 network.
Confirmed I can ping the DSL modem from OPNsense.
Using my laptop I can ping the DSL modem as well as accessing its web management interface.
This basic DSL modem is configured in Modem (bridge) mode, thus static routes cannot be configured on it so the NAT rule is required.
Black-hole routes are configured in OPNsense.
I don't have a spare DSL service so I can't bring up an address on re0 via DHCP.
QuoteRouting tables
Internet:
Destination Gateway Flags Netif Expire
default 10.200.128.1 UGS em0
10.0.0.0/8 link#3 USB lo0
10.200.128.0/24 link#2 U em0
10.200.128.2 link#3 UHS lo0
127.0.0.1 link#3 UH lo0
172.16.0.0/12 link#3 USB lo0
192.168.0.0/16 link#3 USB lo0
192.168.5.0/24 link#7 U bridge0
192.168.5.12 link#3 UHS lo0
QuoteONT interface:
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: ONT (opt1)
options=10<VLAN_HWTAGGING>
ether 58:9c:fc:10:94:50
inet 192.168.5.12 netmask 0xffffff00 broadcast 192.168.5.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
bridge flags=0<>
member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 1 priority 128 path cost 55 vlan protocol 802.1q
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
QuoteWAN interface:
re0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=2038<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,WOL_MAGIC>
ether f4:f2:6d:05:4d:60
inet6 fe80::f6f2:6dff:fe05:4d60%re0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
QuotePacket capture on bridge0:
18:57:48.023370 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:57:59.027252 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:06.132540 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:19.492006 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:39.529253 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:40.574491 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:41.636462 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:41.646456 IP6 fe80::f6f2:6dff:fe05:4d60.546 > ff02::1:2.547: dhcp6 solicit
18:58:42.335115 IP 192.168.5.12 > 192.168.5.1: ICMP echo request, id 43521, seq 26, length 40
18:58:42.335503 IP 192.168.5.1 > 192.168.5.12: ICMP echo reply, id 43521, seq 26, length 40
18:58:43.353259 IP 192.168.5.12 > 192.168.5.1: ICMP echo request, id 43521, seq 27, length 40
18:58:43.353541 IP 192.168.5.1 > 192.168.5.12: ICMP echo reply, id 43521, seq 27, length 40
18:58:43.658233 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:44.360620 IP 192.168.5.12 > 192.168.5.1: ICMP echo request, id 43521, seq 28, length 40
18:58:44.360906 IP 192.168.5.1 > 192.168.5.12: ICMP echo reply, id 43521, seq 28, length 40
18:58:45.375716 IP 192.168.5.12 > 192.168.5.1: ICMP echo request, id 43521, seq 29, length 40
18:58:45.376006 IP 192.168.5.1 > 192.168.5.12: ICMP echo reply, id 43521, seq 29, length 40
18:58:46.720254 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from f4:f2:6d:05:4d:60, length 300
18:58:47.345014 ARP, Request who-has 192.168.5.12 tell 192.168.5.1, length 46
18:58:47.345024 ARP, Reply 192.168.5.12 is-at 58:9c:fc:10:94:50, length 28
I do have a running ISP connection on WAN and I have a MAC set on WAN that is different from the bridge MAC. My symptoms are:
- After a while of inactivity (300 seconds will be enough), a ping does not succeed any more, despite an ARP entry being present and non-expired.
- "arp -d <IP-OF-ONT>" makes the ping work again immediately.
- A continuous ping never stops working.
I have fiddled around with various settings on the bridge and even disabled the igc hardware mac filtering via promiscuous mode, all to no avail.
Are you pinging your ONT from OPNsense or from a network behind it?
Do you have the NAT rule configured in OPNsense for the ONT network?
It does not matter, because OpnSense cannot reach the IP.
Wow. It seems I have found the problem. The timeout, as it turned out, was exactly 300 seconds. What happens is:
The ARP table entry on OpnSense is valid for 1200 seconds first. My ONT seems to have a timeout of 300 seconds. After 301s, OpnSense still thinks it knows the MAC of the ONT and sends the packet - which is received by the ONT.
It seems my ONT has:
a. No "passive ARP learning".
b. An ARP timeout of only 300 seconds.
Thus, there is no valid ARP entry for answering the request. Even if the ONT does ARP then (of which I am not sure), it will be sent to the WAN interface, not the ONT one. That seems to be the case for some chipsets, so it is probably dependent on the ONT brand. It could probably be circumvented if an intermediate switch was used, but I am not sure.
After reaching the ARP timeout of OpnSense or when the ARP entry is manually removed, it forces an ARP broadcast, which immediately heals the condition.
What did help is setting net.link.ether.inet.max_age=300 on OpnSense.
So, it seems this only happens with certain ONTs that have a short ARP timeout and no passive ARP learning and with this specific setup where there are ambiguities of the two network interfaces on the same physical layer. At least this did never happen with virtual IPs.
There are a number of articles relating to net.link.ether.inet.max_age and reducing its value, in particular, to overcome slowness issues.
There is also a patch submission for FreeBSD 16 for the default timeout value to be changed - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=291100
@meyergru - Is your ONT configured in bridge or router mode?
Bridge mode of course. I do not like double NAT. It is a Leox LXT-010H-D, BTW.
I have verified the gc time for the routes on that system to be 300s:
# cat /proc/sys/net/ipv4/route/gc_timeout
300
This would not be a problem in itself, it is just that with this configuration, an ARP request from the ONT will not work.
Thus, reducing the ARP timeout on OpnSense to 300s or below will do the trick.
Quote from: meyergru on July 28, 2026, 10:17:54 AMBridge mode of course
Just as I thought.
Thanks for the info re your ONT.
I would expect your Internet traffic will be communicating using the MAC addresses of your WAN port and your ISP's gateway, and the MAC on the bridge will only be seen when communicating with the ONT network.
It isn't clear in my mind why your ONT behaviour occurs when a bridge is used and is for traffic within its network, as opposed to a VIP. Maybe its simply "It is what it is".
On my DSL modem, I have it configured to synchronise its time from the firewall as well as dispatching events to syslog on an internal server, so there is constant activity from the modem to the firewall.
Quote from: meyergru on July 26, 2026, 09:20:10 PMA fix for this is probably non-trivial, because you would have to delete and re-create the virtual IP when the DHCP finally succeeds and that could break other things.
Yes, it would very likely break the gateway monitor, which is very touchy for some reason. My address-reordering script breaks it, anyway.
Quote from: lmoore on July 26, 2026, 08:46:05 PM- Add a Bridge interface and include WAN as a member
So I could configure the WAN IF as static IP and then have the bridged IF get a SLAAC address, apparently without it getting its own LLA? Does bridge mode create overhead? I've read that it puts its members into promiscuous mode, which I'd rather avoid as it might trigger driver bugs.
Quote from: drosophila on July 29, 2026, 02:50:16 AMSo I could configure the WAN IF as static IP and then have the bridged IF get a SLAAC address
In message #8, I mentioned I only have an IPv4 environment, someone with an IPv6 environment may be able to answer this question.
The WAN IF would be configured as per how you would normally configure it, then add it as a member of the bridge. The bridge is simply allowing another interface to exist for the ONT management network to be attached. In an earlier post I mentioned it is like a "simple switch", more correctly, it's like a hub as all traffic will be seen on the interfaces but with the added benefit that firewall rules can be applied to the interfaces.
Ideally, I would use a virtual Ethernet interface for the ONT network and add it as member of the bridge. The bridge would then remain unnumbered. However, as vether-kmod is not available in OPNsense, nor FreeBSD 15 & 16, putting the IP address on the bridge was my only choice.
Quote from: drosophila on July 29, 2026, 02:50:16 AMDoes bridge mode create overhead?
There will be an overhead, however, with my slow DSL service there is no observed performance issue and buffer-bloat is handled well by the traffic shaper.
Perhaps someone with a high-speed internet service can test and compare performance when a bridge is used and report back.
Quote from: drosophila on July 29, 2026, 02:50:16 AMI've read that it puts its members into promiscuous mode
Message #19 shows the interface details. Yes, the bridge members are put in to promiscuous mode - see ifconfig(8).
In my WAN rules, my last inbound rule will capture any packet arriving which isn't handled by another rule - I've not seen anything untoward from this rule, just the usual probes.
attacking the slow dhcp
-replace the cable
-is the AutoNeg not working properly
-some kind of incompatibility on the network interface side?
Thanks for the clarifications! Initially I got my hopes up that I might be able to use this to get rid of the autogenerated LLA and also get an interface that exclusively receives Privacy Extension addresses. However, at least the latter will probably not work since PE depends on SLAAC and that will automatically give me ULA/GUA addresses on the respective interface that I'd then have to manually separate regardless. Thus I'd still need a script and then it wouldn't change the general situation. :)
Regarding promiscuous mode it would be odd to have attacks through this, since the upstream node will certainly use a switch, so an attacker would have to know your MAC, and even if they do, the likelihood of someone using promiscuous mode without having set up a tight firewall is too low to bother for any common criminal. High-profile criminals like NSA or similar insecurity agencies would be the only ones with sufficient budget and determination if you ended up on their hit-list for criticizing their respective "beloved leader" or say something they don't like.
I just upgraded to the latest and other than system not rebooting automatically after the upgrade, at least this time NAT worked correctly after it got a DHCPv4 lease and IPv6 egress worked correctly once I got a my delegation via DHCPv6.
I'm pretty sure the trick is to go: Firewall -> NAT -> Outbound
And then for every source network, change the NAT Address from `Interface address` to the actual interface name (in my case `ONTWAN address`) listed in Interfaces that you have configured to get an address via DHCPv4.