WAN interface flapping with 22.1.2

[foxmanb]

Hi All,
New here, and not a expert in linux/bsd etc, so please, go easy on me.. lol.

I upgraded to 22.1.2 on release day, and when my router came back, up, the WAN interface would pick up a DHCP address, hold it for about 10 seconds, then drop, renew, hold it for 10 seconds, then drop... rinse, wash, repeat. In those 10 seconds I was able to use the opnsense-revert command to roll back to 22.1.1. The interface is stable under the old version. I am running a custom built box, intel 9500T, Intel x550-t2 NIC. "Gateway" is a Motorola MB8611 cable modem which connects at 2.5G.

This is what I see in the logs over and over again with the new 22.1.2 version.

2022-03-03T07:03:46-05:00   Critical   dhclient   exiting.   
2022-03-03T07:03:46-05:00   Error   dhclient   connection closed   
2022-03-03T07:03:46-05:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic wan(ix1)   
2022-03-03T07:03:41-05:00   Error   opnsense   /usr/local/etc/rc.linkup: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on ix0   
2022-03-03T07:03:41-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: keeping current default gateway '73.134.218.1'   
2022-03-03T07:03:41-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 73.134.218.1   
2022-03-03T07:03:41-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan   
2022-03-03T07:03:40-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'   
2022-03-03T07:03:37-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.   
2022-03-03T07:03:36-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: The WAN_DHCP monitor address is empty, skipping.   
2022-03-03T07:03:36-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '73.134.218.1'   
2022-03-03T07:03:36-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 73.134.218.1   
2022-03-03T07:03:36-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan   
2022-03-03T07:03:36-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'   
2022-03-03T07:03:35-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: On (IP address: 73.134.x.x) (interface: WAN[wan]) (real interface: ix1).   
2022-03-03T07:03:35-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'ix1'   
2022-03-03T07:03:34-05:00   Error   dhclient   send_packet: Network is down   
2022-03-03T07:03:29-05:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic wan(ix1)   
2022-03-03T07:03:28-05:00   Error   opnsense   /usr/local/etc/rc.linkup: Clearing states for stale wan route on ix1   
2022-03-03T07:03:28-05:00   Critical   dhclient   exiting.   
2022-03-03T07:03:28-05:00   Error   dhclient   connection closed   
2022-03-03T07:03:28-05:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic wan(ix1)   
2022-03-03T07:03:23-05:00   Error   opnsense   /usr/local/etc/rc.linkup: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on ix0   
2022-03-03T07:03:23-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: keeping current default gateway '73.134.218.1'   
2022-03-03T07:03:23-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 73.134.218.1   
2022-03-03T07:03:23-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan   
2022-03-03T07:03:22-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'   
2022-03-03T07:03:19-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.   
2022-03-03T07:03:18-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: The WAN_DHCP monitor address is empty, skipping.   
2022-03-03T07:03:18-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '73.134.218.1'   
2022-03-03T07:03:18-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 73.134.218.1   
2022-03-03T07:03:18-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan   
2022-03-03T07:03:18-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'   
2022-03-03T07:03:17-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: On (IP address: 73.134.x.x) (interface: WAN[wan]) (real interface: ix1).   
2022-03-03T07:03:17-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'ix1'   
2022-03-03T07:03:12-05:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic wan(ix1)   
2022-03-03T07:03:11-05:00   Error   opnsense   /usr/local/etc/rc.linkup: Clearing states for stale wan route on ix1   
2022-03-03T07:03:11-05:00   Critical   dhclient   exiting.   
2022-03-03T07:03:11-05:00   Error   dhclient   connection closed   
2022-03-03T07:03:11-05:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic wan(ix1)   
2022-03-03T07:03:06-05:00   Error   opnsense   /usr/local/etc/rc.linkup: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on ix0   
2022-03-03T07:03:05-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: keeping current default gateway '73.134.218.1'   
2022-03-03T07:03:05-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 73.134.218.1   
2022-03-03T07:03:05-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan   
2022-03-03T07:03:05-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'   
2022-03-03T07:03:02-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.   
2022-03-03T07:03:01-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: The WAN_DHCP monitor address is empty, skipping.   
2022-03-03T07:03:01-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '73.134.218.1'   
2022-03-03T07:03:01-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 73.134.218.1   
2022-03-03T07:03:01-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan   
2022-03-03T07:03:01-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'   
2022-03-03T07:03:00-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: On (IP address: 73.134.x.x) (interface: WAN[wan]) (real interface: ix1).   
2022-03-03T07:03:00-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'ix1'   
2022-03-03T07:02:55-05:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic wan(ix1)   
2022-03-03T07:02:54-05:00   Error   opnsense   /usr/local/etc/rc.linkup: Clearing states for stale wan route on ix1   
2022-03-03T07:02:53-05:00   Critical   dhclient   exiting.   
2022-03-03T07:02:53-05:00   Error   dhclient   connection closed   

I can confirm that I did not experience this on previous versions, and rolling back to 22.1.1 gives me a stable system. This was all done using community release versions rather than development versions.

Current stable version is:
OPNsense 22.1.1_3-amd64
FreeBSD 13.0-STABLE
OpenSSL 1.1.1m 14 Dec 2021

Any ideas? I really appreciate any help.

Thank you,
Fox

[peter.vynck]

Same problem here. I have 3 WAN-connections, all get their address from ISP through DHCP. The problematic interface is that one with a DHCP-delivered static address. When I disable that WAN-interface (MAC spoofing) the system becomes stable.
Did a fresh install on another machine with an older version, did the upgrades and face the exact same problem?!
Later today will try 2 things: revert to previous version on the original machine. And try a fresh installation with latest version on the other machine.

[foxmanb]

I look forward to hearing what you find. It's a very strange issue.

[subivoodoo]

Hi,

I bet you use intrusion detection in IPS mode together with MAC spoofing on your WAN?

I figured out the same issue here:
https://forum.opnsense.org/index.php?topic=26672.0

Workaround at the moment for me:
a) disable IPS or intrusion detection
or
b) remove the spoofed MAC from WAN

[foxmanb]

Hmm, I do use a spoofed mac, or at least at one time it was spoofed. I'm assuming I would need to generate my own MAC address rather than spoofing one? Xfinity requires a MAC be presented on the WAN interface.

[subivoodoo]

During my tests it doesn't matter if it's a real MAC or not. As soon as there is some MAC entered on the same interface as intrusion detection with IPS is enabled, the if down/up happens after suricata/netmap is fully loaded.

[aimdev]

I raised this in
https://forum.opnsense.org/index.php?topic=26657.msg128900#msg128900
January 31, 2022, 12:44:15 pm

I have had no issues since.
For operational reasons I have not re-enabled suricata.
This issue did not occur with 21.x
LAN & WAN interfaces are not virtual, and are intel 

em0@pci0:0:31:6:   class=0x020000 rev=0x21 hdr=0x00 vendor=0x8086 device=0x156f subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection I219-LM'
    class      = network
    subclass   = ethernet
igb0@pci0:2:0:0:   class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1539 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'I211 Gigabit Network Connection'
    class      = network
    subclass   = ethernet

MAC not spoofed

[peter.vynck]

Hi, reading the posts here I decided to remove the MAC-spoofing. As a result everything is back to normal...
I downloaded updated rules for Suricata and enabled it: everything stays normal.

Regarding the MAC-spoofing: the MAC-address I had in the config was actually the real physical MAC of the interface. Can that be a reason?

[aimdev]

I would suggest that you keep a lookout on the console to see if the WAN drops.
In addition to my post, mac address was not spoofed.

[smeetst]

I have the same issue, however I have to spoof the mac.

But if I revert freeradius:
opnsense-revert -r 22.1.1 os-freeradius

the problem is gone.
The rest of the stack is up to date.

[peter.vynck]

Hi, personally not using os-freeradius here.

As for my remark about spoofing the real MAC-address: when I tried on another machine the real MAC-address was different. Meaning that it is the spoofing itself that seems to trigger the problem.

I will try to replicate the issue this weekend by re-introducing spoofing 'just for fun'.

[subivoodoo]

In my case I do not have freeradius installed... but I reverted OpnSense to 22.1.1_3 (opnsense-revert -r 22.1.1 opnsense) and until now it also works for me on my clean test install (MAC spoofing + intrusion detection with IPS mode on).

[peter.vynck]

I introduced MAC-spoofing again and immediately the problems start all-over.
What surprises me is that the responsiveness from the router-GUI on the LAN-side gets crippled as well making it hard to change the settings back to 'normal'.


The log looks like this after entering a MAC-address:

2022-03-06T13:35:07   Error   opnsense    /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'igb0'
2022-03-06T13:35:02   Error   opnsense    /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic opt1(igb0)
2022-03-06T13:34:59   Error   opnsense    /usr/local/etc/rc.filter_configure: Ignore down inet6 gateways : WAN1FIXED_DHCP
2022-03-06T13:34:59   Error   opnsense    /usr/local/etc/rc.filter_configure: ROUTING: keeping current default gateway '213.118.192.1'
2022-03-06T13:34:59   Error   opnsense    /usr/local/etc/rc.filter_configure: Ignore down inet gateways : WAN1FIXED_DHCP
2022-03-06T13:34:59   Error   opnsense    /interfaces.php: The WAN1FIXED_DHCP IPv4 interface address is invalid, skipping.
2022-03-06T13:34:59   Error   opnsense    /interfaces.php: Choose to bind WAN1FIXED_DHCP on  since we could not find a proper match.
2022-03-06T13:34:59   Error   opnsense    /interfaces.php: Adding static route for monitor 8.8.8.8 via 213.118.152.1
2022-03-06T13:34:59   Error   opnsense    /interfaces.php: Removing static route for monitor 8.8.8.8 via 213.118.152.1
2022-03-06T13:34:59   Error   opnsense    /interfaces.php: Adding static route for monitor 8.8.4.4 via 213.118.192.1
2022-03-06T13:34:59   Error   opnsense    /interfaces.php: Removing static route for monitor 8.8.4.4 via 213.118.192.1
2022-03-06T13:34:58   Error   opnsense    /interfaces.php: ROUTING: keeping current default gateway '213.118.192.1'
2022-03-06T13:34:58   Error   opnsense    /interfaces.php: ROUTING: setting IPv4 default route to 213.118.192.1
2022-03-06T13:34:58   Error   opnsense    /interfaces.php: ROUTING: IPv4 default gateway set to opt4
2022-03-06T13:34:58   Error   opnsense    /interfaces.php: ROUTING: entering configure using defaults
2022-03-06T13:34:57   Error   opnsense    /usr/local/etc/rc.filter_configure: ROUTING: keeping current default gateway '213.118.192.1'
2022-03-06T13:34:57   Error   opnsense    /interfaces.php: ROUTING: skipping IPv4 default route
2022-03-06T13:34:57   Error   opnsense    /interfaces.php: ROUTING: IPv4 default gateway set to opt4
2022-03-06T13:34:57   Error   opnsense    /interfaces.php: ROUTING: entering configure using 'opt1'
2022-03-06T13:34:57   Error   opnsense    /interfaces.php: The command '/sbin/dhclient -c '/var/etc/dhclient_opt1.conf' -p '/var/run/dhclient.igb0.pid' 'igb0'' returned exit code '15', the output was 'igb0: no link ...'
2022-03-06T13:34:57   Error   opnsense    /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic opt1(igb0)
2022-03-06T13:34:56   Error   opnsense    /interfaces.php: Clearing states for stale opt1 route on igb0
2022-03-06T13:34:56   Critical   dhclient    exiting.
2022-03-06T13:34:56   Error   dhclient    connection closed

[RickNY]

Also experienced WAN issues with 22.1.2_1 here... I was using MAC spoofing on the WAN interface, but removed it for troubleshooting.  I am not using IPS on this device.  One of the issues I have is that if my upstream device (cable modem) reboots, the DHCP client does not appear to get an IP again once the link returns -- the GUI just displays "dhcp" -- I have to manually reboot the entire firewall in order to get an IP on the WAN interface again. 

Second -- on reboots, my Wireguard VPN fails and stays that way because the initialization for that takes place while the WAN is still broken -- and even when it returns, it doesn't attempt to fix itself.. So I have to manually disable Wireguard and re-enable it for it to work.

[Daemotrix]

I have also the same problem as described upper, I'm using HP 600 G2 DM with Intel NIC and also I'm using MAC spoofing. Problems with flapping started after the update, I also tried a fresh install, but without results. With older rls It started to work normally so there must be some bug in this rls.