OPNsense Forum

Archive => 22.1 Legacy Series => Topic started by: foxmanb on March 03, 2022, 01:45:18 pm

Title: WAN interface flapping with 22.1.2
Post by: foxmanb on March 03, 2022, 01:45:18 pm
Hi All,
New here, and not a expert in linux/bsd etc, so please, go easy on me.. lol.

I upgraded to 22.1.2 on release day, and when my router came back, up, the WAN interface would pick up a DHCP address, hold it for about 10 seconds, then drop, renew, hold it for 10 seconds, then drop... rinse, wash, repeat. In those 10 seconds I was able to use the opnsense-revert command to roll back to 22.1.1. The interface is stable under the old version. I am running a custom built box, intel 9500T, Intel x550-t2 NIC. "Gateway" is a Motorola MB8611 cable modem which connects at 2.5G.

This is what I see in the logs over and over again with the new 22.1.2 version.

2022-03-03T07:03:46-05:00   Critical   dhclient   exiting.   
2022-03-03T07:03:46-05:00   Error   dhclient   connection closed   
2022-03-03T07:03:46-05:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic wan(ix1)   
2022-03-03T07:03:41-05:00   Error   opnsense   /usr/local/etc/rc.linkup: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on ix0   
2022-03-03T07:03:41-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: keeping current default gateway '73.134.218.1'   
2022-03-03T07:03:41-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 73.134.218.1   
2022-03-03T07:03:41-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan   
2022-03-03T07:03:40-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'   
2022-03-03T07:03:37-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.   
2022-03-03T07:03:36-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: The WAN_DHCP monitor address is empty, skipping.   
2022-03-03T07:03:36-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '73.134.218.1'   
2022-03-03T07:03:36-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 73.134.218.1   
2022-03-03T07:03:36-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan   
2022-03-03T07:03:36-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'   
2022-03-03T07:03:35-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: On (IP address: 73.134.x.x) (interface: WAN[wan]) (real interface: ix1).   
2022-03-03T07:03:35-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'ix1'   
2022-03-03T07:03:34-05:00   Error   dhclient   send_packet: Network is down   
2022-03-03T07:03:29-05:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic wan(ix1)   
2022-03-03T07:03:28-05:00   Error   opnsense   /usr/local/etc/rc.linkup: Clearing states for stale wan route on ix1   
2022-03-03T07:03:28-05:00   Critical   dhclient   exiting.   
2022-03-03T07:03:28-05:00   Error   dhclient   connection closed   
2022-03-03T07:03:28-05:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic wan(ix1)   
2022-03-03T07:03:23-05:00   Error   opnsense   /usr/local/etc/rc.linkup: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on ix0   
2022-03-03T07:03:23-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: keeping current default gateway '73.134.218.1'   
2022-03-03T07:03:23-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 73.134.218.1   
2022-03-03T07:03:23-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan   
2022-03-03T07:03:22-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'   
2022-03-03T07:03:19-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.   
2022-03-03T07:03:18-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: The WAN_DHCP monitor address is empty, skipping.   
2022-03-03T07:03:18-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '73.134.218.1'   
2022-03-03T07:03:18-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 73.134.218.1   
2022-03-03T07:03:18-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan   
2022-03-03T07:03:18-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'   
2022-03-03T07:03:17-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: On (IP address: 73.134.x.x) (interface: WAN[wan]) (real interface: ix1).   
2022-03-03T07:03:17-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'ix1'   
2022-03-03T07:03:12-05:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic wan(ix1)   
2022-03-03T07:03:11-05:00   Error   opnsense   /usr/local/etc/rc.linkup: Clearing states for stale wan route on ix1   
2022-03-03T07:03:11-05:00   Critical   dhclient   exiting.   
2022-03-03T07:03:11-05:00   Error   dhclient   connection closed   
2022-03-03T07:03:11-05:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic wan(ix1)   
2022-03-03T07:03:06-05:00   Error   opnsense   /usr/local/etc/rc.linkup: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on ix0   
2022-03-03T07:03:05-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: keeping current default gateway '73.134.218.1'   
2022-03-03T07:03:05-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 73.134.218.1   
2022-03-03T07:03:05-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan   
2022-03-03T07:03:05-05:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'   
2022-03-03T07:03:02-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.   
2022-03-03T07:03:01-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: The WAN_DHCP monitor address is empty, skipping.   
2022-03-03T07:03:01-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '73.134.218.1'   
2022-03-03T07:03:01-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 73.134.218.1   
2022-03-03T07:03:01-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan   
2022-03-03T07:03:01-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'   
2022-03-03T07:03:00-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: On (IP address: 73.134.x.x) (interface: WAN[wan]) (real interface: ix1).   
2022-03-03T07:03:00-05:00   Error   opnsense   /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'ix1'   
2022-03-03T07:02:55-05:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic wan(ix1)   
2022-03-03T07:02:54-05:00   Error   opnsense   /usr/local/etc/rc.linkup: Clearing states for stale wan route on ix1   
2022-03-03T07:02:53-05:00   Critical   dhclient   exiting.   
2022-03-03T07:02:53-05:00   Error   dhclient   connection closed   

I can confirm that I did not experience this on previous versions, and rolling back to 22.1.1 gives me a stable system. This was all done using community release versions rather than development versions.

Current stable version is:
OPNsense 22.1.1_3-amd64
FreeBSD 13.0-STABLE
OpenSSL 1.1.1m 14 Dec 2021

Any ideas? I really appreciate any help.

Thank you,
Fox

Title: Re: WAN interface flapping with 22.1.2
Post by: peter.vynck on March 03, 2022, 03:44:34 pm
Same problem here. I have 3 WAN-connections, all get their address from ISP through DHCP. The problematic interface is that one with a DHCP-delivered static address. When I disable that WAN-interface (MAC spoofing) the system becomes stable.
Did a fresh install on another machine with an older version, did the upgrades and face the exact same problem?!
Later today will try 2 things: revert to previous version on the original machine. And try a fresh installation with latest version on the other machine.
Title: Re: WAN interface flapping with 22.1.2
Post by: foxmanb on March 03, 2022, 05:20:33 pm
I look forward to hearing what you find. It's a very strange issue.
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on March 03, 2022, 05:29:47 pm
Hi,

I bet you use intrusion detection in IPS mode together with MAC spoofing on your WAN?

I figured out the same issue here:
https://forum.opnsense.org/index.php?topic=26672.0 (https://forum.opnsense.org/index.php?topic=26672.0)

Workaround at the moment for me:
a) disable IPS or intrusion detection
or
b) remove the spoofed MAC from WAN
Title: Re: WAN interface flapping with 22.1.2
Post by: foxmanb on March 03, 2022, 05:39:55 pm
Hmm, I do use a spoofed mac, or at least at one time it was spoofed. I'm assuming I would need to generate my own MAC address rather than spoofing one? Xfinity requires a MAC be presented on the WAN interface.
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on March 03, 2022, 05:47:01 pm
During my tests it doesn't matter if it's a real MAC or not. As soon as there is some MAC entered on the same interface as intrusion detection with IPS is enabled, the if down/up happens after suricata/netmap is fully loaded.
Title: Re: WAN interface flapping with 22.1.2
Post by: aimdev on March 03, 2022, 06:05:01 pm
I raised this in
https://forum.opnsense.org/index.php?topic=26657.msg128900#msg128900
January 31, 2022, 12:44:15 pm

I have had no issues since.
For operational reasons I have not re-enabled suricata.
This issue did not occur with 21.x
LAN & WAN interfaces are not virtual, and are intel 

em0@pci0:0:31:6:   class=0x020000 rev=0x21 hdr=0x00 vendor=0x8086 device=0x156f subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection I219-LM'
    class      = network
    subclass   = ethernet
igb0@pci0:2:0:0:   class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1539 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'I211 Gigabit Network Connection'
    class      = network
    subclass   = ethernet

MAC not spoofed
Title: Re: WAN interface flapping with 22.1.2
Post by: peter.vynck on March 03, 2022, 07:21:05 pm
Hi, reading the posts here I decided to remove the MAC-spoofing. As a result everything is back to normal...
I downloaded updated rules for Suricata and enabled it: everything stays normal.

Regarding the MAC-spoofing: the MAC-address I had in the config was actually the real physical MAC of the interface. Can that be a reason?
Title: Re: WAN interface flapping with 22.1.2
Post by: aimdev on March 03, 2022, 07:27:16 pm
I would suggest that you keep a lookout on the console to see if the WAN drops.
In addition to my post, mac address was not spoofed.
Title: Re: WAN interface flapping with 22.1.2
Post by: smeetst on March 04, 2022, 08:30:56 am
I have the same issue, however I have to spoof the mac.

But if I revert freeradius:
opnsense-revert -r 22.1.1 os-freeradius

the problem is gone.
The rest of the stack is up to date.
Title: Re: WAN interface flapping with 22.1.2
Post by: peter.vynck on March 04, 2022, 08:41:53 am
Hi, personally not using os-freeradius here.

As for my remark about spoofing the real MAC-address: when I tried on another machine the real MAC-address was different. Meaning that it is the spoofing itself that seems to trigger the problem.

I will try to replicate the issue this weekend by re-introducing spoofing 'just for fun'.
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on March 04, 2022, 07:26:55 pm
In my case I do not have freeradius installed... but I reverted OpnSense to 22.1.1_3 (opnsense-revert -r 22.1.1 opnsense) and until now it also works for me on my clean test install (MAC spoofing + intrusion detection with IPS mode on).
Title: Re: WAN interface flapping with 22.1.2
Post by: peter.vynck on March 06, 2022, 01:53:01 pm
I introduced MAC-spoofing again and immediately the problems start all-over.
What surprises me is that the responsiveness from the router-GUI on the LAN-side gets crippled as well making it hard to change the settings back to 'normal'.


The log looks like this after entering a MAC-address:

2022-03-06T13:35:07   Error   opnsense    /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'igb0'
2022-03-06T13:35:02   Error   opnsense    /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic opt1(igb0)
2022-03-06T13:34:59   Error   opnsense    /usr/local/etc/rc.filter_configure: Ignore down inet6 gateways : WAN1FIXED_DHCP
2022-03-06T13:34:59   Error   opnsense    /usr/local/etc/rc.filter_configure: ROUTING: keeping current default gateway '213.118.192.1'
2022-03-06T13:34:59   Error   opnsense    /usr/local/etc/rc.filter_configure: Ignore down inet gateways : WAN1FIXED_DHCP
2022-03-06T13:34:59   Error   opnsense    /interfaces.php: The WAN1FIXED_DHCP IPv4 interface address is invalid, skipping.
2022-03-06T13:34:59   Error   opnsense    /interfaces.php: Choose to bind WAN1FIXED_DHCP on  since we could not find a proper match.
2022-03-06T13:34:59   Error   opnsense    /interfaces.php: Adding static route for monitor 8.8.8.8 via 213.118.152.1
2022-03-06T13:34:59   Error   opnsense    /interfaces.php: Removing static route for monitor 8.8.8.8 via 213.118.152.1
2022-03-06T13:34:59   Error   opnsense    /interfaces.php: Adding static route for monitor 8.8.4.4 via 213.118.192.1
2022-03-06T13:34:59   Error   opnsense    /interfaces.php: Removing static route for monitor 8.8.4.4 via 213.118.192.1
2022-03-06T13:34:58   Error   opnsense    /interfaces.php: ROUTING: keeping current default gateway '213.118.192.1'
2022-03-06T13:34:58   Error   opnsense    /interfaces.php: ROUTING: setting IPv4 default route to 213.118.192.1
2022-03-06T13:34:58   Error   opnsense    /interfaces.php: ROUTING: IPv4 default gateway set to opt4
2022-03-06T13:34:58   Error   opnsense    /interfaces.php: ROUTING: entering configure using defaults
2022-03-06T13:34:57   Error   opnsense    /usr/local/etc/rc.filter_configure: ROUTING: keeping current default gateway '213.118.192.1'
2022-03-06T13:34:57   Error   opnsense    /interfaces.php: ROUTING: skipping IPv4 default route
2022-03-06T13:34:57   Error   opnsense    /interfaces.php: ROUTING: IPv4 default gateway set to opt4
2022-03-06T13:34:57   Error   opnsense    /interfaces.php: ROUTING: entering configure using 'opt1'
2022-03-06T13:34:57   Error   opnsense    /interfaces.php: The command '/sbin/dhclient -c '/var/etc/dhclient_opt1.conf' -p '/var/run/dhclient.igb0.pid' 'igb0'' returned exit code '15', the output was 'igb0: no link ...'
2022-03-06T13:34:57   Error   opnsense    /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic opt1(igb0)
2022-03-06T13:34:56   Error   opnsense    /interfaces.php: Clearing states for stale opt1 route on igb0
2022-03-06T13:34:56   Critical   dhclient    exiting.
2022-03-06T13:34:56   Error   dhclient    connection closed
Title: Re: WAN interface flapping with 22.1.2
Post by: RickNY on March 06, 2022, 03:45:42 pm

Also experienced WAN issues with 22.1.2_1 here... I was using MAC spoofing on the WAN interface, but removed it for troubleshooting.  I am not using IPS on this device.  One of the issues I have is that if my upstream device (cable modem) reboots, the DHCP client does not appear to get an IP again once the link returns -- the GUI just displays "dhcp" -- I have to manually reboot the entire firewall in order to get an IP on the WAN interface again. 

Second -- on reboots, my Wireguard VPN fails and stays that way because the initialization for that takes place while the WAN is still broken -- and even when it returns, it doesn't attempt to fix itself.. So I have to manually disable Wireguard and re-enable it for it to work.

Title: Re: WAN interface flapping with 22.1.2
Post by: Daemotrix on March 06, 2022, 04:33:42 pm
I have also the same problem as described upper, I'm using HP 600 G2 DM with Intel NIC and also I'm using MAC spoofing. Problems with flapping started after the update, I also tried a fresh install, but without results. With older rls It started to work normally so there must be some bug in this rls.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on March 07, 2022, 04:34:27 pm
Same WAN flapping issue on Protectli 6 port hardware with intel NIC.  No MAC Spoofing enabled.  Suricata enabled only on LAN.  Multi-WAN setup, Primary (flapping) WAN is Static IPv4 via comcast.  Disabling "Gateway monitoring" fixes the issue, but obviously breaks the failover group and is not a long term solution.  Have verified that the monitor IP was still up and responding during "flaps"...



Previous versions of OPNSense did not display this issue.
Title: Re: WAN interface flapping with 22.1.2
Post by: LoZ on March 09, 2022, 11:42:08 am
Just created a forum account to report this too.

I had upgraded to 22.1.2_1 a few days ago and had not seen any real issues.

However when I enabled IPS last night I immediately started seeing the WAN interface start flapping (going up and down, together with the CPU usage continuously going from 0-100% and back again).

I switched off IPS and rebooted the router but this did not resolve the issue.

I had to revert to the previous version using
Code: [Select]
opnsense-revert -r 22.1.1 opnsense - thanks to a poster further up this thread for that!

Immediately after reverting the WAN connection went stable again. I should mention that I am using MAC spoofing.
Title: Re: WAN interface flapping with 22.1.2
Post by: sirdir on March 09, 2022, 09:02:09 pm
I have a similar problem. Just installed Opnsense today. First it worked, then I upgraded the version I downloaded to the newest version and the WAN interface goes UP DOWN UP DOWN all the time. When I configure the WAN as static IP, that doesn’t happen. On the same interface I also get the message on the console:
appresolve: can’t allocate llinfo for x.x.x.x on igb1
(x.x.x.x is the gateway address of the wan ip). This WAN is connected by a WLAN bridge so x.x.x.x actually is not physically on the same network. Still, it’s always reachable and I don’t get this message with pfsense.
Last but not least, I also use MAC spoofing on that interface.
Any ideas?
Title: Re: WAN interface flapping with 22.1.2
Post by: LoZ on March 10, 2022, 10:40:06 am
Further to my post above, I've since (as a result of an error during the nginx plugin install) seen a crash report.

Of interest is the dmesg.boot log:

Code: [Select]
rpresolve: can't allocate llinfo for <WAN_Gateway_IP> on igb3
arpresolve: can't allocate llinfo for <WAN_Gateway_IP> on igb3
arpresolve: can't allocate llinfo for <WAN_Gateway_IP> on igb3
igb3: link state changed to UP
arpresolve: can't allocate llinfo for <WAN_Gateway_IP> on igb3
igb3: link state changed to DOWN
arpresolve: can't allocate llinfo for <WAN_Gateway_IP> on igb3
arpresolve: can't allocate llinfo for <WAN_Gateway_IP> on igb3
arpresolve: can't allocate llinfo for <WAN_Gateway_IP> on igb3
arpresolve: can't allocate llinfo for <WAN_Gateway_IP> on igb3
arpresolve: can't allocate llinfo for <WAN_Gateway_IP> on igb3
arpresolve: can't allocate llinfo for <WAN_Gateway_IP> on igb3
arpresolve: can't allocate llinfo for <WAN_Gateway_IP> on igb3
igb3: link state changed to UP
arpresolve: can't allocate llinfo for <WAN_Gateway_IP> on igb3
igb3: link state changed to DOWN

Hope this is useful information!
Title: Re: WAN interface flapping with 22.1.2
Post by: DocGonzo74 on March 11, 2022, 10:06:52 pm
I've been chasing this same issue for a few days now.  WAN gets flappy when I'm using a spoofed MAC. I'm doing dual DHCP WAN with CARP.  Once I upgraded, my WAN circuit started acting up.  Killing the spoofed MAC did it.  Still chasing an answer as to why. 
Title: Re: WAN interface flapping with 22.1.2
Post by: ToniE on March 14, 2022, 02:13:02 pm
similar problems, the only working solution for me was to reinstall to 21.7.8
Title: Re: WAN interface flapping with 22.1.2
Post by: trijenhout on March 17, 2022, 09:46:12 pm
are there any updates on this issu??running opsense in a vm whit dedicated pashtrue pci card and ii have the same problem.... is it a kernel problem? is there a work around? i do need macspoofing....
Title: Re: WAN interface flapping with 22.1.2
Post by: trijenhout on March 20, 2022, 12:14:05 pm
22.1.3 still have this issue, for me....
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on March 20, 2022, 03:27:51 pm
Also still having issues on 22.1.3, disabling gateway monitoring (which is not an option in my multi-wan environments) fixes the issue since the link is not really down.
Title: Re: WAN interface flapping with 22.1.2
Post by: Supermule on March 20, 2022, 08:27:31 pm
Dpinger problem?
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on March 20, 2022, 11:15:21 pm
Quote
Dpinger problem?
Yes, dpinger logs show missed pings/latency when it doesn't exist.  I have increased my latency threshold and ping frequency to compensate but it still occurs.  Note that if I have a static IP on the connection that's flapping...And replacing the opnsense box with a sonicwall (that's also doing gateway monitoring) and the issue disappears.  There is another thread on the forum about dpinger creating static routes to the gateway monitors and possibly causing issues.

Code: [Select]
2022-03-20T09:45:45-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Alarm latency 19767us stddev 1454us loss 37%
2022-03-20T09:43:46-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 0 RTT: 20989us RTTd: 5447us Loss: 20%)
2022-03-20T09:43:46-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Clear latency 20989us stddev 5447us loss 20%
2022-03-20T09:38:21-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 1 RTT: 27705us RTTd: 11063us Loss: 37%)
2022-03-20T09:38:21-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Alarm latency 27705us stddev 11063us loss 37%
2022-03-20T05:35:41-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 0 RTT: 490329us RTTd: 1441182us Loss: 0%)
2022-03-20T05:35:41-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Clear latency 490329us stddev 1441182us loss 0%
2022-03-20T05:35:31-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 1 RTT: 506742us RTTd: 1463061us Loss: 0%)
2022-03-20T05:35:31-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Alarm latency 506742us stddev 1463061us loss 0%
2022-03-20T01:36:23-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 0 RTT: 24121us RTTd: 12174us Loss: 0%)
2022-03-20T01:36:23-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Clear latency 24121us stddev 12174us loss 0%
2022-03-20T01:35:19-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 1 RTT: 600018us RTTd: 1670674us Loss: 3%)
2022-03-20T01:35:19-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Alarm latency 600018us stddev 1670674us loss 3%
2022-03-19T13:36:19-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 0 RTT: 21108us RTTd: 7988us Loss: 0%)
2022-03-19T13:36:19-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Clear latency 21108us stddev 7988us loss 0%
2022-03-19T13:35:16-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 1 RTT: 814390us RTTd: 2117391us Loss: 0%)
2022-03-19T13:35:16-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Alarm latency 814390us stddev 2117391us loss 0%
2022-03-19T09:37:04-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 0 RTT: 23782us RTTd: 9052us Loss: 23%)
2022-03-19T09:37:04-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Clear latency 23782us stddev 9052us loss 23%
2022-03-19T09:36:23-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 1 RTT: 26176us RTTd: 11155us Loss: 37%)
Title: Re: WAN interface flapping with 22.1.2
Post by: kropotkin on March 21, 2022, 04:33:50 pm
I had the WAN flapping issue with mac spoofing and have resolved by installing the updated FreeBSD 13 em driver, though I don't know if you have intel nics as well?
To generate the driver file I spun up a FreeBSD vm then pkg search intel-em-mod and install. Copied the if_em_updated.ko driver to /boot/modules/ as per Franco's reply in this post https://forum.opnsense.org/index.php?topic=20905.0.
I also disabled suricata on the wan interface and turned off flow control on all NICs.
Now running on a non flapping OPNsense 22.1.3 with WAN DHCP and mac spoofing.
@Franco - can this driver be added to OPNsense as it seems to resolve a number of stability issues.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on March 21, 2022, 07:24:41 pm
Interesting, I am using intel nics (Protecli boxes), but suricata is only on the LAN and WAN 1 is static, WAN 2 is DHCP.  No MAC spoofing...

Looks like the driver your using will be included in FreeBSD 13.1
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on March 21, 2022, 07:57:58 pm
On my systems where I can reproduce this issue I also have Intel NIC's... but not em. I have ixl (X710) or igc (I225).
To test this Intel driver issue theory, I've changed to a virtual WAN interface (vtnet on KVM and virtio) and there was no issue with MAC spoofing + IDP IPS on!!!
Title: Re: WAN interface flapping with 22.1.2
Post by: firewall on March 21, 2022, 07:59:00 pm
Also seeing this issue on a NUC with 6x Intel eth devices, no MAC spoofing whatsoever, and Suricata enabled on LAN/WLAN only. This seems to be a rather prevalent issue--has it been acknowledged by project staff?

EDIT - SOLVED: This issue continued for 2 months until an issue with Intel driver config on FBSD13 was called out. The resolution was posted here and the fix described worked for me (thanks @tracerrx!):
https://forum.opnsense.org/index.php?topic=27299.msg137350#msg137350

EDIT 2: called it too soon. still broken.
Title: Re: WAN interface flapping with 22.1.2
Post by: DocGonzo74 on March 23, 2022, 08:44:33 pm
Been messing with this on and off over the last week..  I stabilized my dual wan HA setup by removing the mac spoofing and hostname from the WAN interfaces on the primary firewall.  Once I did that, the flapping stopped completely for the primary.  08-setwanmac

Whenever I move to my backup, the backup's WAN interfaces would flap and my WANs would take turns going up and down.

Tried a bunch of different things along what you all have tried, then I thought about it and added a simple script to the following directory:

/usr/local/etc/rc.syshook.d/start/08-setwanmac

08-setwanmac contains this:

#!/bin/sh

# Change WAN MAC addresses
ifconfig igb4 ether yy:yy:yy:yy:yy:yx
ifconfig igb5 ether xx:xx:xx:xx:xx:xy

the 08-setwanmac is silly.. just using ifconfig to change the MAC to the desired MAC (a clone of my primary firewall NIC MAC addresses). 

Super static and simple, but it's survived quite a few reboots and forced swaps with minimal packet loss and zero flaps.   I just inserted the MAC change prior to the newwanip script, thinking that the mac change would occur before the newwanip script.   Working out so far.

Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on March 24, 2022, 06:29:53 pm
For me the only fix is either disable gateway monitor (not really an option) or to change the gateway monitor from 8.8.8.8 to 8.8.4.4 then back again each time it goes "down"..
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on March 24, 2022, 07:11:38 pm
Well in any case you seem to have overlapping DNS servers for the different interfaces, either set manually, by ISP or gateway monitor. In some cases ISPs push Google servers which is pretty mean since it also pins a route for it through their interface.


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on March 24, 2022, 07:19:33 pm
@franco so the dns servers under settings-->general cannot overlap with gateway monitor IP's?  I use piholes for DNS and push the piholes ip's out via DHCP to all clients. 
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on March 24, 2022, 07:36:17 pm
Every one of those creates a host route if you select a gateway for it. If these host routes conflict with the use in the gateway monitoring (most of the time because at least one host route overlaps multiple interfaces or the whole config is reversed there) you get the gateway flapping when the wrong interface comes back as the monitor uses the wrong gateway to monitor another.


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on March 24, 2022, 08:14:02 pm
@franco I think this fixed the issue!  Thanks!  When you try to add two gateway monitors that overlap, the GUI alerts you and will not save it.  However it will allow you to add a gateway monitor with the same IP as a DNS server specified in settings-->General without a warning.  FWIW, my dns servers in settings->general were assigned to NONE
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on March 31, 2022, 11:01:53 pm
@franco after about a week, the exact problem has returned.. I have ensured there are no over-lapping dns entries.  Switching the monitor from 8.8.8.8 to 75.75.75.75 and the interface immediately returns to up.

Code: [Select]
2022-03-31T16:57:42-04:00 Warning dpinger send_interval 2000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 55% dest_addr 75.75.75.75 bind_addr 173.9.169.97 identifier "WAN_Comcast_GWv4 "
2022-03-31T16:57:42-04:00 Warning dpinger exiting on signal 15
2022-03-31T16:57:42-04:00 Warning dpinger exiting on signal 15
2022-03-31T16:57:26-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 0 RTT: 20033us RTTd: 4707us Loss: 40%)
2022-03-31T16:57:26-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Clear latency 20033us stddev 4707us loss 40%
2022-03-31T16:54:13-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 1 RTT: 20596us RTTd: 1985us Loss: 58%)
2022-03-31T16:54:13-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Alarm latency 20596us stddev 1985us loss 58%
2022-03-31T16:53:40-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 0 RTT: 20997us RTTd: 1835us Loss: 41%)
2022-03-31T16:53:40-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Clear latency 20997us stddev 1835us loss 41%
2022-03-31T16:51:34-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 1 RTT: 29015us RTTd: 10285us Loss: 58%)
2022-03-31T16:51:34-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Alarm latency 29015us stddev 10285us loss 58%
2022-03-31T16:48:17-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 0 RTT: 40731us RTTd: 9156us Loss: 54%)
2022-03-31T16:48:17-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Clear latency 40731us stddev 9156us loss 54%
2022-03-31T16:47:35-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 8.8.8.8 Alarm: 1 RTT: 0us RTTd: 0us Loss: 100%)
2022-03-31T16:47:35-04:00 Warning dpinger WAN_Comcast_GWv4 8.8.8.8: Alarm latency 0us stddev 0us loss 100%
2022-03-31T16:47:33-04:00 Warning dpinger send_interval 15000ms loss_interval 4000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 60000ms latency_alarm 2500ms loss_alarm 80% dest_addr 1.1.1.1 bind_addr 192.168.42.135 identifier "WAN_HNETIPV4 "
2022-03-31T16:47:33-04:00 Warning dpinger send_interval 2000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 55% dest_addr 8.8.8.8 bind_addr 173.9.169.97 identifier "WAN_Comcast_GWv4 "
2022-03-31T16:47:33-04:00 Warning dpinger exiting on signal 15
2022-03-31T16:47:33-04:00 Warning dpinger exiting on signal 15
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on April 01, 2022, 06:26:35 pm
Is it possible that Monit (setup to monitor an ipsec VPN with "failed ping4 count 10 address XXX") is causing the flapping issue?  Seems like disabling my monitors for multiple ipsec vpns restores stability to the gateway.
Title: Re: WAN interface flapping with 22.1.2
Post by: foxmanb on April 03, 2022, 02:33:39 am
I can confirm I do not have monit running and did experience this issue.
Title: Re: WAN interface flapping with 22.1.2
Post by: firewall on April 04, 2022, 06:32:49 pm
Disabling Suricata last month (less than ideal) reduced the frequency of WAN drop but it's still happening on occasion. Is anyone else still seeing this behavior?
Title: Re: WAN interface flapping with 22.1.2
Post by: jeremias.lubberger on April 05, 2022, 02:29:09 pm
I have also experienced WAN flapping with v22.1.4. (All Intel NICs, if that's relevant)
Disabling MAC Spoofing and/or IPS did not resolve the issue, neither did a rollback to 22.1.1.
Finally I had to re-install 21.7 to reach stability again.
Title: Re: WAN interface flapping with 22.1.2
Post by: binoix on April 10, 2022, 09:38:09 pm
Hello,

It seems I'm experiencing similar issues with loss of WAN. I can't find relevant logs so far, everything seems ok according to web interface, and I need to ifconfig down && up to restore connectivity.
I have Intel NIC (I210), but don't use mac spoofing nor suricata nor monitoring.
I will try downgrading to 21.7 (using 22.1.5, and had the same issues with 22.1.4 and below).

Any ideas of logs I can check to investigate?

Regards
Title: Re: WAN interface flapping with 22.1.2
Post by: trijenhout on April 10, 2022, 10:02:31 pm
no luck whit 22.1.5 hope for a nice kernel update i guese...?
Title: Re: WAN interface flapping with 22.1.2
Post by: foxmanb on April 12, 2022, 03:23:55 pm
Hi Opnsense folks. Any progress on this one? Will it be addressed in a future release?

Thank you!
Title: Re: WAN interface flapping with 22.1.2
Post by: Edwin70 on April 14, 2022, 01:32:06 pm
Hi Opnsense folks. Any progress on this one? Will it be addressed in a future release?

Thank you!

+1  :)
Title: Re: WAN interface flapping with 22.1.2
Post by: Scuro on April 14, 2022, 04:39:52 pm
As a new OPNSense user I can confirm that MAC spoofing causes the WAN to flap.
I am running a J4125 unit with Intel I225-V NICs on 22.1.6.

Logs:
Code: [Select]
2022-04-14T07:11:05-07:00 Error opnsense /interfaces.php: The command '/sbin/dhclient -c '/var/etc/dhclient_wan.conf' -p '/var/run/dhclient.igc0.pid' 'igc0'' returned exit code '1', the output was 'igc0: no link .............. giving up'
2022-04-14T07:10:22-07:00 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic wan(igc0)
2022-04-14T07:10:21-07:00 Error opnsense /usr/local/etc/rc.linkup: Warning! dhcpd_dhcp4_configure() found no suitable IPv4 address on opt1
2022-04-14T07:10:21-07:00 Error opnsense /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'

If I disable MAC spoofing it no longer flaps the WAN.
Title: Re: WAN interface flapping with 22.1.2
Post by: grunge on April 15, 2022, 08:59:24 pm
I need to spoof the mac address so still waiting for a fix :'(
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on April 17, 2022, 01:14:35 pm
I also need it... a fix would be nice!

Or is there at least an idea of the real issue? Intel driver [only?] issue with mac spoofing???
Title: Re: WAN interface flapping with 22.1.2
Post by: devhunter55 on April 17, 2022, 03:35:28 pm
Thx all for the very good hint about "mac spoofing" .. i did not have got the chance to test it.
1rst of all, i wrote my ISP if "mac spoofing" is mandatory from his side.

on my side - UNBOUND is not working anymore since 22.1.2 or since UNBOUND version: unbound 1.15.0

Yesterday, i gave it a try again - and upgraded to 22.1.6.

I've got a lot of UNBOUND "overrides" in place & BlockLists.
I did disable the BlockLists - but this didn't help.

DNS is not working anymore - it seems that the DNS Resolver will switch from offline to online & vice versa in a very short time (what others called "flapping")

between i can connect to the WEB for a short time (but resolution is very, very slow).
..getting a WAN DHCP ip & and then it disappears again.

The whole machine gets very hot, CPU & unbound is about 100% - restart UNBOUND does not fix this issue.

Fortunately i'm using ZFS, so Restore is quick & easy - but full RESTORE was needed in every upgrade after version: 22.1.1 (with UNBOUND 1.15.0).

I tried the upgrades also with different hardware - same result - no chance to get UNBOUND working again - and DNS - of course is fundamental.

messages:
------------
<11>1 2022-04-15T20:59:19+02:00 opnsense-host opnsense 88624 - [meta sequenceId="32"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic wan(igb1)
<27>1 2022-04-15T20:59:19+02:00 opnsense-host dhclient 96268 - [meta sequenceId="33"] dhclient already running, pid: 86990.
<26>1 2022-04-15T20:59:19+02:00 opnsense-host dhclient 96268 - [meta sequenceId="34"] exiting.
<11>1 2022-04-15T20:59:19+02:00 opnsense-host opnsense 88624 - [meta sequenceId="35"] /usr/local/etc/rc.linkup: The command '/sbin/dhclient -c '/var/etc/dhclient_wan.conf' -p '/var/run/d
hclient.igb1.pid' 'igb1'' returned exit code '1', the output was 'dhclient already running, pid: 86990. exiting.'
<11>1 2022-04-15T20:59:19+02:00 opnsense-host opnsense 88624 - [meta sequenceId="36"] /usr/local/etc/rc.linkup: Accept router advertisements on interface igb1
<13>1 2022-04-15T20:59:19+02:00 opnsense-host dhcp6c 3104 - [meta sequenceId="37"] RTSOLD script - Sending SIGHUP to dhcp6c
<11>1 2022-04-15T20:59:19+02:00 opnsense-host opnsense 88624 - [meta sequenceId="38"] /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'
<11>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="39"] /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to lan
<11>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="40"] /usr/local/etc/rc.linkup: ROUTING: skipping IPv4 default route
<11>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="41"] /usr/local/etc/rc.linkup: ROUTING: IPv6 default gateway set to wan
<11>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="42"] /usr/local/etc/rc.linkup: ROUTING: skipping IPv6 default route
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="43"] plugins_configure ipsec (,wan)
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="44"] plugins_configure ipsec (execute task : ipsec_configure_do(,wan))
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="45"] plugins_configure dhcp ()
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="46"] plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="47"] plugins_configure dns ()
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="48"] plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="49"] plugins_configure dns (execute task : unbound_configure_do())
<11>1 2022-04-15T20:59:20+02:00 opnsense-host opnsense 88624 - [meta sequenceId="50"] /usr/local/etc/rc.linkup: warning: ignoring missing default tunable request: debug.pfftpproxy
<13>1 2022-04-15T20:59:24+02:00 opnsense-host dhcp6c 64283 - [meta sequenceId="51"] RTSOLD script - Sending SIGHUP to dhcp6c
<27>1 2022-04-15T20:59:36+02:00 opnsense-host upsmon 39698 - [meta sequenceId="52"] UPS [ups@localupsip]: connect failed: Connection failure: Operation timed out
<13>1 2022-04-15T20:59:49+02:00 opnsense-host configctl 68544 - [meta sequenceId="53"] event @ 1650049188.54 msg: Apr 15 20:59:48 opnsense-host config[87361]: [2022-04-15T20:59:48+02
:00][info] config-event: new_config /conf/backup/config-1650049188.5364.xml
<13>1 2022-04-15T20:59:49+02:00 opnsense-host configctl 68544 - [meta sequenceId="54"] event @ 1650049188.54 exec: system event config_changed
<27>1 2022-04-15T21:00:56+02:00 opnsense-host upsmon 39698 - [meta sequenceId="1"] UPS [ups@localupsip]: connect failed: Connection failure: Operation timed out
<29>1 2022-04-15T21:00:56+02:00 opnsense-host upsmon 39698 - [meta sequenceId="2"] UPS ups@localupsip is unavailable
<11>1 2022-04-15T21:01:19+02:00 opnsense-host configctl 87822 - [meta sequenceId="3"] error in configd communication  Traceback (most recent call last):   File "/usr/local/sbin/configctl
", line 66, in exec_config_cmd     line = sock.recv(65536).decode() socket.timeout: timed out
<11>1 2022-04-15T21:01:19+02:00 opnsense-host opnsense 99032 - [meta sequenceId="4"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic wan(igb1)
<11>1 2022-04-15T21:01:19+02:00 opnsense-host opnsense 99032 - [meta sequenceId="5"] /usr/local/etc/rc.linkup: Clearing states for stale wan route on igb1
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="6"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic wan(igb1)
<27>1 2022-04-15T21:01:20+02:00 opnsense-host dhclient 23026 - [meta sequenceId="7"] dhclient already running, pid: 86990.
<26>1 2022-04-15T21:01:20+02:00 opnsense-host dhclient 23026 - [meta sequenceId="8"] exiting.
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="9"] /usr/local/etc/rc.linkup: The command '/sbin/dhclient -c '/var/etc/dhclient_wan.conf' -p '/var/run/dh
client.igb1.pid' 'igb1'' returned exit code '1', the output was 'dhclient already running, pid: 86990. exiting.'
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="10"] /usr/local/etc/rc.linkup: Accept router advertisements on interface igb1
<13>1 2022-04-15T21:01:20+02:00 opnsense-host dhcp6c 30500 - [meta sequenceId="11"] RTSOLD script - Sending SIGHUP to dhcp6c
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="12"] /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="13"] /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to lan
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="14"] /usr/local/etc/rc.linkup: ROUTING: skipping IPv4 default route
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="15"] /usr/local/etc/rc.linkup: ROUTING: IPv6 default gateway set to wan
<11>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="16"] /usr/local/etc/rc.linkup: ROUTING: skipping IPv6 default route
<13>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="17"] plugins_configure ipsec (,wan)
<13>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="18"] plugins_configure ipsec (execute task : ipsec_configure_do(,wan))
<13>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="19"] plugins_configure dhcp ()
<13>1 2022-04-15T21:01:20+02:00 opnsense-host opnsense 16955 - [meta sequenceId="20"] plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2022-04-15T21:01:21+02:00 opnsense-host opnsense 16955 - [meta sequenceId="21"] plugins_configure dns ()
<13>1 2022-04-15T21:01:21+02:00 opnsense-host opnsense 16955 - [meta sequenceId="22"] plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2022-04-15T21:01:21+02:00 opnsense-host opnsense 16955 - [meta sequenceId="23"] plugins_configure dns (execute task : unbound_configure_do())
<11>1 2022-04-15T21:01:21+02:00 opnsense-host opnsense 16955 - [meta sequenceId="24"] /usr/local/etc/rc.linkup: warning: ignoring missing default tunable request: debug.pfftpproxy
<13>1 2022-04-15T21:01:24+02:00 opnsense-host dhcp6c 88779 - [meta sequenceId="25"] RTSOLD script - Sending SIGHUP to dhcp6c
Title: Re: WAN interface flapping with 22.1.2
Post by: devhunter55 on April 17, 2022, 03:41:16 pm
https://forum.opnsense.org/index.php?topic=27372.15
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on April 18, 2022, 10:37:28 am
In my case it's really only an issue on Intel NIC's... switched my WAN to the onboard Realtek NIC and all is working now (mac spoofing with IDP on in IPS mode, OPNsense 22.1.6)

Is there any kernel or driver fix underway?
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on April 18, 2022, 04:42:33 pm
I still have to change the gateway monitor address on multiple firewalls every 3-4 days (all dual wan) or the gateway start to flap...
Title: Re: WAN interface flapping with 22.1.2
Post by: bugvito on April 19, 2022, 07:55:51 pm
In my case it's really only an issue on Intel NIC's... switched my WAN to the onboard Realtek NIC and all is working now (mac spoofing with IDP on in IPS mode, OPNsense 22.1.6)

Is there any kernel or driver fix underway?

What would be your intel NIC model? I'm having WAN issues (spoofing MACs on WAN vlans) when upgrading from 22.1.2 to 22.1.5-6 with the dreaded i225, but I'm not clear at this point the actual issue.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on April 19, 2022, 08:08:06 pm
Code: [Select]
sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.igb.5.%desc: Intel(R) I211 (Copper)
dev.igb.4.%desc: Intel(R) I211 (Copper)
dev.igb.3.%desc: Intel(R) I211 (Copper)
dev.igb.2.%desc: Intel(R) I211 (Copper)
dev.igb.1.%desc: Intel(R) I211 (Copper)
dev.igb.0.%desc: Intel(R) I211 (Copper)

sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.igb.3.%desc: Intel(R) I210 Flashless (Copper)
dev.igb.2.%desc: Intel(R) I210 Flashless (Copper)
dev.igb.1.%desc: Intel(R) I210 Flashless (Copper)
dev.igb.0.%desc: Intel(R) I210 Flashless (Copper)
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on April 20, 2022, 07:14:35 am
I have this issue on a X710 and on another system with I225... and no issues on a test VM (unraid/KVM virtio NIC) and also no issue on some old crappy realtek onboard NIC.
Title: Re: WAN interface flapping with 22.1.2
Post by: foxmanb on April 21, 2022, 01:02:09 am
@Franco any update on this one? Will we see it resolved in a future release (soon hopefully?).
Title: Re: WAN interface flapping with 22.1.2
Post by: devhunter55 on April 21, 2022, 09:49:35 am
@Franco - yes - we would all appreciate if this could fixed soon  ;)

(knocked out since 22.1.1 (with UNBOUND 1.15.0))
Title: Re: WAN interface flapping with 22.1.2
Post by: bugvito on April 24, 2022, 08:45:11 pm
I have this issue on a X710 and on another system with I225... and no issues on a test VM (unraid/KVM virtio NIC) and also no issue on some old crappy realtek onboard NIC.

Same here; my VM with VirtIO is fine for WAN with VLANs and MAC spoofing (2 USB dongles on proxmox, AQC111U), my bare metal with I225 has issues with this WAN+vlan+spoof setup.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on April 24, 2022, 09:43:52 pm
Everyone having this issue seems to be using intel nics, so maybe this is just a FreeBSD 13 driver issue? Anyone have a howto to install the updated intel drivers that are included in FreeBSD 13.1?
Title: Re: WAN interface flapping with 22.1.2
Post by: devhunter55 on April 24, 2022, 10:29:03 pm
this may be true in my case, too (but i think - as long we don't know the reason for - it is an assumption):

root@opnsense:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'

dev.igb.5.%desc: Intel(R) I210 (Copper)
dev.igb.4.%desc: Intel(R) I210 (Copper)
dev.igb.3.%desc: Intel(R) I210 (Copper)
dev.igb.2.%desc: Intel(R) I210 (Copper)
dev.igb.1.%desc: Intel(R) I210 (Copper)
dev.igb.0.%desc: Intel(R) I210 (Copper)


---------------------------------

Intel® Ethernet Controller I210
I210 controllers support speeds up to 1GbE on a single port with advanced features such as Audio-Video Bridging (AVB), IEEE 802.1AS precision timestamping, Error Correcting Code (ECC) Packet Buffers, and Enhanced Management Interface options.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on April 24, 2022, 11:01:39 pm
A quick google search turns up a past issue between the intel igb driver/freebsd and people using coreboot bios... I do use coreboot, anyone else?  We need to find some sort of commonality for @franco and team to replicate. Nothing has changed in my setups with the exception of upgrading from 21 to 22.

My setup (multiple locations that have this problem )
- Dual WAN (Comcast/Xfinity [Primary] + Satellite [starlink or hughesnet] )  Some are static IP others are DHCP
- WAN setup for failover
- Surricata on the Lan
- Intel igb210 & igb211
- No MAC spoofing
- Multiple vLans on LAN
- Multiple OpenVPN site to site tunnels
- Wiregaurd (go) on primary WAN
- Protectli FW4 + FW6 boxes running Coreboot Bios
- Gateway monitoring ON
- MDNS Repeater on LAN/vLAN
- Wake on LAN on LAN
- No IPv6
- PowerD enabled w/ hiadaptive
- Monit Gateway alerting enabled
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on April 25, 2022, 08:12:44 am
Yes it is an assumption that it is Intel NIC only... but I can see here issue reports for Intel only (drivers em, igb, igc, ix and ixl) together with MAC spoofing and there is an earlier post here fixing this issue with updated Intel em drivers:

I had the WAN flapping issue with mac spoofing and have resolved by installing the updated FreeBSD 13 em driver, though I don't know if you have intel nics as well?
To generate the driver file I spun up a FreeBSD vm then pkg search intel-em-mod and install. Copied the if_em_updated.ko driver to /boot/modules/ as per Franco's reply in this post https://forum.opnsense.org/index.php?topic=20905.0.
I also disabled suricata on the wan interface and turned off flow control on all NICs.
Now running on a non flapping OPNsense 22.1.3 with WAN DHCP and mac spoofing.
@Franco - can this driver be added to OPNsense as it seems to resolve a number of stability issues.

I see 2 main issue (at the moment with intel drivers only since 22.1.2):
- VLAN's + MAC spoofing
- IDP with IPS mode + MAC spoofing

By the way, here easy reproducable (https://forum.opnsense.org/index.php?topic=26672.0 (https://forum.opnsense.org/index.php?topic=26672.0)):
- new clean OPNsense 22.1.2 or higher (on Intel NIC only?)
- enter some spoofed MAC
- enable IDP with IPS and wait 2 minutes until Suricata is fully loaded
=> link is flapping with DOWN/UP messages on monitor console
- delete spoofed MAC from this interface and hit "save"
- DOWN/UP monitor console messages immediately disappears ("Apply" not yet pressed!)
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on April 27, 2022, 03:19:12 pm
@subivoodoo Except I am having this issue and I don't use any MAC Spoofing.
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on April 27, 2022, 07:48:33 pm
@tracerrx different or additional issue than all the others here?
Title: Re: WAN interface flapping with 22.1.2
Post by: Grossartig on April 29, 2022, 05:52:31 pm
It was pointed out to me that my issue (https://forum.opnsense.org/index.php?topic=28158.0 (https://forum.opnsense.org/index.php?topic=28158.0)) which I had reported separately a few hours ago may be the same as the one discussed in this thread.

To which I want to add that my box is not using coreboot but AMI, and it's using Realtek Ethernet controllers. Also, disabling IPS seems to allow my box to obtain a WAN IP again (but unsure for how long -- currently testing here). Also, no MAC spoofing, suricata was configured on WAN (not LAN), no VLANs, no IPv6. Also, only a single LAN, no multi WAN here.

More system details here, for completeness (network controller details at bottom):

Code: [Select]
# pciconf -lv
hostb0@pci0:0:0:0: class=0x060000 rev=0x0b hdr=0x00 vendor=0x8086 device=0x5af0 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:2:0: class=0x030000 rev=0x0b hdr=0x00 vendor=0x8086 device=0x5a85 subvendor=0x19da subdevice=0xb325
    vendor     = 'Intel Corporation'
    device     = 'HD Graphics 500'
    class      = display
    subclass   = VGA
hdac0@pci0:0:14:0: class=0x040300 rev=0x0b hdr=0x00 vendor=0x8086 device=0x5a98 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series Audio Cluster'
    class      = multimedia
    subclass   = HDA
none0@pci0:0:15:0: class=0x078000 rev=0x0b hdr=0x00 vendor=0x8086 device=0x5a9a subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series Trusted Execution Engine'
    class      = simple comms
ahci0@pci0:0:18:0: class=0x010601 rev=0x0b hdr=0x00 vendor=0x8086 device=0x5ae3 subvendor=0x19da subdevice=0xb325
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller'
    class      = mass storage
    subclass   = SATA
pcib1@pci0:0:19:0: class=0x060400 rev=0xfb hdr=0x01 vendor=0x8086 device=0x5ad8 subvendor=0x19da subdevice=0xb325
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:19:2: class=0x060400 rev=0xfb hdr=0x01 vendor=0x8086 device=0x5ada subvendor=0x19da subdevice=0xb325
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:19:3: class=0x060400 rev=0xfb hdr=0x01 vendor=0x8086 device=0x5adb subvendor=0x19da subdevice=0xb325
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:20:0: class=0x060400 rev=0xfb hdr=0x01 vendor=0x8086 device=0x5ad7 subvendor=0x19da subdevice=0xb325
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port B'
    class      = bridge
    subclass   = PCI-PCI
xhci0@pci0:0:21:0: class=0x0c0330 rev=0x0b hdr=0x00 vendor=0x8086 device=0x5aa8 subvendor=0x19da subdevice=0xb325
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI'
    class      = serial bus
    subclass   = USB
sdhci_pci0@pci0:0:28:0: class=0x080501 rev=0x0b hdr=0x00 vendor=0x8086 device=0x5acc subvendor=0x19da subdevice=0xb325
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series eMMC Controller'
    class      = base peripheral
    subclass   = SD host controller
sdhci_pci1@pci0:0:30:0: class=0x080501 rev=0x0b hdr=0x00 vendor=0x8086 device=0x5ad0 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series SDIO Controller'
    class      = base peripheral
    subclass   = SD host controller
isab0@pci0:0:31:0: class=0x060100 rev=0x0b hdr=0x00 vendor=0x8086 device=0x5ae8 subvendor=0x19da subdevice=0xb325
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series Low Pin Count Interface'
    class      = bridge
    subclass   = PCI-ISA
none1@pci0:0:31:1: class=0x0c0500 rev=0x0b hdr=0x00 vendor=0x8086 device=0x5ad4 subvendor=0x19da subdevice=0xb325
    vendor     = 'Intel Corporation'
    device     = 'Celeron N3350/Pentium N4200/Atom E3900 Series SMBus Controller'
    class      = serial bus
    subclass   = SMBus
iwm0@pci0:1:0:0: class=0x028000 rev=0x81 hdr=0x00 vendor=0x8086 device=0x3165 subvendor=0x8086 subdevice=0x4010
    vendor     = 'Intel Corporation'
    device     = 'Wireless 3165'
    class      = network
re0@pci0:2:0:0: class=0x020000 rev=0x0c hdr=0x00 vendor=0x10ec device=0x8168 subvendor=0x10ec subdevice=0x0123
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
re1@pci0:4:0:0: class=0x020000 rev=0x0c hdr=0x00 vendor=0x10ec device=0x8168 subvendor=0x10ec subdevice=0x0123
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on April 29, 2022, 06:02:00 pm
@subivoodoo Sam issues as the others here... dpinger will state that the gateway is down, remove it from the fail-over group from 10  - 120 seconds, then put it back.  Happens repeatedly until you go in and change the gateway monitor IP, then it stabilizes for 3-4 days.  Rinse repeat.  However when dpinger says gateway is down, it is in fact not down.  I have increased all my thresholds but problem still occurs.
Title: Re: WAN interface flapping with 22.1.2
Post by: Edwin70 on May 04, 2022, 12:08:42 pm
Just to keep this issue under attention a small update from my side. Checkup on losing WAN connection for the last couple of days:
Code: [Select]
2022-04-29T02:35:36 Notice configd.py [9a0eae11-df9a-417b-b714-eb723d44fd0a] Linkup stopping igb0
2022-04-27T09:05:55 Notice configd.py [64f504d3-3e5d-4b62-9976-f2316296d9d9] Linkup stopping igb0
2022-04-25T01:19:10 Notice configd.py [e164f86e-bc77-426c-886c-d01f40b3da50] Linkup stopping igb0
So still every couple of days, for no reason, the WAN connection is lost. MAC spoofing is off and no IPS. Current OPSense version: 22.1.6 (amd64) on Protectli FW4B with Intel NICs.
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on May 04, 2022, 01:40:17 pm
Note that configd action for "Linkup stopping igb0" is called by devd in the operating system, likely due to reacting to either a hardware or software event detaching the interface. We can't do much about hardware flapping, and for software flapping there is only netmap responsible either by IPS mode intrusion detection or zenamor. The software flap may also be introduced by intrusion detection rules update. ;)


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 04, 2022, 02:28:25 pm
@franco It seems to be related to dpinger. 

All my links have a primary low latency connection, and a secondary satellite connection that's higher latency (starlink 40-100ms or hughesnet 700-1500ms).  Even though I have accounted for the higher latency in the gateway monitor settings for each satellite connection, the high latency on the secondary link seems to be making dpinger miss responses on the low latency primary link.  This did NOT occur in the 21.x series, and was introduced in 22.x (FreeBSD 13).  Disabling gateway monitoring on the secondary connections seems to resolve the flapping of the primary link.

FWIW, these are all Protectli FW4 and FW6, no mac spoofing, Suricata on LAN, Dual WAN
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on May 04, 2022, 02:32:26 pm
But dpinger won't do a linkdown/up as far as I know.

Latency is tricky and needs to be accounted for in advanced monitoring settings per gateway. Gateway monitoring can provide false-positives indeed.


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 04, 2022, 02:47:18 pm
@franco I understand the latency and the settings below have always worked in the past for hughesnet/viasat connection monitoring.  But previous to 22.x the high latency on wan2 never caused wan1 to be removed from the wan group. 

Latency thresholds: 1000 - 2500
Packet Loss thresholds: 30 - 60
Probe Interval: 15
Alert Interval: 60
Time Period: 60
Loss Interval: 4
Title: Re: WAN interface flapping with 22.1.2
Post by: Edwin70 on May 04, 2022, 03:20:41 pm
For me anyway no IPS or Zenarmor.

If the OS is the problem, it is probably a driver issue which was introduced in OPNSense 22 with the introduction of FreeBSD 13. In 21 series I also had no problem. This driver issue has been mentioned before.
Any news on a new release with updated drivers?
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 04, 2022, 03:35:43 pm
@edwin70 I believe the updated drivers are only for the intel em not intel igb
Title: Re: WAN interface flapping with 22.1.2
Post by: Edwin70 on May 04, 2022, 03:59:13 pm
@tracerrx Thanks for the info. I did not know that. In that case I wait for other solutions to come. Hopefully soon, as I'm back on the 21.x version. :(
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 05, 2022, 09:52:15 pm
@edwin70 Don't tell my wife but I was wrong.. Just looked it up.. em drivers do support i211 and i210 in freebsd 13...

Code: [Select]

The em(4) driver supports Gigabit Ethernet adapters based on the Intel 82540, 82541ER, 82541PI, 82542, 82543, 82544, 82545, 82546, 82546EB, 82546GB, 82547, 82571, 82572, 82573, 82574, 82575, 82576, and 82580 controller chips:

    Intel Gigabit ET Dual Port Server Adapter (82576)

    Intel Gigabit VT Quad Port Server Adapter (82575)

    Intel Single, Dual and Quad Gigabit Ethernet Controller (82580)

    Intel i210 and i211 Gigabit Ethernet Controller

    Intel i350 and i354 Gigabit Ethernet Controller

    Intel PRO/1000 CT Network Connection (82547)

    Intel PRO/1000 F Server Adapter (82543)

    Intel PRO/1000 Gigabit Server Adapter (82542)

    Intel PRO/1000 GT Desktop Adapter (82541PI)

    Intel PRO/1000 MF Dual Port Server Adapter (82546)

    Intel PRO/1000 MF Server Adapter (82545)

    Intel PRO/1000 MF Server Adapter (LX) (82545)

    Intel PRO/1000 MT Desktop Adapter (82540)

    Intel PRO/1000 MT Desktop Adapter (82541)

    Intel PRO/1000 MT Dual Port Server Adapter (82546)

    Intel PRO/1000 MT Quad Port Server Adapter (82546EB)

    Intel PRO/1000 MT Server Adapter (82545)

    Intel PRO/1000 PF Dual Port Server Adapter (82571)

    Intel PRO/1000 PF Quad Port Server Adapter (82571)

    Intel PRO/1000 PF Server Adapter (82572)

    Intel PRO/1000 PT Desktop Adapter (82572)

    Intel PRO/1000 PT Dual Port Server Adapter (82571)

    Intel PRO/1000 PT Quad Port Server Adapter (82571)

    Intel PRO/1000 PT Server Adapter (82572)

    Intel PRO/1000 T Desktop Adapter (82544)

    Intel PRO/1000 T Server Adapter (82543)

    Intel PRO/1000 XF Server Adapter (82544)

    Intel PRO/1000 XT Server Adapter (82544)

Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 05, 2022, 10:35:15 pm
Looks like the intel em driver currently included is 7.6.1, but most recent intel in 7.7.8

Code: [Select]
dev.igb.0.iflib.driver_version: 7.6.1-k
However if we compile our own we would have to re-compile every time there is a kernel update I believe.
Title: Re: WAN interface flapping with 22.1.2
Post by: Edwin70 on May 06, 2022, 10:27:11 am
@edwin70 Don't tell my wife but I was wrong.. Just looked it up.. em drivers do support i211 and i210 in freebsd 13...

Don’t worry, I won’t tell. Thank you for looking into it. It gives me hope this issue might be resolved with the drivers. Although there might be more to it.
Title: Re: WAN interface flapping with 22.1.2
Post by: rum on May 06, 2022, 08:05:46 pm

When can we expect a release with the current drivers?
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 07, 2022, 12:28:19 am
@rum you can't the drivers are part of the FreeBSD kernel... I'm sure that they have back ported any security patches into the current drivers.  To update the drivers either the freebsd team would need to make the changes, or you need to manually re-compile and re-install the drivers.

Unfortunately I believe you have to re-compile every time the freebsd kernel is updated. What the opnsense team has done in the past with realtek drivers was make the updated drivers a plugin, so they did all the re-compilation on the back end for us.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 07, 2022, 05:34:41 pm
@franco can we just
Code: [Select]
pkg install intel-em-kmod and then set if_em_updated_load=YES?

Title: Re: WAN interface flapping with 22.1.2
Post by: franco on May 09, 2022, 04:21:25 pm
We don't carry the driver in our repo but basically that's what you have to do, yes.


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 09, 2022, 04:36:01 pm
For anyone looking for this .. most of the protectli devices use the intel IGB drivers (Specifically i210/i211 NIC's).... This has fixed the majority of the WAN flapping issues I was having...Note that these instructions are a little different from those listed in other posts referencing the Intel EM drivers.

From the command line of your opnsense box:
Code: [Select]
pkg install git
pkg install wget
cd /usr
git clone https://github.com/opnsense/plugins
git clone https://github.com/opnsense/ports
git clone https://github.com/opnsense/src
cd src/
git checkout
git checkout stable/22.1
cd /tmp
wget https://downloadmirror.intel.com/682705/igb-2.5.21.tar.gz
tar xzf igb-2.5.21.tar.gz
cd igb-2.5.21/src
make
cp if_igb.ko /boot/modules/if_igb_updated.ko

From the opnsense GUI:
System=>Settings=>Tunables
Code: [Select]
Tunable => if_igb_updated_load
Value => YES

You need to reboot the opnsense box for the changes to take effect.. afterwords when you run
Code: [Select]
sysctl -a | grep dev.igb you should see the new driver version
Title: Re: WAN interface flapping with 22.1.2
Post by: Edwin70 on May 09, 2022, 06:01:42 pm
@tracerrx Again, thank for taking the time to dive into this issue. I'm a bit confused. You suggested two possible routes to update the driver.

I presume it is the extended command line steps I have to do?

And @franco There drivers in the release you use. Why can't you update them?
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 09, 2022, 07:43:51 pm
@edwin70 if you have a protectli, you cannot use the pkg install method because those are intel EM drivers, you need intel IGB drivers and I don;t see a package for intel IGB.

The team at opnsense could make these drivers a plugin if they wanted, however it was not the opnsense team that broke them.. Either the drivers regressed between freebsd v12 and v13, or there were updated/patched drivers included in hardened bsd 12.. They were never included in opnsense
Title: Re: WAN interface flapping with 22.1.2
Post by: Edwin70 on May 10, 2022, 11:32:53 am
@tracerrx It is a bit above my "nerd-level", but I will give it a go. And I presume that every time there is an OPNSense update, I have to do it again. It probably depends, but just to be on the safe side.
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on May 10, 2022, 12:12:31 pm
We don't want to diverge from FreeBSD unless we really have to. It could be that the default Intel driver even with the version currently used in FreeBSD 13 would work fine and we are looking at something that changed in FreeBSD in particular, not in the Intel driver.

Maintaining and ensuring compatibility of drivers for the same drivers already present in FreeBSD base is rather difficult, see the whole Realtek dilemma over the years... It's a lot of work to put up with both sides.


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: peter.vynck on May 11, 2022, 11:10:40 am
Just to clarify: this 'issue' is not solved in release 22.1.7.

Meaning that when you spoof the actual physically present MAC your interface will have strange behaviour.
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on May 11, 2022, 12:12:47 pm
Since the kernel last changed in 22.1.5 changes in 22.1.6 and 22.1.7 are unlikely. That also goes for later 22.1.x unless we get a clarification to which FreeBSD upstream commit we should actually include. I'm open for ideas.


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: peter.vynck on May 11, 2022, 05:15:27 pm
I am just a simple user and have idea about upstream commits.
But the logic behind it seems pretty simple: if the spoofed MAC address is the same as the physical MAC address just ignore the spoofed address.
Title: Re: WAN interface flapping with 22.1.2
Post by: bugvito on May 11, 2022, 07:55:05 pm
Is moving to 13.1 a possibility for 22.7? Assuming that this issue would be resolved?
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 11, 2022, 08:09:50 pm
If something is fixed in 13.1, that's not currently in 13, let @franco know which commit (That's what he is asking above). If you search through the recent commits to FreeBSD 13.1 https://github.com/freebsd/freebsd-src/commits/releng/13.1 (https://github.com/freebsd/freebsd-src/commits/releng/13.1) there are a quite a few that mention ethernet drivers and/or MAC addresses.  But I don't see anything that looks like it effects the particular issues we've been experiencing in this thread.
Title: Re: WAN interface flapping with 22.1.2
Post by: atxx on May 13, 2022, 03:47:07 pm
I registered in order to note that I'm experiencing this issue on Broadcom BCM5720 (Microserver Gen8) therefore this is probably not Intel specific as speculated.

I updated to 22.1.6 three days ago and my WAN interface started flapping next day.
After a restart it was fine, until two days later (today) when the flapping started happening again.
Issue still present on 22.1.7_1.

IPS is on, mac spoofing is active, and gateway monitor is disabled. ifdown/up doesn't solve the issue, I need to restart the device for this to get resolved; this issue is quite disruptive.

I do feel obliged to say that Opnsense has pretty much been smooth sailing up until now. Thank you for your good work!
Title: Re: WAN interface flapping with 22.1.2
Post by: Grossartig on May 13, 2022, 04:37:26 pm
I updated to 22.1.6 three days ago and my WAN interface started flapping next day.
After a restart it was fine, until two days later (today) when the flapping started happening again.
Issue still present on 22.1.7_1.

IPS is on, mac spoofing is active, and gateway monitor is disabled. ifdown/up doesn't solve the issue, I need to restart the device for this to get resolved; this issue is quite disruptive.

Can you see if it's more stable when IPS is disabled? I had to disable it on my end for WAN to stabilize.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 13, 2022, 04:54:32 pm
IPS is on, mac spoofing is active, and gateway monitor is disabled. ifdown/up doesn't solve the issue, I need to restart the device for this to get resolved; this issue is quite disruptive.

Are you running IPS on the LAN or WAN?
Title: Re: WAN interface flapping with 22.1.2
Post by: atxx on May 13, 2022, 05:34:07 pm
@Grossartig After I posted, I contacted my ISP and gave them my opnsense device's MAC address therefore I turned mac spoofing off. If the issue appears again I'll try turning IPS off and report back.

@tracerrx WAN
Title: Re: WAN interface flapping with 22.1.2
Post by: Vesalius on May 13, 2022, 06:16:51 pm
Is moving to 13.1 a possibility for 22.7? Assuming that this issue would be resolved?
using 13.1 for 22.7 is the plan regardless.

https://forum.opnsense.org/index.php?topic=28302.msg137463#msg137463
Title: Re: WAN interface flapping with 22.1.2
Post by: bugvito on May 13, 2022, 06:55:35 pm
Good find Vesalius,

I had time to test a bit last night, and the steps posted to compile the IGB drivers did solve my WAN problems. I have another issue so my testing only lasted about 30-60 minutes, but without any WAN issues.

I will remove the tunable setting specifying the compiled IGB driver, and potentially test 22.7 if all goes well.

None of this helps identifying/resolve the real cause, unfortunately.

Thanks!
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on May 13, 2022, 07:11:49 pm
I did a quick test with 22.7.pre3 as described in this post... it doesn't seem to work for me  :'( (MAC spoofing + IPS).

Console screenshots attached... before and after the same issue.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 13, 2022, 08:33:29 pm
@bugvito you will need to re-compile the IGB driver again for 22.7 (and for every kernel update).  What brand hardware are you using?
Title: Re: WAN interface flapping with 22.1.2
Post by: Dantichrist on May 14, 2022, 06:54:01 am
@tracerrx

I finally got some time to play with updating the NIC driver, and it works like a charm. I've had the MAC spoofed for about 12 hours now with no issues at all. Before the interface would drop within 15 to 30 min.

I just wanted to thank you for posting a fix!
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 14, 2022, 07:08:17 am
@dantichrist you will need to re-compile the IGB driver again for 22.7 (and for every kernel update there after).  What brand hardware are you using?
Title: Re: WAN interface flapping with 22.1.2
Post by: Dantichrist on May 14, 2022, 09:17:43 am
@tracerrx It's just an old i7-4790 box that I'm using. I'm using a dual intel 82576 (igb) card in it/bare metal. No vlans or anything fancy but I am running suricata/zenarmor. I'm planning on doing this after each update until the driver in BSD is sorted out. Now I just have to get os-ddclient working with godaddy and I'll be all set.  :)
Title: Re: WAN interface flapping with 22.1.2
Post by: firewall on May 14, 2022, 11:30:59 am
after a pleasant 12 or so hours with the driver from intel i'm right back to square one--and its seemingly worse than before. also a pleasant few years with opnsense absolutely rekt by 2 months of unreliable connectivity.
Title: Re: WAN interface flapping with 22.1.2
Post by: bugvito on May 14, 2022, 01:49:44 pm
@bugvito you will need to re-compile the IGB driver again for 22.7 (and for every kernel update).  What brand hardware are you using?

I'm aware. I was wondering if I could find a way to execute a command on a new kernel install event to recompile the driver. With that said, the driver still loaded and worked fine under 22.7pre3 without recompiling, but is a lucky/risky thing to do/try. This would still be an unviable solution, as this may cause problems down the road.

I did disable my compiled igd driver (confirmed not loaded with kldstat) and updated to 22.7.pre3, and to my pleasant surprise, everything is working fine for me (after 1h).
The test box is a cheap HSUNG RS34g with 4xi225 (I believe that the board itself is popular: 1090np-12).

Unfortunately this does not help identifying the real issue at hand for this topic, and as some already reported, 22.7 did not resolve their issue, while the original issue may not be intel nic specific.

Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 14, 2022, 03:51:47 pm
@bugvito you should only need to re-compile the driver when you get a kernel update. I would not disable the built in igb driver, if you want to stop using the newly compiled driver just change your tunable to "if_igb_updated_load=NO"
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on May 19, 2022, 08:30:37 am
Another test on my main rig with manually compiled new Intel driver for ixl => it does not work for me...

MAC spoofing + IPS + Intel ixl drivers from march 2022 = if down/up
Title: Re: WAN interface flapping with 22.1.2
Post by: Edwin70 on May 20, 2022, 11:27:42 am
As there is still no definite cause for this issue, let alone a solution, I’m still running on 21.7.

Are there any security issues in the 21.7 release I should be aware off? I run a basic setup, but with a WireGuard VPN running.
Title: Re: WAN interface flapping with 22.1.2
Post by: atxx on May 21, 2022, 02:05:02 pm
Continuing from my last post, removing mac spoofing (after coordinating with the ISP), and disabling IPS does not solve the issue. Reverting to 22.1.1 does not solve the issue either. Reminder that my controller is a Broadcom BCM5720. I'm at a loss and not sure what to do as I've spent countless hours trying to solve this after upgrading to 22.1.7_1. My WAN interface keeps flapping every few minutes/hours. Any advice is welcome.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 21, 2022, 04:29:11 pm
@atxx is the gateway stable under 21.x?
Title: Re: WAN interface flapping with 22.1.2
Post by: atxx on May 21, 2022, 11:36:05 pm
@atxx is the gateway stable under 21.x?
I'll try downgrading and get back to you. I've had issues with 22.1.1 on the LAN side after reverting - vlan interfaces appeared as "down" (GUI) and were missing when running ifconfig -a. Any ideas how I can resolve that in order to downgrade to 21.x?

//edit: I ended up starting again from scratch, from 21.7.8. So far so good, I'll report back soon.

//edit2: Everything was stable on 21.7.8 for almost 24h, then I tried mac spoofing (to test something with the ISP) and the WAN interface started constantly flapping, again disrupting everything. Turning it off & restarting didn't resolve the issues. I noticed that the port was negotiating at 100BASE-TX and after cleaning all cable and port contacts with alcohol and compressed air, the interface is stable again and negotiating gbit speeds. I want to upgrade to 22.x just to check that the issue was with the cables/ports at some point.
Title: Re: WAN interface flapping with 22.1.2
Post by: firewall on May 31, 2022, 05:48:27 am
there are many threads both here and reddit of people reporting wan connectivity issues--all of which likely related to the same issue but perhaps misunderstood / mischaracterized. has this actually been researched by opnsense devs vs. outright disregarded as a "possible issue with intel drivers"? certainly i'm not the only person who feels like opnsense bug reports are sometimes dropped in the same way packets are with pf..
Title: Re: WAN interface flapping with 22.1.2
Post by: devhunter55 on May 31, 2022, 11:29:50 am
After running into these issues, i did revert to OPNsense 22.1.1_3.

This version is running very stable - no flapping interfaces anymore.

I'm thinking to wait for a more stable Freebsd version with running Intel drivers.
May be with FreeBSD 13.1 or/and Opnsense 23.1.x ?
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on May 31, 2022, 04:45:42 pm
FWIW, the intel IGB drivers do survive the 22.1.8_1 update without having to be re-installed/compiled.
Title: Re: WAN interface flapping with 22.1.2
Post by: devhunter55 on May 31, 2022, 08:48:32 pm
@tracerrx - that's a very good Information, indeed - many thx for it !
Title: Re: WAN interface flapping with 22.1.2
Post by: foxmanb on June 03, 2022, 03:47:33 pm
I finally dug into this a bit and was able to resolve my issue.

I am an xfinity cable sub and was always under the impression that I had to use a cloned MAC address on my WAN connection.

Today I removed the cloned MAC, powered everything down, and rebooted. System picked up a new WAN IP address because my MAC changed to the MAC on my NIC. My assumption of having to use a cloned MAC was incorrect, I had been doing that for the last 15 years...

It was successful. Once I did this, I upgraded to the latest and greatest version and it is stable.
Currently running OPNsense 22.1.8_1 (amd64/OpenSSL) at Fri Jun  3 09:40:31 EDT 2022

IDS is enabled and there is no flap.
Title: Re: WAN interface flapping with 22.1.2
Post by: firewall on June 04, 2022, 08:06:38 am
It was successful. Once I did this, I upgraded to the latest and greatest version and it is stable.
Currently running OPNsense 22.1.8_1 (amd64/OpenSSL) at Fri Jun  3 09:40:31 EDT 2022

IDS is enabled and there is no flap.

give it time
Title: Re: WAN interface flapping with 22.1.2
Post by: foxmanb on June 04, 2022, 05:34:52 pm
It was successful. Once I did this, I upgraded to the latest and greatest version and it is stable.
Currently running OPNsense 22.1.8_1 (amd64/OpenSSL) at Fri Jun  3 09:40:31 EDT 2022

IDS is enabled and there is no flap.

give it time

So far so good.
Title: Re: WAN interface flapping with 22.1.2
Post by: TheeDude on June 10, 2022, 03:48:00 am
I was having theese "flapping" issues also when I upgraded to 22.1.2, and were forced to downgrade to 22.1.1_3.
This version works great.

I have a Qotom machine, with intel network cards, and I am also using mac spoofing.
What do you think, is it safe upgrading to 22.7 when it launches?
Is this issue resolved with FreeBSD 13.1?
I did notice that there is a testing thread, they didn´t seem to talk about this issue so much..

Title: Re: WAN interface flapping with 22.1.2
Post by: Supermule on June 10, 2022, 10:21:12 am
You could test if its the Intel drivers thats the cause by running the troubled versions in a VM instead of bare metal.

Then it would be obvious where the culprit is located.
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 11, 2022, 02:08:50 am
 Mine  is doing the same thing but why only on the wan interface with intel in particular? My lan and wlan are also on igb nics but they do not do this. why is the interface cycling only on the WAN?

 If you want to monitor, here are a few examples, just grep which nic you wish to watch.   

tail -f /var/log/configd/latest.log | grep em0

tail -f /var/log/configd/latest.log | grep igb3

 You get the idea.

Edit: Found out exactly why, it was DNS overlaps between the two WANS using the same DNS servers (Google) and two Gateways using one of each of the two Google DNS servers. I have since changed it so each Gateway has four DNS servers that are NOT Google and in the Gateway monitoring, used two completely different DNS servers that are not duplicated in the DNS settings so absolutely nothing matches. Of course for the Gateway monitoring you can use anything that allows pinging, even the default ISP gateway.
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 11, 2022, 03:01:20 am
I registered in order to note that I'm experiencing this issue on Broadcom BCM5720 (Microserver Gen8) therefore this is probably not Intel specific as speculated.

I do feel obliged to say that Opnsense has pretty much been smooth sailing up until now. Thank you for your good work!

This speaks volumes. It would suggest that it is not an Intel issue at all. Notice that LAN, WLAN etc don't cycle even if using  Intel nics? My WAN2 interface which I had to create from an OPT and rename it does not do it either.

Edit: Back to using the built in WAN, it was DNS overlaps between two WANS using the same DNS servers (Google) and two Gateways using one of each of the two Google DNS. I've decided I don't want any of my traffic going through Google. Considering making my own DNS servers and synchronizing them often.
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 11, 2022, 03:40:23 am
I am curious, rc.newwanip was inherited from pfsense but it was changed in may 2022, last month of course. As far as I know, this is only used on the wan.

Edit: This wasn't related to the problem at all, it was DNS overlaps between two WANS using the same DNS servers (Google) and two Gateways using one of each of the two Google DNS. I've decided I don't want any of my traffic going through Google.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on June 11, 2022, 03:52:49 am
@Davesworld its possible it's not an intel only issue.. However installing the newer intel IGB drivers solved the issue for me and others. 
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on June 11, 2022, 03:56:34 am
@davesworld  It looks like there have been a lot of changes to rc.newwanip since pfsense see: https://github.com/opnsense/core/commits/master/src/etc/rc.newwanip

edit.. Disregard..the above.. apparently i'm sleepy or cant read...
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 11, 2022, 04:16:59 am
@Davesworld its possible it's not an intel only issue.. However installing the newer intel IGB drivers solved the issue for me and others.

 How long has it been stable? The reason I ask is I have more than one WAN and the other WANS do not cycle with the stock kernel intel drivers. Only the built in WAN that is preselected like LAN is, uses rc.newwanip. It's the only interface that uses it. Why is it even needed? When you add new WAN interfaces, they are OPT interfaces and one can name them WAN2 WAN3 etc. Since I have several static IPs, I added yet a third WAN and used one of my unused static IPs. My default WAN still goes up and down none of the other interfaces including lan, also using the stock intel driver, never go down. Someone is also having the same problem with a broadcom nic. I'm very surprised an out of kernel driver really fixed it for you and others if the others are as lucky as you.

 When creating another WAN, one has to click on the block private networks and block bogon networks. ALL traffic is blocked by the firewall by default unless you add rules allow anything in so other than that rc.newwanip it behaves exactly as the built in WAN and doesn't cycle. 

Edit: No new WAN creation needed, I'm back to the built in wan, it was DNS overlaps between the two WANS using the same DNS servers (Google) and two Gateways using one of each of the two Google DNS. I've decided I don't want any of my traffic going through Google.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on June 11, 2022, 04:26:02 am
This is interesting... I will try moving my primary comcast (static IP) on my protectli WAN to one of the opt interfaces, revert the drivers and see if I get flapping... Most of the protectli devices use Intel IGB drivers...
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 11, 2022, 05:24:38 am
This is interesting... I will try moving my primary comcast (static IP) on my protectli WAN to one of the opt interfaces, revert the drivers and see if I get flapping... Most of the protectli devices use Intel IGB drivers...

I created an interface called WAN_ALT and changed my gateway to that interface and disabled the WAN interface. Since WAN_ALT is also on my fiber link the gateway had priority of 1 and since that priority gateway is now on WAN_ALT, it started routing over WAN_ALT as soon as I disabled WAN. Just rememember to block bogons and non routable ips reserved for lans. I'm watching the dynamic logs.

Edit: No new WAN creation needed, I'm back to the built in wan, it was DNS overlaps between the two WANS using the same DNS servers (Google) and two Gateways using one of each of the two Google DNS. I am no longer using Goggle's DNS servers at the moment.
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 11, 2022, 11:26:19 am
The interface that is assigned to wan even though wan is disabled is still going up and down. It's on autopilot.

em0: link state changed to DOWN
em0: link state changed to UP
em0: link state changed to DOWN
em0: link state changed to UP
em0: link state changed to DOWN
em0: link state changed to UP
em0: link state changed to DOWN
em0: link state changed to UP
em0: link state changed to DOWN
em0: link state changed to UP
em0: link state changed to DOWN
em0: link state changed to UP

 But it can't hurt me due to my WAN_ALT. Yep, there's a problem in a recent update

Edit: No new WAN creation needed, I'm back to the built in wan, it was DNS overlaps between the two WANS using the same DNS servers (Google) and two Gateways using one of each of the two Google DNS as I have stated in other edits here.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on June 11, 2022, 03:27:32 pm
When it flaps, is it down for ~2 minutes?
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 11, 2022, 06:51:11 pm
When it flaps, is it down for ~2 minutes?

Just about that, sometimes longer.

root@thor:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.iflib.driver_version:'
dev.em.0.iflib.driver_version: 7.6.1-k
dev.igb.6.iflib.driver_version: 7.6.1-k
dev.igb.5.iflib.driver_version: 7.6.1-k
dev.igb.4.iflib.driver_version: 7.6.1-k
dev.igb.3.iflib.driver_version: 7.6.1-k
dev.igb.2.iflib.driver_version: 7.6.1-k
dev.igb.1.iflib.driver_version: 7.6.1-k
dev.igb.0.iflib.driver_version: 7.6.1-k

This driver hadn't been changed recently. This is the stock kernel driver.

Edit: No new WAN creation needed and definitely not the driver, I'm back to the built in wan, it was DNS overlaps between the two WANS using the same DNS servers (Google) and two Gateways using one of each of the two Google DNS as I have stated in other edits here. This was bad but I got a wakeup call for doing that. I'm avoiding Google's DNS servers for now and maybe forever.

Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 11, 2022, 08:02:42 pm
This is interesting... I will try moving my primary comcast (static IP) on my protectli WAN to one of the opt interfaces, revert the drivers and see if I get flapping... Most of the protectli devices use Intel IGB drivers...

 Most devices that are purpose built for a firewall/router use intel. I have never seen realtek.
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 11, 2022, 11:31:52 pm
My method of creating another wan is just a bandaid. Took Franco's advice about overlapping DNS entry. I had been using google DNS for gateway monitoring, and dns for each gateway as I have two wans. I made them all different. We'll see.
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 12, 2022, 05:55:47 am
Well in any case you seem to have overlapping DNS servers for the different interfaces, either set manually, by ISP or gateway monitor. In some cases ISPs push Google servers which is pretty mean since it also pins a route for it through their interface.


Cheers,
Franco

Mine was flapping too, my primary gateway. Since I have two WANS, the instructions you told someone else as far as DNS being set to none doesn't sound right for multiple wans with their own gateways. Each gateway I have two DNS addresses. Before I was using both google dns addresses on each gateway and then using one google dns for monitoring one gateway and the second google dns to monitor the second gateway so now I am back to using the real wan after having added a third one and moving my cable to it temporarily. Also I have completely different primary and secondary dns settings for each gateway. No two match now. If I understood you correctly, we can't have DNS's that are the same for two gateways and then using one of each to monitor the gateway?  I just set my gateways to just use their respective ISP gateway to monitor for now. If I understand you correctly on another note, avoid google dns. How am I doing so far?

 I did make a third WAN temporarily as I indicated above and moved my fiber(main internet connection) to it and it was rock solid. I am using the proper WAN again that is already in the distro with no dns overlaps and so far it's not flapping. I believe this is all documented somewhere but it's been a while. Am I correct in asserting that the the DNS entries should only be assigned gateways when there are more than one WAN? I never suspected the driver as others have in this thread as it's at least 4 years old and it didn't cause LAN and WLAN etc to flap if using all Intel nics and would not have just now started causing trouble. Sorry if too many questions at once.

Update, a day later and it's rock solid after removing the overlaps. I may have even went overboard but there are a lot of high quality DNS servers out there.
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 13, 2022, 11:02:35 pm
@Davesworld its possible it's not an intel only issue.. However installing the newer intel IGB drivers solved the issue for me and others.

 I have no way of knowing what other changes were made so I do not know if the driver could possibly have fixed their issue. If it was a driver issue you would also lose other igb interfaces which is clearly not the case. I have difficulty blaming a driver that we have used with rock hard stability since at least 2018. Out of kernel hardware drivers should be avoided if possible and it steals attention away from other issues.

 If only one interface using the same hardware as the other interfaces is cycling, there is most certainly another issue beside the driver, I discovered mine, it was DNS overlap plain and simple and it was right in front of me the whole time. It should be noted that some PFsense users have the same issues from time to time, even years back, and it usually ends up being the same misconfiguration. If you use Google DNS for gateway monitors and dns, it's a recipe for more flapping than the 1920s flapper craze. The possibility that some ISPs are routing traffic through google without telling us certainly doesn't help.

The reason why adding an extra wan also solved my problem is because the gateways and the DNS entries were no longer pointed at a specific gateway which I discovered later while employing Franco's advice about DNS overlap.

 It is certainly possible that a recent update caused the system to react to the misconfiguration that was ignored before.

 I'm sticking with the in kernel drivers as much as possible.
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on June 14, 2022, 12:43:27 am
I had this DNS overlap on 1 device originally, and fixing it definitely made the problem better, however i still got flapping on the wan every 2-3 days until I replaced the driver.  All of my primary WANs are Comcast (some residential DHCP  others business static), so it's possible that Comcast has made a change to their systems that's sending something funny.

Either way, shouldn't the Opnsense GUI prevent you from using overlapping DNS/Gateway Monitors to prevent this?  And why did this work on 21.x and not in 22.x if it's always been the case?
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on June 14, 2022, 08:03:16 am
> Either way, shouldn't the Opnsense GUI prevent you from using overlapping DNS/Gateway Monitors to prevent this?  And why did this work on 21.x and not in 22.x if it's always been the case?

In all of the years the code existed it never prevented it, it never attempted to create visibility for the situation either. There are 4 potential sources for static host routes which can all overlap.

22.1 cleaned up some of the undefined behaviour WRT which route wins which caused issues with people's setups, which formerly was last one configured wins but now it tries to deduplicate the host routes to be created to DNS servers and shows them in the GUI (Interfaces: Overview).

There is, however, still no larger picture or any structure in place that ties together static routes, ISP DNS servers, manual DNS servers and gateway monitor routes. The amount of work due to initial lack of design is the main reason for that. At least now in 22.1.x we have a new tool "ifctl" that registers DNS information from ISPs persistently and the dynamic address scripts don't try to flood the system with the routes that they have just gotten, which is now handled by the main DNS reload code in an orderly fashion, but still has no relation to implied gateway host routes and static routes set by the user.


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 14, 2022, 10:09:46 am
I had this DNS overlap on 1 device originally, and fixing it definitely made the problem better, however i still got flapping on the wan every 2-3 days until I replaced the driver.  All of my primary WANs are Comcast (some residential DHCP  others business static), so it's possible that Comcast has made a change to their systems that's sending something funny.

I would think that if there were a driver issue, the interface would fail and the interface would never come back up on it's own. I would want to see a kernel log entry that points to the driver causing anything. A driver failure is not easy if even possible to recover from without a reboot. They just don't automatically reload and bring back up the interface as far as I know.

Rather than throwing things at the problem, I created a diff file between the in kernel source and the latest BSD driver from Intel which is not that new either. The trouble is that the diff file is 1MB in size and I cannot attach it here. There is more in common between the two than there isn't.

Here is a snip that even contains the command I used:

diff -Naur /home/dave/src-release/13.1.0/sys/dev/e1000/e1000_80003es2lan.c /home/dave/em-7.7.8/src/e1000_80003es2lan.c
--- /home/dave/src-release/13.1.0/sys/dev/e1000/e1000_80003es2lan.c   2022-05-11 16:59:24.000000000 -0700
+++ /home/dave/em-7.7.8/src/e1000_80003es2lan.c   2020-04-08 08:13:17.000000000 -0700
@@ -1,32 +1,31 @@
 /******************************************************************************
-  SPDX-License-Identifier: BSD-3-Clause
 
-  Copyright (c) 2001-2020, Intel Corporation
+  Copyright (c) 2001-2019, Intel Corporation
   All rights reserved.

Yes, I know that the current kernel is 13.0 but the intel driver code even in 13.1 is dated from 2020 and that's just the copyright, the driver itself goes back much further.

The file is much too long to paste in here. If I could get the attachment size permission raised to 1MB I could attach it here. The + and - lines are what is added and subtracted to the old source source code in this instance to make the new driver. The in kernel driver is older than I thought so there should be no new issues with it. Even the next version newer than the in kernel version was released in 2016.
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on June 14, 2022, 11:19:55 am
If we want to throw diffs around maybe start with the most obvious:

Our stable/22.1 branch differences against the main FreeBSD branch with the latest and greatest code for the em(4) driver:

% git diff --stat upstream/main sys/dev/e1000 
 sys/dev/e1000/e1000_phy.c |  2 +-
 sys/dev/e1000/em_txrx.c   | 13 ++++++++-----
 sys/dev/e1000/if_em.c     | 32 +++++++++++++++-----------------
 sys/dev/e1000/igb_txrx.c  | 21 ++++++++++++---------
 4 files changed, 36 insertions(+), 32 deletions(-)

As such I doubt that the current driver situation gets much better than what we have with FreeBSD 13 right now and additional driver updates even from Intel are out of the question for direct release inclusion (kmod packages can be used but that's all there is).


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 14, 2022, 07:00:45 pm
If we want to throw diffs around maybe start with the most obvious:

Our stable/22.1 branch differences against the main FreeBSD branch with the latest and greatest code for the em(4) driver:

% git diff --stat upstream/main sys/dev/e1000 
 sys/dev/e1000/e1000_phy.c |  2 +-
 sys/dev/e1000/em_txrx.c   | 13 ++++++++-----
 sys/dev/e1000/if_em.c     | 32 +++++++++++++++-----------------
 sys/dev/e1000/igb_txrx.c  | 21 ++++++++++++---------
 4 files changed, 36 insertions(+), 32 deletions(-)

As such I doubt that the current driver situation gets much better than what we have with FreeBSD 13 right now and additional driver updates even from Intel are out of the question for direct release inclusion (kmod packages can be used but that's all there is).


Cheers,
Franco

 This is the most meaningful diff, thanks for that, I haven't git pulled them and ran a diff on them. Has there ever been consideration into using deltas? I know there are good reasons for doing a full download and good reasons for using a delta but the delta can only upgrade the most recent version before the update it provides. Just curious.

 As far as Intel goes, they have not updated their out of kernel driver source in two years and probably no need to. Nobody has shown me a log that points to the Intel or Broadcom driver (there was a reported flapping with Broadcom) when their wan flaps and if it was indicated, I just do not see how an interface would be able to bring itself back up once the kernel throws a driver error for that device so the driver would be the last place I would have looked. I've never seen a NIC module recover the hardware once a kernel error is thrown without recycling power to the nic which we have no way to do on a running system. All I have seen is igb up and igb down or em up or em down with zero kernel driver logs from anyone. My case was simply overlap caused by misconfiguration that now is rightfully caught by the upgraded system and WAN hasn't cycled a single time since.

Title: Re: WAN interface flapping with 22.1.2
Post by: firewall on June 15, 2022, 08:16:02 pm
> 22.1 cleaned up some of the undefined behaviour WRT which route wins which caused issues with people's setups, which formerly was last one configured wins but now it tries to deduplicate the host routes to be created to DNS servers and shows them in the GUI (Interfaces: Overview).

> At least now in 22.1.x we have a new tool "ifctl" that registers DNS information from ISPs persistently and the dynamic address scripts don't try to flood the system with the routes that they have just gotten, which is now handled by the main DNS reload code in an orderly fashion, but still has no relation to implied gateway host routes and static routes set by the user.

Given the lack of meaningful consistency amongst those reporting this flapping issue, along with the fact these problems were not reported prior to 22.1.x, is it possible that the above-referenced changes are the root cause? If so, and should it be straightforward to back them out with a patch, I'll gladly be a guinea pig.
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on June 15, 2022, 09:32:43 pm
You can test any 22.1.x and the initial 22.1 to see if something changed there (opnsense-revert). If not it would indicate the FreeBSD 13 kernel. To rule out changes before 22.1 release and after 21.7.8 use 21.7.8 and switch to development version which is the same core as 22.1 without the FreeBSD 13 kernel.

I'm still suspecting the kernel has a hand in this which makes it difficult to nail to some single change/component.

Going backwards on complex changes such as DNS registration behaviour is not easily possible due to larger code changes involved, but also not relevant given the ways to pin this down to a clear confirmation (is it core or is it kernel).


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 16, 2022, 06:19:59 am

I'm still suspecting the kernel has a hand in this which makes it difficult to nail to some single change/component.

Going backwards on complex changes such as DNS registration behaviour is not easily possible due to larger code changes involved, but also not relevant given the ways to pin this down to a clear confirmation (is it core or is it kernel).


Cheers,
Franco

 Well, the kernel is temporary. I wonder who else besides me had this issue caused purely by DNS overlap and now are no longer flapping? I've been solid since I got rid of the overlap. I also wonder if those who used compiled out of kernel drivers and say it stopped flapping, did not also make configuration changes that in themselves may have actually fixed it by getting rid of overlap. The other thing that raises eyebrows is that only that one interface was involved, not the others as many people have igb nics on all interfaces and they also use that same driver module. 
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on June 16, 2022, 08:36:05 am
> Well, the kernel is temporary.

This doesn't make any sense.


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on June 16, 2022, 06:23:03 pm
Hi everybody,

My case could probably help here???

- IDP with IPS mode on + MAC spoofing
- 2 different system, issue is every time reporducable (also with a new clean OpnSense install):
 * Intel igc driver (on a testing VM with NIC passthrough)
 or
 * Intel ixl driver (on my live firewall hardware)

My testing results:
- ixl driver compiled to newest from intel => still flapping
- tested 22.7.pre3 => still flapping
- opnsense-revert -r 22.1.1 opnsense => FIXED my issues!!!
- changed from intel to realtek NIC => FIXED my issues on all OpnSense versions!!!

Other known working workarounds:
- Remove MAC spoofing
or
- Disable IPS mode

So my guesses:
It isn't an explicit driver issue but it depends on certain NIC's and/or drivers + kernel or changes after 22.1.1... so some complex combination  :(

Regards
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on June 17, 2022, 08:43:57 am
Thanks for adding data points! The only changes in 22.1.2 that seem to be relevant at first glance are:

o interfaces: simplify device destroy code https://github.com/opnsense/core/commit/84cd38adb558
o interfaces: avoid use legacy_get_interface_addresses() in MAC address read https://github.com/opnsense/core/commit/13388839e7e

But upon inspection it doesn't look like these could change the rules of MAC address assignments in terms of making links flap. Second opinions?


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on June 17, 2022, 10:48:19 am
Hi Franco,

I can guarantee you 100% that with the settings (see documentation screenshots from my testing VM attached) up to and with 22.1.1 everything worked fine. From 22.1.2 on it just doesn't work anymore on all my Intel NIC's with IPS + MAC spoofing.

I can also do/send you more logs if needed... the issue is easily reproducible for me in the test VM.

Greetings
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on June 17, 2022, 11:12:51 am
We're definitely missing some sort of logging information from "opnsense-log system" and "opnsense-log gateways" at the time of the link events. Some script has to be responsible or at least react to linkup which makes this worse than before.


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on June 17, 2022, 11:46:47 am
Hi again,

Logs after IPS enabled attached, Gateway log is/was empty.

Regards
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on June 17, 2022, 07:24:03 pm
Logs, ignore the DHCP error, this is a known issue for starlink currently.  Here is a comcast (with static IP) flap... No Mac spoofing, IDS enabled on LAN, IPS disabled.


Code: [Select]
2022-06-14T20:52:01-04:00 Notice /update_tables.py remove old alias __automatic_3a953935_0
2022-06-14T20:51:39-04:00 Error opnsense /usr/local/etc/rc.filter_configure: ROUTING: creating /tmp/igb0_defaultgw using 'REDACTED'
2022-06-14T20:51:39-04:00 Error opnsense /usr/local/etc/rc.filter_configure: ROUTING: removing /tmp/igb3_defaultgw
2022-06-14T20:51:32-04:00 Notice dhclient Creating resolv.conf
2022-06-14T20:51:32-04:00 Error dhclient unknown dhcp option value 0x52
2022-06-14T20:51:24-04:00 Error opnsense /usr/local/etc/rc.filter_configure: Ignore down inet6 gateways : WAN_Comcast_GWv4
2022-06-14T20:51:24-04:00 Error opnsense /usr/local/etc/rc.filter_configure: ROUTING: creating /tmp/igb3_defaultgw using '100.64.0.1'
2022-06-14T20:51:24-04:00 Error opnsense /usr/local/etc/rc.filter_configure: ROUTING: removing /tmp/igb0_defaultgw
2022-06-14T20:51:24-04:00 Error opnsense /usr/local/etc/rc.filter_configure: Ignore down inet gateways : WAN_Comcast_GWv4

Code: [Select]
2022-06-14T20:51:36-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 75.75.75.75 Alarm: 0 RTT: 20292us RTTd: 2949us Loss: 10%)
2022-06-14T20:51:36-04:00 Warning dpinger WAN_Comcast_GWv4 75.75.75.75: Clear latency 20292us stddev 2949us loss 10%
2022-06-14T20:51:23-04:00 Notice dpinger GATEWAY ALARM: WAN_Comcast_GWv4 (Addr: 75.75.75.75 Alarm: 1 RTT: 20080us RTTd: 2568us Loss: 11%)
2022-06-14T20:51:23-04:00 Warning dpinger WAN_Comcast_GWv4 75.75.75.75: Alarm latency 20080us stddev 2568us loss 11%
Title: Re: WAN interface flapping with 22.1.2
Post by: Davesworld on June 17, 2022, 07:27:50 pm
> Well, the kernel is temporary.

This doesn't make any sense.


Cheers,
Franco

 The version.
Title: Re: WAN interface flapping with 22.1.2
Post by: firewall on June 18, 2022, 12:05:23 am
My contribution to a game of "lets spot the common factor".

Current version: OPNsense 22.1.1_3-amd64 (reverted as far back as I could)
Interfaces: Intel I211 (x6)
Possible influencing configuration(s) when first encountered:
    - MAC Spoofing: No
    - IDS (+IPS): Yes (W/LAN only)
 
Other potential commonalities:
    - wireguard-kmod (4 in use)
    - Dual WAN w/ cellular hotspot (this was setup since WAN flapping arose. the usage bill resulting from WAN going down frequently absolutely sucks.)
 
Attempted resolution:
    - Disable IDS (+IPS): down/up continues.
    - Revert version to earliest 22.1 Production: down/up continues.
        https://wiki.opnsense.org/manual/opnsense_tools.html#example (https://wiki.opnsense.org/manual/opnsense_tools.html#example)
    - Compile / install latest IGB drivers from Intel: down/up continues.
        https://forum.opnsense.org/index.php?topic=27299.msg137350#msg137350 (https://forum.opnsense.org/index.php?topic=27299.msg137350#msg137350)
    - Confirming no "DNS overlap": none exists.
        https://forum.opnsense.org/index.php?topic=27299.msg139635#msg139635 (https://forum.opnsense.org/index.php?topic=27299.msg139635#msg139635)
     
Next: Revert to 21.7.8  development version (22.1 w/o FBSD 13 kernel)
        https://forum.opnsense.org/index.php?topic=27299.msg139876#msg139876 (https://forum.opnsense.org/index.php?topic=27299.msg139876#msg139876)
       
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on June 18, 2022, 01:56:12 am
Site 1
Current version: OPNsense 22.1.8_1-amd64 / FreebSD 13
Interfaces: Intel I210 (x6)
Device Type: FW6D - 6 Port Intel i5 (8250U)
MAC Spoofing: NO
IDS: LAN
IPS: NO
Wiregaurd: GO on primary WAN
OpenVPN: NO
IPSEC: Yes, multiple site-site tunnels on primary wan
Multi WAN: Yes (Comcast-static/Starlink-dhcp/Hughesnet-dhcp)
Plugins: os-api-backup, os-ddclient, os-mdns-repeater, os-speedtest-community, os-wireguard, os-wol
Verified No DNS/Gateway Overlaps: YES
Updated Drivers Resolves Issue?:  Mostly

-----------------------------------------

Site 2
Current version: OPNsense 22.1.8_1-amd64 / FreebSD 13
Interfaces: Intel I210 (x4)
Device Type: FW4B - 4 Port Intel J3160
MAC Spoofing: NO
IDS: LAN
IPS: NO
Wiregaurd: GO on primary WAN
OpenVPN: NO
IPSEC: Yes, single site-site tunnel on primary wan
Multi WAN: Yes (Comcast-dhcp/Starlink-dhcp)
Plugins: os-api-backup, os-ddclient, os-mdns-repeater, os-speedtest-community, os-wireguard, os-wol
Verified No DNS/Gateway Overlaps: YES
Updated Drivers Resolves Issue?:  Mostly
Title: Re: WAN interface flapping with 22.1.2
Post by: buecker on June 20, 2022, 09:56:24 pm
Add me to the list.  My router has 6x* Intel I211-AT  with the latest updates.

I rebuilt it from scratch and did the bare minimum setup. Everything was good with the first few devices added to the switch but then it went bezerk  when I added everything else. It especially didn't like my Intel NUC.

My 5 port Intel I225-V box with the latest opnsense updates does not have this issue.
Title: Re: WAN interface flapping with 22.1.2
Post by: stefan21 on July 09, 2022, 10:15:18 am
Running on hardware:

# sysctl -a | grep -E 'dev.(igb|ix|em|bg).*.%desc:'
dev.em.0.%desc: Intel(R) Legacy PRO/1000 MT 82540EM
dev.bge.0.%desc: Broadcom NetLink Gigabit Ethernet Controller, ASIC rev. 0x5784100

Had to revert back to

OPNsense 21.7.8-amd64
FreeBSD 12.1-RELEASE-p22-HBSD
OpenSSL 1.1.1m 14 Dec 2021

Changed the intel nic to LAN (before it was on the WAN)
Seems to run stable as before.

In the 21.7.8 system log:

kernel   em0: link state changed to DOWN
opnsense[81347]   /usr/local/etc/rc.linkup: Hotplug event detected for LAN(lan) but ignoring since interface is configured with static IP (192.x.x.x ::)
kernel      em0: link state changed to UP
opnsense[34693]   /usr/local/etc/rc.linkup: Hotplug event detected for LAN(lan) but ignoring since interface is configured with static IP (192.x.x.x ::)
opnsense[50625]   /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'em0'
opnsense[50625]   /usr/local/etc/rc.newwanip: On (IP address: 192.x.x.x) (interface: LAN[lan]) (real interface: em0).

The sense ist sitting behind a Vodafone cable modem. Configured as exposed host. The cable modem should still run in bridged mode. There have been updates from vodafone, therefore I'm not quite sure if there's something messed up in the config. I'll check this on monday.

I'm running another sense (also Hardware) with the latest version. No intel nics, mainly the same configuration. No problem with this machine. Does not loose WAN nor wireguard.

regrads,
stefan

Edit: both machines are configured with IDS and IPS on WAN and ZENARMOR on LAN
Edit: NO mac spoofing
Title: Re: WAN interface flapping with 22.1.2
Post by: crissi on July 10, 2022, 01:44:24 pm
Hi,

Current Version: OPNsense 22.1.10-amd64 (on BareMetal)
Interface: Intel I211
Suricata / Sensei: deactivated
Gateway Monitoring: disabled
MAC Spoofing: Yes

As soon is i add the spoofed MAC to the WAN Interface, the flipping starts (up / down every 2-3 seconds).

Code: [Select]
2022-07-10T13:32:54 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic wan(igb0)
2022-07-10T13:32:53 Error opnsense /usr/local/etc/rc.linkup: Clearing states for stale wan route on igb0
2022-07-10T13:32:53 Critical dhclient exiting.
2022-07-10T13:32:53 Error dhclient connection closed

When I remove the spoofed MAC Address, all is fine and connection is stable.



UPDATE:

Installed now for futher testing my 3 year old Realtek USB GbE Ethernet Adapter, and configured new WAN Port,
and added there my spoofed MAC Address.

Result: No flapping, all fine, connection is stable!

Next i enabled Intrusion Detection IDS / IPS and Promiscuous mode.

Result: No flapping, all fine, connection still stable!


As it seems to be Driver Related as the old Realtek is working without any issue, why is the Intel 211 Driver not working as it should be a Standard Driver? Is FreeBSD using old Drivers?

What to do?
Title: Re: WAN interface flapping with 22.1.2
Post by: stefan21 on July 11, 2022, 02:26:20 pm
The sense ist sitting behind a Vodafone cable modem. Configured as exposed host. The cable modem should still run in bridged mode. There have been updates from vodafone, therefore I'm not quite sure if there's something messed up in the config. I'll check this on monday.

On monday morning the interface with the intel nic didn't work any longer. I removed the nic and put a realtek in the machine. Interestingly no error shows up in any log which may lead to a defunct nic.

The last vodafone update messed up my configuration. It's a Fritz!Box 6591 Cable BH. Exposed host was disabled, realtime prio was disabled and also self port opening. I changed the settings back.

I'll report if the errors are gone.

regards,
stefan

Title: Re: WAN interface flapping with 22.1.2
Post by: iMx on July 12, 2022, 08:15:55 am
Been seeing the same on 22.1.10, somehow it seems worse since I upgraded to .10 - although I've certainly been fighting this for a while - but might be a coincidence.

WAN interface: Intel(R) Ethernet Controller X710 for 10GbE SFP+

- Disabled MAC spoofing, did not fix things
- Disabled gateway monitoring, seemed to improve things but did not resolve
- Upgraded the 710 firmware, using stock drivers, did not fix things
- This morning, I have now loaded the Intel updated drivers (1.12.35)

I am also using RSS, so this is maybe the next thing for me to rule out.

I do also note, that the latest Intel drivers are not iflib, so the various tunings have now changed (tx/rx ring buffer, queues).  There are details in the readme.txt.

I did also try the updated IGB driver (out of curiosity) as I also have the below, although these ports are NOT on the WAN and did not see the problem.

Intel(R) I211 (Copper)

But this lead to 'weird' things.  For example, the HAproxy instance running on opnsense could not health check my Home Assistant server (to provide SSL externally) TCP port.  The Home Assistant physical port on opnsense, is on the I211.  Although traffic could pass from LAN -> Home Assistant through the firewall...the firewall itself could not reach the Home Assistant TCP port.

I could see the SYN from opnsense HAProxy -> Home Assistant on the server port, and the SYN,ACK reply reach the firewall, but for some reason it was being dropped.  I did not have time to look into this further, so rolled it back leaving just the WAN X710 using the Intel drivers.  This instantly resolved the HAproxy/Home Assistant issue.
Title: Re: WAN interface flapping with 22.1.2
Post by: Scuro on July 14, 2022, 03:49:51 am
My 5 port Intel I225-V box with the latest opnsense updates does not have this issue.

My 4 port I225-V b3 box is having this issue.

Details:
No spoofing, no IDS. Just IPv4, VLANS, and a weighted upload.
WAN will randomly disconnect and show the following repeated:

Code: [Select]
2022-07-13T18:03:33-07:00 Critical dhclient exiting.
2022-07-13T18:03:33-07:00 Error dhclient connection closed
2022-07-13T18:03:33-07:00 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic wan(igc0)

Most times it will come back up on its own, other times it will refuse to pick up DHCP from WAN.
Turning interface on/off doesn't fix. Requires reboot or physically unplugging port.
Title: Re: WAN interface flapping with 22.1.2
Post by: iMx on July 15, 2022, 07:12:08 pm
Been seeing the same on 22.1.10, somehow it seems worse since I upgraded to .10 - although I've certainly been fighting this for a while - but might be a coincidence.

....

- This morning, I have now loaded the Intel updated drivers (1.12.35)

Whilst I'm still running the updated drivers, I'm pretty sure my issue was 4 dodgy wall ports where my cable modem connects - 2 ports per 1 gang box - so when I swapped the port, even to a different box, I still saw the problem...on all 4 ports.  In the end, I ran a 15m cable direct from the opnense box to the cable modem (and got moaned at with the cable going through the house/hallway) which fixed the problem.

Long story short...I replaced the wall modules and re-terminated the cabling and I think this has resolved my problem.  Over the next week, I'll start rolling back the various changes such as the updated drivers.

I've re-enabled gateway monitoring, this was in some cases causing the problem when the port flapped - although interestingly often the flaps weren't shown in the switch logs, so whilst in duration they were short they were long/frequent enough to cause disruption when gateway monitoring was enabled.
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on July 16, 2022, 04:27:17 pm
Hi all,

I did a new test with a fresh install of the current RC 22.7 (22.7.r1_8) with my Intel igc + MAC spoofing + IPS => still flapping  :(

But then I changed from DHCP to static IP => NO flapping!!!

Retested this with 22.1.10 (same Intel igc + MAC spoofing + IPS) but with static IP instead of DHCP => also no flapping! Sadly the static IP is not possible on my internet connection... just in my test VM's.

Another strange behavior:
As soon as I remove the spoofed MAC and hit "Save" (not yet "Apply changes" pressed), the flapping stops immediately.

So I think the "Intel + MAC spoof + IPS" issue must be something around DHCP within "a script"... but on Intel only??????

Regards
Title: Re: WAN interface flapping with 22.1.2
Post by: l0stnyc on July 23, 2022, 07:04:20 pm
Since I recently experienced the same issue, I just wanted to throw in my data points.  My situation is a bit different in that I haven’t really changed my config since 17.x.  However, recently I decided to turn off suricata (for no reason other than it was causing my system to run about 7C higher on average).  For two hours it worked fine.  Then the WAN link started to go up and down.  I came across this post in the past but I didn’t put the two together right away.  Instead, I called my ISP and swapped out my cable modem, which solved the issue for about 18 hours, then it started to flap again.  I eventually had to enable suricata again and it has been fine ever since.  so for me, if i disable suricata the WAN starts to act up.

Current version: OPNsense 22.1.10
Interfaces: Intel pro 1000
MAC Spoofing: YES
IDS: LAN
IPS: YES

Title: Re: WAN interface flapping with 22.1.2
Post by: Patrick M. Hausen on July 23, 2022, 09:42:51 pm
The common denominator in this thread seems to be MAC spoofing. Why is this necessary?
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on July 24, 2022, 12:18:13 am
Some people need to use MAC spoofing because of the way their internet provider (cable), locks the IP to the devices MAC. 

Common denominator seems to be a mix of MAC Spoofing OR intel i210/i211 devices OR IPS/IDS with promiscuous mode.

Title: Re: WAN interface flapping with 22.1.2
Post by: axsdenied on July 24, 2022, 01:03:34 am
As a heads up, usually most providers allow you to switch devices if you call them. You simply need to give them your MAC address and it works.
Title: Re: WAN interface flapping with 22.1.2
Post by: crissi on July 25, 2022, 09:46:05 am
Installed for further testing now pfsense 2.6.0. Added spoofed MAC address to WAN Interface (Intel I211 NIC), and voila no interface flapping at all!

Driver Version is on opnsense and pfsense identical

Code: [Select]
[2.6.0-RELEASE][root@pf.home.arpa]/root: sysctl -a | grep -E 'dev.(igb|ix|em).*.iflib.driver_version:'
dev.igb.5.iflib.driver_version: 7.6.1-k
dev.igb.4.iflib.driver_version: 7.6.1-k
dev.igb.3.iflib.driver_version: 7.6.1-k
dev.igb.2.iflib.driver_version: 7.6.1-k
dev.igb.1.iflib.driver_version: 7.6.1-k
dev.igb.0.iflib.driver_version: 7.6.1-k


Code: [Select]
root@opn:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.iflib.driver_version:'
dev.igb.5.iflib.driver_version: 7.6.1-k
dev.igb.4.iflib.driver_version: 7.6.1-k
dev.igb.3.iflib.driver_version: 7.6.1-k
dev.igb.2.iflib.driver_version: 7.6.1-k
dev.igb.1.iflib.driver_version: 7.6.1-k
dev.igb.0.iflib.driver_version: 7.6.1-k
Title: Re: WAN interface flapping with 22.1.2
Post by: subivoodoo on July 25, 2022, 07:49:53 pm
As a heads up, usually most providers allow you to switch devices if you call them. You simply need to give them your MAC address and it works.
Not the provider I have   >:(

My solution right now is: I have changed the MAC address within the eprom of the WAN NIC. There exists a tool called eeupdate.exe (can be found via Google, it's not official by Intel) which can do this on many Intel NIC's. Now with the "physical" changed MAC, I don't need to spoof it any longer and so no flapping...
Title: Re: WAN interface flapping with 22.1.2
Post by: firewall on July 28, 2022, 06:56:53 pm
The common denominator in this thread seems to be MAC spoofing. Why is this necessary?

many, myself included, have experienced this issue since the 22.x upgrade despite not using MAC spoofing.
Title: Re: WAN interface flapping with 22.1.2
Post by: os914964619 on August 05, 2022, 05:11:41 am
I just found this post and figured it would be helpful to people here who are watching it for any updates / progress.

For people who are doing MAC spoofing, the port flapping was caused by a bug that was introduced in 22.1.2.

It's been fixed as of this commit thanks to Franco:

https://github.com/opnsense/core/commit/d19cd6cdbf4da581f71e5483b279f82fb4396bec
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on August 05, 2022, 07:32:37 am
For details and patch command: https://forum.opnsense.org/index.php?topic=29691.msg143440#msg143440
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on August 09, 2022, 03:47:15 pm
Just confirming that 22.7.1 seems to fix this issue for me (I was NOT using MAC spoofing)...
Title: Re: WAN interface flapping with 22.1.2
Post by: crissi on August 09, 2022, 05:56:48 pm
Issue is for me with 22.7.1 (with MAC spoofing) as well fixed
Title: Re: WAN interface flapping with 22.1.2
Post by: TheHesster on September 08, 2022, 07:08:00 pm
I'm having an issue at one of my sites with flapping of the wan interface.
I am NOT using IPS or MAC Spoofing.

The two sites use an identical Protectli VP-2410. One site has the issue, one site does not. Basically the exact same config on the two sites.

Any one have any information on how I can troubleshoot this to see if it's an issue with the firewall or with the cable modem?
Title: Re: WAN interface flapping with 22.1.2
Post by: tracerrx on September 11, 2022, 09:29:40 pm
Try upgrading to 22.7.4.... This has fixed all the WAN Flapping (or flipping and not returning) for me...
Title: Re: WAN interface flapping with 22.1.2
Post by: ToniE on October 09, 2022, 07:08:56 pm
Yesterday I upgraded to 22.7.5 and lost WAN connection 3 times since then. Today I went back to 21.7.8.
This is still a problem.
I have a Gigabyte GA-J3455N-D3H motherboard with integrated 2x Realtek NICs.
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on October 10, 2022, 08:29:32 am
The original issue reported here was indeed introduced in 22.1.2 (check the title) and 100% confirmed fixed in 22.7.1. Today, this thread is so convoluted and overloaded with different issues that posting here with "me too" is the least effective way to get to the bottom of any other issue that looks the same on the surface.


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: ToniE on October 10, 2022, 09:16:29 am
Yes I also had the same issue when 22.1.2 was introduced as I wrote about at the beginning of this thread when I had to go back to 21.7.8 same as now.
Title: Re: WAN interface flapping with 22.1.2
Post by: franco on October 10, 2022, 09:59:39 am
Yep, and as I said if you have this issue on 22.7.1 or above it's NOT that issue even if it looks like it, or you have two issues one of which was fixed.

What this episode teaches us is that network card drivers can suck when spoofing MAC addresses... maybe in conjunction with specific chipsets and the issue could be more prevailing in FreeBSD 13 (22.x and up) where we don't have direct influence on.


Cheers,
Franco
Title: Re: WAN interface flapping with 22.1.2
Post by: ToniE on October 13, 2022, 03:46:34 pm
I did a clean install (22.7.5) so now it works without problems. :)
Title: Re: WAN interface flapping with 22.1.2
Post by: SolarCzar on January 07, 2023, 01:15:38 pm
I found this post and adding that my WAN started "flapping" and I have been searching desperately how to fix, as it's knocking down my entire network.  I am NOT doing Mac spoofing. 
I came on with OPNSense around 22.7.3 and have not experienced it until 22.7.10.  I tried to revert to 22.7.9, but the flapping was more frequent.  The only consistent log when it occurs is this...

Code: [Select]
2023-01-07T05:51:05-06:00 Critical dhclient exiting.
2023-01-07T05:49:18-06:00 Critical dhclient exiting.
2023-01-07T05:14:50-06:00 Critical dhclient exiting.
2023-01-07T03:01:36-06:00 Critical dhclient exiting.
2023-01-07T00:45:08-06:00 Critical dhclient exiting.

Any help is appreciated