WAN interface flapping with 22.1.2

[subivoodoo]

Another test on my main rig with manually compiled new Intel driver for ixl => it does not work for me...

MAC spoofing + IPS + Intel ixl drivers from march 2022 = if down/up

[Edwin70]

As there is still no definite cause for this issue, let alone a solution, I'm still running on 21.7.

Are there any security issues in the 21.7 release I should be aware off? I run a basic setup, but with a WireGuard VPN running.

[atxx]

Continuing from my last post, removing mac spoofing (after coordinating with the ISP), and disabling IPS does not solve the issue. Reverting to 22.1.1 does not solve the issue either. Reminder that my controller is a Broadcom BCM5720. I'm at a loss and not sure what to do as I've spent countless hours trying to solve this after upgrading to 22.1.7_1. My WAN interface keeps flapping every few minutes/hours. Any advice is welcome.

[tracerrx]

@atxx is the gateway stable under 21.x?

[atxx]

@atxx is the gateway stable under 21.x?

I'll try downgrading and get back to you. I've had issues with 22.1.1 on the LAN side after reverting - vlan interfaces appeared as "down" (GUI) and were missing when running ifconfig -a. Any ideas how I can resolve that in order to downgrade to 21.x?

//edit: I ended up starting again from scratch, from 21.7.8. So far so good, I'll report back soon.

//edit2: Everything was stable on 21.7.8 for almost 24h, then I tried mac spoofing (to test something with the ISP) and the WAN interface started constantly flapping, again disrupting everything. Turning it off & restarting didn't resolve the issues. I noticed that the port was negotiating at 100BASE-TX and after cleaning all cable and port contacts with alcohol and compressed air, the interface is stable again and negotiating gbit speeds. I want to upgrade to 22.x just to check that the issue was with the cables/ports at some point.

[firewall]

there are many threads both here and reddit of people reporting wan connectivity issues--all of which likely related to the same issue but perhaps misunderstood / mischaracterized. has this actually been researched by opnsense devs vs. outright disregarded as a "possible issue with intel drivers"? certainly i'm not the only person who feels like opnsense bug reports are sometimes dropped in the same way packets are with pf..

[devhunter55]

After running into these issues, i did revert to OPNsense 22.1.1_3.

This version is running very stable - no flapping interfaces anymore.

I'm thinking to wait for a more stable Freebsd version with running Intel drivers.
May be with FreeBSD 13.1 or/and Opnsense 23.1.x ?

[tracerrx]

FWIW, the intel IGB drivers do survive the 22.1.8_1 update without having to be re-installed/compiled.

[devhunter55]

@tracerrx - that's a very good Information, indeed - many thx for it !

[foxmanb]

I finally dug into this a bit and was able to resolve my issue.

I am an xfinity cable sub and was always under the impression that I had to use a cloned MAC address on my WAN connection.

Today I removed the cloned MAC, powered everything down, and rebooted. System picked up a new WAN IP address because my MAC changed to the MAC on my NIC. My assumption of having to use a cloned MAC was incorrect, I had been doing that for the last 15 years...

It was successful. Once I did this, I upgraded to the latest and greatest version and it is stable.
Currently running OPNsense 22.1.8_1 (amd64/OpenSSL) at Fri Jun  3 09:40:31 EDT 2022

IDS is enabled and there is no flap.

[firewall]

It was successful. Once I did this, I upgraded to the latest and greatest version and it is stable.
Currently running OPNsense 22.1.8_1 (amd64/OpenSSL) at Fri Jun  3 09:40:31 EDT 2022

IDS is enabled and there is no flap.

give it time

[foxmanb]

It was successful. Once I did this, I upgraded to the latest and greatest version and it is stable.
Currently running OPNsense 22.1.8_1 (amd64/OpenSSL) at Fri Jun  3 09:40:31 EDT 2022

IDS is enabled and there is no flap.

give it time

So far so good.

[TheeDude]

I was having theese "flapping" issues also when I upgraded to 22.1.2, and were forced to downgrade to 22.1.1_3.
This version works great.

I have a Qotom machine, with intel network cards, and I am also using mac spoofing.
What do you think, is it safe upgrading to 22.7 when it launches?
Is this issue resolved with FreeBSD 13.1?
I did notice that there is a testing thread, they didn´t seem to talk about this issue so much..

[Supermule]

You could test if its the Intel drivers thats the cause by running the troubled versions in a VM instead of bare metal.

Then it would be obvious where the culprit is located.

[Davesworld]

 Mine  is doing the same thing but why only on the wan interface with intel in particular? My lan and wlan are also on igb nics but they do not do this. why is the interface cycling only on the WAN?

If you want to monitor, here are a few examples, just grep which nic you wish to watch.   

tail -f /var/log/configd/latest.log | grep em0

tail -f /var/log/configd/latest.log | grep igb3

You get the idea.

Edit: Found out exactly why, it was DNS overlaps between the two WANS using the same DNS servers (Google) and two Gateways using one of each of the two Google DNS servers. I have since changed it so each Gateway has four DNS servers that are NOT Google and in the Gateway monitoring, used two completely different DNS servers that are not duplicated in the DNS settings so absolutely nothing matches. Of course for the Gateway monitoring you can use anything that allows pinging, even the default ISP gateway.