OPNSense MultiWAN - port flapping?

[misterbaz]

OPNSense 23.1_6 is running on a Supermicro X11SDV-4C-TP8F D-2123IT. Ethernet ports being used are the Intel I350-AM4 chipset.

Code: [Select]

dev.igb.3.%desc: Intel(R) I350 (Copper)
dev.igb.2.%desc: Intel(R) I350 (Copper)
dev.igb.1.%desc: Intel(R) I350 (Copper)
dev.igb.0.%desc: Intel(R) I350 (Copper)

Driver version 7.6.1-k

I've got AT&T Fiber running on igb0 with my BGW210-700 in passthrough mode no problem.  I've been running this for years.  Running dual-stack.

My fiber seems to get cut quarterly, so I purchased an Inseego X Pro 5G UW hotspot as a failover WAN solution.  It has an Ethernet port, and it supports IP Passthrough - which I configured.

I've followed the OPNSense documentation for multiwan.  It seems to be configured correctly.  Here is the problem:  When I mark the primary AT&T gateway as down (force a failover to the hotspot), the Internet works for a little bit (maybe 30 seconds? - traceroute proves traffic is indeed going through the hotspot), and then the interface just flaps (up/down - DEVD thinks there is a detach event or whatever, which causes dhclient to get angry).  Unchecking the AT&T as being marked down makes it failover back to the primary gateway with no problem, AND the hotspot interface stops flapping.

I can run a traceroute through the hotspot gateway without failure, but it seems putting an actual traffic load on it causes the interface to flap.

I'm thinking I could have a similar issue as these:

• https://forum.opnsense.org/index.php?topic=32299.0
• https://forum.opnsense.org/index.php?topic=27299.0
• https://forum.opnsense.org/index.php?topic=32315.0

I'm not using MAC spoofing, but I do have ZenArmor running.  It's currently in permissive mode while I'm trying to get Multi-WAN to work.

I've also tried setting the MTU lower to 1420, but that didn't make any difference.

UPDATE:
Here is some information from the logs:

Code: [Select]

2023-02-06T14:28:50-06:00 Error dhclient unknown dhcp option value 0x78
2023-02-06T14:28:48-06:00 Error dhclient unknown dhcp option value 0x78
2023-02-06T14:28:45-06:00 Error opnsense /usr/local/etc/rc.linkup: The command '/sbin/dhclient -c '/var/etc/dhclient_opt2.conf' -p '/var/run/dhclient.igb3.pid' 'igb3'' returned exit code '11', the output was 'DHCPREQUEST on igb3 to 255.255.255.255 port 67 DHCPNAK from 192.168.98.1 DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 1 DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 1 DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 2 DHCPOFFER from 192.168.98.1 unknown dhcp option value 0x78 DHCPOFFER from 192.168.98.1 DHCPOFFER already seen. DHCPREQUEST on igb3 to 255.255.255.255 port 67 igb3 link state up -> down DHCPREQUEST on igb3 to 255.255.255.255 port 67 DHCPREQUEST on igb3 to 255.255.255.255 port 67 igb3 link state down -> up DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 1 DHCPOFFER from 192.168.98.1 unknown dhcp option value 0x78'
2023-02-06T14:28:45-06:00 Critical dhclient exiting.
2023-02-06T14:28:45-06:00 Error dhclient connection closed
2023-02-06T14:28:43-06:00 Error dhclient unknown dhcp option value 0x78
2023-02-06T14:28:42-06:00 Error dhclient send_packet: Network is down
2023-02-06T14:28:40-06:00 Error dhclient send_packet: Network is down
2023-02-06T14:28:37-06:00 Error dhclient unknown dhcp option value 0x78
2023-02-06T14:28:30-06:00 Critical dhclient exiting.
2023-02-06T14:28:30-06:00 Error dhclient connection closed
2023-02-06T14:28:23-06:00 Error dhclient unknown dhcp option value 0x78
2023-02-06T14:28:21-06:00 Error dhclient unknown dhcp option value 0x78
2023-02-06T14:28:17-06:00 Critical dhclient exiting.
2023-02-06T14:28:17-06:00 Error dhclient connection closed
2023-02-06T14:13:27-06:00 Error dhclient unknown dhcp option value 0x78

2023-02-06T08:15:47-06:00 Error opnsense /usr/local/etc/rc.newwanip: The command '/bin/pkill -'TERM' -F '/var/run/dpinger_5GWAN_DHCP.pid'' returned exit code '3', the output was 'pkill: Cannot open pidfile `/var/run/dpinger_5GWAN_DHCP.pid': No such file or directory'

Interestingly enough, now it will only flap a few times, then STICK.  I'm not certain what is going on to make it flap and now it will eventually just stick and keep working.  80Mbps/8Mbps on average (I'm not in a great area for 5G coverage).

[axsdenied]

I would honestly suspect the Inseego X Pro 5G device as the source of the issue then I would OPNsense.  I did a quick glance at reviews on that device and suffice it to say, they were not stellar.    It's a low powered device not really designed for this situation.

FWIW I run a similar scenario as my primary internet goes down more often than I'd like so I have tmobile (in the U.S.) home internet device as my backup.  It works well with no flapping.

[misterbaz]

I would honestly suspect the Inseego X Pro 5G device as the source of the issue then I would OPNsense.  I did a quick glance at reviews on that device and suffice it to say, they were not stellar.    It's a low powered device not really designed for this situation.

FWIW I run a similar scenario as my primary internet goes down more often than I'd like so I have tmobile (in the U.S.) home internet device as my backup.  It works well with no flapping.

I don't have many other great options, since all fiber/coax is sent down the same road through the same pipe.  It all gets cut at the same time.  I'm needing a wireless solution, and a 5G modem/hotspot is my best option.

I remember finding a post somewhere that a person was using this exact device as their failover WAN, but they were using a Netgear NightHawk something-rather.  So, I know it is possible, just not sure the perfect config for OPNSense to make it work more reliably.

[axsdenied]

No I get it, my Tmobile device IS a 5G wireless device.  I'm suggesting it's a device issue. Is there a higher powered option?

[misterbaz]

No I get it, my Tmobile device IS a 5G wireless device.  I'm suggesting it's a device issue. Is there a higher powered option?

It's directly powered via USB-C, and it's the best option from my wireless carrier (Verizon).

[axsdenied]

FYI the "unknown dhcp option value 0x78" issue generally means OPNsense side is trying to get a DHCP address from your wireless device and doesn't understand the option it's using.

If everything was working this can usually be ignored but, in your case, may be related.

I assume, because you were following the guidance documentation that your Firewall rules for your second WAN are default and your NAT:Outbound is also default?

No block of private networks setting on in the interface?

[tracerrx]

Are you still having this issue? 

It seems like I am now getting this happening on 2 sites (that were stable on 22.x).  Both sites have comcast/xfinity DHCP as primary WAN (thats flapping) and starlink DHCP as secondary WAN.  I have other sites that have comcast static IP as primary and starlink DHCP as secondary and the flapping does not occur.  It seems to be related to the primary WAN DHCP.  When primary is marked as down, a simple gateway edit/save/apply brings it back up immediately.

All sites are using intel nics.