Messages

No I get it, my Tmobile device IS a 5G wireless device.  I'm suggesting it's a device issue. Is there a higher powered option?

It's directly powered via USB-C, and it's the best option from my wireless carrier (Verizon).

I would honestly suspect the Inseego X Pro 5G device as the source of the issue then I would OPNsense.  I did a quick glance at reviews on that device and suffice it to say, they were not stellar.  ;)  It's a low powered device not really designed for this situation.

FWIW I run a similar scenario as my primary internet goes down more often than I'd like so I have tmobile (in the U.S.) home internet device as my backup.  It works well with no flapping.

I don't have many other great options, since all fiber/coax is sent down the same road through the same pipe.  It all gets cut at the same time.  I'm needing a wireless solution, and a 5G modem/hotspot is my best option.

I remember finding a post somewhere that a person was using this exact device as their failover WAN, but they were using a Netgear NightHawk something-rather.  So, I know it is possible, just not sure the perfect config for OPNSense to make it work more reliably.

OPNSense 23.1_6 is running on a Supermicro X11SDV-4C-TP8F D-2123IT. Ethernet ports being used are the Intel I350-AM4 chipset.

Code Select Expand


dev.igb.3.%desc: Intel(R) I350 (Copper)
dev.igb.2.%desc: Intel(R) I350 (Copper)
dev.igb.1.%desc: Intel(R) I350 (Copper)
dev.igb.0.%desc: Intel(R) I350 (Copper)


Driver version 7.6.1-k

I've got AT&T Fiber running on igb0 with my BGW210-700 in passthrough mode no problem.  I've been running this for years.  Running dual-stack.

My fiber seems to get cut quarterly, so I purchased an Inseego X Pro 5G UW hotspot as a failover WAN solution.  It has an Ethernet port, and it supports IP Passthrough - which I configured.

I've followed the OPNSense documentation for multiwan.  It seems to be configured correctly.  Here is the problem:  When I mark the primary AT&T gateway as down (force a failover to the hotspot), the Internet works for a little bit (maybe 30 seconds? - traceroute proves traffic is indeed going through the hotspot), and then the interface just flaps (up/down - DEVD thinks there is a detach event or whatever, which causes dhclient to get angry).  Unchecking the AT&T as being marked down makes it failover back to the primary gateway with no problem, AND the hotspot interface stops flapping.

I can run a traceroute through the hotspot gateway without failure, but it seems putting an actual traffic load on it causes the interface to flap.

I'm thinking I could have a similar issue as these:

• https://forum.opnsense.org/index.php?topic=32299.0
• https://forum.opnsense.org/index.php?topic=27299.0
• https://forum.opnsense.org/index.php?topic=32315.0

I'm not using MAC spoofing, but I do have ZenArmor running.  It's currently in permissive mode while I'm trying to get Multi-WAN to work.

I've also tried setting the MTU lower to 1420, but that didn't make any difference.

UPDATE:
Here is some information from the logs:

Code Select Expand


2023-02-06T14:28:50-06:00 Error dhclient unknown dhcp option value 0x78
2023-02-06T14:28:48-06:00 Error dhclient unknown dhcp option value 0x78
2023-02-06T14:28:45-06:00 Error opnsense /usr/local/etc/rc.linkup: The command '/sbin/dhclient -c '/var/etc/dhclient_opt2.conf' -p '/var/run/dhclient.igb3.pid' 'igb3'' returned exit code '11', the output was 'DHCPREQUEST on igb3 to 255.255.255.255 port 67 DHCPNAK from 192.168.98.1 DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 1 DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 1 DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 2 DHCPOFFER from 192.168.98.1 unknown dhcp option value 0x78 DHCPOFFER from 192.168.98.1 DHCPOFFER already seen. DHCPREQUEST on igb3 to 255.255.255.255 port 67 igb3 link state up -> down DHCPREQUEST on igb3 to 255.255.255.255 port 67 DHCPREQUEST on igb3 to 255.255.255.255 port 67 igb3 link state down -> up DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 1 DHCPOFFER from 192.168.98.1 unknown dhcp option value 0x78'
2023-02-06T14:28:45-06:00 Critical dhclient exiting.
2023-02-06T14:28:45-06:00 Error dhclient connection closed
2023-02-06T14:28:43-06:00 Error dhclient unknown dhcp option value 0x78
2023-02-06T14:28:42-06:00 Error dhclient send_packet: Network is down
2023-02-06T14:28:40-06:00 Error dhclient send_packet: Network is down
2023-02-06T14:28:37-06:00 Error dhclient unknown dhcp option value 0x78
2023-02-06T14:28:30-06:00 Critical dhclient exiting.
2023-02-06T14:28:30-06:00 Error dhclient connection closed
2023-02-06T14:28:23-06:00 Error dhclient unknown dhcp option value 0x78
2023-02-06T14:28:21-06:00 Error dhclient unknown dhcp option value 0x78
2023-02-06T14:28:17-06:00 Critical dhclient exiting.
2023-02-06T14:28:17-06:00 Error dhclient connection closed
2023-02-06T14:13:27-06:00 Error dhclient unknown dhcp option value 0x78

2023-02-06T08:15:47-06:00 Error opnsense /usr/local/etc/rc.newwanip: The command '/bin/pkill -'TERM' -F '/var/run/dpinger_5GWAN_DHCP.pid'' returned exit code '3', the output was 'pkill: Cannot open pidfile `/var/run/dpinger_5GWAN_DHCP.pid': No such file or directory'


Interestingly enough, now it will only flap a few times, then STICK.  I'm not certain what is going on to make it flap and now it will eventually just stick and keep working.  80Mbps/8Mbps on average (I'm not in a great area for 5G coverage).