Help with USB NIC + LTE Modem for redundant WAN

[ciaduck]

I'm having an issue with the second WAN. Seems like the device and NIC work fine. It seems to be able to pick up a DHCP config when the router boots. But if I reconnect the cable while opnsense is running I get continuous dhclient attempts, but no actual response. It's almost as if something about the firewall is preventing DHCP on the "wan2" interface.

Code Select Expand


<13>1 2024-03-10T20:56:08-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="27"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for opt3(ue0)
<13>1 2024-03-10T20:56:09-06:00 OPNsense.home.arpa dhclient 69737 - [meta sequenceId="28"] dhclient-script: Reason PREINIT on ue0 executing
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 76044 - [meta sequenceId="1"] dhclient-script: Reason TIMEOUT on ue0 executing
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 77695 - [meta sequenceId="2"] dhclient-script: New IP Address (ue0): 48.X.X.X
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 78719 - [meta sequenceId="3"] dhclient-script: New Subnet Mask (ue0): 255.255.255.0
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 80979 - [meta sequenceId="4"] dhclient-script: New Broadcast Address (ue0): 48.255.255.255
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 82814 - [meta sequenceId="5"] dhclient-script: New Routers (ue0): 48.X.X.X
<13>1 2024-03-10T20:57:28-06:00 OPNsense.home.arpa dhclient 86364 - [meta sequenceId="6"] dhclient-script: New Routers (ue0): 48.X.X.X
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa dhclient 15693 - [meta sequenceId="7"] dhclient-script: Reason FAIL on ue0 executing
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="8"] /usr/local/etc/rc.linkup: ROUTING: entering configure using 'opt3'
<11>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="9"] /usr/local/etc/rc.linkup: ROUTING: not a valid opt3 interface gateway address: 'missing'
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="10"] /usr/local/etc/rc.linkup: ROUTING: configuring inet default gateway on wan
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="11"] /usr/local/etc/rc.linkup: ROUTING: keeping inet default route to 76.X.X.X
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="12"] /usr/local/etc/rc.linkup: ROUTING: configuring inet6 default gateway on wan
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="13"] /usr/local/etc/rc.linkup: ROUTING: keeping inet6 default route to fe80::XXXXXX%igb0
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="14"] /usr/local/etc/rc.linkup: plugins_configure monitor (,WAN2_DHCP)
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="15"] /usr/local/etc/rc.linkup: plugins_configure monitor (execute task : dpinger_configure_do(,WAN2_DHCP))
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="16"] /usr/local/etc/rc.linkup: plugins_configure ipsec (,opt3)
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="17"] /usr/local/etc/rc.linkup: plugins_configure ipsec (execute task : ipsec_configure_do(,opt3))
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="18"] /usr/local/etc/rc.linkup: plugins_configure dhcp ()
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="19"] /usr/local/etc/rc.linkup: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2024-03-10T20:57:31-06:00 OPNsense.home.arpa opnsense 13518 - [meta sequenceId="20"] /usr/local/etc/rc.newwanip: Failed to detect IP for interface opt3
<13>1 2024-03-10T20:57:40-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="21"] /usr/local/etc/rc.linkup: plugins_configure dns ()
<13>1 2024-03-10T20:57:40-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="22"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2024-03-10T20:57:40-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="23"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : unbound_configure_do())
<13>1 2024-03-10T20:57:49-06:00 OPNsense.home.arpa opnsense 34870 - [meta sequenceId="24"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for opt3(ue0)
<27>1 2024-03-10T20:57:49-06:00 OPNsense.home.arpa dhclient 68039 - [meta sequenceId="25"] connection closed
<26>1 2024-03-10T20:57:49-06:00 OPNsense.home.arpa dhclient 68039 - [meta sequenceId="26"] exiting.
<13>1 2024-03-10T20:57:50-06:00 OPNsense.home.arpa opnsense 39292 - [meta sequenceId="27"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for opt3(ue0)


It just repeats these attempts in an endless loop until I disable the interface or reboot.

EDIT: Manually running dhclient on the interface gives the same results.

Code Select Expand


root@OPNsense:~ # dhclient -d ue0
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 5
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 14
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 16
My address (48.X.X.X) was re-added
My address (48.X.X.X) was deleted, dhclient exiting


Thank you for your time.

[ciaduck]

Just as an update. There appears to be issues with these realtek USB NICs, and FreeBSD.

I thought by turning off HW features (Interfaces -> Hardware settings -> Overwrite global settings) for CRC, TSO, and LRO, there was an improvement. But it was shortlived.

I noticed these devices tend to flap UP and DOWN a lot. Perhaps there are kernel driver issues. I thought this would be a cheap and easy way to add a NIC to my platform which is already fully utilized. Unfortunately it doesn't seem to work.

If anyone has any suggestions for getting these USB NICs to function properly, please let me know.

Code Select Expand


<13>1 2024-03-11T23:07:50-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="510"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:53-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="511"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:53-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="512"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="513"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="514"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:55-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="515"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:55-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="516"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:56-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="517"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:56-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="518"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:57-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="519"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:57-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="520"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:58-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="521"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:58-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="522"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:59-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="523"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:59-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="524"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:08:00-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="525"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:08:00-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="526"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:08:08-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="527"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:08:08-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="528"] <6>ue0: link state changed to UP


I tried both an Anker and a Ugreen NIC. Same results.

Code Select Expand


<13>1 2024-03-11T20:23:49-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="9"] ugen0.2: <Realtek USB 10/100/1000 LAN> at usbus0 (disconnected)
<13>1 2024-03-11T20:23:49-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="10"] ure0: at uhub1, port 1, addr 1 (disconnected)
<13>1 2024-03-11T20:23:49-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="11"] rgephy0: detached
<13>1 2024-03-11T20:23:49-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="12"] miibus0: detached
<13>1 2024-03-11T20:23:49-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="13"] ure0: detached
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="14"] ugen0.2: <Realtek USB 10/100/1000 LAN> at usbus0
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="15"] ure0 on uhub1
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="16"] ure0: <Realtek USB 10/100/1000 LAN, class 0/0, rev 3.00/31.00, addr 1> on usbus0
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="17"] miibus0: <MII bus> on ure0
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="18"] rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 0 on miibus0
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="19"] rgephy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="20"] <6>ue0: <USB Ethernet> on ure0

<13>1 2024-03-11T23:03:46-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="1"] ugen0.2: <Realtek USB 10/100/1000 LAN> at usbus0 (disconnected)
<13>1 2024-03-11T23:03:46-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="2"] ure0: at uhub1, port 2, addr 1 (disconnected)
<13>1 2024-03-11T23:03:46-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="3"] rgephy0: detached
<13>1 2024-03-11T23:03:46-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="4"] miibus0: detached
<13>1 2024-03-11T23:03:46-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="5"] ure0: detached
<13>1 2024-03-11T23:03:59-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="6"] ugen0.2: <ASIX AX88179A> at usbus0
<13>1 2024-03-11T23:03:59-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="7"] axge0 on uhub1
<13>1 2024-03-11T23:03:59-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="8"] axge0: <NetworkInterface> on usbus0
<13>1 2024-03-11T23:04:00-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="9"] miibus0: <MII bus> on axge0
<13>1 2024-03-11T23:04:00-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="10"] ukphy0: <Generic IEEE 802.3u media interface> PHY 3 on miibus0
<13>1 2024-03-11T23:04:00-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="11"] ukphy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
<13>1 2024-03-11T23:04:00-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="12"] <6>ue0: <USB Ethernet> on axge0


[Greg_E]

In my experience, the USB devices tend to overheat. I've had some luck clamping the usb device to a large sheet of metal to help move the heat out. But in the end, I was still getting a lot of times when the link was down. To me they are a better than nothing emergency fix.

[ciaduck]

Thanks, but it's not heat. Both of these devices have aluminum casings and feel fine to the touch.

I did find this on the FreeBSD forums. I'll try it out when I have time later.
https://forums.freebsd.org/threads/kernel-upgrade-from-12-2-stable-to-12-4-release-sshd-could-not-bind.90114/#post-620622

Is there a recommended method for settings? Is it appropriate to modify /etc/rc.conf ?

[ciaduck]

So I think I've figured out what the problem is, but I have no idea how to actually fix it.

netwait didn't seem to matter or make any difference. I tried adding it to the boot config:

Code Select Expand


root@OPNsense:~ # cat /usr/local/etc/rc.syshook.d/early/21-netwait

echo "NETWAIT"
netwait_enable="YES"
netwait_timeout="10"
netwait_ip="${defaultrouter}"
netwait_if="ue0"


On to my diagnosis/findings

Steps to reproduce:
Plug in USB NIC and uplink.
Configure/Enable interface in the GUI.
Watch the flapping in /var/log/system/latest.log

What I think is actually going on:
I think devd is somehow "fencing" on the USB NIC device/category, being both an ethernet NIC, and a USB. This will happen when the interface is configured in the GUI, but may be subverted by booting with the interface config already enabled. However hotplugging the USB also seems to cause the issue. (I actually had it resolve the issue once as well, the interface was "ON", and I unplugged and replugged it, but this doesn't always work.)

This appears to be an issue with OPNsense, and not FreeBSD. I was able to get into the CLI and manually run

Code Select Expand


root@OPNsense:~ # ifconfig ue0 down
root@OPNsense:~ # ifconfig ue0 up
root@OPNsense:~ # dhclient -d ue0
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPACK from 48.X.X.X
bound to 48.X.X.X -- renewal in 21600 seconds.
^C
root@OPNsense:~ # ifconfig ue0
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN2 (opt3)
        options=80008<VLAN_MTU,LINKSTATE>
        ether 20:7b:d2:d8:29:56
        inet 48.X.X.X netmask 0xffffff00 broadcast 48.255.255.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


Clearly manually configuring the interface from the shell works. FreeBSD seems to be happy about the NIC.

This also does not cause the flapping the the logs. Something about OPNsense devd is causing it to constantly up/down/configure/reconfigure the interface. It will do this in an endless loop.

Not sure where to go next. Maybe I'll get on github with a bug report?

Thanks.

[ciaduck]

Seems like it IS a FreeBSD issue. And seems like all USB Ethernet devices seem to be broken on a kernel level.

So much for my devd fencing idea.

I just found this bug report via reddit:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252165

The reporter outlines the exact symptom.

https://www.reddit.com/r/OPNsenseFirewall/comments/12kv3ns/random_ue0_link_state_change_to_down_ue_link/

And with the duration of this bug, and the current status of the bug report, I have no hope that this will ever be fixed. Time to find a different (more expensive) solution. *sigh*

[Greg_E]

Yeah, general rule is that USB NICs do not work reliably. I've tried several and even lived with one for a while as the WAN connection which was a little better than having it as LAN. But everything I've read is that USB NIC on BSD is nothing but headache, and my personal experience says this is true. Sad because it could open up some use cases.

This also reflects on "don't use Realtek NIC" on pfsense, seems OPN has a fix for these with a plugin or just working straight off. Again another limitation with drivers. Hedging bets, I bought Intel i350 cards for my test and production firewalls. I should probably buy a Realtek card for a test machine.

[ciaduck]

It really bothers me the answer is "Don't do this" when it could just as easily be "Let's fix it."

In the spirit of open source, I've decided to take it to the FreeBSD kernel people and see if we can get something going.

I guess I'll update this topic if there ever is a fix. My experience with open source projects has been hit or miss though. So we will see if I can get any traction.