Can I get away with USB ethernet for WAN?

[wuwos]

Noob here. So I have an old Intel NUC (nuc6i3syk) which comes with only one ethernet port. As far as I know, there is unfortunately no extra PCIe slot left on the motherboard.

I'm wondering if I can get away with using a USB ethernet adapter for WAN (1Gbps).

I know USB ethernet adapter (especially those with Realtek chipset) don't work really well with FreeBSD. But I would like to know if it's really that bad.
Meanwhile I do have a managed switch lying around. Is it better to just plug WAN into the switch, assign VLAN there and connect it to the NUC instead?

[bartjsmit]

I run OPNsense on a NUC7i3BNH with ESXi 7. This avoids juggling VLAN's in OPNsense since they are trunked as VMware port groups and end up as simple NIC's from the firewall's point of view.

There is a VMware fling to run ethernet over USB, which ends up as a VMXNET3 interface on the firewall as well.

You will lose some resource overhead to the hypervisor but there is far less drama if you decide to upgrade the hardware at some point. Just remember to add the os-vmware plugin to OPNsense for better management.


Bart...

[wuwos]

Thank you for the suggestion Bart.
If I understand correctly, even with the resource overhead, you are essentially recommending the virtualization of VLAN processing and USB driver in ESXi because they work better that way?

[bartjsmit]

Yes, the compatibility with VMware is much better than with "real" hardware. Your NUC looks well supported, although not officially: https://www.virten.net/2016/01/vmware-homeserver-esxi-on-6th-gen-intel-nuc/

I didn't bother with USB ethernet since I have DMZ, guest, IoT, LAN, WAN and a few other VLANs. Trunking them all from the managed switch was just easier and saves cables.

Once the firewall is virtual you can migrate it to new hardware, Hyper-V, Proxmox or whatever else without having to do a fresh install.

For data protection I run the community edition of Veeam with nightly backups: https://www.veeam.com/virtual-machine-backup-solution-free.html?ad=downloads and I snapshot the VM before doing upgrades or massive firewall rule changes for easy roll-back. Remember to always delete a snapshot when you're done or it will slowly eat up all your disk space.

Bart...

[wuwos]

Thanks for the information. It took me a while to resolve the performance issue and now I'm back with new issues.

My Opnsense VM is associated with 2 existing USB ethernet adapters (vusb0, vusb1) and they are both working as expected. The traffic on both adapters are completely untagged.
In ESXi I created a new virtual switch TestVLAN which connects to a third USB ethernet adapter vusb2 as uplink. The adapter links to a TP-Link router which tags the traffic with VLAN ID 12, 21 & 22.
Then I created 3 port groups (TestVLAN12, TestVLAN21, TestVLAN22), each of which is assigned a VLAN ID (12, 21, 22 respectively).
Opnsense is associated with the new port groups. DHCP service is running on each interface. Firewall configured to allow traffic.
But none of the devices connected to the new virtual switch is getting IP address from DHCP (sometimes an IP is assigned but from the wrong VLAN). Actually it breaks vusb0 & vusb1 too. No traffic gets through any of the adapters (old and new).

What am I doing wrong?

[bartjsmit]

If you build a vSwitch with two uplinks, they will load balance. Build a vSwitch per USB adaptor to keep the traffic separate.

You will need to trunk through your physical NIC's, so all TP-link ports connected to an uplink need to tag all the VLAN's that you have assigned port groups for on the vSwitch.

Have a look through the guide if you haven't already: https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-703-networking-guide.pdf

It's a bit of a tome, but you'll only need chapter 2  ;)

Bart...

[wuwos]

In my setup each adapter has their own vSwitch.

This is how the VLANs are configured on TP-Link router (running OpenWRT):

VLANID	CPU	LAN1	LAN2	LAN3	LAN4	WAN
status	✅	✅ vusb1	❌	❌	✅ vusb2	❌
1	tagged	off	off	off	off	off
2	tagged	off	off	off	off	untagged
11	tagged	untagged	untagged	untagged	off	off
12	tagged	off	off	off	tagged	off
21	tagged	off	off	off	tagged	off
22	tagged	off	off	off	tagged	off

(VLAN 1 & 2 are built-in. The only change I made was turn off all LAN ports for VLAN 1 since LAN ports 1-3 are assigned to VLAN 11 and LAN 4 is for VLAN 12, 21 & 22.)

• LAN2 & 3 and WAN are not plugged in. They should be irrelevant in this case.
• LAN1 connects to vusb1 on ESXi host. I can connect to the SSID associated with VLAN 11. Everything works.
• But once LAN4 connects to vusb2, and I connect to any SSID associated with VLAN 12, 21 or 22, everything including vusb0 vusb1 & vusb2 stops working.

[wuwos]

Actually the issue seems to be opnsense on ESXi is unable to handle more than 3 virtual NIC? Let me explain:
(vSwitch x 3, each associated with only 1 USB ethernet adapter)

This setup works: 3 port groups  (2 without any VLAN tagging, 1 with VLAN ID 12)

Code Select Expand

portgroupWAN    <-> vSwitchWAN  <-> vusb0
portgroupLAN    <-> vSwitchLAN  <-> vusb1
portgroupVLAN12 <-> vSwitchVLAN <-> vusb2

This does NOT work: 4 port groups (2 without any VLAN tagging, 1 with VLAN ID 12, 1 with VLAN ID 21)

Code Select Expand

portgroupWAN    <-> vSwitchWAN  <-> vusb0
portgroupLAN    <-> vSwitchLAN  <-> vusb1
portgroupVLAN12 <-> vSwitchVLAN <-> vusb2
portgroupVLAN21 <-> vSwitchVLAN <-> vusb2

[bartjsmit]

Actually the issue seems to be opnsense on ESXi is unable to handle more than 3 virtual NIC

Mine runs with six vNIC's - all through the single vSwitch and uplink though. Can you try with just the onboard NUC ethernet?

[wuwos]

Thanks for the suggestion. I haven't switched it to the onboard NIC yet. But there are 3 USB ethernet adapters (RTL8153 x 2 + RTL8156 x 1) connected to the host.
Once I swapped the adapter (RTL8153) which used to handle VLAN traffic for the adapter (RTL8156) which used to handle WAN traffic, VLAN seems to be working.
Now neither RTL8153 is working. ::)
I'm suspect it's an ESXi USB driver issue.

[bartjsmit]

Well done! You're always on thin ice with unsupported kit, so I would leave it as is - quit while you're ahead