Anybody out there who is using OPNsense as a switch?

[Layer8]

Hey all,

my old 16x 1G SRW2016 Linksys switch is driving me nuts, because there is no CLI and the WebUI can only be opend with <=IE8 on WinXP (i have a WinXP-VM to manage this switch). I decided to upgrade to 10G, but i dont know what to do.

Upgrading to a 10G Switch would be a safe bank. But i also could install a 4x 10G PCIe-card in my OPNsense (which is virtualized and running on a Ryzen 4750G). I also have 2x 10GbE (Intel X550-AT2) and (2x 1GbE Intel i210) onboard, so there would be enough ports in this machine to connect all needed devices.

Does anyone have experience running the OPNsense as a switch?

I mean, it should be possible to bridge phys NIC interfaces and let them act like a single broadcast domain.

But will it work reliable? And is there some kind of hardware (depending on used NIC of course) offload support or would all traffic flow over the CPU even if its only Layer2 traffic in a broadcast domain?

Thanks for sharing your experiences.

[mimugmail]

I dont have practical experience but it runs in software, I would only use it for Gigabit, not more

[Patrick M. Hausen]

The FreeBSD bridge is solid, reliable, and has only recently seen a performance improvement by a factor of at least five. I honestly do not know if it can switch at Gigabit or beyond. I'd be interested in measurments. 

https://freebsdfoundation.org/blog/500-if_bridge-performance-improvement/

[Layer8]

I dont have practical experience but it runs in software, I would only use it for Gigabit, not more

OPNsense runs in software and computes L3 traffic which i would suggest is more computeintensiv than L2. Even if you have physical NICs attached to your OPNsense hardware acceleration is disabled per default. Also, nearly every Hypervisor is running its network stack in software without hardware acceleration. So, thats not an argument anymore i think.

The only argument against SDN with OPNsense would be if FreeBSD is very inefficient in switching.

[Layer8]

The FreeBSD bridge is solid, reliable, and has only recently seen a performance improvement by a factor of at least five. I honestly do not know if it can switch at Gigabit or beyond. I'd be interested in measurments. 

https://freebsdfoundation.org/blog/500-if_bridge-performance-improvement/

I dont use the 10G interfaces in my server at the moment. So i could add these to a OPNsense VM and bridge both. Then i will connect a PC with 10G NIC to each of the bridged cards and do some iperf tests. Jep, i think this would be a good idea. Thanks for being my muse. :-)

[mimugmail]

I dont have practical experience but it runs in software, I would only use it for Gigabit, not more

OPNsense runs in software and computes L3 traffic which i would suggest is more computeintensiv than L2. Even if you have physical NICs attached to your OPNsense hardware acceleration is disabled per default. Also, nearly every Hypervisor is running its network stack in software without hardware acceleration. So, thats not an argument anymore i think.

The only argument against SDN with OPNsense would be if FreeBSD is very inefficient in switching.

Everytime when I talk with Ad about bridges they dont recommend using it

[Layer8]

Ok, sounds like a FreeBSD thing. Ill test it. But who is Ad ?

[Patrick M. Hausen]

Ad Schellevis, one of the main OPNsense developers.

I have a data centre worth of VNET jails on FreeBSD all bridged. So do give it a try. Make sure to enable spanning tree. It's in the UI. The FreeBSD default is off, which can lead to "interesting" effects. 

[Vexz]

I use an 8 port machine for my OPNsense where 7 ports are bridged. I can confirm after almost a year now that it's working perfectly fine and I never even once had any problem.

[hkais]

I am using virtualized OPNsense atop of a kvm virtualizer. Kvm is using also bridges either openswitch or direct kernel bridges. Both work like a charm compared to previous hardware solution.
So do not see any issue to do like you planned (SDN)
So having multiple VLANs for the internal and external networks and OPNsense is the router/firewall/NAT in between all of them

Anyway one flaw in my message, KVM is based in my case on linux, so cannot say for sure for BSD/OPNsense if it will perform also so well there.

[Layer8]

Thanks Vexz and hkais for sharing your experiences.

Can you please tell something about bandwith and load?

Vexz: If i understand you right, you realized the bridges inside OPNsense, right? Do you have a bare metal installation or are you running you installation on a hypervisor? What are your hardware specs?

hkais: So, you realized bridges in the hypervisor, right? I have to figure out if this is possible with (free) ESXi v7 or v8. I think in default, the virtual switch of ESXi filters such traffic.

[Vexz]

Can you please tell something about bandwith and load?

Feels exactly like a simple L2-Switch with a FW rule set to "any allow" so you know what to expect about bandwidth and load. I've never seen the CPU load go above 15%.

Vexz: If i understand you right, you realized the bridges inside OPNsense, right?

Yes.

Do you have a bare metal installation or are you running you installation on a hypervisor? What are your hardware specs?

It's a bare metal installation. Intel Core i3-8130U, 8GB RAM, 128GB NVMe SSD, 8x Intel i211-AT NICs.