Slow LAN speed in OPNsense vs pfSense (w/ Proxmox)

[nimaim]

So I recently got a fitlet2 w/ J3455 processor (and put in 16GB RAM). I am running Proxmox on it so I can virtualize some things. IOMMU is supported but unfortunately, even with ACS override, a ton of devices belong to the same group and I cannot pass them through individually. I guess that's the risk you take with these small embedded devices. As a result, I'm forced to use the virtualized VirtIO net drivers. That's fine: I bridged each port in Proxmox and passed them through for WAN and LAN and it worked fine.

I set up OPNsense with 8GB RAM, host processor, VirtIO block disk and VirtIO net, and all the default opn options (so hardware offloading all disabled) and let it update to 20.7.4. I then installed iperf3 and ran a test with another PC on the same gigabit switch. I average ~780 Mbit/sec which seems rather slow. My Proxmox hypervisor gets 950 Mbit/sec. I then installed pfSense just to check and with the default options there, I pretty much get the same as Proxmox, 940 Mbit/sec average. CPU load is ~45% when flooding it.

Any ideas what I can tune? I would love to use OPNsense but this would break it for me. Thanks.

[mimugmail]

You can update to 2.5.0 pfsense and will see same performance drop. Problem with FreeBSD 12

[nimaim]

Really? In regards to what exactly? VM/VirtIO driver performance? Is there a changelog showing these changes?

I updated to pfSense 2.5.0a (FreeBSD 12) just to check and performance is just slightly lower .. no where near as bad as OPN. Screenshot attached for proof.

And just for completeness sake, I posted the best run I got from OPN as well, though it usually stays < 800 Mbit/s. Results are all over the place with OPN.

[mimugmail]

Can you also test against 20.1.9?

[nimaim]

Sure, give me a sec to set up another VM.

[nimaim]

Nop, basically the same or worse on 20.1.9. ... I'm guessing it's either a driver issue (unless it's the same exact driver as pfSense's) or some other tuning parameter. The values also tend to fluctuate all over the place. Not sure what else to try.

[mimugmail]

Iperf running in the Firewall itself is known to be run badly. Best would be to run iperf through it

[nimaim]

How else am I supposed to test the LAN speed within it? This is from a default install with no rules enabled, no packages installed. And if that was the case, I would be seeing the same thing in pfSense, no? Clearly, there is something inherently different in OPN causing a 100-200 Mbit/s speed decrease. Unfortunately, that's tough to overlook.

EDIT: Just tested in an Untangle VM (yes I know it's paid, but just did a trial) and got the full max line rate of 950 Mbit/s steady from also within the VM. I wanted to see how well Debian VirtIO drivers worked vs. BSD's. So this is also working.

[nimaim]

Any other suggestions before I give up on this (besides running it bare metal)?

[mimugmail]

No, I dont have the time for a lab to reproduce. Only thing I can say is that when 80% of max throughput is too slow for you, you maybe should design for 10G and higher equipment. No experience about Proxmox, only bare metal

[nimaim]

I'm not complaining that I'm not getting max throughout (there is always some loss when virtualizing), I'm complaining that OPN is not giving me the same results as just about everything else I've tried, which is close to line rate. I'm just trying to figure out what that is and fix it. My WAN is also gigabit fiber, so yes it's a big deal for me.

Designing for 10G is not really a valid answer: I'd have to redo my entire network for that, get completely new equipment, a different appliance device, etc. etc. and who's to say I get even close to saturating that with OPN? I'm sorry but I also don't have thousands of $ to spend to upgrade just to fix what seems to be a software/driver issue.

[devilkin]

What's the difference wrt cpu usage, and how if you stay with freebsd 12 based builds?

Sent from my IN2023 using Tapatalk

[nimaim]

I mentioned it above .. ~45% with OPN when flooding it with iperf packets. pfSense 2.4.5 hovers ~25% but that's comparing apples to oranges as that's running FreeBSD 11.x. But I got similar numbers with pfSense 2.5.0a (I nuked that particular VM since they release dev snapshots everyday). Is it normal for HardenedBSD to be more CPU intensive?

[mimugmail]

Maybe iperf performs worse with HBSD, thats why it's better tocmeasure through the firewall

[marios]

@nimaim did you ever get OPN running on Proxmox with decent performance?