20Gb WAN - Slow with OpnSense

[benyamin]

Not much to do then methinks...

I hesitate to think that adding a hypervisor layer would improve performance, but I am certainly willing to try that. It seems like an additional bottleneck layer.

Adding the hypervisor layer might not improve performance, but if it does, it might highlight a potential driver / other issue in FreeBSD. Your CPU is supported on all current versions of ESXi. QEMU/KVM-based, e.g. Proxmox, should be fine too.

You might also want to check for BIOS (a new one just got released) and CPU microcode updates too.

Might want to be sure these are the latest, especially the CPU microcode. The most recent update for your CPU at the time of posting is 10 August 2022, revision 02006e05. The microcode update *should* be in a suitably recent DELL BIOS, but you may want to check...

[Supermule]

If the hardware matches, the hypervisor can push 20gbit/s.

It adds almost nothing and if you are worried then use passthrough on the NIC's

[benyamin]

...2x Mellanox Connectx-5 Dual 100Gb Nics.
I have my WAN configured as Lagg0-Vlan300, and Lan as Lagg1-Vlan10
I have tested with iperf3 on LAN ... and can never exceed 3000Mbs.
When booting the system on debian live, and configuring the interfaces to match the OpnSense I am able to get ... 930Gb from LAN using ntttcp.

@rebdbarrow73, just looking at your first post again...

What LAGG protocol did you use and did you match the configuration on the switch? In any case, i.e. no matter which protocol you do use, you will not be able to exceed 100Gbps, i.e. the speed of a single interface, per connection. Did you also bond the NICs on the Debian Live system? Presuming you haven't already done so, I would recommend unbonding the LAGG for the purpose of initial testing and use a single NIC for now.
 
You mention 930Gb, but this seems ambiguous to me. I presumed you meant 93Gbps of throughput, which is very close to wire speed. Also, how many connections did you use in your test?

I note that you are using two different test programs, iperf3 and ntttcp. If possible, it would be best to use the same program on each test platform (or alternatively both programs on each platform).

If the hardware matches, the hypervisor can push 20gbit/s.
It adds almost nothing and if you are worried then use passthrough on the NIC's

The idea for the test would be to homogenise the hardware of the two different test platforms by way of virtualisation. If the results are the same between the two, then the underlying problem is likely due to a driver issue. For that reason you shouldn't use passthrough for the test.

That being said, per @Supermule's post, your production use case might find hypervisor limits sufficient and you could switch to passthrough NICs (at least some of them) in production if necessary - although in your case, I suspect this would result in SPOFs.

It is also worth noting that using a firewall "on top" of a hypervisor results in additional multiple potential attack vectors, especially if there is no bastion host between the public internet and OPNsense (or any other firewall for that matter).

[ilya_rt]

This is what I got from ConnectX-3 and FreeBSD-12. 3.2Gbit\sec and no more. And it could hang up Mellanox NIC and only power reset could help. My issues were IB drivers and I could not resolve it, so went for CentOS. On Linux I'v got about 20+ Gbit\sec on the same hardware. It was not NAT , but direct NIC-to-NIC tests. So I see that even today FreeBSD with Mellanox is not a good mix.

[Coldfirex]

12.3 is a bit old.  Can you see if the results change with 13.2 and/or the upcoming 14?

[benyamin]

My issues were IB drivers...

I'm pretty sure we are dealing with the Ethernet here and not InfiniBand too.
Like comparing apples with oranges...
Same with FreeBSD 12.3 vs. 13.1.
Same with ConnectX-3 vs. ConnectX-5 (different driver base).