Can OPNsense handle 100Gbit in a VMware environment? Tips to test Bandwith?

[Layer8]

Hey,

we are building a new data center a the moment. During installation and testing phase, we use a OPNsense in a VMware VM as router and firewall. Before we go live, we will switch from OPNsense to NSX-T.

Our VM Hosts are Dell PowerEdge R740XD with 100Gbit interfaces (2x dual 25Gbit fibreNICs, attached over two port channels to two physical switches, which are stacked as single logical switch with 2x100Gbit links in between).

We would like to do some benchmarking/stresstesting of the environment in january but we absolutely dont know if its possible to reach 100Gbit/s in this environment.

We thought about the following test setup:

Host1 = Win or Linux on baremetal with a 25Gbit interface
ESXi1 = VMware Host with 100GBit Interface
ESXi2 = VMware Host with 100Gbit Interface
OPNsense1 = VM on ESXi1 with a vmxnet3 interface on a distributed switch connected to the 100Gbit LAG
OPNsense2 = VM on ESXi2 with a vmxnet3 interface on a distributed switch connected to the 100Gbit LAG 

OPNsense1 and OPNsense2 will be configured with 11 VLANs and static routing, to multiply a 20Gbit-Stream from Host1 to reach a combined traffic of 100Gbit.

Host1 should send a 20Gbit iperf stream over LAN to
OPNsense1 -> VLAN1 -> OPNsense2  -> VLAN2 ->
OPNsense1 -> VLAN3 -> OPNsense2  -> VLAN4 ->
...
OPNsense1 -> VLAN9 -> OPNsense2  -> VLAN10 -> OPNsense1

Do you guys have any experiences with such a setup? Are there any bootle necks by design? Like limitations in OPNsense or undocumented throughpout-limitations in VMware?

Thanks for your thoughts.

[opnfwb]

I'm not sure this would be possible on a FreeBSD 12.x based version of OPNsense due to the throughput issues that a lot of folks have reported with vmxnet3 NICs (an issue more with FreeBSD than with OPNsense). If you do attempt this, I think your best bet would be a NIC passthrough however that doesn't sound like an option if all of the hosts only have two physical uplinks (2x25GB).

I've loadsim'd virtual environments before and one of the neat tools I used for our last vSAN buildout is a free one called HCIbench. This may not be exactly what you're looking for because its more geared to test compute and storage at the same time. But it hammers the network pretty good and can give you an idea of your throughput capabilities with a loaded host.

If you just want to test raw network throughput and ensure that the physical layer can really push those speeds I think your best bet would be some Linux VMs and iperf. Even then I doubt you'll get 100GB of throughput from a single VM, you'll likely need a pool of them pushing traffic back and forth to each other. If you aren't in production yet I'd also consider booting the hosts with a Linux bootable ISO (you can easily do this via the iDRAC virtual ISO mount) and running your throughput tests on the bare hardware just to see what native physical can push.

Sorry this isn't really your answer but with this much bandwidth it can be a challenge for a lot of firewalls.

[lilsense]

The true 100Gig thruput should be with a 100GigE card, since a bundle can only max at a 25gig single stream.

you'd use over a 10TB+ data for your RW test.

[mimugmail]

You cant achieve more than 18Gbit with 21.x, in theory around 45Gbit should be possible with 22.x

[Layer8]

Thanks for all your replies.

@opnfwb: Yes, we noticed the vmxnet3 problem this week. We only have around 600-700Mbit/s routed throughput with opnsense installed in a VMware VM with vmxnet3.

Is there a workaround for this issue?

@mimugmail: Thats interesting, thanks. Whats the reason for this 18Gbit-Limitation?

@lilsense: Yes, you are also right. Its not possible to aggregate 4x25Gbit NICs to a single 100Gbit/s TCP-Stream. Thats only possible with a single physical 100G-NIC.

So, we noticed that esxi-hypervisor comes with iperf. Its located in /usr/lib/vmware/vsan/bin/iperf3 and its internaly used to test vsan performance.
Note: Its not possible to start this original file over shell, you have to copy it first to /usr/lib/vmware/vsan/bin/iperf3.copy for example and start the copy. You also have to disable the esxi firewall.

What we see with iperf3 is, that we have full throughout of 25Gbit between all esxi-hosts.

[johndchch]

@opnfwb: Yes, we noticed the vmxnet3 problem this week. We only have around 600-700Mbit/s routed throughput with opnsense installed in a VMware VM with vmxnet3.

Is there a workaround for this issue?

that’s an issue with your esxi setup, not opnsense - I’m saturating a 1gbit link just fine running under esxi7

what is the cpu in your host? there’s plenty of bits in 21.7 that are single threaded - hence you need a cpu with decent single core speed, throwing more cores at the vm won’t help

also you need to lock the opnsense image in ram

obviously what NIC you’re using in your host also matters….generally if you want decent performance you want intel NICs in the host

[mimugmail]

Thanks for all your replies.

@opnfwb: Yes, we noticed the vmxnet3 problem this week. We only have around 600-700Mbit/s routed throughput with opnsense installed in a VMware VM with vmxnet3.

Is there a workaround for this issue?

@mimugmail: Thats interesting, thanks. Whats the reason for this 18Gbit-Limitation?

@lilsense: Yes, you are also right. Its not possible to aggregate 4x25Gbit NICs to a single 100Gbit/s TCP-Stream. Thats only possible with a single physical 100G-NIC.

So, we noticed that esxi-hypervisor comes with iperf. Its located in /usr/lib/vmware/vsan/bin/iperf3 and its internaly used to test vsan performance.
Note: Its not possible to start this original file over shell, you have to copy it first to /usr/lib/vmware/vsan/bin/iperf3.copy for example and start the copy. You also have to disable the esxi firewall.

What we see with iperf3 is, that we have full throughout of 25Gbit between all esxi-hosts.

Bad locking in pf handling