esxi 6.7 virtual nic performance dreadful! Suggestions?

[sparticle]

We have today migrated our OPNSense router to a VMware ESXI 6.7 VM.

Install went well despite config import losing all PPPoE settings.

We had to reinstall suricata and a few other things.

It was up and running pretty quick.

However, the network performance is dreadful.

When creating the VM the closest option we could find was Other FREEBSD12 or later 64 bit

The vnic options were e1000e or VMXNET3

I had read somewhere that e1000e was the right choice so that is what we chose.

iperf3 run shows this:

Code Select Expand


Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  38.1 MBytes   320 Mbits/sec   57    624 KBytes       
[  5]   1.00-2.00   sec  35.0 MBytes   294 Mbits/sec    0    697 KBytes       
[  5]   2.00-3.00   sec  35.0 MBytes   294 Mbits/sec    0    751 KBytes       
[  5]   3.00-4.00   sec  33.8 MBytes   283 Mbits/sec    2    571 KBytes       
[  5]   4.00-5.00   sec  35.0 MBytes   294 Mbits/sec    0    611 KBytes       
[  5]   5.00-6.00   sec  32.5 MBytes   273 Mbits/sec    0    652 KBytes       
[  5]   6.00-7.00   sec  33.8 MBytes   283 Mbits/sec    0    690 KBytes       
[  5]   7.00-8.00   sec  33.8 MBytes   283 Mbits/sec    0    727 KBytes       
[  5]   8.00-9.00   sec  36.2 MBytes   304 Mbits/sec    1    540 KBytes       
[  5]   9.00-10.00  sec  33.8 MBytes   283 Mbits/sec    0    618 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   347 MBytes   291 Mbits/sec   60             sender
[  5]   0.00-10.02  sec   345 MBytes   288 Mbits/sec                  receiver
CPU Utilization: local/sender 2.0% (0.2%u/1.7%s), remote/receiver 40.5% (11.3%u/29.2%s)
snd_tcp_congestion cubic
rcv_tcp_congestion newreno


Any other VM on the esxi host run pretty much at the full GB of the vswitch uplinks.

example from the lan server.

Code Select Expand


Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   111 MBytes   933 Mbits/sec                 
[  5]   1.00-2.00   sec   112 MBytes   940 Mbits/sec                 
[  5]   2.00-3.00   sec   112 MBytes   940 Mbits/sec                 
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec                 
[  5]  10.00-10.00  sec   334 KBytes   900 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate
[  5] (sender statistics not available)
[  5]   0.00-10.00  sec  1.09 GBytes   940 Mbits/sec                  receiver
rcv_tcp_congestion cubic
iperf 3.9


Can anyone assist with better settings or config changes please.

Cheers
Spart

[sparticle]

A bit more information.

If we disable suricata then we get upto approx. 730Mb/sec

We have tried the VMXNET drivers also.

CPU and Memory for the VM are low even with Suricata switched on.

Really need some help with this please.

Cheers
Spart

[sparticle]

Talking to myself I know but after a full day of testing and reading endless posts/kb's etc. I am  not really any nearer to a solution.

This is the latest release of OPNSense installed clean on a new esxi 6.7 VM with 4VCPU and 8GB memory. Host is a dell R720 with dual 2650 v2 and a 4 port Broadcom BCM5720.

The test machine is a I7 8700K 12 CPU's and 32Gb ram.

All linux VM's run at full 1GB wire speed across the lan with iperf3 testing.

The OPNSense VM varies in speed but is between 250 - 350 Mb/s slower. Best speeds seems to be with LRO on and tunable hw.pci.honor_msi_blacklist = 0

With LRO off and the tunable removed then speed is roughly a 1/3 of the linux machines some of which are running old versions like 16.04 for instance.

This system is in the sticks not in civilisation so every Mb/s counts when we are doing remote backups etc.

Cheers
Spart

[phoenix]

Did you actually try the VMXNET3 driver? Unfortunately I only have an ADSL connection here in the UK but when I lived in France I had a full Gigabit fibre connection and I ran the VMXNET3 drivers on OPNsense for about 8 years and never had a slow download  and always the full speed that was also on ESXi 6.7 and also updated to ESXi 7.

[sparticle]

Did you actually try the VMXNET3 driver? Unfortunately I only have an ADSL connection here in the UK but when I lived in France I had a full Gigabit fibre connection and I ran the VMXNET3 drivers on OPNsense for about 8 years and never had a slow download  and always the full speed that was also on ESXi 6.7 and also updated to ESXi 7.

Yes, it is running with that adaptor now but performance is not optimal. The issue it seems has nothing to do with VMWare it is the FBSD driver. There are many open 'bugs' and no action from the dev team.

We are limited in our options on this one. There is no hardware to spin up at the site. The e1000 is worse than the VMXNET3.

We may have an option of installing a dual port NIC apart from the standard quad port netXtreme already in it. We could in theory pass it through esxi to the OPNSense VM and use that.

Looking through the HCL for FreeBSD 13.1 (https://www.freebsd.org/releases/13.1R/hardware/) the vmx driver is not even listed. SO maybe they dropped support for it quietly!


Cheers