We have today migrated our OPNSense router to a VMware ESXI 6.7 VM.
Install went well despite config import losing all PPPoE settings.
We had to reinstall suricata and a few other things.
It was up and running pretty quick.
However, the network performance is dreadful.
When creating the VM the closest option we could find was Other FREEBSD12 or later 64 bit
The vnic options were e1000e or VMXNET3
I had read somewhere that e1000e was the right choice so that is what we chose.
iperf3 run shows this:
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 38.1 MBytes 320 Mbits/sec 57 624 KBytes
[ 5] 1.00-2.00 sec 35.0 MBytes 294 Mbits/sec 0 697 KBytes
[ 5] 2.00-3.00 sec 35.0 MBytes 294 Mbits/sec 0 751 KBytes
[ 5] 3.00-4.00 sec 33.8 MBytes 283 Mbits/sec 2 571 KBytes
[ 5] 4.00-5.00 sec 35.0 MBytes 294 Mbits/sec 0 611 KBytes
[ 5] 5.00-6.00 sec 32.5 MBytes 273 Mbits/sec 0 652 KBytes
[ 5] 6.00-7.00 sec 33.8 MBytes 283 Mbits/sec 0 690 KBytes
[ 5] 7.00-8.00 sec 33.8 MBytes 283 Mbits/sec 0 727 KBytes
[ 5] 8.00-9.00 sec 36.2 MBytes 304 Mbits/sec 1 540 KBytes
[ 5] 9.00-10.00 sec 33.8 MBytes 283 Mbits/sec 0 618 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 347 MBytes 291 Mbits/sec 60 sender
[ 5] 0.00-10.02 sec 345 MBytes 288 Mbits/sec receiver
CPU Utilization: local/sender 2.0% (0.2%u/1.7%s), remote/receiver 40.5% (11.3%u/29.2%s)
snd_tcp_congestion cubic
rcv_tcp_congestion newreno
Any other VM on the esxi host run pretty much at the full GB of the vswitch uplinks.
example from the lan server.
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 111 MBytes 933 Mbits/sec
[ 5] 1.00-2.00 sec 112 MBytes 940 Mbits/sec
[ 5] 2.00-3.00 sec 112 MBytes 940 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec
[ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec
[ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec
[ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec
[ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec
[ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec
[ 5] 10.00-10.00 sec 334 KBytes 900 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate
[ 5] (sender statistics not available)
[ 5] 0.00-10.00 sec 1.09 GBytes 940 Mbits/sec receiver
rcv_tcp_congestion cubic
iperf 3.9
Can anyone assist with better settings or config changes please.
Cheers
Spart
A bit more information.
If we disable suricata then we get upto approx. 730Mb/sec
We have tried the VMXNET drivers also.
CPU and Memory for the VM are low even with Suricata switched on.
Really need some help with this please.
Cheers
Spart
Talking to myself I know but after a full day of testing and reading endless posts/kb's etc. I am not really any nearer to a solution.
This is the latest release of OPNSense installed clean on a new esxi 6.7 VM with 4VCPU and 8GB memory. Host is a dell R720 with dual 2650 v2 and a 4 port Broadcom BCM5720.
The test machine is a I7 8700K 12 CPU's and 32Gb ram.
All linux VM's run at full 1GB wire speed across the lan with iperf3 testing.
The OPNSense VM varies in speed but is between 250 - 350 Mb/s slower. Best speeds seems to be with LRO on and tunable hw.pci.honor_msi_blacklist = 0
With LRO off and the tunable removed then speed is roughly a 1/3 of the linux machines some of which are running old versions like 16.04 for instance.
This system is in the sticks not in civilisation so every Mb/s counts when we are doing remote backups etc.
Cheers
Spart
Did you actually try the VMXNET3 driver? Unfortunately I only have an ADSL connection here in the UK but when I lived in France I had a full Gigabit fibre connection and I ran the VMXNET3 drivers on OPNsense for about 8 years and never had a slow download and always the full speed that was also on ESXi 6.7 and also updated to ESXi 7.
Quote from: phoenix on October 17, 2022, 10:22:53 PM
Did you actually try the VMXNET3 driver? Unfortunately I only have an ADSL connection here in the UK but when I lived in France I had a full Gigabit fibre connection and I ran the VMXNET3 drivers on OPNsense for about 8 years and never had a slow download and always the full speed that was also on ESXi 6.7 and also updated to ESXi 7.
Yes, it is running with that adaptor now but performance is not optimal. The issue it seems has nothing to do with VMWare it is the FBSD driver. There are many open 'bugs' and no action from the dev team.
We are limited in our options on this one. There is no hardware to spin up at the site. The e1000 is worse than the VMXNET3.
We may have an option of installing a dual port NIC apart from the standard quad port netXtreme already in it. We could in theory pass it through esxi to the OPNSense VM and use that.
Looking through the HCL for FreeBSD 13.1 (https://www.freebsd.org/releases/13.1R/hardware/) the vmx driver is not even listed. SO maybe they dropped support for it quietly!
Cheers