Some suggestions for my new OPNSense (on ESXi host)

[weust]

I would have to check Hyper-V, but inside the VM I did.

[framura]

I resume this (old) thread because I have again same problem.

Recently I upgraded my ISP connection to 1Gbps and I repeat same tests: now I get 95% ISP's speed when I use directly my iMac with ISP's router but (only) 60% when I use my iMac with OPNSense router (obviously with same conditions).

During these tests CPU is between 35%-45%.

Where can I investigate?

TIA

[phoenix]

You shouldn't really tag on a message to an (very) old thread, a new one would have been better.

How about giving us some details of your current configuration, OPNsense version, ESXi version, which drivers are you using for the NICs, which OPNsense services are enabled (Suricata, SNMPD, etc.) etc., etc.?

FWIW, I've also recent gone to a 1Gb fibre connection and run my OPNsense on ESXi 6.5 with the VMXNET3 drivers and don't see any problem in download speed.

[weust]

I'm testing a bit at home with ESXi 6.5.0 U1, and using VMXnet3 cards made the download speed go back to ~120 Mbps. Haven't tested with E1000 cards yet.
Bare metal I do get the full 400 Mbps. Same hardware.

Creating a new VM and choosing Other/FreeBSD 64 bit selects E1000. Not VMXnet3.
Checking information on support is vague. It's supported by FreeBSD, but  not 100%.

[framura]

Actually I use OPNSense 17.7.7_1 on ESXi 6.0u2 and I use VMXNET3 drivers.

I don't have any particular service enabled, I use only some VPN connections )OPNsense is a VPN client) but not in this case because I inserted a specific firewall rule for these tests (also, if I switch off these VPN connections I get same results).

I will try to go back to E1000 drivers but it's seems really strange to me.

[weust]

I'm running with E1000 now, and speeds are normal now.
410Mbps down and 40 up.

[xinnan]

Try your setup with only 2 v-cores and see if you notice a loss at all.  If you have time. 
Also, you didn't mention any packages that are memory hungry.  Wondering if 2GB or even just 1GB wouldn't be enough ram?  Just thinking about saving untapped resources for other things if possible. 

[MasterXBKC]

Hi Allesandro,

If you assign four vCPU's to a VMware guest it will only be able to run when four physical cores are available on the host.

Between the four cores for Ubuntu and four for OPNsense, there are no free cores for ESXi itself. If you only assign two cores to OPNsense, and perhaps two cores to Ubuntu, it will be much easier for the VM's to run simultaneously with the Hypervisor.

Bart...

Disregard everything Bart is saying here, he has no idea of that which he speaks, no offense bart, but i oversee a quite large vmware vsphere cluster and have been working with vmware since back in the vmware server 2.0 days years and years ago.   These statements have no basis in fact.   An 8 core box can have all 8 cores assigned to 2, or even 15 virtual machines simultaneously as long as the cpu cores arent being constantly peaked.   This is what makes vmware so successfull, instead of having idle virtual machines wasting the power of their cpu cores that are idle, all cpu cores simply share their cycles with all the virtual machines they are assigned to, this is one of the main benefits of virtualization.

Im not speaking on opinion, we are actually a vmware partner.

[xinnan]

Overprovisioning is pretty normal.  It's a good way to make the most of the hardware.  Still, no need to be harsh.  It's not intuitive.  I'd google a little about maximizing resources and include 2017 in the search since things change.  The closer you come to keeping your CPU maxed without impairing performance, the more you are getting your money's worth.  Cycles idle are cycles wasted. 

[framura]

Thanks for these suggestions,

I will try with less RAM and with E1000 drivers (with these I already tried but WAN interface doesn't works, really strange).

Else I haven't any other idea.

[MasterXBKC]

i actually happen to have an opnsense 17.7 running in our cluster as ive been using it to develop and test an opnsense extention, the cluster is on gigabit fiber inside our datacenter directly connected to Level 3's pipeline.

I know i set it up using vmxnet3 interfaces as well and its been working flawlessly, however i dont believe ive ever done a speedtest through it, if time avails i will do a few, and drop them here for comparison, it has none of the extra content filtering, etc turned on, its just a default install with a wan and lan interface.

The cluster itself is the following:
2 x Dell R710s
|-2 x Xeon X5650 Hex Core CPUs
|-144 GB DDR3 ECC Registered Buffered RAM
|-Quad Port Intel Gigabit NICs
|-8 x 4TB Drives in RAID 10

2 x Dell R720XDs
|-2 x Xeon E5 8 Core CPUs
|-256GB DDR3 ECC Registered Buffered RAM
|-Quad Port Intel Gigabit NICs
|-24 x 4TB Drives seperated into 6 seperate 4 drive RAID 10 Arrays

4 x HP DL360 G7s
|-2 x Xeon E5620 Quad Core CPUs
|-48 GB DDR3 ECC Registered Buffered RAM
|-Quad Port Intel Gigabit NICs
|-8 x 1TB Drives in RAID 10

[MasterXBKC]

One thing i know can help from testing with a pfsense, is set the virtual machine to reserve/lock all of its ram so that vmware doesnt page it around thus causing some minor lag and delays occasionally, it wasnt a night and day difference, but i did notice some latency change, and generally a tiny bit more snappyness.

Another thought that crossed my mind, is hyperthreading enabled on your ESXI box?   If so, each core is loosely like a half a core, so to assign the equivalent of 2 full cpu cores, you actually need to assign 4.

A screenshot from your vsphere client summary tab of the host would help us identify if hyperthreading is enabled.

[framura]

Hi,

I tried following:

1) Fresh (and basic) OPNSense 17.7.7 installation, with VMXNET3 drivers: same results
2) Fresh (and basic) PFSENSE 2.4 installation with VMNETX3 drivers: same results

With E1000 drivers performances drop to 550Mbps (in both cases).

For other questions: I don't use hyperthreading, I have a C2758 CPU with 4 cores for OPNSense VM (see attachment).

Thanks for your help.

[MasterXBKC]

Do you have virtualization enabled in the bios, there should be a VT-D, and a Virtualization setting in the bios under the cpu section most likely, if these are not enabled, it will force the system to use software emulation, and this could severely rob your performance and throughput as well.

It is also possible that, assuming your running vmware on this C2758 CPU, that it could be too much for such a CPU, the virtualization overhead could be making that difference, but ive never run vmware on such a low end CPU.

[xinnan]

I have.  It will work, but you will hit a cap on bandwidth.  Probably just a 5% hit if the VM is working right.

How big a hassle would it be for you to install opnsense on the machine directly to test the limits of the physical machine and THEN compare it to opnsense in VM?