Some suggestions for my new OPNSense (on ESXi host)

[framura]

Hi,

yesterday I just installed OPNSense (16.1.5) on a VM inside ESXi 6 host.

I use a Supermicro A1SRM-2758F motherboard, with 16GB RAM and 256 GB SSD: this MB has 4 Ethernet ports (+1 for IPMI), CPU is 2,40Ghz with 8 cores, 6 SATA ports: on this machine I installed VMWare ESXi 6.


At this moment, I installed OPNSense on a VM with 4GB RAM, 16GB disk space (on the SSD), 4 cores, with 2 Ethernet ports, one for LAN traffic and one for WAN traffic (at this moment only IPv4 but I plan to add IPv6): in next few days I will add a second VM, where I will install Ubuntu Server (I will use it as file server with 4 SATA HDDs), with 4 cores and 2 LAN ports.

I will use OPNSense machine as firewall/router, DHCP server (on the LAN side), proxy server, VPN gateway (i.e. OPNSense will be connect to a VPN Provider for encrypt Internet traffic, I don't need VPN on LAN side).

I need some advice on OPNSense:

1) I think 16GB disk space are sufficient (maybe exaggerated), but disk space is not a problem
2) I have some doubt about RAM (4GB) and core's number (4), in particular about VPN traffic: my WAN speed is actually 100Mbps but in near future I will upgrade to 300Mbps and I would like to not slow down Internet speed with VPN.

In the next few days I'll do some performance tests but your suggestions on this configuration are welcome.

Thanks in advance

Alessandro

P.S.: I would like to contribute to OPNSense (many thanks for your product): I will donate but I would also like to participate more concretely, for example with translation (my native language is italian).

[phoenix]

What you haven't mentioned is the load you will have on this firewall, is this replacing another firewall or what? Is this in a business environment or a home LAN, how many users and what sort of traffic? I'd also suggest you are likely to be allocating too many vCPUs to the firewall and I'd guess you may need more disk space if you have many users and/or growing log file requirements.

[framura]

Hi phoenix,

you are right, I forgot to mention some information.

My environment is a home LAN (20 users max) but we use it also for our work: traffic is diversified, from classic Web surfing to streaming services, file transfer and remote control of infrastructure of our clients.

I am replacing my previous firewall (ASUS RT-N16, with Tomato firmware): this was the bottleneck with VPN (lack of CPU power).

Why do you suggest I am using too much vCPUs?

Thanks
Thanks

[bartjsmit]

Hi Allesandro,

If you assign four vCPU's to a VMware guest it will only be able to run when four physical cores are available on the host.

Between the four cores for Ubuntu and four for OPNsense, there are no free cores for ESXi itself. If you only assign two cores to OPNsense, and perhaps two cores to Ubuntu, it will be much easier for the VM's to run simultaneously with the Hypervisor.

You should only assign more vCPU's to a VM if it becomes CPU starved. VMware has a good PDF on performance: https://pubs.vmware.com/vsphere-60/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-60-monitoring-performance-guide.pdf

Also, if your logs are important, you should send them to a remote server.

Bart...

[framura]

Hi,

I made some tests (with 4 vCPUs) and today I will repeat with 2 vCPUs.

I am not sure about AES-NI usage (with Openvpn): I must to configure Cryptodev in OpenVPN and AES-NI into system-settings (if I remember correctly) or AES-NI is always used independently from these two configurations (obviously if CPU has AES-NI capabilities)?

To maximize performance (as indicated in vmware document) I will also try to disable any CPU power management in the BIOS and to configure ESX with "High Performance" profile: do you think is worth the effort?

Thanks

[franco]

AES-NI for OpenVPN only works when properly configured as you stated (System Settings AES-NI and OpenVPN cryptodev). Please note that it doesn't work for LibreSSL due to the cryptodev engine removal.

[framura]

Thanks.

Another question: I think AES-NI works with AES-128(256)-CBC encryption algorithm but I am not sure about BF-CBC algorithm.

I haven't found explicit informations about this.

Thanks

Alessandro

[cruxv]

If you assign four vCPU's to a VMware guest it will only be able to run when four physical cores are available on the host.

That's not correct. The ESXi CPU scheduler will actually run an instruction on any available core unless you specifically state to use CPU affinity on a VM - even then it won't "reserve" all the core for a single VM.

http://www.vmware.com/files/pdf/techpaper/VMware-vSphere-CPU-Sched-Perf.pdf

[framura]

Hi,

I resume this thread for my new problem.

Recently I upgraded my WAN from 100Mbps to 500Mbps: obviously I made some tests with my opnsense setup (VM on VMWare ESXi 6 Update 2) but I am not satisfied with speed.

First of all I tested WAN real speed (web surfing, torrent, ftp, usenet) without OPNSense (my iMac connected directly to Internet), no VPN and I found I get 480Mbps (for example with ftp file transfer or usenet) in some cases: I am impressed.

I repeated same tests (obviously under same conditions) with OPNSense (16.1.15 version) but I get an 30% average decrease: not good.

Then I checked OPNSense cfg, but I LRO, TSO and checksum was already disabled: on ESXi side I suspect TSO/LRO is enabled and I used E1000 driver on OPNSense VM machine (maybe VMXNET3 is better?).

Can you suggest me right direction to investigate?

Thanks in advance

Alessandro

[weust]

Why would you use E1000 cards instead of VMXNET3 cards?
Always use VMXNET3 unless you can't.

I would think that will remove your issues.
When I ran ESXi 6 I had no problem going up to 200Mbit. The max I had at the time.

[framura]

Thanks weust,

I used E1000 as suggested on opnsense wiki.

Now I will try to change drivers (WAN and LAN)  and I will post my results.

[weust]

Also seems to be mainly directed to Traffic Shaping.
If you don't use that, stick to VMXNET3.

Never read the wiki. Wasn't there when I started with OPNsense 15.1 in early 2015 :-)

[framura]

Thanks,

just tried: I disabled old WAN and LAN network adapters (with E1000 drivers) from ESXi and created two new adapters with VMXNET3 drivers.

After setup these new two adapters also on OPNSense (one for WAN and one for LAN), named vmx0 and vmx1, I tried some transfer tests and I get 450Mbps as peak speed: not bad

Perhaps it is enough (?).

Thanks for your help

[weust]

I have the same motherboard running Hyper-V 2012 R2 an reaching my 300Mbit/s is not a problem at all.
It feels to me you should be able to reach at least the 480 you got with iMac.

Btw, my VM runs with two cores and 2GB RAM. 1 is enough if I don't use IPS/IDS.

[framura]

I have 4 cores inside OPNSense VM, only useful when I use VPN.

On Hyper-V have you disabled LSO, TSO, etc?

I disabled inside OPNSense but on ESXi are enabled (by default).