OPNsense Forum

English Forums => Hardware and Performance => Topic started by: framura on March 04, 2016, 10:49:35 am

Title: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on March 04, 2016, 10:49:35 am
Hi,

yesterday I just installed OPNSense (16.1.5) on a VM inside ESXi 6 host.

I use a Supermicro A1SRM-2758F motherboard, with 16GB RAM and 256 GB SSD: this MB has 4 Ethernet ports (+1 for IPMI), CPU is 2,40Ghz with 8 cores, 6 SATA ports: on this machine I installed VMWare ESXi 6.


At this moment, I installed OPNSense on a VM with 4GB RAM, 16GB disk space (on the SSD), 4 cores, with 2 Ethernet ports, one for LAN traffic and one for WAN traffic (at this moment only IPv4 but I plan to add IPv6): in next few days I will add a second VM, where I will install Ubuntu Server (I will use it as file server with 4 SATA HDDs), with 4 cores and 2 LAN ports.

I will use OPNSense machine as firewall/router, DHCP server (on the LAN side), proxy server, VPN gateway (i.e. OPNSense will be connect to a VPN Provider for encrypt Internet traffic, I don't need VPN on LAN side).

I need some advice on OPNSense:

1) I think 16GB disk space are sufficient (maybe exaggerated), but disk space is not a problem
2) I have some doubt about RAM (4GB) and core's number (4), in particular about VPN traffic: my WAN speed is actually 100Mbps but in near future I will upgrade to 300Mbps and I would like to not slow down Internet speed with VPN.

In the next few days I'll do some performance tests but your suggestions on this configuration are welcome.

Thanks in advance

Alessandro

P.S.: I would like to contribute to OPNSense (many thanks for your product): I will donate but I would also like to participate more concretely, for example with translation (my native language is italian).

Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: phoenix on March 04, 2016, 10:54:55 am
What you haven't mentioned is the load you will have on this firewall, is this replacing another firewall or what? Is this in a business environment or a home LAN, how many users and what sort of traffic? I'd also suggest you are likely to be allocating too many vCPUs to the firewall and I'd guess you may need more disk space if you have many users and/or growing log file requirements.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on March 04, 2016, 12:56:49 pm
Hi phoenix,

you are right, I forgot to mention some information.

My environment is a home LAN (20 users max) but we use it also for our work: traffic is diversified, from classic Web surfing to streaming services, file transfer and remote control of infrastructure of our clients.

I am replacing my previous firewall (ASUS RT-N16, with Tomato firmware): this was the bottleneck with VPN (lack of CPU power).

Why do you suggest I am using too much vCPUs?

Thanks
Thanks
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: bartjsmit on March 04, 2016, 05:17:08 pm
Hi Allesandro,

If you assign four vCPU's to a VMware guest it will only be able to run when four physical cores are available on the host.

Between the four cores for Ubuntu and four for OPNsense, there are no free cores for ESXi itself. If you only assign two cores to OPNsense, and perhaps two cores to Ubuntu, it will be much easier for the VM's to run simultaneously with the Hypervisor.

You should only assign more vCPU's to a VM if it becomes CPU starved. VMware has a good PDF on performance: https://pubs.vmware.com/vsphere-60/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-60-monitoring-performance-guide.pdf

Also, if your logs are important, you should send them to a remote server.

Bart...
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on March 07, 2016, 12:44:58 pm
Hi,

I made some tests (with 4 vCPUs) and today I will repeat with 2 vCPUs.

I am not sure about AES-NI usage (with Openvpn): I must to configure Cryptodev in OpenVPN and AES-NI into system-settings (if I remember correctly) or AES-NI is always used independently from these two configurations (obviously if CPU has AES-NI capabilities)?

To maximize performance (as indicated in vmware document) I will also try to disable any CPU power management in the BIOS and to configure ESX with "High Performance" profile: do you think is worth the effort?

Thanks
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: franco on March 07, 2016, 01:40:20 pm
AES-NI for OpenVPN only works when properly configured as you stated (System Settings AES-NI and OpenVPN cryptodev). Please note that it doesn't work for LibreSSL due to the cryptodev engine removal.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on March 08, 2016, 09:55:29 am
Thanks.

Another question: I think AES-NI works with AES-128(256)-CBC encryption algorithm but I am not sure about BF-CBC algorithm.

I haven't found explicit informations about this.

Thanks

Alessandro


Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: cruxv on May 14, 2016, 10:54:56 pm

If you assign four vCPU's to a VMware guest it will only be able to run when four physical cores are available on the host.


That's not correct. The ESXi CPU scheduler will actually run an instruction on any available core unless you specifically state to use CPU affinity on a VM - even then it won't "reserve" all the core for a single VM.

http://www.vmware.com/files/pdf/techpaper/VMware-vSphere-CPU-Sched-Perf.pdf (http://www.vmware.com/files/pdf/techpaper/VMware-vSphere-CPU-Sched-Perf.pdf)
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on June 06, 2016, 09:53:33 pm
Hi,

I resume this thread for my new problem.

Recently I upgraded my WAN from 100Mbps to 500Mbps: obviously I made some tests with my opnsense setup (VM on VMWare ESXi 6 Update 2) but I am not satisfied with speed.

First of all I tested WAN real speed (web surfing, torrent, ftp, usenet) without OPNSense (my iMac connected directly to Internet), no VPN and I found I get 480Mbps (for example with ftp file transfer or usenet) in some cases: I am impressed.

I repeated same tests (obviously under same conditions) with OPNSense (16.1.15 version) but I get an 30% average decrease: not good.

Then I checked OPNSense cfg, but I LRO, TSO and checksum was already disabled: on ESXi side I suspect TSO/LRO is enabled and I used E1000 driver on OPNSense VM machine (maybe VMXNET3 is better?).

Can you suggest me right direction to investigate?

Thanks in advance

Alessandro
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: weust on June 06, 2016, 10:30:03 pm
Why would you use E1000 cards instead of VMXNET3 cards?
Always use VMXNET3 unless you can't.

I would think that will remove your issues.
When I ran ESXi 6 I had no problem going up to 200Mbit. The max I had at the time.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on June 06, 2016, 10:36:58 pm
Thanks weust,

I used E1000 as suggested on opnsense wiki.

Now I will try to change drivers (WAN and LAN)  and I will post my results.

Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: weust on June 06, 2016, 11:01:55 pm
Also seems to be mainly directed to Traffic Shaping.
If you don't use that, stick to VMXNET3.

Never read the wiki. Wasn't there when I started with OPNsense 15.1 in early 2015 :-)
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on June 06, 2016, 11:32:57 pm
Thanks,

just tried: I disabled old WAN and LAN network adapters (with E1000 drivers) from ESXi and created two new adapters with VMXNET3 drivers.

After setup these new two adapters also on OPNSense (one for WAN and one for LAN), named vmx0 and vmx1, I tried some transfer tests and I get 450Mbps as peak speed: not bad :)

Perhaps it is enough (?).

Thanks for your help
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: weust on June 06, 2016, 11:49:31 pm
I have the same motherboard running Hyper-V 2012 R2 an reaching my 300Mbit/s is not a problem at all.
It feels to me you should be able to reach at least the 480 you got with iMac.

Btw, my VM runs with two cores and 2GB RAM. 1 is enough if I don't use IPS/IDS.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on June 07, 2016, 11:32:47 am
I have 4 cores inside OPNSense VM, only useful when I use VPN.

On Hyper-V have you disabled LSO, TSO, etc?

I disabled inside OPNSense but on ESXi are enabled (by default).
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: weust on June 07, 2016, 03:23:23 pm
I would have to check Hyper-V, but inside the VM I did.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on November 08, 2017, 03:07:47 pm
I resume this (old) thread because I have again same problem.

Recently I upgraded my ISP connection to 1Gbps and I repeat same tests: now I get 95% ISP's speed when I use directly my iMac with ISP's router but (only) 60% when I use my iMac with OPNSense router (obviously with same conditions).

During these tests CPU is between 35%-45%.

Where can I investigate?

TIA

Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: phoenix on November 08, 2017, 03:57:01 pm
You shouldn't really tag on a message to an (very) old thread, a new one would have been better.

How about giving us some details of your current configuration, OPNsense version, ESXi version, which drivers are you using for the NICs, which OPNsense services are enabled (Suricata, SNMPD, etc.) etc., etc.?

FWIW, I've also recent gone to a 1Gb fibre connection and run my OPNsense on ESXi 6.5 with the VMXNET3 drivers and don't see any problem in download speed.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: weust on November 08, 2017, 04:02:52 pm
I'm testing a bit at home with ESXi 6.5.0 U1, and using VMXnet3 cards made the download speed go back to ~120 Mbps. Haven't tested with E1000 cards yet.
Bare metal I do get the full 400 Mbps. Same hardware.

Creating a new VM and choosing Other/FreeBSD 64 bit selects E1000. Not VMXnet3.
Checking information on support is vague. It's supported by FreeBSD, but  not 100%.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on November 08, 2017, 05:56:06 pm
Actually I use OPNSense 17.7.7_1 on ESXi 6.0u2 and I use VMXNET3 drivers.

I don't have any particular service enabled, I use only some VPN connections )OPNsense is a VPN client) but not in this case because I inserted a specific firewall rule for these tests (also, if I switch off these VPN connections I get same results).

I will try to go back to E1000 drivers but it's seems really strange to me.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: weust on November 09, 2017, 12:31:00 am
I'm running with E1000 now, and speeds are normal now.
410Mbps down and 40 up.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: xinnan on November 09, 2017, 04:06:50 am
Try your setup with only 2 v-cores and see if you notice a loss at all.  If you have time. 
Also, you didn't mention any packages that are memory hungry.  Wondering if 2GB or even just 1GB wouldn't be enough ram?  Just thinking about saving untapped resources for other things if possible. 
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: MasterXBKC on November 09, 2017, 06:23:50 am
Hi Allesandro,

If you assign four vCPU's to a VMware guest it will only be able to run when four physical cores are available on the host.

Between the four cores for Ubuntu and four for OPNsense, there are no free cores for ESXi itself. If you only assign two cores to OPNsense, and perhaps two cores to Ubuntu, it will be much easier for the VM's to run simultaneously with the Hypervisor.

Bart...

Disregard everything Bart is saying here, he has no idea of that which he speaks, no offense bart, but i oversee a quite large vmware vsphere cluster and have been working with vmware since back in the vmware server 2.0 days years and years ago.   These statements have no basis in fact.   An 8 core box can have all 8 cores assigned to 2, or even 15 virtual machines simultaneously as long as the cpu cores arent being constantly peaked.   This is what makes vmware so successfull, instead of having idle virtual machines wasting the power of their cpu cores that are idle, all cpu cores simply share their cycles with all the virtual machines they are assigned to, this is one of the main benefits of virtualization.

Im not speaking on opinion, we are actually a vmware partner.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: xinnan on November 09, 2017, 06:30:28 am
Overprovisioning is pretty normal.  It's a good way to make the most of the hardware.  Still, no need to be harsh.  It's not intuitive.  I'd google a little about maximizing resources and include 2017 in the search since things change.  The closer you come to keeping your CPU maxed without impairing performance, the more you are getting your money's worth.  Cycles idle are cycles wasted. 
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on November 09, 2017, 12:20:13 pm
Thanks for these suggestions,

I will try with less RAM and with E1000 drivers (with these I already tried but WAN interface doesn't works, really strange).

Else I haven't any other idea.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: MasterXBKC on November 09, 2017, 03:08:02 pm
i actually happen to have an opnsense 17.7 running in our cluster as ive been using it to develop and test an opnsense extention, the cluster is on gigabit fiber inside our datacenter directly connected to Level 3's pipeline.

I know i set it up using vmxnet3 interfaces as well and its been working flawlessly, however i dont believe ive ever done a speedtest through it, if time avails i will do a few, and drop them here for comparison, it has none of the extra content filtering, etc turned on, its just a default install with a wan and lan interface.

The cluster itself is the following:
2 x Dell R710s
|-2 x Xeon X5650 Hex Core CPUs
|-144 GB DDR3 ECC Registered Buffered RAM
|-Quad Port Intel Gigabit NICs
|-8 x 4TB Drives in RAID 10

2 x Dell R720XDs
|-2 x Xeon E5 8 Core CPUs
|-256GB DDR3 ECC Registered Buffered RAM
|-Quad Port Intel Gigabit NICs
|-24 x 4TB Drives seperated into 6 seperate 4 drive RAID 10 Arrays

4 x HP DL360 G7s
|-2 x Xeon E5620 Quad Core CPUs
|-48 GB DDR3 ECC Registered Buffered RAM
|-Quad Port Intel Gigabit NICs
|-8 x 1TB Drives in RAID 10
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: MasterXBKC on November 09, 2017, 03:18:19 pm
One thing i know can help from testing with a pfsense, is set the virtual machine to reserve/lock all of its ram so that vmware doesnt page it around thus causing some minor lag and delays occasionally, it wasnt a night and day difference, but i did notice some latency change, and generally a tiny bit more snappyness.

Another thought that crossed my mind, is hyperthreading enabled on your ESXI box?   If so, each core is loosely like a half a core, so to assign the equivalent of 2 full cpu cores, you actually need to assign 4.

A screenshot from your vsphere client summary tab of the host would help us identify if hyperthreading is enabled.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on November 10, 2017, 03:12:55 pm
Hi,

I tried following:

1) Fresh (and basic) OPNSense 17.7.7 installation, with VMXNET3 drivers: same results
2) Fresh (and basic) PFSENSE 2.4 installation with VMNETX3 drivers: same results

With E1000 drivers performances drop to 550Mbps (in both cases).

For other questions: I don't use hyperthreading, I have a C2758 CPU with 4 cores for OPNSense VM (see attachment).

Thanks for your help.


Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: MasterXBKC on November 11, 2017, 07:19:48 am
Do you have virtualization enabled in the bios, there should be a VT-D, and a Virtualization setting in the bios under the cpu section most likely, if these are not enabled, it will force the system to use software emulation, and this could severely rob your performance and throughput as well.

It is also possible that, assuming your running vmware on this C2758 CPU, that it could be too much for such a CPU, the virtualization overhead could be making that difference, but ive never run vmware on such a low end CPU.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: xinnan on November 11, 2017, 07:22:27 am
I have.  It will work, but you will hit a cap on bandwidth.  Probably just a 5% hit if the VM is working right.

How big a hassle would it be for you to install opnsense on the machine directly to test the limits of the physical machine and THEN compare it to opnsense in VM?
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on November 11, 2017, 09:14:58 am
I think to try some other firewall distro, like LEDE or Untangle (always on a VM) and repeat my tests with these distro: do you think is a good idea?

Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: xinnan on November 11, 2017, 10:03:26 am
No - You must only try opnsense!   

Try them.  Its smart to test your choices.  You will miss the features of opnsense sooner or later though. 
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: franco on November 11, 2017, 10:52:09 am
LEDE is pretty good. The Linux underneath will give you less issues probably.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on November 11, 2017, 11:12:04 am
I have already tried with Untangle (version 13.1.0): basic installation on a VM with 2 cores and 2 GB RAM.

VM it's configured as Linux 2.6.x 64bit, VMXNET3 drivers.

I get 90% ISP's speed, ie 900Mbps.....aaargghhh......much better than OPNSense.

Now I will try this Untangle VM with same OPNSense's cfg (cores and RAM).

Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: weust on November 11, 2017, 11:39:32 am
The Atom C2000 series does not support VT-d, so forget that part.

I've been running Hyper-V 2012 R2 first, and now 2016 for over a year on my C2758F.
It may not be the fastest CPU, but for home usage is more then fine.

Can't do IDS/IPS in the VM and my ISP speed, but 2 vCores and 150Mbit/s is doable.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on November 11, 2017, 11:49:21 am
With 4 cores and 4GB RAM on Untangle VM, I get same results: 90% ISP's speed, ie. 900Mbps

So, I start to think the problem relies on FreeBSD: what do you think?
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: phoenix on November 11, 2017, 12:21:32 pm
With 4 cores and 4GB RAM on Untangle VM, I get same results: 90% ISP's speed, ie. 900Mbps

So, I start to think the problem relies on FreeBSD: what do you think?
I don't think that is likely be the case. I run my OPNsense in a VM with a single vCPU (although I did use 2vCPUs) with a gigabit FTTH connection and achieve about 700-800Mb download speed with IDS/IPS enabled. I'm currently on the 18.1b version and I found no difference in performance with 2 vCPUs and no difference on the previous 17.7.x versions of OPNsense so I would rule out any 'problem' with FreeBSD in my environment.

The hardware I use is an Asrock MB and intel i340/i350 NICs with ESXi 6.5-U1 virtualization is enabled on the motherboard and I have no problems using the VMXNET3 NICs in any of my VMs and the VM Tools installed on all of them, including OPNsense.

I don't think you've given any details of your hardware, have you?
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on November 11, 2017, 03:18:19 pm
I have a C2758 supermicro mb, 8 cores 2.4ghz, 16gb ram.

If isn’t an hardware problem or VMware problem ( with untangle, in a similar vm, I get 900mbps) where can I investigate?

Thanks in advance


Sent from my iPad using Tapatalk
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: phoenix on November 11, 2017, 04:32:57 pm
Do you have the VMware Tools installed in OPNsense?

[edit]Which motherboard (model number) are you actually using? Does it have Intel NICs and if not, what are they? As it's a problem with download speed I'd suggest you investigate the NICs on your M/B.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on November 11, 2017, 05:39:55 pm
I have Supermicro A1SRM-2758F, Atom C2758 CPU with 8 cores, 2.4Ghz x core, 16GB RAM.

I installed vmware tools (directly from opnsense) and on this mobo there are Intel NICs on this mobo (I354 controller).



Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: phoenix on November 11, 2017, 08:07:36 pm
What services do you have enabled? Do you have the SNMP service enabled, if you do then which modules are enabled? Have you disabled all the NIC offload functions in OPNsense? Have you also checked if all the NICs in ESXi have TSO enabled as per this article: https://nielshagoort.com/2017/10/19/tcp-segmentation-offload-esxi-explained Have you taken any snapshots of this VM and if so, how long have they been there?
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on November 12, 2017, 12:09:06 pm
Try to summarize my tests and my situation.

I always used OPNSense as my router/firewall on vmware virtual machine (ESXi 6.0u2) and I think it is a great product.

Until now I had 500Mbps as internet bandwidth and my router I get always very good performance. Recently I upgraded to 1Gbps and I retried some old tests to measure performances.

So using some file transfer and usenet downloads as test, I get:

1) If I exec these tests from a computer directly connected to ISP I get (90-100)% ISP's speed
2) If I exec these tests from the same computer connected to my OPNSense router (on a VM), I get 60% ISP's speed
3) If I exec these tests from the same computer connected to a fresh installation of OPNSense 17.7.7 (on a VM), I get 60% ISP's speed
4) If I exec these tests from the same computer connected to a fresh installation of PFSense 2.4 (on a VM), I get 60% ISP's speed
5) If I exec these tests from the same computer connected to a fresh installation of LEDE 17 (on a VM), I get (90-100)% ISP's speed
6) If I exec these tests from the same computer connected to a fresh installation of Untangle 13 (on a VM), I get (90-100)% ISP's speed

Obviously above VM is the same (CPU, RAM) between different tests: I tried with 2core-4GB RAM, 2core-2GB RAM, 4core-4GB RAM, also with CPU and RAM reservation.

On OPNSense/PFSense I disabled all NIC offload functions.

So this is why I think my problem was on FreeBSD side, maybe on vmx drivers.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: phoenix on November 12, 2017, 02:12:51 pm
I fully understand why you think it may be a Freebsd problem (and I'm not disputing that) but I can say that I'm not seeing that, you also didn't really answer my questions about what services you have and whether the TSO settings on ESXi are enabled or not. I don't really see why a change to a 1Gb connection would have problems whereas your previous connection did not. As you're on an older version of ESXi, have you considered upgrading to the latest 6.5 release? Have you also considered asking on the VMware Community forums (or even the Freebsd lists) whether there are any problems with Freebsd on your version of ESXi?

I can't really offer much advice other than to say again that my system doesn't have those problems  on ESXi 6.5U1.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on November 12, 2017, 05:38:01 pm
Hi Bill,

you are right, I don't answer to your question: I checked TSO on ESXi side and I found it enabled on vmnic0 and vmnic1 and also on vmkernel layer (it's disabled on OPNsense).

Now, I will try to disable TSO on ESXi (as stated on article you posted) and then to upgrade to ESXi 6.5u1 version (but I am skeptical), before to ask help on VMWare or FreeBSD community.
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: framura on November 12, 2017, 07:04:39 pm
Tried but no success.

I tried before disabling TSO on ESXI side, same results: so I upgraded to ESXi 6.5u1, same results (TSO is already OFF on ESXI side).

Now I will try to get some help on vmware forum: on what FreeBSD list can I ask for help?

In the meantime I start to investigate some cfg vmnic parameters.

Alex
Title: Re: Some suggestions for my new OPNSense (on ESXi host)
Post by: franco on November 14, 2017, 05:07:02 am
A good starting point is https://bugs.freebsd.org/bugzilla/ to look for similar reports or open a case with enough info for others to investigate / inspect potential issues either in configuration or code.


Cheers,
Franco