Some suggestions for my new OPNSense (on ESXi host)

[framura]

I think to try some other firewall distro, like LEDE or Untangle (always on a VM) and repeat my tests with these distro: do you think is a good idea?

[xinnan]

No - You must only try opnsense!   

Try them.  Its smart to test your choices.  You will miss the features of opnsense sooner or later though. 

[franco]

LEDE is pretty good. The Linux underneath will give you less issues probably.

[framura]

I have already tried with Untangle (version 13.1.0): basic installation on a VM with 2 cores and 2 GB RAM.

VM it's configured as Linux 2.6.x 64bit, VMXNET3 drivers.

I get 90% ISP's speed, ie 900Mbps.....aaargghhh......much better than OPNSense.

Now I will try this Untangle VM with same OPNSense's cfg (cores and RAM).

[weust]

The Atom C2000 series does not support VT-d, so forget that part.

I've been running Hyper-V 2012 R2 first, and now 2016 for over a year on my C2758F.
It may not be the fastest CPU, but for home usage is more then fine.

Can't do IDS/IPS in the VM and my ISP speed, but 2 vCores and 150Mbit/s is doable.

[framura]

With 4 cores and 4GB RAM on Untangle VM, I get same results: 90% ISP's speed, ie. 900Mbps

So, I start to think the problem relies on FreeBSD: what do you think?

[phoenix]

With 4 cores and 4GB RAM on Untangle VM, I get same results: 90% ISP's speed, ie. 900Mbps

So, I start to think the problem relies on FreeBSD: what do you think?

I don't think that is likely be the case. I run my OPNsense in a VM with a single vCPU (although I did use 2vCPUs) with a gigabit FTTH connection and achieve about 700-800Mb download speed with IDS/IPS enabled. I'm currently on the 18.1b version and I found no difference in performance with 2 vCPUs and no difference on the previous 17.7.x versions of OPNsense so I would rule out any 'problem' with FreeBSD in my environment.

The hardware I use is an Asrock MB and intel i340/i350 NICs with ESXi 6.5-U1 virtualization is enabled on the motherboard and I have no problems using the VMXNET3 NICs in any of my VMs and the VM Tools installed on all of them, including OPNsense.

I don't think you've given any details of your hardware, have you?

[framura]

I have a C2758 supermicro mb, 8 cores 2.4ghz, 16gb ram.

If isn't an hardware problem or VMware problem ( with untangle, in a similar vm, I get 900mbps) where can I investigate?

Thanks in advance


Sent from my iPad using Tapatalk

[phoenix]

Do you have the VMware Tools installed in OPNsense?

[edit]Which motherboard (model number) are you actually using? Does it have Intel NICs and if not, what are they? As it's a problem with download speed I'd suggest you investigate the NICs on your M/B.

[framura]

I have Supermicro A1SRM-2758F, Atom C2758 CPU with 8 cores, 2.4Ghz x core, 16GB RAM.

I installed vmware tools (directly from opnsense) and on this mobo there are Intel NICs on this mobo (I354 controller).

[phoenix]

What services do you have enabled? Do you have the SNMP service enabled, if you do then which modules are enabled? Have you disabled all the NIC offload functions in OPNsense? Have you also checked if all the NICs in ESXi have TSO enabled as per this article: https://nielshagoort.com/2017/10/19/tcp-segmentation-offload-esxi-explained Have you taken any snapshots of this VM and if so, how long have they been there?

[framura]

Try to summarize my tests and my situation.

I always used OPNSense as my router/firewall on vmware virtual machine (ESXi 6.0u2) and I think it is a great product.

Until now I had 500Mbps as internet bandwidth and my router I get always very good performance. Recently I upgraded to 1Gbps and I retried some old tests to measure performances.

So using some file transfer and usenet downloads as test, I get:

1) If I exec these tests from a computer directly connected to ISP I get (90-100)% ISP's speed
2) If I exec these tests from the same computer connected to my OPNSense router (on a VM), I get 60% ISP's speed
3) If I exec these tests from the same computer connected to a fresh installation of OPNSense 17.7.7 (on a VM), I get 60% ISP's speed
4) If I exec these tests from the same computer connected to a fresh installation of PFSense 2.4 (on a VM), I get 60% ISP's speed
5) If I exec these tests from the same computer connected to a fresh installation of LEDE 17 (on a VM), I get (90-100)% ISP's speed
6) If I exec these tests from the same computer connected to a fresh installation of Untangle 13 (on a VM), I get (90-100)% ISP's speed

Obviously above VM is the same (CPU, RAM) between different tests: I tried with 2core-4GB RAM, 2core-2GB RAM, 4core-4GB RAM, also with CPU and RAM reservation.

On OPNSense/PFSense I disabled all NIC offload functions.

So this is why I think my problem was on FreeBSD side, maybe on vmx drivers.

[phoenix]

I fully understand why you think it may be a Freebsd problem (and I'm not disputing that) but I can say that I'm not seeing that, you also didn't really answer my questions about what services you have and whether the TSO settings on ESXi are enabled or not. I don't really see why a change to a 1Gb connection would have problems whereas your previous connection did not. As you're on an older version of ESXi, have you considered upgrading to the latest 6.5 release? Have you also considered asking on the VMware Community forums (or even the Freebsd lists) whether there are any problems with Freebsd on your version of ESXi?

I can't really offer much advice other than to say again that my system doesn't have those problems  on ESXi 6.5U1.

[framura]

Hi Bill,

you are right, I don't answer to your question: I checked TSO on ESXi side and I found it enabled on vmnic0 and vmnic1 and also on vmkernel layer (it's disabled on OPNsense).

Now, I will try to disable TSO on ESXi (as stated on article you posted) and then to upgrade to ESXi 6.5u1 version (but I am skeptical), before to ask help on VMWare or FreeBSD community.

[framura]

Tried but no success.

I tried before disabling TSO on ESXI side, same results: so I upgraded to ESXi 6.5u1, same results (TSO is already OFF on ESXI side).

Now I will try to get some help on vmware forum: on what FreeBSD list can I ask for help?

In the meantime I start to investigate some cfg vmnic parameters.

Alex