ESXi Shaper 1Gb download performance

[FlangeMonkey]

Hi Guys,

I have a couple of shapers, one for inbound and one for outbound on my WAN interface.  I am running a virtual firewall, the external WAN interface is now passthrough using igb, which has improved things a lot.

When enabling any shapers using vmxnet3 on the LAN interface, my download performance drops from 800-900Mbit to 600Mbit. 

I therefore changed to E1000e for the LAN, and that increased performance without shapers to between 850-950Mbit.  However, I am still getting a performance drop with shapers to around 700Mbit (during testing, I am increasing the shaper to 1800Mbit to eliminate queue size).

Any Suggestions?

Thanks,

[opn_nwo]

Somebody correct me if I'm wrong, but I think the shaper is not currently supported in VM setups.

[FlangeMonkey]

I've been using it for years without issues, its just at this speed, its hitting some form of bottle neck.  I know there are card related issues and recommendations for shapers in a virtual machine, such as using e1000 cards over vmxnet3.

I've done a bunch of testing on to try and resolve it and found various things regarding Hz and tickrates on ipfw, I think thats a dead end.  I've ended up dropping in pfsense to see how that performs as a comparison.  That is working very well and although I'm not a fan of pfsense, I'll keep with it until I find the cause.

Edit:  Although I have shapers working in pfsense, the other interesting observation is the performance without shapers, I am noticing a marked performance increase over opnsense.

[FlangeMonkey]

For anyone else looking, it appears to Spectre and Meltdown Mitigation.  This is a subtle difference between pfSense and opnsense.  I only needed to tweak 'hw.ibrs_disable' to 1, within Tunables.

https://docs.opnsense.org/troubleshooting/hardening.html

[dinguz]

Check the kern.hz sysctl, in virtualised setups it sometimes gets set to 100 which is too low for traffic shaping to function properly. If this is the case, set it to 1000 and try again.