Anyone test Opnsense running as a VMWare Guest?

[s4rs]

I decided to test Opnsense running under a hypervisor and am running into performance issues. I tested SmarOS and ESXi 6.5 with the same results. If I run iperf from Opnsense/BSD as a client pointing to an iperf server on the wan or lan side I get good performance IE approx 550Mb/s. If I setup iperf as a server on Opnsense/BSD and point a client from either the wan or lan I get horrible performance IE 70Mb/s.

I tested a server on the wan and a client on the lan and vise-verse with similar results. It seems like when running under a hypervisor receiving packets are an issue. Anyone seen this and have a tweak? Its not a resource issue, CPU is low 15% and plenty of memory available.

BTW I get very similar results if I run vyos. So there is something specific about routers in a virtual setting.

[werner]

What interface type are you using in VMWare ?

Can't say much about OPNSense performance virtual but I do have a few VyOS routers on VMWare backed by 10Gbit connections, performance has never been an issue. Using VMXNet3...

[phoenix]

I do run OPNsense in a VM on ESXi 6.0 and I also use the VMXNET3 driver (although I've also use the e1000) without any performance problems whatsoever.

[bobbythomas]

I'm running OPNsense on my Proxmox VM, e1000 is the adapter I am using and I am getting around 100mbps over wifi on a Wifi AC network. Haven't tested LAN speeds, will test it next week and will share the results.

Sent from my ONE A2003 using Tapatalk

[s4rs]

I also tested vyos and it had the exact same poor network performance as opnsense. I have a theory that the issue is with network bridge created by the hypervisor.

From searching the internet I have seen posts on people running pfsense in vmware. They always post iperf results from the appliance to other devices and not lan to wan (or wan to lan) tests from connected systems. I would love to see others results from running like this. This is where I see the issue. In fact if you setup iperf server on the opnsense or vyos appliance and run a client to it you will see the same poor performance. Outbound traffic is fine its inbound traffic that gets slammed.

 I think most people don't see performance issues since their wan links are much slower than the bottle neck I am seeing. In fact I could run like this since my wan is currently set to 50Mb/s. The obvious issue is if I add more wan bandwidth.

[s4rs]

After doing numerous searches I could not find any bottlenecks so I decided the issue was with the system I was using. I created a vyos vm on an 8 core xeon server running SmartOS I use for most of my home stuff. Running the iperf tests there everything ran great. I was getting pretty close to line speed IE 900Mb/s outside in and inside out.

So it looks like virtualization does not add much overhead.

I need to check the small systems BIOS CPU settings and make sure everything is setup for virtualization.

[s4rs]

This was an interesting bug. I went back to bare metal and did another test and I saw the same horrible performance as I was seeing while running under a hypervisor. I decided to do a BIOS reset and performance recovered. I then re-tested running under a hypervisor on the mini computer and the performance was up to 500Mb/s. So something strange happened to the system that a power cycle wouldn't fix. Live and learn.