Poor Performance with OpnSense 23.1 and Hyper-V 2019

[Neo]

Apologies if I've missied an existing solution somewhere... I did do searches on this and found a thread back around the 22.1 RC timeframe that 'might' be related but did not seem to offer a conclusive remediation and might not be the same issue I'm experiencing...

Background: I have been running OpnSense as a VPN ("only") gateway for the past couple years on a single NIC Intel NUC so everything other than the interface assigned to LAN is a tagged vLAN... It has multiple WAN links (1gb, 300mb, LTE) and multiple VPN links (one for each WAN) all handled by an L2 vLAN switch... The performance has been excellent with full gigabit throughput from a physical PC on the LAN to internet hosts with consistent speed test results on 21.1->23.1

I am now building out a Hyper-V VM with a slightly different configuration (1 WAN link, 2 VPN tunnels, LAN + several additional vLANs that will be firewalled and have limited or no access between them)... The WAN and LAN are on separate virtual NICs defined on the VM at the HV level and I have a Win 10 VM with a single vNIC on the LAN side on the same vSwitch as well as a physical PC on the LAN side to test from...

Upload speeds are fine but download speed is about 25-30% of what I would expect...

Details:
* Win 10 VM is on the same virtual "10 gb" switch as all the OpnSense vNICs/vLANs
* vSwitch tied to a physical 3 NIC "team" (LAG) between the host server and L2 vLAN switch in the server rack
* Rack switch has 1gb uplink to main L2 vLAN Switch near ISP router
* ISP router has 1gb connection to main L2 vLAN Switch
* both physical switches have plenty of backplane bandwidth and are not handling excessive traffic

Thoughts:
* no bottleneck on 10gb vSwitch
* no bottleneck on 3gb LAG
* data transfer between PC on main switch and other (Win Server 2019) VMs on the same vSwitch/Physical switch are fast
* some "potential" limitations of 1gb fiber link between switches but should not limit downlaod to 25-30% of normal

My gut says there is something about OpnSense or FreeBSD that isn't working well with my Hyper-V setup as I've done many other things with this host and set of switches (even using multiple vLANs and other virtual router configs) -- I have not done a lot of deep granular tweaking of Hyper-V network settings other than turning off VMQ on the physical NICs (they are Broadcom and turning that off has long been recommend on these NICs) and I'm not very familiar with low level settings on OpnSense or the underlying networking of Hardened BSD...

Hoping someone else has already experienced this and has a fix for me or that this does in fact relate to whatever changed (and caused issues) in 22.1 RC and there is a remedy via tweaks on OpnSense or HV (or both)...

Please advise!

[opnfwb]

Hello, I also run OPNsense on Hyper-V (just in my LAB as a VM) and had a similar issue.

The below commands allowed me to check for RSC and also disable it. You'll need to run this command on the "external" or "shared" switch than the WAN side interface of OPNsense is connected to. I didn't have to run this for any of the other vSwitches, just the one I was using for WAN on the OPNsense VM.

To check RSC status:

Code: [Select]

Get-VMSwitch -Name "your wan switch" | Select-Object *RSC*
To disable RSC on the vSwitch:

Code: [Select]

Set-VMSwitch -Name "your wan switch" -EnableSoftwareRsc $false
It's been over a year since I read the thread on it but I recall this being due to some issues between Hyper-V and FreeBSD 13.

[Neo]

Thank you for your response.

I did see that in the original thread from 22.1 RC and I did try that... it did not seem to make a difference... However, all interfaces are attached to the same vSwitch in my case which is and "external" switch connected to a 3 NIC Team on the host (which plugs into a LAG on the physical switch). The vNIC for the OpnSense LAN interface is native (vLAN 0) but all other interfaces on the OpnSense VM are on tagged vLANs (set a the VM level)... not sure if that makes a difference or not...

I can try disabling RSC again and see if that makes a difference... unfortunately I only have the 3 physical NICS to work with and 3 critical VMs (Server 2019) using that already so room for experimentation is somewhat limited...

[Neo]

To narrow this down a bit further I built a plain OpnSense with just LAN and WAN on a fresh VM with WAN connected to my main production vSwitch and LAN connected to an isolated private vSwitch and turned on an iperf3 server on a Windows 2019 VM... I tested the speed using both a Ubuntu (22.04) VM and a Win 10 VM to the Win 2019 server (all connected to same vSwitch so nothing actually touching an external switch in the real world) and then moved both to the isolated vSwitch on the LAN side of the OpnSense and tested against... Results were similar in both cases... I only get about 30% of the expected data transfer when routing through the OpnSense vs having no opnsense in between...

[Neo]

iperf3 outputs (before and after):

***


PS C:\temp\iperf-3.1.3-win64> .\iperf3.exe -c 10.99.1.2 -p 7777 -bidir
Connecting to host 10.99.1.2, port 7777
[  4] local 192.168.1.101 port 49713 connected to 10.99.1.2 port 7777
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  84.6 MBytes   709 Mbits/sec
[  4]   1.00-2.00   sec  93.9 MBytes   786 Mbits/sec
[  4]   2.00-3.00   sec  83.8 MBytes   704 Mbits/sec
[  4]   3.00-4.00   sec  96.9 MBytes   812 Mbits/sec
[  4]   4.00-5.00   sec  87.8 MBytes   737 Mbits/sec
[  4]   5.00-6.00   sec  90.4 MBytes   758 Mbits/sec
[  4]   6.00-7.00   sec  94.1 MBytes   789 Mbits/sec
[  4]   7.00-8.00   sec  88.2 MBytes   741 Mbits/sec
[  4]   8.00-9.00   sec  97.0 MBytes   813 Mbits/sec
[  4]   9.00-10.00  sec  97.4 MBytes   818 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   914 MBytes   767 Mbits/sec                  sender
[  4]   0.00-10.00  sec   914 MBytes   767 Mbits/sec                  receiver

iperf Done.
PS C:\temp\iperf-3.1.3-win64> .\iperf3.exe -c 10.99.1.2 -p 7777 -bidir
Connecting to host 10.99.1.2, port 7777
[  4] local 10.99.1.106 port 49715 connected to 10.99.1.2 port 7777
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   351 MBytes  2.94 Gbits/sec
[  4]   1.00-2.00   sec   413 MBytes  3.46 Gbits/sec
[  4]   2.00-3.00   sec   377 MBytes  3.16 Gbits/sec
[  4]   3.00-4.00   sec   417 MBytes  3.50 Gbits/sec
[  4]   4.00-5.00   sec   412 MBytes  3.45 Gbits/sec
[  4]   5.00-6.00   sec   425 MBytes  3.56 Gbits/sec
[  4]   6.00-7.00   sec   402 MBytes  3.37 Gbits/sec
[  4]   7.00-8.00   sec   416 MBytes  3.49 Gbits/sec
[  4]   8.00-9.00   sec   430 MBytes  3.61 Gbits/sec
[  4]   9.00-10.00  sec   415 MBytes  3.48 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.96 GBytes  3.40 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.96 GBytes  3.40 Gbits/sec                  receiver



***

... sure feels like an OpnSense/FreeBSD interaction with Hyper-V to me... Don't know if this can be resolved or how to approach tweaking it further... I've already plated with VMQ and RSC... at this point its simply 2 endpoints on 1 or 2 virtual switches with or without OpnSense in between not much traffic on the target server hosting the iperf3 target either...

Are there settings I should look at tweaking within OpnSense and/or BSD at this point? I'm getting a bit beyond my comfort zone there... Are there any guides anywhere on optimal configuration of OpnSense in Hyper-V 2019? I need to know if there is a solution I can chase or if I'm running into a limitation of this combination that will just have me chasing my tail... It's likely not going to work for me unless I can get this figured out...

[oneplane]

Have you tried without windows? (i.e. Xen, KVM, ESX) That would be the fastest/easiest test to know for sure if it is the microsoft layer or something else (i.e. cpu).

[Neo]

No... I don't have a quick/easy way to do that... This is a Dell T320 with 16xSAS RAID 10, 4x1G NICs, and plenty of RAM/CPU... The OpnSense VM was given 3x vCPU and 4GB RAM and showed no indication of resource constraint on the dashboard... I have been thinking about toying with ProxMox but don't have that built yet and the hardware would be nothing compared to a T320... The host is not that loaded but it is hosting 4 VMs that are critical so I can't be doing a lot of rebooting the host... Assuming I could prove this was not an issue on a ProxMox box, where would I go from there?

I am wondering if there are others here getting full performance out of OpnSense on a Hyper-VM and would share their configuration... I do have Broadcom NICs and am not using VMQ but that does not seem to hold anything else back other than OpnSense... I have other Linux stuff (not BSD) running without issues (like Ubuntu, Docker with PiHole/Cloudflaird, Home Assistant, etc.) which is why my gut is saying BSD vs HV... but I sure could use some corroboration from elsewhere...

[zhuoerh]

I built an opnsense VM in Hyper-v on a Server 2019 recently. VM has 4G ram, 4 vcore.
Opnsense LAN is a trunk port in Server 2019, with breakouts done at Opnsense VLAN level (not Hyper-V level). This LAN is connected to an external vswitch that links to a physical 82599 10g nic.

When routing through opnsense, performance is close to your observation:
iperf single stream, spped varies widely 350Mbps ~ 800Mbps, CPU 50-75%
iperf 5 streams, speed varies widely 1.3Gbps ~ 2.4Gbps, CPU 80%-100%
iperf 10 streams, speed varies widely 1.5Gbps ~ 2.8Gbps, CPU 90%-100%

I tried changing RSC parameter as previous poster suggested but no change.

I am inclined to move it to a physical box instead.

UPDATE:
Did some more digging and testing, turning off VMQ on the host network adapter and turning off RSC on the virtual switch imporves throughput by ~30%. I can now get:
iperf 1 tream, 550Mbps ~ 1.0Gbps
iperf 5 streams, 1.7Gbps ~ 2.5Gbps
iperf 10 streams, 1.9Gbps ~ 2.9Gbps

Still a long shot from 10G.