22.1rc1 slow in Hyper-V

[Com DAC]

I thought I'd spin up a second vm to test 22.1 RC 1. It installed without issue and boots significantly faster. Restored configuration backup from previous version (21.7.7) without issue. All seems to technically work except my throughput is horrible. If I spin up the 21.7.7 vm I can get 300mbps (my full internet speed). Shut down 21.7.7 and spin up 22.1 RC 1 and I can only get 1 - 2 mbps. Anyone else had a chance to test in Hyper-V? Anyone else noticing this?

Incase it matters both VM's are Gen 2. All the hardware offloading is turned off (though I've also tried turning it on).

I've now also tried a compeltely clean install (without restoring my configuration backup) and still seeing the same slowdown.

[8191]

I run two VMs in Asure. I assume that's also Hyper-V running in the background. Didn't recognize any performance issues so far, but as those are dev. VMs they are not heavily used.

[franco]

Slow speed just indicates mismatch in NIC handling/checksum failures, etc.


Cheers,
Franco

[Com DAC]

Slow speed just indicates mismatch in NIC handling/checksum failures, etc.


Cheers,
Franco

Ok sounds logical. Though all the settings I can see between v21.7.7 and 22.1 RC are the same both inside OPNsense and in hyper-v. This is why I posted to see if anyone else has seen this. I've tried all the hardware offload settings can you suggest any other settings to try?

Thank you,

[franco]

I'm not sure, it might also be some sort of regression in Hyper-V or its support. I don't think this is related but as an example this was issued last week:

https://www.freebsd.org/security/advisories/FreeBSD-EN-22:03.hyperv.asc

There's a number of bugs open here but skimming through the list doesn't have anything sticking out:

https://bugs.freebsd.org/bugzilla/buglist.cgi?quicksearch=hyper-v

For the time being I don't have actionable advice.


Cheers,
Franco

[Com DAC]

Just as another test I've tried upgrading my 21.7.7 to 22.1. Once it upgraded to FreeBSD 13 the network slowed down. Everything I've tested seems to be pointing to an issue in the FreeBSD 13 Hyper-V network drivers. Though the web interface is fast and responsive it's just getting through OPNsense to the internet that is slow.

Thank you,

[8191]

Cannot confirm your experience on an Azure VM:

Code Select Expand

mf@opnsense-dev:~ % speedtest

   Speedtest by Ookla

     Server: Claranet Benelux B.V. - Amsterdam (id = 30847)
        ISP: Microsoft Corporation
    Latency:     2.53 ms   (0.26 ms jitter)
   Download:  3093.84 Mbps (data used: 3.2 GB )                               
     Upload:   956.58 Mbps (data used: 864.6 MB )                               
Packet Loss:     0.0%
Result URL: https://www.speedtest.net/result/c/abd6703b-9231-498c-97b6-98533a3df037
mf@opnsense-dev:~ % uname -r
13.0-STABLE

[Com DAC]

I've been testing in Hyper-V on Windows 11. Maybe Azure is running a slightly different version of Hyper-V. I'll try spinning up a test VM on a different version of Hyper-V and see if it still happens.

Thank you,

[opnfwb]

I too am able to duplicate some odd performance issues in Hyper-V with the 22.1RC.

I've tried this with a Gen1 Hyper-V VM and also in VirtualBox with virtio NICs. I agree with the findings that the OPNsense 22.1RC seems to have odd issues in Hyper V. In virtualbox, I'll get at most 70mbits of bandwidth and the VM will freeze up during a speedtest. I will get odd errors reported at the console, AHCI timeouts, etc. etc. If I swap the VirtualBox NICs back to emulated E1000 NICs, the will get full throughput of my connection (500/500).

For the Gen1 Hyper V VM, the download speed will hit maybe 21mbits/sec. However, the upload speed will be a bit better at around 200mbits/sec. Still nowhere near the full capability of the connection. CPU usage while running the tests does spike higher, in some cases around 90% briefly but it never completely utilizes either core.

Specs of VMs:
Running on Server 2019
2 vCPUs
2048MB RAM
All running on SSD datastore

Also for testing purposes, I used the following tunables:
vm.pmap.pti = 0
hw.ibrs_disable = 0
hw.pci.honor_msi_blacklist = 0

[Com DAC]

Updated to RC2 (took hours with the reduced speed) and the problem still persists.

[opnfwb]

I've done more testing with mine and it's quite an odd issue. I can push Iperf3 traffic through the firewall at over 1gbit/sec. Usually 1.1 to 1.4gbit/sec. This is using a client and server VM sitting on the WAN and LAN side of the router, so it's actually pushing and pulling routed traffic through.

However, from the same VM on the client side I cannot run a download speedtest (fast.com, speedtest.net, etc.) and get anything higher than about 10mbit on the download. On the upload, it will go somewhat higher, at around 200mbit. Very odd and I'm still troubleshooting to determine the cause.

So far I've tried disabling RSS. This had no effect. I've also tried lowering the MTU on the OPNsense VM, this also didn't make a difference. I'll keep digging and see what I can figure out but its a very odd issue. This is only present in HyperV or VirtualBox with virtio NICs, bare metal installs with igb NICs do just fine and get good throughput.

[Com DAC]

I just tried doing a clean install of freeBSD 13 and then bootstrapping OPNsense. The freeBSD installed fine. Bootstrapping OPNsense went smooth and fast ( https://github.com/opnsense/update#opnsense-bootstrap ).  Once it rebooted and OPNsense was running it's access to the internet slowed the same as with the standard OPNsense install.

[franco]

Thanks, it either points to a configuration issue that our system does (might also be traffic normalization or the like) or there's an issue in the FreeBSD code between 13.0 and 13-STABLE as we use it. The latter would be more problematic since that code is supposed to be 13.1 in a short while.


Cheers,
Franco

[Com DAC]

not that I expected anything to have changed in such a short period of time but I've tested against the full release 22.1 and the problem still exists.

[Vesalius]

As another data point, upgraded a proxmox 21.7.8 OPNsense VM to 22.1 and getting full speed on my symmetrical 1Gb fiber using virtio Nics from proxmox 7.1.