22.1rc1 slow in Hyper-V

[mnaim]

I can confirm that. Hyper-V on Server 2022 (on premise)
21.7
Download 800Mbps
Upload 881Mbps
22.1
Download 829Mbps
Upload 1.57Mbps

After rollback (thanks to this post snapshot was taken) speeds are ok.

[franco]

Can someone record a packet capture on 22.1 with thew Hyper-V slowdown and tell us if it's bad checksums, missing packets or out of order traffic? At first glance there is no mention in FreeBSD bug tracker but having full download speed but poor upload seems to indicate outgoing checksum issues?


Cheers,
Franco

[Com DAC]

Here is a full all interface packet capture while using speedtest.net. Both my upload and download are being impacted.

Packet Capture: REDACTED BY ADMIN

Thank you,

[franco]

Thanks I grabbed it and redacted the URL.


Cheers,
Franco

[dandickson]

I snapshotted one of my production systems and upgraded it, i'm seeing issues on the download traffic, the upload seems normal.

I had to revert the machine back however if it would help, I can provide a temporary test system for you to run tests against franco

[AllPurposeGeek]

I am having this same issue on the published release.
Host is Windows Server 2022
NIC is a Intel(R) 82576 Gigabit.
Virtual "Switch" is configured with "Enable single-root I/O virtualization SR-IOV"


Speed is 1-2 Megabits with or without TCP offloading. Had to roll back to 21.7

[franco]

I'm looking at https://github.com/opnsense/src/commit/cdc59163ff8e8f2a which isn't on FreeBSD 13.0-RELEASE. Is that a positive on "Windows Server 2019 hosts and later"? Maybe RSC can be disabled in the host to test this theory?


Cheers,
Franco

[Com DAC]

I've setup two new test vswitches (one for WAN and one for LAN). Disabled software RSC on both and now my download speeds are great but my upload speeds are still unusable.

I think this might be on to something.

[Maurice]

Upgraded a production system to 22.1. Throughput dropped from several hundred Mbps to less than 5 Mbps (up and down). Came here to learn it's a known issue. Windows Server 2022, Gen2 VM, Intel NIC, no special settings.
Can't provide more information, sorry. Not back in the debugging / development game (yet). Just need a working system, so already rolled back to 21.7.

Maurice

[N0ah]

Same here, upgraded from 21.7 to 22.1, speeds dropped from ~190 / 40 mbps to 3-15mbps.
Server 2019 Core, Gen 2, v9 VM; WAN vSwitch is based on a Realtek Card, LAN vSwitch on Multiplex / 2x Intel. No SR-IOV.

Could provide tests, logs or whatnot.

Reverted back to 21.7, but kept a snapshot of 22.1.

Cheers

[franco]

I'm building a kernel without the RSC support now to try.

EDIT: just to report back on the packet capture analysis... it didn't show obvious stack issues so I assumed that this is some sort of packet disappearing act which made me look at changes between FreeBSD 13.0 and 13-STABLE we are using.


Cheers,
Franco

[franco]

So when on 22.1 install this kernel and reboot:

# opnsense-update -zkr 22.1-hyperv
# yes | opnsense-shell reboot

(might take a bit to sync the kernel to all mirrors... the default mirror should have it)


Cheers,
Franco

[Com DAC]

that seems to have done it. Speeds are good with the updated kernel.

[Maurice]

Upgraded to 22.1 again and installed the patched kernel. First impression: Performance "feels" like 21.7.

Thanks, Franco.

Maurice

[Sisyphe]

Hi,

I'm experiencing similar slowdowns with 22.1 final on ESXi 7.0 Update 3. Reverting to 21.7 resolved the issue.
I haven't tried the patched kernel yet as I assume this may only be relevant for Hyper-V.

Cheers