22.1rc1 slow in Hyper-V

[N0ah]

So when on 22.1 install this kernel and reboot:

# opnsense-update -zkr 22.1-hyperv
# yes | opnsense-shell reboot

(might take a bit to sync the kernel to all mirrors... the default mirror should have it)


Cheers,
Franco

Can confirm this works, speeds are back to normal.
Thanks a lot!

[franco]

Replaced default upgrade kernel so that should be it. For reference the following commits were added removing the feature...

https://github.com/opnsense/src/commit/e91d90c0ac
https://github.com/opnsense/src/commit/2077062afc

Thanks all for helping confirm this. I'll try to follow up with FreeBSD/Microsoft on Monday.


Cheers,
Franco

[franco]

So I'm trying to look at this closer with Microsoft and there are a few open questions that would require someone to load the bad kernel and provide more information:

1. Does the throughput drop affect TCP / UDP or both? We suspect TCP but unsure about UDP...

2. We need the output from "sysctl -a | grep rsc" after a speed test with bad throughput run.


Thanks,
Franco

[mnaim]

I can help, how to load "bad kernel"?

[franco]

Good question. Let me get back in a second.

EDIT: load the following kernel

# opnsense-update -zkr 22.1

(reboot)


Cheers,
Franco

[Com DAC]

Attached are the results of the sysctl -a | grep rsc from before the speed test and then after the speed test. Also attached are the results of the speed test.

Does anyone have any guidance on how to test udp?

Thank you,

[mnaim]

dev.hn.1.rx.7.rsc_drop: 0
dev.hn.1.rx.7.rsc_pkts: 15
dev.hn.1.rx.6.rsc_drop: 0
dev.hn.1.rx.6.rsc_pkts: 224
dev.hn.1.rx.5.rsc_drop: 0
dev.hn.1.rx.5.rsc_pkts: 13
dev.hn.1.rx.4.rsc_drop: 2
dev.hn.1.rx.4.rsc_pkts: 286
dev.hn.1.rx.3.rsc_drop: 0
dev.hn.1.rx.3.rsc_pkts: 15
dev.hn.1.rx.2.rsc_drop: 1
dev.hn.1.rx.2.rsc_pkts: 349
dev.hn.1.rx.1.rsc_drop: 0
dev.hn.1.rx.1.rsc_pkts: 11
dev.hn.1.rx.0.rsc_drop: 0
dev.hn.1.rx.0.rsc_pkts: 501
dev.hn.0.rx.7.rsc_drop: 0
dev.hn.0.rx.7.rsc_pkts: 0
dev.hn.0.rx.6.rsc_drop: 0
dev.hn.0.rx.6.rsc_pkts: 0
dev.hn.0.rx.5.rsc_drop: 0
dev.hn.0.rx.5.rsc_pkts: 0
dev.hn.0.rx.4.rsc_drop: 0
dev.hn.0.rx.4.rsc_pkts: 0
dev.hn.0.rx.3.rsc_drop: 0
dev.hn.0.rx.3.rsc_pkts: 0
dev.hn.0.rx.2.rsc_drop: 0
dev.hn.0.rx.2.rsc_pkts: 0
dev.hn.0.rx.1.rsc_drop: 0
dev.hn.0.rx.1.rsc_pkts: 0
dev.hn.0.rx.0.rsc_drop: 0
dev.hn.0.rx.0.rsc_pkts: 0

[mnaim]

Ok, got interesting finding.
Installed iperf3 and used public fastest server from iperf site.

On bad kernel

On server behind opnsense (NAT)

iperf3 -c paris.testdebit.info -t 20 -f m -i 1 -P 10 -p 9240 -R

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  11.8 MBytes  99.0 Mbits/sec
[  7]   0.00-1.00   sec  10.7 MBytes  89.8 Mbits/sec
[  9]   0.00-1.00   sec  12.8 MBytes   107 Mbits/sec
[ 11]   0.00-1.00   sec  9.02 MBytes  75.6 Mbits/sec
[ 13]   0.00-1.00   sec   590 KBytes  4.83 Mbits/sec
[ 15]   0.00-1.00   sec  13.2 MBytes   111 Mbits/sec
[ 17]   0.00-1.00   sec  13.7 MBytes   115 Mbits/sec
[ 19]   0.00-1.00   sec   612 KBytes  5.01 Mbits/sec
[ 21]   0.00-1.00   sec   537 KBytes  4.40 Mbits/sec
[ 23]   0.00-1.00   sec  11.5 MBytes  96.6 Mbits/sec
[SUM]   0.00-1.00   sec  84.4 MBytes   708 Mbits/sec

Reverse direction (see that packets are lost for few seconds)

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec
[  7]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec
[  9]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec
[ 11]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec
[ 13]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec
[ 15]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec
[ 17]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec
[ 19]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec
[ 21]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec
[ 23]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec
[SUM]   0.00-1.00   sec  2.50 MBytes  21.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 Mbits/sec
[  7]   1.00-2.00   sec  0.00 Bytes  0.00 Mbits/sec
[  9]   1.00-2.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 11]   1.00-2.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 13]   1.00-2.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 15]   1.00-2.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 17]   1.00-2.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 19]   1.00-2.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 21]   1.00-2.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 23]   1.00-2.00   sec  0.00 Bytes  0.00 Mbits/sec
[SUM]   1.00-2.00   sec  0.00 Bytes  0.00 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 Mbits/sec
[  7]   2.00-3.00   sec  0.00 Bytes  0.00 Mbits/sec
[  9]   2.00-3.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 11]   2.00-3.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 13]   2.00-3.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 15]   2.00-3.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 17]   2.00-3.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 19]   2.00-3.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 21]   2.00-3.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 23]   2.00-3.00   sec  0.00 Bytes  0.00 Mbits/sec
[SUM]   2.00-3.00   sec  0.00 Bytes  0.00 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 Mbits/sec
[  7]   3.00-4.00   sec  0.00 Bytes  0.00 Mbits/sec
[  9]   3.00-4.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 11]   3.00-4.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 13]   3.00-4.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 15]   3.00-4.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 17]   3.00-4.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 19]   3.00-4.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 21]   3.00-4.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 23]   3.00-4.00   sec  0.00 Bytes  0.00 Mbits/sec
[SUM]   3.00-4.00   sec  0.00 Bytes  0.00 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 Mbits/sec
[  7]   4.00-5.00   sec  0.00 Bytes  0.00 Mbits/sec
[  9]   4.00-5.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 11]   4.00-5.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 13]   4.00-5.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 15]   4.00-5.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 17]   4.00-5.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 19]   4.00-5.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 21]   4.00-5.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 23]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec
[SUM]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   128 KBytes  1.05 Mbits/sec
[  7]   5.00-6.00   sec  0.00 Bytes  0.00 Mbits/sec
[  9]   5.00-6.00   sec   128 KBytes  1.05 Mbits/sec
[ 11]   5.00-6.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 13]   5.00-6.00   sec   128 KBytes  1.05 Mbits/sec
[ 15]   5.00-6.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 17]   5.00-6.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 19]   5.00-6.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 21]   5.00-6.00   sec  0.00 Bytes  0.00 Mbits/sec
[ 23]   5.00-6.00   sec  0.00 Bytes  0.00 Mbits/sec
[SUM]   5.00-6.00   sec   384 KBytes  3.15 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -         

So I got idea if NAT is not somehow involved

Installed iperf3 on opnsense itself

iperf3 -c paris.testdebit.info -t 20 -f m -i 1 -P 10 -p 9240

[  5]   3.00-4.00   sec  11.4 MBytes  95.8 Mbits/sec    0    290 KBytes       
[  7]   3.00-4.00   sec   602 KBytes  4.93 Mbits/sec    0   65.5 KBytes       
[  9]   3.00-4.00   sec  8.31 MBytes  69.7 Mbits/sec    0    221 KBytes       
[ 11]   3.00-4.00   sec  5.31 MBytes  44.5 Mbits/sec    3   83.9 KBytes       
[ 13]   3.00-4.00   sec  19.1 MBytes   161 Mbits/sec    0   6.81 MBytes       
[ 15]   3.00-4.00   sec  11.0 MBytes  92.5 Mbits/sec    0    282 KBytes       
[ 17]   3.00-4.00   sec  13.6 MBytes   114 Mbits/sec    0   1.37 MBytes       
[ 19]   3.00-4.00   sec  5.21 MBytes  43.7 Mbits/sec    1   39.8 KBytes       
[ 21]   3.00-4.00   sec  1.40 MBytes  11.8 Mbits/sec    0    168 KBytes       
[ 23]   3.00-4.00   sec  10.1 MBytes  85.1 Mbits/sec    0   1.09 MBytes       
[SUM]   3.00-4.00   sec  86.1 MBytes   722 Mbits/sec    4   

In reverse direction same speed       

So yes it has to be NAT (or some kind of packet processing)

(on good kernel all directions NAT/direct ok)

[whu]

Thanks for running more tests. In order to further narrow down the issue, I'd like to understand more about the settings of the test. Would you let me know if the iperf3 client and opnsense guest are running on the same Windows Hyper-V host or not?

The other question is how the NAT was configured on the opnsense guest? What is the IP address of the forwarding interface on the Opnserver and how the iperf3 client IP address is configured in NAT to send to iperf3 server over the internet?
Is it possible to use FreeBSD 13 guest to replace the opnsense just for the troubleshooting purpose? I am more familiar to stock FreeBSD source so if it can be reproduced on stock FreeBSD, it will be very helpful for me to quickly narrow down the problem.

Some more information about the RSC feature which seems to cause this problem. When RSC is turned on, Hyper-V vSwitch tries to coalesce small TCP packets into a larger one and send it the receiving guest. From my experience, it happens only when the sender and receiver are on same Hyper-V host. When Opnsense is acting as a packet forwarder, vSwitch on Hyper-V is not supposed to perform RSC. But in this case it seems the RSC is happening on the Opnsense guest for forwarding traffic. So there might be some config setting which confuses vSwitch.

[mnaim]

Yes, I my case its single hyper-v host that has opnsense vm acting as FW for that host (and other VMs). So WAN is ethernet card and LAN is virtual card connected to vSwitch.

WAN has public ip, LAN private, NAT between on opnsense. In case of iperf on opnsense (direct) it runs directly on WAN port as client. In case of iperf on Host (or other VMs) connected to vSwitch iperf communicate on LAN (vswitch) -> GW (opnsense NAT) -> WAN.

Someone in this post wrote, that he installed stock Freebsd and bootstrapped opnsense and issue was still there.

Based on commits that were rolledback, there has to be something in these 2 commits. Are any new settings involved in these commits?

[y2kw]

So when on 22.1 install this kernel and reboot:

# opnsense-update -zkr 22.1-hyperv
# yes | opnsense-shell reboot

(might take a bit to sync the kernel to all mirrors... the default mirror should have it)


Cheers,
Franco

Thanks Franco you're the best!!
I had an issue with the upload. the kernel update fix it!
What a great and fast support, that's the reason I love this community and your product.
BR

[franco]

I built a FreBSD 13-STABLE kernel just to verify this is present there as well. I've used upstream commit 4ee9fbcd853 which is what we used in the 22.1 release as well. But this one does not have any OPNsense modifications.

# opnsense-update -zkr 22.1-fbsd

(reboot)


Cheers,
Franco

[mnaim]

Tested vanilla freebsd kernel - opnsense-update -zkr 22.1-fbsd
Issue is still there :/

[whu]

Thanks folks for testing on FreeBSD. Would you outline the detailed NAT configuration on the FreeBSD VM which handles the packet forwarding between Iperf3 client (on the same Hyper-V host) and Iperf3 server (in WAN)? The more details like the number of hn NIC, etc, the better for me to reproduce. Thanks in advance.

[franco]

I did forget one additional data point since we're not using GENERIC and there might be something interfering in RSS option (which is off by default though). I sort of doubt it has an impact but we never know until we try...

# opnsense-update -zkr 22.1-fbsd-generic

(reboot)

Thanks in advances for the test drive!  :)


Cheers,
Franco