Messages

I was able to reproduce on FreeBSD VMs running on same Hyper-V host, with one of which being the iperf client and the other NAT server. So the problem seems to be that vswitch is L2 device. It doesn't check the IP address of packets so it always turns on RSC/LRO whenever it can. User really needs to manually disable RSC/LRO when the system is forwarding traffic. On Linux, system automatically turns off RSC/LRO when ip_forward is set to 1 in /etc/sysctl.conf. There seem no similar way in FreeBSD to do so. User will have to remember to disable RSC/LRO when the system is acting as a router.

Of course the current RSC implementation lacks the way to disable it on hn interface. This is something that I need to do next. The problem is that LRO in FreeBSD seems to be the GRO in Linux, which the packet aggregation happens in software above the NIC driver. There is no direct feature bit for NIC in FreeBSD to represent the true RSC/LRO, which the packet aggregation happens in hardware on NIC. I will need to consult FreeBSD network community for this. For now, reverting the RSC patch seems to be the right thing to do.

I will update later when I have fix ready for this.

Thanks folks for testing on FreeBSD. Would you outline the detailed NAT configuration on the FreeBSD VM which handles the packet forwarding between Iperf3 client (on the same Hyper-V host) and Iperf3 server (in WAN)? The more details like the number of hn NIC, etc, the better for me to reproduce. Thanks in advance.

Thanks for running more tests. In order to further narrow down the issue, I'd like to understand more about the settings of the test. Would you let me know if the iperf3 client and opnsense guest are running on the same Windows Hyper-V host or not?

The other question is how the NAT was configured on the opnsense guest? What is the IP address of the forwarding interface on the Opnserver and how the iperf3 client IP address is configured in NAT to send to iperf3 server over the internet?
Is it possible to use FreeBSD 13 guest to replace the opnsense just for the troubleshooting purpose? I am more familiar to stock FreeBSD source so if it can be reproduced on stock FreeBSD, it will be very helpful for me to quickly narrow down the problem.

Some more information about the RSC feature which seems to cause this problem. When RSC is turned on, Hyper-V vSwitch tries to coalesce small TCP packets into a larger one and send it the receiving guest. From my experience, it happens only when the sender and receiver are on same Hyper-V host. When Opnsense is acting as a packet forwarder, vSwitch on Hyper-V is not supposed to perform RSC. But in this case it seems the RSC is happening on the Opnsense guest for forwarding traffic. So there might be some config setting which confuses vSwitch.