OPNsense Forum
Archive => 22.1 Legacy Series => Topic started by: Com DAC on January 15, 2022, 03:42:32 pm
-
I thought I'd spin up a second vm to test 22.1 RC 1. It installed without issue and boots significantly faster. Restored configuration backup from previous version (21.7.7) without issue. All seems to technically work except my throughput is horrible. If I spin up the 21.7.7 vm I can get 300mbps (my full internet speed). Shut down 21.7.7 and spin up 22.1 RC 1 and I can only get 1 - 2 mbps. Anyone else had a chance to test in Hyper-V? Anyone else noticing this?
Incase it matters both VM's are Gen 2. All the hardware offloading is turned off (though I've also tried turning it on).
I've now also tried a compeltely clean install (without restoring my configuration backup) and still seeing the same slowdown.
-
I run two VMs in Asure. I assume that’s also Hyper-V running in the background. Didn’t recognize any performance issues so far, but as those are dev. VMs they are not heavily used.
-
Slow speed just indicates mismatch in NIC handling/checksum failures, etc.
Cheers,
Franco
-
Slow speed just indicates mismatch in NIC handling/checksum failures, etc.
Cheers,
Franco
Ok sounds logical. Though all the settings I can see between v21.7.7 and 22.1 RC are the same both inside OPNsense and in hyper-v. This is why I posted to see if anyone else has seen this. I've tried all the hardware offload settings can you suggest any other settings to try?
Thank you,
-
I'm not sure, it might also be some sort of regression in Hyper-V or its support. I don't think this is related but as an example this was issued last week:
https://www.freebsd.org/security/advisories/FreeBSD-EN-22:03.hyperv.asc
There's a number of bugs open here but skimming through the list doesn't have anything sticking out:
https://bugs.freebsd.org/bugzilla/buglist.cgi?quicksearch=hyper-v
For the time being I don't have actionable advice.
Cheers,
Franco
-
Just as another test I've tried upgrading my 21.7.7 to 22.1. Once it upgraded to FreeBSD 13 the network slowed down. Everything I've tested seems to be pointing to an issue in the FreeBSD 13 Hyper-V network drivers. Though the web interface is fast and responsive it's just getting through OPNsense to the internet that is slow.
Thank you,
-
Cannot confirm your experience on an Azure VM:
mf@opnsense-dev:~ % speedtest
Speedtest by Ookla
Server: Claranet Benelux B.V. - Amsterdam (id = 30847)
ISP: Microsoft Corporation
Latency: 2.53 ms (0.26 ms jitter)
Download: 3093.84 Mbps (data used: 3.2 GB )
Upload: 956.58 Mbps (data used: 864.6 MB )
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/abd6703b-9231-498c-97b6-98533a3df037
mf@opnsense-dev:~ % uname -r
13.0-STABLE
-
I've been testing in Hyper-V on Windows 11. Maybe Azure is running a slightly different version of Hyper-V. I'll try spinning up a test VM on a different version of Hyper-V and see if it still happens.
Thank you,
-
I too am able to duplicate some odd performance issues in Hyper-V with the 22.1RC.
I've tried this with a Gen1 Hyper-V VM and also in VirtualBox with virtio NICs. I agree with the findings that the OPNsense 22.1RC seems to have odd issues in Hyper V. In virtualbox, I'll get at most 70mbits of bandwidth and the VM will freeze up during a speedtest. I will get odd errors reported at the console, AHCI timeouts, etc. etc. If I swap the VirtualBox NICs back to emulated E1000 NICs, the will get full throughput of my connection (500/500).
For the Gen1 Hyper V VM, the download speed will hit maybe 21mbits/sec. However, the upload speed will be a bit better at around 200mbits/sec. Still nowhere near the full capability of the connection. CPU usage while running the tests does spike higher, in some cases around 90% briefly but it never completely utilizes either core.
Specs of VMs:
Running on Server 2019
2 vCPUs
2048MB RAM
All running on SSD datastore
Also for testing purposes, I used the following tunables:
vm.pmap.pti = 0
hw.ibrs_disable = 0
hw.pci.honor_msi_blacklist = 0
-
Updated to RC2 (took hours with the reduced speed) and the problem still persists.
-
I've done more testing with mine and it's quite an odd issue. I can push Iperf3 traffic through the firewall at over 1gbit/sec. Usually 1.1 to 1.4gbit/sec. This is using a client and server VM sitting on the WAN and LAN side of the router, so it's actually pushing and pulling routed traffic through.
However, from the same VM on the client side I cannot run a download speedtest (fast.com, speedtest.net, etc.) and get anything higher than about 10mbit on the download. On the upload, it will go somewhat higher, at around 200mbit. Very odd and I'm still troubleshooting to determine the cause.
So far I've tried disabling RSS. This had no effect. I've also tried lowering the MTU on the OPNsense VM, this also didn't make a difference. I'll keep digging and see what I can figure out but its a very odd issue. This is only present in HyperV or VirtualBox with virtio NICs, bare metal installs with igb NICs do just fine and get good throughput.
-
I just tried doing a clean install of freeBSD 13 and then bootstrapping OPNsense. The freeBSD installed fine. Bootstrapping OPNsense went smooth and fast ( https://github.com/opnsense/update#opnsense-bootstrap ). Once it rebooted and OPNsense was running it's access to the internet slowed the same as with the standard OPNsense install.
-
Thanks, it either points to a configuration issue that our system does (might also be traffic normalization or the like) or there's an issue in the FreeBSD code between 13.0 and 13-STABLE as we use it. The latter would be more problematic since that code is supposed to be 13.1 in a short while.
Cheers,
Franco
-
not that I expected anything to have changed in such a short period of time but I've tested against the full release 22.1 and the problem still exists.
-
As another data point, upgraded a proxmox 21.7.8 OPNsense VM to 22.1 and getting full speed on my symmetrical 1Gb fiber using virtio Nics from proxmox 7.1.
-
I can confirm that. Hyper-V on Server 2022 (on premise)
21.7
Download 800Mbps
Upload 881Mbps
22.1
Download 829Mbps
Upload 1.57Mbps
After rollback (thanks to this post snapshot was taken) speeds are ok.
-
Can someone record a packet capture on 22.1 with thew Hyper-V slowdown and tell us if it's bad checksums, missing packets or out of order traffic? At first glance there is no mention in FreeBSD bug tracker but having full download speed but poor upload seems to indicate outgoing checksum issues?
Cheers,
Franco
-
Here is a full all interface packet capture while using speedtest.net. Both my upload and download are being impacted.
Packet Capture: REDACTED BY ADMIN
Thank you,
-
Thanks I grabbed it and redacted the URL.
Cheers,
Franco
-
I snapshotted one of my production systems and upgraded it, i'm seeing issues on the download traffic, the upload seems normal.
I had to revert the machine back however if it would help, I can provide a temporary test system for you to run tests against franco
-
I am having this same issue on the published release.
Host is Windows Server 2022
NIC is a Intel(R) 82576 Gigabit.
Virtual "Switch" is configured with "Enable single-root I/O virtualization SR-IOV"
Speed is 1-2 Megabits with or without TCP offloading. Had to roll back to 21.7
-
I'm looking at https://github.com/opnsense/src/commit/cdc59163ff8e8f2a which isn't on FreeBSD 13.0-RELEASE. Is that a positive on "Windows Server 2019 hosts and later"? Maybe RSC can be disabled in the host to test this theory?
Cheers,
Franco
-
I've setup two new test vswitches (one for WAN and one for LAN). Disabled software RSC on both and now my download speeds are great but my upload speeds are still unusable.
I think this might be on to something.
-
Upgraded a production system to 22.1. Throughput dropped from several hundred Mbps to less than 5 Mbps (up and down). Came here to learn it's a known issue. Windows Server 2022, Gen2 VM, Intel NIC, no special settings.
Can't provide more information, sorry. Not back in the debugging / development game (yet). Just need a working system, so already rolled back to 21.7.
Maurice
-
Same here, upgraded from 21.7 to 22.1, speeds dropped from ~190 / 40 mbps to 3-15mbps.
Server 2019 Core, Gen 2, v9 VM; WAN vSwitch is based on a Realtek Card, LAN vSwitch on Multiplex / 2x Intel. No SR-IOV.
Could provide tests, logs or whatnot.
Reverted back to 21.7, but kept a snapshot of 22.1.
Cheers
-
I'm building a kernel without the RSC support now to try.
EDIT: just to report back on the packet capture analysis... it didn't show obvious stack issues so I assumed that this is some sort of packet disappearing act which made me look at changes between FreeBSD 13.0 and 13-STABLE we are using.
Cheers,
Franco
-
So when on 22.1 install this kernel and reboot:
# opnsense-update -zkr 22.1-hyperv
# yes | opnsense-shell reboot
(might take a bit to sync the kernel to all mirrors... the default mirror should have it)
Cheers,
Franco
-
that seems to have done it. Speeds are good with the updated kernel.
-
Upgraded to 22.1 again and installed the patched kernel. First impression: Performance "feels" like 21.7.
Thanks, Franco.
Maurice
-
Hi,
I'm experiencing similar slowdowns with 22.1 final on ESXi 7.0 Update 3. Reverting to 21.7 resolved the issue.
I haven't tried the patched kernel yet as I assume this may only be relevant for Hyper-V.
Cheers
-
So when on 22.1 install this kernel and reboot:
# opnsense-update -zkr 22.1-hyperv
# yes | opnsense-shell reboot
(might take a bit to sync the kernel to all mirrors... the default mirror should have it)
Cheers,
Franco
Can confirm this works, speeds are back to normal.
Thanks a lot!
-
Replaced default upgrade kernel so that should be it. For reference the following commits were added removing the feature...
https://github.com/opnsense/src/commit/e91d90c0ac
https://github.com/opnsense/src/commit/2077062afc
Thanks all for helping confirm this. I'll try to follow up with FreeBSD/Microsoft on Monday.
Cheers,
Franco
-
So I'm trying to look at this closer with Microsoft and there are a few open questions that would require someone to load the bad kernel and provide more information:
1. Does the throughput drop affect TCP / UDP or both? We suspect TCP but unsure about UDP...
2. We need the output from "sysctl -a | grep rsc" after a speed test with bad throughput run.
Thanks,
Franco
-
I can help, how to load "bad kernel"?
-
Good question. Let me get back in a second.
EDIT: load the following kernel
# opnsense-update -zkr 22.1
(reboot)
Cheers,
Franco
-
Attached are the results of the sysctl -a | grep rsc from before the speed test and then after the speed test. Also attached are the results of the speed test.
Does anyone have any guidance on how to test udp?
Thank you,
-
dev.hn.1.rx.7.rsc_drop: 0
dev.hn.1.rx.7.rsc_pkts: 15
dev.hn.1.rx.6.rsc_drop: 0
dev.hn.1.rx.6.rsc_pkts: 224
dev.hn.1.rx.5.rsc_drop: 0
dev.hn.1.rx.5.rsc_pkts: 13
dev.hn.1.rx.4.rsc_drop: 2
dev.hn.1.rx.4.rsc_pkts: 286
dev.hn.1.rx.3.rsc_drop: 0
dev.hn.1.rx.3.rsc_pkts: 15
dev.hn.1.rx.2.rsc_drop: 1
dev.hn.1.rx.2.rsc_pkts: 349
dev.hn.1.rx.1.rsc_drop: 0
dev.hn.1.rx.1.rsc_pkts: 11
dev.hn.1.rx.0.rsc_drop: 0
dev.hn.1.rx.0.rsc_pkts: 501
dev.hn.0.rx.7.rsc_drop: 0
dev.hn.0.rx.7.rsc_pkts: 0
dev.hn.0.rx.6.rsc_drop: 0
dev.hn.0.rx.6.rsc_pkts: 0
dev.hn.0.rx.5.rsc_drop: 0
dev.hn.0.rx.5.rsc_pkts: 0
dev.hn.0.rx.4.rsc_drop: 0
dev.hn.0.rx.4.rsc_pkts: 0
dev.hn.0.rx.3.rsc_drop: 0
dev.hn.0.rx.3.rsc_pkts: 0
dev.hn.0.rx.2.rsc_drop: 0
dev.hn.0.rx.2.rsc_pkts: 0
dev.hn.0.rx.1.rsc_drop: 0
dev.hn.0.rx.1.rsc_pkts: 0
dev.hn.0.rx.0.rsc_drop: 0
dev.hn.0.rx.0.rsc_pkts: 0
-
Ok, got interesting finding.
Installed iperf3 and used public fastest server from iperf site.
On bad kernel
On server behind opnsense (NAT)
iperf3 -c paris.testdebit.info -t 20 -f m -i 1 -P 10 -p 9240 -R
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 11.8 MBytes 99.0 Mbits/sec
[ 7] 0.00-1.00 sec 10.7 MBytes 89.8 Mbits/sec
[ 9] 0.00-1.00 sec 12.8 MBytes 107 Mbits/sec
[ 11] 0.00-1.00 sec 9.02 MBytes 75.6 Mbits/sec
[ 13] 0.00-1.00 sec 590 KBytes 4.83 Mbits/sec
[ 15] 0.00-1.00 sec 13.2 MBytes 111 Mbits/sec
[ 17] 0.00-1.00 sec 13.7 MBytes 115 Mbits/sec
[ 19] 0.00-1.00 sec 612 KBytes 5.01 Mbits/sec
[ 21] 0.00-1.00 sec 537 KBytes 4.40 Mbits/sec
[ 23] 0.00-1.00 sec 11.5 MBytes 96.6 Mbits/sec
[SUM] 0.00-1.00 sec 84.4 MBytes 708 Mbits/sec
Reverse direction (see that packets are lost for few seconds)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec
[ 7] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec
[ 9] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec
[ 11] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec
[ 13] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec
[ 15] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec
[ 17] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec
[ 19] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec
[ 21] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec
[ 23] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec
[SUM] 0.00-1.00 sec 2.50 MBytes 21.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 7] 1.00-2.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 9] 1.00-2.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 11] 1.00-2.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 13] 1.00-2.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 15] 1.00-2.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 17] 1.00-2.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 19] 1.00-2.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 21] 1.00-2.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 23] 1.00-2.00 sec 0.00 Bytes 0.00 Mbits/sec
[SUM] 1.00-2.00 sec 0.00 Bytes 0.00 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 7] 2.00-3.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 9] 2.00-3.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 11] 2.00-3.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 13] 2.00-3.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 15] 2.00-3.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 17] 2.00-3.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 19] 2.00-3.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 21] 2.00-3.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 23] 2.00-3.00 sec 0.00 Bytes 0.00 Mbits/sec
[SUM] 2.00-3.00 sec 0.00 Bytes 0.00 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 7] 3.00-4.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 9] 3.00-4.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 11] 3.00-4.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 13] 3.00-4.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 15] 3.00-4.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 17] 3.00-4.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 19] 3.00-4.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 21] 3.00-4.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 23] 3.00-4.00 sec 0.00 Bytes 0.00 Mbits/sec
[SUM] 3.00-4.00 sec 0.00 Bytes 0.00 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 7] 4.00-5.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 9] 4.00-5.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 11] 4.00-5.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 13] 4.00-5.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 15] 4.00-5.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 17] 4.00-5.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 19] 4.00-5.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 21] 4.00-5.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 23] 4.00-5.00 sec 128 KBytes 1.05 Mbits/sec
[SUM] 4.00-5.00 sec 128 KBytes 1.05 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec
[ 7] 5.00-6.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 9] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec
[ 11] 5.00-6.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 13] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec
[ 15] 5.00-6.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 17] 5.00-6.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 19] 5.00-6.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 21] 5.00-6.00 sec 0.00 Bytes 0.00 Mbits/sec
[ 23] 5.00-6.00 sec 0.00 Bytes 0.00 Mbits/sec
[SUM] 5.00-6.00 sec 384 KBytes 3.15 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
So I got idea if NAT is not somehow involved
Installed iperf3 on opnsense itself
iperf3 -c paris.testdebit.info -t 20 -f m -i 1 -P 10 -p 9240
[ 5] 3.00-4.00 sec 11.4 MBytes 95.8 Mbits/sec 0 290 KBytes
[ 7] 3.00-4.00 sec 602 KBytes 4.93 Mbits/sec 0 65.5 KBytes
[ 9] 3.00-4.00 sec 8.31 MBytes 69.7 Mbits/sec 0 221 KBytes
[ 11] 3.00-4.00 sec 5.31 MBytes 44.5 Mbits/sec 3 83.9 KBytes
[ 13] 3.00-4.00 sec 19.1 MBytes 161 Mbits/sec 0 6.81 MBytes
[ 15] 3.00-4.00 sec 11.0 MBytes 92.5 Mbits/sec 0 282 KBytes
[ 17] 3.00-4.00 sec 13.6 MBytes 114 Mbits/sec 0 1.37 MBytes
[ 19] 3.00-4.00 sec 5.21 MBytes 43.7 Mbits/sec 1 39.8 KBytes
[ 21] 3.00-4.00 sec 1.40 MBytes 11.8 Mbits/sec 0 168 KBytes
[ 23] 3.00-4.00 sec 10.1 MBytes 85.1 Mbits/sec 0 1.09 MBytes
[SUM] 3.00-4.00 sec 86.1 MBytes 722 Mbits/sec 4
In reverse direction same speed
So yes it has to be NAT (or some kind of packet processing)
(on good kernel all directions NAT/direct ok)
-
Thanks for running more tests. In order to further narrow down the issue, I'd like to understand more about the settings of the test. Would you let me know if the iperf3 client and opnsense guest are running on the same Windows Hyper-V host or not?
The other question is how the NAT was configured on the opnsense guest? What is the IP address of the forwarding interface on the Opnserver and how the iperf3 client IP address is configured in NAT to send to iperf3 server over the internet?
Is it possible to use FreeBSD 13 guest to replace the opnsense just for the troubleshooting purpose? I am more familiar to stock FreeBSD source so if it can be reproduced on stock FreeBSD, it will be very helpful for me to quickly narrow down the problem.
Some more information about the RSC feature which seems to cause this problem. When RSC is turned on, Hyper-V vSwitch tries to coalesce small TCP packets into a larger one and send it the receiving guest. From my experience, it happens only when the sender and receiver are on same Hyper-V host. When Opnsense is acting as a packet forwarder, vSwitch on Hyper-V is not supposed to perform RSC. But in this case it seems the RSC is happening on the Opnsense guest for forwarding traffic. So there might be some config setting which confuses vSwitch.
-
Yes, I my case its single hyper-v host that has opnsense vm acting as FW for that host (and other VMs). So WAN is ethernet card and LAN is virtual card connected to vSwitch.
WAN has public ip, LAN private, NAT between on opnsense. In case of iperf on opnsense (direct) it runs directly on WAN port as client. In case of iperf on Host (or other VMs) connected to vSwitch iperf communicate on LAN (vswitch) -> GW (opnsense NAT) -> WAN.
Someone in this post wrote, that he installed stock Freebsd and bootstrapped opnsense and issue was still there.
Based on commits that were rolledback, there has to be something in these 2 commits. Are any new settings involved in these commits?
-
So when on 22.1 install this kernel and reboot:
# opnsense-update -zkr 22.1-hyperv
# yes | opnsense-shell reboot
(might take a bit to sync the kernel to all mirrors... the default mirror should have it)
Cheers,
Franco
Thanks Franco you're the best!!
I had an issue with the upload. the kernel update fix it!
What a great and fast support, that's the reason I love this community and your product.
BR
-
I built a FreBSD 13-STABLE kernel just to verify this is present there as well. I've used upstream commit 4ee9fbcd853 which is what we used in the 22.1 release as well. But this one does not have any OPNsense modifications.
# opnsense-update -zkr 22.1-fbsd
(reboot)
Cheers,
Franco
-
Tested vanilla freebsd kernel - opnsense-update -zkr 22.1-fbsd
Issue is still there :/
-
Thanks folks for testing on FreeBSD. Would you outline the detailed NAT configuration on the FreeBSD VM which handles the packet forwarding between Iperf3 client (on the same Hyper-V host) and Iperf3 server (in WAN)? The more details like the number of hn NIC, etc, the better for me to reproduce. Thanks in advance.
-
I did forget one additional data point since we're not using GENERIC and there might be something interfering in RSS option (which is off by default though). I sort of doubt it has an impact but we never know until we try...
# opnsense-update -zkr 22.1-fbsd-generic
(reboot)
Thanks in advances for the test drive! :)
Cheers,
Franco
-
bug still present :/
I did forget one additional data point since we're not using GENERIC and there might be something interfering in RSS option (which is off by default though). I sort of doubt it has an impact but we never know until we try...
# opnsense-update -zkr 22.1-fbsd-generic
(reboot)
Thanks in advances for the test drive! :)
opnsense-update -zkr 22.1-fbsd-generic
Cheers,
Franco
-
Thanks folks for testing on FreeBSD. Would you outline the detailed NAT configuration on the FreeBSD VM which handles the packet forwarding between Iperf3 client (on the same Hyper-V host) and Iperf3 server (in WAN)? The more details like the number of hn NIC, etc, the better for me to reproduce. Thanks in advance.
of course, just tell me, which commands would you like to run
ifconfig
hn0: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
ether MAC_ADD_REDACTED
inet x.x.x.x(PUBLIC_IP_REDACTED) netmask 0xffffff80 broadcast PUBLIC_GW_IP_REDACTED
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
hn1: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
ether MAC_ADD_REDACTED
inet 192.168.0.254 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pfctl -sn
no nat proto carp all
nat on hn0 inet from 127.0.0.0/8 to any port = isakmp -> (hn0:0) static-port
nat on hn0 inet from 127.0.0.0/8 to any -> (hn0:0) port 1024:65535
nat on hn0 inet from 192.168.0.0/24 to any port = isakmp -> (hn0:0) static-port
nat on hn0 inet from 192.168.0.0/24 to any -> (hn0:0) port 1024:65535
no rdr proto carp all
rdr on hn0 inet proto tcp from <IP_LIST1> to (hn0) port = ms-wbt-server -> 192.168.0.1 port 3389
hn0 = WAN - on hyper-v is configured as external network, connected to physical network card (allow management operating system to share this netwrok adapter - DISABLED)
hn1 = LAN - on hyperv- is configured as internal only network (192.168.0.254 is opnsense VM LAN and 192.168.0.1 is windows host internal LAN IP)
-
on hyper-v host
Get-VMSwitch | fl
Name : Internet
Id : 41192bac-682d-4434-804e-f00fad5d33db
Notes :
Extensions : {Microsoft Windows Filtering Platform, Microsoft Azure VFP Switch Ex
tension, Microsoft NDIS Capture}
BandwidthReservationMode : None
PacketDirectEnabled : False
EmbeddedTeamingEnabled : False
AllowNetLbfoTeams : False
IovEnabled : True
SwitchType : External
AllowManagementOS : False
NetAdapterInterfaceDescription : HP NC362i Integrated DP Gigabit Server Adapter
NetAdapterInterfaceDescriptions : {HP NC362i Integrated DP Gigabit Server Adapter}
NetAdapterInterfaceGuid : {31c16232-44e6-430f-ae7f-7dd8bff83306}
IovSupport : False
IovSupportReasons : {}
AvailableIPSecSA : 0
NumberIPSecSAAllocated : 0
AvailableVMQueues : 0
NumberVmqAllocated : 0
IovQueuePairCount : 0
IovQueuePairsInUse : 0
IovVirtualFunctionCount : 0
IovVirtualFunctionsInUse : 0
PacketDirectInUse : False
DefaultQueueVrssEnabledRequested : True
DefaultQueueVrssEnabled : False
DefaultQueueVmmqEnabledRequested : True
DefaultQueueVmmqEnabled : False
DefaultQueueVrssMaxQueuePairsRequested : 16
DefaultQueueVrssMaxQueuePairs : 0
DefaultQueueVrssMinQueuePairsRequested : 1
DefaultQueueVrssMinQueuePairs : 0
DefaultQueueVrssQueueSchedulingModeRequested : StaticVrss
DefaultQueueVrssQueueSchedulingMode : Dynamic
DefaultQueueVrssExcludePrimaryProcessorRequested : False
DefaultQueueVrssExcludePrimaryProcessor : False
SoftwareRscEnabled : True
RscOffloadEnabled : False
BandwidthPercentage : 0
DefaultFlowMinimumBandwidthAbsolute : 0
DefaultFlowMinimumBandwidthWeight : 0
CimSession : CimSession: .
ComputerName : xx
IsDeleted : False
DefaultQueueVmmqQueuePairs : 0
DefaultQueueVmmqQueuePairsRequested : 16
Name : Hyper-V
Id : 73270709-1ae1-4f20-bdca-d05b400a7dca
Notes :
Extensions : {Microsoft Windows Filtering Platform, Microsoft Azure VFP Switch Extension, Microsoft NDIS Capture}
BandwidthReservationMode : Absolute
PacketDirectEnabled : False
EmbeddedTeamingEnabled : False
AllowNetLbfoTeams : False
IovEnabled : False
SwitchType : Internal
AllowManagementOS : True
NetAdapterInterfaceDescription :
NetAdapterInterfaceDescriptions :
NetAdapterInterfaceGuid :
IovSupport : False
IovSupportReasons :
AvailableIPSecSA : 0
NumberIPSecSAAllocated : 0
AvailableVMQueues : 0
NumberVmqAllocated : 0
IovQueuePairCount : 0
IovQueuePairsInUse : 0
IovVirtualFunctionCount : 0
IovVirtualFunctionsInUse : 0
PacketDirectInUse : False
DefaultQueueVrssEnabledRequested : True
DefaultQueueVrssEnabled : False
DefaultQueueVmmqEnabledRequested : True
DefaultQueueVmmqEnabled : False
DefaultQueueVrssMaxQueuePairsRequested : 16
DefaultQueueVrssMaxQueuePairs : 0
DefaultQueueVrssMinQueuePairsRequested : 1
DefaultQueueVrssMinQueuePairs : 0
DefaultQueueVrssQueueSchedulingModeRequested : StaticVrss
DefaultQueueVrssQueueSchedulingMode : Dynamic
DefaultQueueVrssExcludePrimaryProcessorRequested : False
DefaultQueueVrssExcludePrimaryProcessor : False
SoftwareRscEnabled : True
RscOffloadEnabled : False
BandwidthPercentage : 0
DefaultFlowMinimumBandwidthAbsolute : 0
DefaultFlowMinimumBandwidthWeight : 0
CimSession : CimSession: .
ComputerName : xx
IsDeleted : False
DefaultQueueVmmqQueuePairs : 0
DefaultQueueVmmqQueuePairsRequested : 16
-
Hi, I've run into the same issue on a new server and also could reproduced it locally. My input regarding this:
Some more information about the RSC feature which seems to cause this problem. When RSC is turned on, Hyper-V vSwitch tries to coalesce small TCP packets into a larger one and send it the receiving guest. From my experience, it happens only when the sender and receiver are on same Hyper-V host. When Opnsense is acting as a packet forwarder, vSwitch on Hyper-V is not supposed to perform RSC. But in this case it seems the RSC is happening on the Opnsense guest for forwarding traffic. So there might be some config setting which confuses vSwitch.
It only seems to happen if the data sender is on the same Hyper-V host (or the host it self) as the Opnsense VM. For the same connection in the opposite direction, it is not a problem. Also I tested without NAT, just routing packets.
-
I was able to reproduce on FreeBSD VMs running on same Hyper-V host, with one of which being the iperf client and the other NAT server. So the problem seems to be that vswitch is L2 device. It doesn't check the IP address of packets so it always turns on RSC/LRO whenever it can. User really needs to manually disable RSC/LRO when the system is forwarding traffic. On Linux, system automatically turns off RSC/LRO when ip_forward is set to 1 in /etc/sysctl.conf. There seem no similar way in FreeBSD to do so. User will have to remember to disable RSC/LRO when the system is acting as a router.
Of course the current RSC implementation lacks the way to disable it on hn interface. This is something that I need to do next. The problem is that LRO in FreeBSD seems to be the GRO in Linux, which the packet aggregation happens in software above the NIC driver. There is no direct feature bit for NIC in FreeBSD to represent the true RSC/LRO, which the packet aggregation happens in hardware on NIC. I will need to consult FreeBSD network community for this. For now, reverting the RSC patch seems to be the right thing to do.
I will update later when I have fix ready for this.
-
Same issue with our esxi 6.7 opnsense box (upload is at tens of Mbps not Gbps like our ISP speed - download is not affected). Is there any solution to stay with 22.1? Is it the same problem like in hyper-v? we are using vmxnet nics
-
No this is a Hyper-V hn(4) driver issue only.
Cheers,
Franco
-
Sorry - apparently red herrings here. In fact - I had very slow (0.2Mbps) upload to web on my PBR squid. Not behaving like this it while not throught the PBRed proxy. So I looked everywhere. One of the last resort was set-up a priority level in the PBR rule up to level 3. Now I have ~930Mbps download and 800-900Mbps upload. Surricata on the wan only and zenarmor on the local ifaces.
In the process I've enabled RSS, net.isr.bindthreads etc. (which I already have had from 21.7).
Nothing seem to work - as we are crucialy dependent on the web - it's a real pain in the a..
-
I'm not sure what your point is besides this being the place to discuss it.
> Now I have ~930Mbps download and 800-900Mbps upload.
> Nothing seem to work
You need to structure your reports better and you can do so in a new topic.
Thanks,
Franco
-
Thanks Franco - this is going to the another direction, unfortunately - I was celebrating too early. I will summarize it in my own guestion-thread
-
Any update on this?
-
Upgrades to 22.1 were fixed already. 22.1.1 will switch the kernel also. If you install from 22.1 images and hit the issue just reinstall kernel from firmware packages tab.
(22.1.1 on Wedesday.)
Cheers,
Franco
-
Cheers - just competed the upgrade and so far so good!
-
Hi Franco,
I too am having these issues with an aggravated situation. I cannot download the updates...it takes forever!!!
Is there an ISO image with the corrections? I suppose it`s version 22.1.1 right?
Thanks
-
Are you even using Hyper-V? There is just one scenario where you can catch the faulty kernel and that is install from 22.1 image. If you use the 21.7 image it's fine...
And even then it should just work to reinstall the kernel on 22.1.
But I doubt you are having that particular issue described here.
Cheers,
Franco
-
Is there any kernel fix which can solve similar issue on qemu/kvm installation? I'm suffering from bad WAN upload speed since upgrading to 22.1
-
This change helped me to solve my upload issue:
https://forum.opnsense.org/index.php?topic=26602.msg128862#msg128862
If you have a VLAN but HAVE NOT assigned the parent interface please do so, enable it and then all the hardware features are properly turned off to get your speed back.
-
Hello,
we have the same problem with 22.1.9 on our Hyper-V Cluster.
If the vServers and the Opnsense VM are on the same Host there is perfect speed.
But if the VMs are on different Hosts than the Opnsense VM the upload speed is horrible.
Is there any fix we can try?
-
Hello,
we have the same problem with 22.1.9 on our Hyper-V Cluster.
If the vServers and the Opnsense VM are on the same Host there is perfect speed.
But if the VMs are on different Hosts than the Opnsense VM the upload speed is horrible.
Is there any fix we can try?
The fix that worked for me was to disable RSC on the shared physical interface:
Set-VMSwitch -Name "your-switch-name" -EnableSoftwareRsc $false
IIRC this is a known bug in FreeBSD. It may be worth trying the beta of OPNsense coming out in July to see if this resolves the issue?
-
I tried this right now but the problem remains.
Is there any opnsense-patch we can install for trying to solve this issue?
Or maybe switch back to an older kernel which doesn't include the bug?
-
@Franco
Is it safe for me to try this?
Thanks in advance!
So when on 22.1 install this kernel and reboot:
# opnsense-update -zkr 22.1-hyperv
# yes | opnsense-shell reboot
(might take a bit to sync the kernel to all mirrors... the default mirror should have it)
Cheers,
Franco
-
You are very late to this particular party. None of discussed applies to 22.1.x installations.
You have an error, but certainly not this one.
Cheers,
Franco
-
But if I change back to version 21.7.x the problem is gone and the speed is perfect over different hosts ::)
The fix Set-VMSwitch -Name "your-switch-name" -EnableSoftwareRsc $false
didn't work for me...
-
Yes, different issue as I said. Please don't hijack this thread.
Microsoft changed a couple of things in FreeBSD 13 that weren't optimal it seems...
Cheers,
Franco