22.1rc1 slow in Hyper-V

[mnaim]

bug still present :/

I did forget one additional data point since we're not using GENERIC and there might be something interfering in RSS option (which is off by default though). I sort of doubt it has an impact but we never know until we try...

# opnsense-update -zkr 22.1-fbsd-generic

(reboot)

Thanks in advances for the test drive! 

opnsense-update -zkr 22.1-fbsd-generic

Cheers,
Franco

[mnaim]

Thanks folks for testing on FreeBSD. Would you outline the detailed NAT configuration on the FreeBSD VM which handles the packet forwarding between Iperf3 client (on the same Hyper-V host) and Iperf3 server (in WAN)? The more details like the number of hn NIC, etc, the better for me to reproduce. Thanks in advance.

of course, just tell me, which commands would you like to run

ifconfig

hn0: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
        ether MAC_ADD_REDACTED
        inet x.x.x.x(PUBLIC_IP_REDACTED) netmask 0xffffff80 broadcast PUBLIC_GW_IP_REDACTED
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
hn1: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
        ether MAC_ADD_REDACTED
        inet 192.168.0.254 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

pfctl -sn

no nat proto carp all
nat on hn0 inet from 127.0.0.0/8 to any port = isakmp -> (hn0:0) static-port
nat on hn0 inet from 127.0.0.0/8 to any -> (hn0:0) port 1024:65535
nat on hn0 inet from 192.168.0.0/24 to any port = isakmp -> (hn0:0) static-port
nat on hn0 inet from 192.168.0.0/24 to any -> (hn0:0) port 1024:65535
no rdr proto carp all
rdr on hn0 inet proto tcp from <IP_LIST1> to (hn0) port = ms-wbt-server -> 192.168.0.1 port 3389


hn0 = WAN - on hyper-v is configured as external network, connected to physical network card (allow management operating system to share this netwrok adapter - DISABLED)
hn1 = LAN - on hyperv- is configured as internal only network (192.168.0.254 is opnsense VM LAN and 192.168.0.1 is windows host internal LAN IP)

[mnaim]

on hyper-v host

Get-VMSwitch | fl

Name                                             : Internet
Id                                               : 41192bac-682d-4434-804e-f00fad5d33db
Notes                                            :
Extensions                                       : {Microsoft Windows Filtering Platform, Microsoft Azure VFP Switch Ex
                                                   tension, Microsoft NDIS Capture}
BandwidthReservationMode                         : None
PacketDirectEnabled                              : False
EmbeddedTeamingEnabled                           : False
AllowNetLbfoTeams                                : False
IovEnabled                                       : True
SwitchType                                       : External
AllowManagementOS                                : False
NetAdapterInterfaceDescription                   : HP NC362i Integrated DP Gigabit Server Adapter
NetAdapterInterfaceDescriptions                  : {HP NC362i Integrated DP Gigabit Server Adapter}
NetAdapterInterfaceGuid                          : {31c16232-44e6-430f-ae7f-7dd8bff83306}
IovSupport                                       : False
IovSupportReasons                                : {}
AvailableIPSecSA                                 : 0
NumberIPSecSAAllocated                           : 0
AvailableVMQueues                                : 0
NumberVmqAllocated                               : 0
IovQueuePairCount                                : 0
IovQueuePairsInUse                               : 0
IovVirtualFunctionCount                          : 0
IovVirtualFunctionsInUse                         : 0
PacketDirectInUse                                : False
DefaultQueueVrssEnabledRequested                 : True
DefaultQueueVrssEnabled                          : False
DefaultQueueVmmqEnabledRequested                 : True
DefaultQueueVmmqEnabled                          : False
DefaultQueueVrssMaxQueuePairsRequested           : 16
DefaultQueueVrssMaxQueuePairs                    : 0
DefaultQueueVrssMinQueuePairsRequested           : 1
DefaultQueueVrssMinQueuePairs                    : 0
DefaultQueueVrssQueueSchedulingModeRequested     : StaticVrss
DefaultQueueVrssQueueSchedulingMode              : Dynamic
DefaultQueueVrssExcludePrimaryProcessorRequested : False
DefaultQueueVrssExcludePrimaryProcessor          : False
SoftwareRscEnabled                               : True
RscOffloadEnabled                                : False
BandwidthPercentage                              : 0
DefaultFlowMinimumBandwidthAbsolute              : 0
DefaultFlowMinimumBandwidthWeight                : 0
CimSession                                       : CimSession: .
ComputerName                                     : xx
IsDeleted                                        : False
DefaultQueueVmmqQueuePairs                       : 0
DefaultQueueVmmqQueuePairsRequested              : 16

Name                                             : Hyper-V
Id                                               : 73270709-1ae1-4f20-bdca-d05b400a7dca
Notes                                            :
Extensions                                       : {Microsoft Windows Filtering Platform, Microsoft Azure VFP Switch Extension, Microsoft NDIS Capture}
BandwidthReservationMode                         : Absolute
PacketDirectEnabled                              : False
EmbeddedTeamingEnabled                           : False
AllowNetLbfoTeams                                : False
IovEnabled                                       : False
SwitchType                                       : Internal
AllowManagementOS                                : True
NetAdapterInterfaceDescription                   :
NetAdapterInterfaceDescriptions                  :
NetAdapterInterfaceGuid                          :
IovSupport                                       : False
IovSupportReasons                                :
AvailableIPSecSA                                 : 0
NumberIPSecSAAllocated                           : 0
AvailableVMQueues                                : 0
NumberVmqAllocated                               : 0
IovQueuePairCount                                : 0
IovQueuePairsInUse                               : 0
IovVirtualFunctionCount                          : 0
IovVirtualFunctionsInUse                         : 0
PacketDirectInUse                                : False
DefaultQueueVrssEnabledRequested                 : True
DefaultQueueVrssEnabled                          : False
DefaultQueueVmmqEnabledRequested                 : True
DefaultQueueVmmqEnabled                          : False
DefaultQueueVrssMaxQueuePairsRequested           : 16
DefaultQueueVrssMaxQueuePairs                    : 0
DefaultQueueVrssMinQueuePairsRequested           : 1
DefaultQueueVrssMinQueuePairs                    : 0
DefaultQueueVrssQueueSchedulingModeRequested     : StaticVrss
DefaultQueueVrssQueueSchedulingMode              : Dynamic
DefaultQueueVrssExcludePrimaryProcessorRequested : False
DefaultQueueVrssExcludePrimaryProcessor          : False
SoftwareRscEnabled                               : True
RscOffloadEnabled                                : False
BandwidthPercentage                              : 0
DefaultFlowMinimumBandwidthAbsolute              : 0
DefaultFlowMinimumBandwidthWeight                : 0
CimSession                                       : CimSession: .
ComputerName                                     : xx
IsDeleted                                        : False
DefaultQueueVmmqQueuePairs                       : 0
DefaultQueueVmmqQueuePairsRequested              : 16

[Helvetic]

Hi, I've run into the same issue on a new server and also could reproduced it locally. My input regarding this:

Some more information about the RSC feature which seems to cause this problem. When RSC is turned on, Hyper-V vSwitch tries to coalesce small TCP packets into a larger one and send it the receiving guest. From my experience, it happens only when the sender and receiver are on same Hyper-V host. When Opnsense is acting as a packet forwarder, vSwitch on Hyper-V is not supposed to perform RSC. But in this case it seems the RSC is happening on the Opnsense guest for forwarding traffic. So there might be some config setting which confuses vSwitch.

It only seems to happen if the data sender is on the same Hyper-V host (or the host it self) as the Opnsense VM. For the same connection in the opposite direction, it is not a problem. Also I tested without NAT, just routing packets.

[whu]

I was able to reproduce on FreeBSD VMs running on same Hyper-V host, with one of which being the iperf client and the other NAT server. So the problem seems to be that vswitch is L2 device. It doesn't check the IP address of packets so it always turns on RSC/LRO whenever it can. User really needs to manually disable RSC/LRO when the system is forwarding traffic. On Linux, system automatically turns off RSC/LRO when ip_forward is set to 1 in /etc/sysctl.conf. There seem no similar way in FreeBSD to do so. User will have to remember to disable RSC/LRO when the system is acting as a router.

Of course the current RSC implementation lacks the way to disable it on hn interface. This is something that I need to do next. The problem is that LRO in FreeBSD seems to be the GRO in Linux, which the packet aggregation happens in software above the NIC driver. There is no direct feature bit for NIC in FreeBSD to represent the true RSC/LRO, which the packet aggregation happens in hardware on NIC. I will need to consult FreeBSD network community for this. For now, reverting the RSC patch seems to be the right thing to do.

I will update later when I have fix ready for this.

[Lukas L.]

Same issue with our esxi 6.7 opnsense box (upload is at tens of Mbps not Gbps like our ISP speed - download is not affected). Is there any solution to stay with 22.1? Is it the same problem like in hyper-v? we are using vmxnet nics

[franco]

No this is a Hyper-V hn(4) driver issue only.


Cheers,
Franco

[Lukas L.]

Sorry - apparently red herrings here. In fact - I had very slow (0.2Mbps) upload to web on my PBR squid. Not behaving like this it while not throught the PBRed proxy. So I looked everywhere. One of the last resort was set-up a priority level in the PBR rule up to level 3. Now I have ~930Mbps download and 800-900Mbps upload. Surricata on the wan only and zenarmor on the local ifaces.
In the process I've enabled RSS, net.isr.bindthreads etc. (which I already have had from 21.7).
Nothing seem to work - as we are crucialy dependent on the web - it's a real pain in the a..

[franco]

I'm not sure what your point is besides this being the place to discuss it.

> Now I have ~930Mbps download and 800-900Mbps upload.

> Nothing seem to work

You need to structure your reports better and you can do so in a new topic.


Thanks,
Franco

[Lukas L.]

Thanks Franco - this is going to the another direction, unfortunately - I was celebrating too early. I will summarize it in my own guestion-thread

[Aergan]

Any update on this?

[franco]

Upgrades to 22.1 were fixed already. 22.1.1 will switch the kernel also. If you install from 22.1 images and hit the issue just reinstall kernel from firmware packages tab.

(22.1.1 on Wedesday.)


Cheers,
Franco

[Aergan]

Cheers - just competed the upgrade and so far so good!

[Fedo_Araruva]

Hi Franco,

I too am having these issues with an aggravated situation. I cannot download the updates...it takes forever!!!
Is there an ISO image with the corrections? I suppose it`s version 22.1.1 right?

Thanks

[franco]

Are you even using Hyper-V? There is just one scenario where you can catch the faulty kernel and that is install from 22.1 image. If you use the 21.7 image it's fine...

And even then it should just work to reinstall the kernel on 22.1.

But I doubt you are having that particular issue described here.


Cheers,
Franco