Slow download thoughput from WAN on ESXi 7.0 after update to 22.1, upload OK

[Sisyphe]

Hi,

I've updated OPNsense running on ESXI 7.0 update 3 (build 18905247) from 21.7.8 to 22.1 and I'm experiencing a drastic drop in download throughput from WAN.

21.7.8:

Code Select Expand


   Speedtest by Ookla

     Server: ORANGE FRANCE - Paris (id = 24215)
        ISP: Orange
    Latency:     1.52 ms   (0.48 ms jitter)
   Download:  1778.08 Mbps (data used: 1.4 GB )                               
     Upload:   598.07 Mbps (data used: 270.1 MB )                               
Packet Loss:     0.0%


22.1:

Code Select Expand


Speedtest by Ookla

     Server: ORANGE FRANCE - Paris (id = 24215)
        ISP: Orange
    Latency:     2.22 ms   (0.10 ms jitter)
   Download:     2.41 Mbps (data used: 2.8 MB )                               
     Upload:   597.61 Mbps (data used: 269.8 MB )                               
Packet Loss:     0.0%


Configuration:

• Qlogic 57810 pass-trough for WAN (if_bx0.ko)
• VMXNET 3 card for LAN
• Hardware CRC, TSO, LRO and VLAN Filtering are off.

Reverting to 21.7.8 fixed the issue.

Any idea on where to look?

Thanks.

[cmaxwell]

I have the same exact problem.

Qlogic 57810 on bare metal for me.

So maybe it is an if_bxe issue...hmm.

I also have Hardware CRC, TSO, LRO and VLAN filtering off.

My WAN is a VLAN - so wondering if that is the issue too... :-/

[johndchch]

no slowdown on 22.1 under exsi 7.0 here and no performance differences compared to 2.1.7.1 - but I'm using intel 210s and _not_ passing them thru ( just each one assigned to a dedicated vswitch for wan and lan)

could be worth going from pasthru to vmxnet3 just to see if it makes a difference?

[bolmsted]

A few of us in Bell Canada land are noticing the same thing and suspect its VLAN on WAN as I'm virtualizing OPNsense on Proxmox (using virtio as I switched from passthrough of bxe0) and someone else is running on baremetal.

[johndchch]

A few of us in Bell Canada land are noticing the same thing and suspect its VLAN on WAN as I'm virtualizing OPNsense on Proxmox (using virtio as I switched from passthrough of bxe0) and someone else is running on baremetal.

I'm on vlan tagged wan as well - however I'm having my switch handle the tagging so opnsense doesn't have to handle it. If you've got a suitable switch should be easy to test this and see if it cures your issue

[Chapter]

I have the same problem with a Virtualized OPNSense under Proxmox.

What I have also found is that if I run speedtest-cli from the OPNSense VM I get a normal download speed.

Edit: I also have VLANs on the WAN side

[s4rs]

I have a physical system and when I do happen to be able to connect to the internet with 22.1 I see the the same download/upload speeds. Full download 1Mb upload. I am using vlans on the physical nic.

[bolmsted]

A few of us in Bell Canada land are noticing the same thing and suspect its VLAN on WAN as I'm virtualizing OPNsense on Proxmox (using virtio as I switched from passthrough of bxe0) and someone else is running on baremetal.

I'm on vlan tagged wan as well - however I'm having my switch handle the tagging so opnsense doesn't have to handle it. If you've got a suitable switch should be easy to test this and see if it cures your issue

Well we are using SFP ports for our internet connection so if I were to buy a Ubiquiti switch that support 2.5Gbps and cost $800+ I probably wouldn't be using OPNsense to do this  :(

I was working before the 22.1 upgrade and I'm doing this on a fresh install

[franco]

If you have a VLAN but HAVE NOT assigned the parent interface please do so, enable it and then all the hardware features are properly turned off to get your speed back.


Cheers,
Franco

[Sisyphe]

Thank you Franco, it works!

Code Select Expand

   Speedtest by Ookla

     Server: ORANGE FRANCE - Paris (id = 24215)
        ISP: Orange
    Latency:     2.16 ms   (0.44 ms jitter)
   Download:  1804.37 Mbps (data used: 1.9 GB )                               
     Upload:   595.86 Mbps (data used: 269.2 MB )                               
Packet Loss:     0.0%

Is this change permanent or will the 21.7 behavior be restored in coming versions?

[franco]

This change is permanent. Previously if you had more than one VLAN and modified settings from it that affected the parent all the VLANS tried to apply their settings to the parent which is undefined behaviour solely dependent on the order of the interfaces in the configuration.

For MAC addresses the situation was even worse... ;)


Cheers,
Franco

[Sisyphe]

Thank you for the confirmation and explanation, it makes sense.

Cheers

[kasper93]

Related to ESXi and VMX driver. Tunables like

Code Select Expand

hw.vmx.txndesc
hw.vmx.rxndesc doesn't work anymore. Need to set it up per device

Code Select Expand

dev.vmx.0.iflib.override_nrxds
dev.vmx.0.iflib.override_ntxds

You can read here what values are expected for this override. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237166

This apply to all hw.vmx tunables that you might have had before.

I still have higher CPU usage and as a result reduces throughput, but didn't manage to find the culprit yet.

[TheHellSite]

EDIT: Wrong Thread.