YouTube and Google Play Store poor performance on OPNSense

[Crate2729]

I bought an Intel X710-DA2 to replace the X520-DA2 as final resolution after tuning everything possible.
And voila, the issue is gone!   
All Android devices are now able to update all the previously stucked apps, and the YT videos previously unplayable became playable again.
WOW, a massive headache now gone!

Why X710? It uses the i40e driver and it's PCIe Gen3, while the X520 used ixgbe and it's PCIe Gen2, and I wanted something that is different in both hardware (the X710 is a lot newer) and in driver as well. I don't know what the original issue was but I wanted to solve it once and for good. Maybe this was an issue with the HTTP/3 UDP-based QUIC protocol on the old hardware with an unlucky combination of host and guest kernels? We'll never know, but it's now solved at last.

If anybody else faces the issue, here are the details of the cards for reference:

X520-DA2

lspci

Code: [Select]

2d:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
        Subsystem: Intel Corporation 10GbE 2P X520 Adapter
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

lshw

Code: [Select]

       description: Ethernet interface
       product: Ethernet 10G 2P X520 Adapter
       vendor: Intel Corporation
       physical id: 0.1
       bus info: pci@0000:2d:00.1
       logical name: enp45s0f1
       version: 01
       size: 10Gbit/s
       capacity: 10Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical fibre 10000bt-fd
       configuration: autonegotiation=off broadcast=yes driver=ixgbe driverversion=6.5.11-8-pve duplex=full
 firmware=0x8000042f latency=0 link=yes multicast=yes port=fibre speed=10Gbit/s
       resources: irq:204 memory:fbd80000-fbdfffff ioport:f000(size=32) memory:fbf00000-fbf03fff
 memory:fbd00000-fbd7ffff memory:c0200000-c02fffff memory:c0300000-c03fffff

X710-DA2

lspci

Code: [Select]

2d:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
        Subsystem: Intel Corporation Ethernet 10G 2P X710 Adapter
        Kernel driver in use: i40e
        Kernel modules: i40e

lshw

Code: [Select]

       description: Ethernet interface
       product: Ethernet Controller X710 for 10GbE SFP+
       vendor: Intel Corporation
       physical id: 0.1
       bus info: pci@0000:2d:00.1
       logical name: enp45s0f1
       version: 02
       size: 10Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical fibre autonegotiation
       configuration: autonegotiation=off broadcast=yes driver=i40e driverversion=6.5.11-8-pve duplex=full
 firmware=6.80 0x80003d72 18.8.9 latency=0 link=yes multicast=yes port=fibre speed=10Gbit/s
       resources: irq:76 memory:f8000000-f8ffffff memory:fa000000-fa007fff memory:fcd00000-fcd7ffff


[queenkjuul]

I am extremely disheartened to learn that I will have to spend even more money on hardware, after already barely keeping my whole 2.5gb LAN upgrade under budget by buying Realtek NICs, which work just fine in all my other machines (which are all Linux or Windows) . However, I guess I can finally stop banging my head against the wall trying to solve my OPNsense config, so that's good.

[tb_one]

We do have intel cards, and xeon cpus and most time QUIC/UDP443 not passing the gateway.
sometimes we get some handshake packets back, and then we have big problem. browsers or apps do not switch back to tcp, so everything locks up. if theres just absolute no response loading websites takes 2-3 seconds longer, but as soon as the client does receive some packets back its locked to QUIC.

New Hardware is not a Solution, since we do have kind of high end hardware.

[andrebrait]

I registered just to respond to the comment above saying hardware isn't the solution: yes, you're right. It's software. But the drivers between the two cards do differ.

I once had an issue like this with an Intel X550-T2 on pfSense 2.7.0, using the ixgbe driver. I had a constant CPU usage or about 10% and some general slowdown and I couldn't understand why. I went back to my previous i225-LM and everything went back to normal. I tried yet another X550-T2 and had the same issue. I updated the X550-T2's firmware and same issue. I ultimately went back to the two onboard Intel Gigabit NICs my computer's motherboard has.

So, just chiming in, it's not their imagination. The ixgbe drivers are weird.

###################
EDIT: my original report on the pfSense subreddit

DHCPv6 server "high" CPU usage on Intel X550-T2, but not on Intel i225-LM

I'm running pfSense 2.7.0 CE and I noticed that, after replacing my 2 port QNAP i225-LM with an X550-T2, not only the CPU usage is generally slightly higher, but the DHCPv6 server process seems to constantly use 3%+ CPU, whereas is used almost 0% with the i225-LM. Disabling DHCPv6 server gets rid of that, but general CPU usage is still a little higher.

All settings are the same. TCP offloading is ON, all other offloading settings are disabled.

Just wondering if anyone here knows of anything (be it in the ixgbe / igb drivers or whatnot) that could cause that?

I also tried updating the NVM on the X550-T2 to version 3.60 (the card came with version 1.00 installed) and, other than losing ASPM for some reason, nothing changed.

EDIT: I switched back to the i225-LM and everything is back to normal. I switched to a second X550-T2 and it displayed the same behavior again.