After upgrade from 25.1.12 to 25.7.2: High response times when having traffic

[rainerle]

Hi,
after the upgrade on Tuesday the uplink delay went up from 1.2ms to 24ms.

Where can I look to fix this?

The measurement is taken from another machine behind the opnsense.

CPU usage between 2% and 12%,
More than 50% Memory free,
no idea where to look further...

Thanks
Rainer
 

[rainerle]

dmesg

Code Select Expand

root@opnsense01:~ # grep ixl2 /var/run/dmesg.boot
[1] ixl2: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem 0x21ffb000000-0x21ffbffffff,0x21fff008000-0x21fff00ffff irq 36 at device 0.2 numa-domain 0 on pci6
[1] ixl2: fw 3.10.53077 api 1.5 nvm 4.00 etid 8000183c oem 1.263.0
[1] ixl2: PF-ID[2]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
[1] ixl2: Using 1024 TX descriptors and 1024 RX descriptors
[1] ixl2: Using 8 RX queues 8 TX queues
[1] ixl2: Using MSI-X interrupts with 9 vectors
[1] ixl2: Ethernet address: 7c:d3:0a:d8:34:a2
[1] ixl2: Allocating 8 queues for PF LAN VSI; 8 queues active
[1] ixl2: SR-IOV ready
[1] ixl2: netmap queues/slots: TX 8/1024, RX 8/1024
[1] ixl2: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
[1] ixl2: link state changed to UP
root@opnsense01:~ #

pciconf -lv

Code Select Expand

...
ixl0@pci0:10:0:0:       class=0x020000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x37d3 subvendor=0x17aa subdevice=0x4021
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X722 for 10GbE SFP+'
    class      = network
    subclass   = ethernet
ixl1@pci0:10:0:1:       class=0x020000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x37d3 subvendor=0x17aa subdevice=0x4021
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X722 for 10GbE SFP+'
    class      = network
    subclass   = ethernet
ixl2@pci0:10:0:2:       class=0x020000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x37d3 subvendor=0x17aa subdevice=0x4021
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X722 for 10GbE SFP+'
    class      = network
    subclass   = ethernet
ixl3@pci0:10:0:3:       class=0x020000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x37d3 subvendor=0x17aa subdevice=0x4021
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X722 for 10GbE SFP+'
    class      = network
    subclass   = ethernet
...