26.1.3 and Intel X710 (ixl0)

[felipe0123]

After upgrading to 26.1.3 I'm experiencing issues with the intel x710 interface. I wonder if anyone has seem something similar or have any advice.

Less than 1 minute after upgrading to 26.1.3, OPNsense lost all connectivity. It's like the network stack crashes completely, no traffic arrives at any of its interfaces (I226 and X710). Rebooting fixes it for 1 minute or so. About half the time, it will require power cycling as the soft reboot won't compete with the shutdown.

Console shows the following message a few times during shutdown:

ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_FULL, error OK


That made me suspect the ixl0 interface, so I unplugged it and the issue went away immediately. Has anyone experienced something similar? Any advice?

[1] ixl0: <Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.3-k> mem 0x82000000-0x827fffff,0x83000000-0x83007fff at device 0.0 on pci1
[1] ixl0: fw 9.156.79020 api 1.15 nvm 9.56 etid 800100fb oem 0.0.0
[1] ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C
[1] ixl0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ixl0: Using 4 RX queues 4 TX queues
[1] ixl0: Using MSI-X interrupts with 5 vectors
[1] ixl0: Ethernet address: 64:62:66:xx:xx:xx
[1] ixl0: Allocating 4 queues for PF LAN VSI; 4 queues active
[1] ixl0: PCI Express Bus: Speed 8.0GT/s Width x4
[1] ixl0: SR-IOV ready
[1] ixl0: netmap queues/slots: TX 4/1024, RX 4/1024

 

[pfry]

[...]Has anyone experienced something similar?[...]

Nope. Similar setup, sans i226s:

Code Select Expand

root@fw:/home/user # dmesg | grep ixl0
[1] ixl0: <Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.3-k> mem 0xf5000000-0xf57fffff,0xf5a18000-0xf5a1ffff at device 0.0 on pci1
[1] ixl0: fw 9.152.77998 api 1.15 nvm 9.50 etid 8000f4ab oem 1.270.0
[1] ixl0: PF-ID[0]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
[1] ixl0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ixl0: Using 8 RX queues 8 TX queues
[1] ixl0: Using MSI-X interrupts with 9 vectors
[1] ixl0: Ethernet address: 3c:fd:fe:e7:2d:88
[1] ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active
[1] ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
[1] ixl0: SR-IOV ready
[1] ixl0: netmap queues/slots: TX 8/1024, RX 8/1024
[19] ixl0: promiscuous mode enabled
root@fw:/home/user #

Interesting differences. I should be running all defaults. Could be due to driver selection differences, based on channel width or device (-DA4, in my case). Not likely relevant. Are you using other ports on the card?

[...]ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_FULL, error OK[...]

Huh. That would seem to suggest use of hardware VLAN filters, specifically a MAC VLAN... which seems unlikely; far likelier a generic, misleading message. (I'm not ambitious enough to dig into the driver.) Just for the heck of it, what do you have set under "Interfaces: Settings -> Network Interfaces"?

[felipe0123]

> Are you using other ports on the card?

device has 2x x710 + 2x i226. Only one of each was in use.


> what do you have set under "Interfaces: Settings -> Network Interfaces"

"disable ... offload" checkboxes all checked. vlan hardware filtering set to disabled too.

[pfry]

[solved] intel x710 - relevant?

[felipe0123]

Thanks, I've seen that. I doesn't seem relevant, that looks like an issue with virtualized servers, I'm not using that. Running on a Protectli

[OPNenthu]

This must be the VP2440. Did you recently install coreboot v0.9.1-rc3 (the one that fixes the i226-v ASPM issue)?  I wonder if that firmware maybe introduced a new issue.

[felipe0123]

It's the vp2440, but I did not update coreboot (ever) or NIC firmware (in the last few months). Other than some RX errors on the v226 connected to ONT, it has been running smoothly for a long while. The issue started immediately after I upgraded to OpnSense 26.1.3. (And of course I forgot to take a ZFS snapshot before upgrading and after moving to the new firewall rules interface)

[lechterpolntrien]

I've been struggling with intermittent instability on my VP2440 since I got it, but I'm still on 25.7. I currently have a 25 day uptime, which is the longest uptime I've had since I got it - I'm sure it will fall over tonight now that I've thought about it.


I was considering a 26.1 upgrade to help with these problems...