26.1.3 and Intel X710 (ixl0)

[felipe0123]

After upgrading to 26.1.3 I'm experiencing issues with the intel x710 interface. I wonder if anyone has seem something similar or have any advice.

Less than 1 minute after upgrading to 26.1.3, OPNsense lost all connectivity. It's like the network stack crashes completely, no traffic arrives at any of its interfaces (I226 and X710). Rebooting fixes it for 1 minute or so. About half the time, it will require power cycling as the soft reboot won't compete with the shutdown.

Console shows the following message a few times during shutdown:

ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_FULL, error OK


That made me suspect the ixl0 interface, so I unplugged it and the issue went away immediately. Has anyone experienced something similar? Any advice?

[1] ixl0: <Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.3-k> mem 0x82000000-0x827fffff,0x83000000-0x83007fff at device 0.0 on pci1
[1] ixl0: fw 9.156.79020 api 1.15 nvm 9.56 etid 800100fb oem 0.0.0
[1] ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, I2C
[1] ixl0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ixl0: Using 4 RX queues 4 TX queues
[1] ixl0: Using MSI-X interrupts with 5 vectors
[1] ixl0: Ethernet address: 64:62:66:xx:xx:xx
[1] ixl0: Allocating 4 queues for PF LAN VSI; 4 queues active
[1] ixl0: PCI Express Bus: Speed 8.0GT/s Width x4
[1] ixl0: SR-IOV ready
[1] ixl0: netmap queues/slots: TX 4/1024, RX 4/1024

 

[pfry]

[...]Has anyone experienced something similar?[...]

Nope. Similar setup, sans i226s:

Code Select Expand

root@fw:/home/user # dmesg | grep ixl0
[1] ixl0: <Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.3-k> mem 0xf5000000-0xf57fffff,0xf5a18000-0xf5a1ffff at device 0.0 on pci1
[1] ixl0: fw 9.152.77998 api 1.15 nvm 9.50 etid 8000f4ab oem 1.270.0
[1] ixl0: PF-ID[0]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
[1] ixl0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ixl0: Using 8 RX queues 8 TX queues
[1] ixl0: Using MSI-X interrupts with 9 vectors
[1] ixl0: Ethernet address: 3c:fd:fe:e7:2d:88
[1] ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active
[1] ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
[1] ixl0: SR-IOV ready
[1] ixl0: netmap queues/slots: TX 8/1024, RX 8/1024
[19] ixl0: promiscuous mode enabled
root@fw:/home/user #

Interesting differences. I should be running all defaults. Could be due to driver selection differences, based on channel width or device (-DA4, in my case). Not likely relevant. Are you using other ports on the card?

[...]ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_FULL, error OK[...]

Huh. That would seem to suggest use of hardware VLAN filters, specifically a MAC VLAN... which seems unlikely; far likelier a generic, misleading message. (I'm not ambitious enough to dig into the driver.) Just for the heck of it, what do you have set under "Interfaces: Settings -> Network Interfaces"?

[felipe0123]

> Are you using other ports on the card?

device has 2x x710 + 2x i226. Only one of each was in use.


> what do you have set under "Interfaces: Settings -> Network Interfaces"

"disable ... offload" checkboxes all checked. vlan hardware filtering set to disabled too.

[pfry]

[solved] intel x710 - relevant?

[felipe0123]

Thanks, I've seen that. I doesn't seem relevant, that looks like an issue with virtualized servers, I'm not using that. Running on a Protectli

[OPNenthu]

This must be the VP2440. Did you recently install coreboot v0.9.1-rc3 (the one that fixes the i226-v ASPM issue)?  I wonder if that firmware maybe introduced a new issue.

[felipe0123]

It's the vp2440, but I did not update coreboot (ever) or NIC firmware (in the last few months). Other than some RX errors on the v226 connected to ONT, it has been running smoothly for a long while. The issue started immediately after I upgraded to OpnSense 26.1.3. (And of course I forgot to take a ZFS snapshot before upgrading and after moving to the new firewall rules interface)

[lechterpolntrien]

I've been struggling with intermittent instability on my VP2440 since I got it, but I'm still on 25.7. I currently have a 25 day uptime, which is the longest uptime I've had since I got it - I'm sure it will fall over tonight now that I've thought about it.


I was considering a 26.1 upgrade to help with these problems...

[felipe0123]

I've been struggling with intermittent instability on my VP2440 since I got it, but I'm still on 25.7. I currently have a 25 day uptime, which is the longest uptime I've had since I got it - I'm sure it will fall over tonight now that I've thought about it.


I was considering a 26.1 upgrade to help with these problems...

Out of curiosity, what kind of issues do you see? VP2440 was never really stable for me until the last 25 release + all NIC firmware updates. 26 made it unstable again.

[lechterpolntrien]

I've been struggling with intermittent instability on my VP2440 since I got it, but I'm still on 25.7. I currently have a 25 day uptime, which is the longest uptime I've had since I got it - I'm sure it will fall over tonight now that I've thought about it.


I was considering a 26.1 upgrade to help with these problems...

Out of curiosity, what kind of issues do you see? VP2440 was never really stable for me until the last 25 release + all NIC firmware updates. 26 made it unstable again.

It's as you described: all interfaces suddenly lose connectivity. Im only using the x710 interfaces for wan & lan, but Ive tried connecting to the ethernet interfaces when this happens and they are affected too.

I have not touched the firmware on thes interfaces or coreboot since I got it. This latest 25.7 release is also the first time its been up for more than a week.

[OPNenthu]

Not to overstate the obvious, but have either of you contacted the vendor?  What was the response?

[felipe0123]

@OPNenthu

This must be the VP2440. Did you recently install coreboot v0.9.1-rc3 (the one that fixes the i226-v ASPM issue)?  I wonder if that firmware maybe introduced a new issue.

FYI, since I started experiencing the issue with x710, I moved my LAN connection to i226. Once I did that I started to experience a different issue, from time to time one of the rx-queues would just stop processing traffic. I tried forcing a single queue and the issue happened with the single queue as well. So I decided to apply the coreboot v0.9.1-rc3.

It's too early to be completely sure, but once I applied the update Ierrs on both igc0 and igc1 dropped to zero (igc0, WAN) had IErrs > 0 almost immediately after boot. As a test I had two internal hosts concurrently running speediest-cli in loop and Errs on WAN is still zero, LAN is 103 but on a deeper inspection it seems to be due to queue saturation probably because I'm still running with a single queue, I will revert that once I get more confident the coreboot update helped.

Although this is not related to the original issue I posted about for x710, I'm posting this here so people facing issues with Protectli VP2440 and v226 interfaces are aware the update helps. Before the update I had hw.pci.enable_aspm set to zero, but that workaround clearly wasn't enough.

[OPNenthu]

Glad that seems to have helped, at least for the I226-V interfaces.

I'm still curious what's going on with the X710 ports?  I don't currently own one but I'm interested in this model and from what I read those are supposed to have a solid driver under FreeBSD (firmware depending).  The only downside of the -BM2 variant on the VP2440 is that it only negotiates at 1/10GbE.

The other oddity is that Protectli runs it at x4 rather than x8, but on their site they claim this is still enough and they have tested it at full line rate in both directions, port-to-port.  https://kb.protectli.com/kb/vp2440-hardware-overview/  (see section on SFP+ Ports)

[felipe0123]

I will try to flip traffic to x710 again to make some tests and report back here.

UPDATE:
1. flipped traffic to X710, pushed 1.5Gb/s traffic with iperf, crashed pretty quickly. Under 3 minutes, likely under 2.
2. I noticed the pciconf -lce pci0:1:0:0 would report ASPM enabled, despite aspm=0 with sysctl
3. forced ASPM off using pciconf
4. pushed 3.5Gb/s of sustained traffic for around 10 minutes, no issue so far.

I'm optimist that may be the issue, I'm running coreboot and it doesn't have an option do disable ASPM at that level, maybe that can be accomplished if running AMI BIOS and no pciconf workaround would be needed. I will contact Protectli support

[OPNenthu]

, I'm running coreboot and it doesn't have an option do disable ASPM at that level, maybe that can be accomplished if running AMI BIOS

At least on the V1410 model, this is correct.  The coreboot image forces ASPM enabled and that's why my attempts to disable it from the OS in the past have failed.  Fortunately this model isn't affected by it (all NICs report 0 errors and I don't experience dropouts).

I will contact Protectli support

Thank you for driving this and keeping us informed.