Intel E810 fails to link on WAN after 26.1 → 26.7 upgrade due to DDP

[homelabber]

Long time lurker, first time caller.

After upgrading from 26.1.x to 26.7, my WAN interface (ice0, Intel E810-XXV for SFP, 25GBase-LR to my ISP) stopped establishing a link entirely. It appears that link-negotiation fails when the ice driver loads its DDP package. On 26.7 there is no fallback when that negotiation fails, unlike on 26.1. Forcing "Safe Mode" (DDP package not loaded, 1 queue; ice_ddp_load="YES" forced via /boot/loader.conf.local) gives a reliable link on both branches. Updating to 26.7.3_8 gave me multiple retry/re-attach cycles instead of giving up after one but did not fully fix it. Without DDP, no 25Gbit fun :(

Components involved:

• NIC: Intel Ethernet Controller E810-XXV for SFP, genuine Intel reference card (not an OEM-customized part)
• Driver: iflib ice, driver_version 1.43.3-k (identical across all OPNsense versions tested)
• Firmware: fw 7.8.2, api 1.7, nvm 4.80 (identical across all tests)
• DDP: ICE OS Default Package version 1.3.41.0, track id 0xc0000001

I am wondering if...

• anyone else has seen E810 fail to negotiate link in full-DDP mode specifically after upgrading to 26.7, where Safe Mode links fine?
• the queue-reconfiguration-triggered link flap on the DDP-loaded attach is a known iflib/ice interaction, or worth its own upstream report?
• anyone has any recommendation on forcing requested_fec at boot to test the theory above?

Happy to pull more diagnostics.

[pfry]

No link with Intel E810 on Opnsense 26.7 / FreeBSD 15 when DDP loaded

Bug is not assigned... The issue is not completely characterized (it's a bit of a hassle to test multiple optics and DAC cables). I wonder if a plugin driver (a la old Realtek) would be worthwhile.

[HenrikS]

Same problem with no link on 25GbE, running both E810 and E823 and DDP after upgrading to 26.7
Without loading DDP we get link and about 1Gbps throughput on each NIC :/

E810-XXV fw 7.6.2 api 1.7 nvm 4.60
E823-C fw 7.1.6 api 1.7 nvm 3.26
ICE OS Default Package version 1.3.41.0

[tuto2]

I tested this on all the Deciso appliance variants running the E810 with 10G-SR and 25G-SR fiber optics. All link and communicate just fine with the DDP package loaded. The link may take a few cycles to come up, but this shouldn't be a problem.

Reading all of the reports in here, as well as https://forum.opnsense.org/index.php?topic=52405 I'm seeing quite some variablity in reported firmware version and indeed https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296989 also reports versions >= 4.60. The only report in the forum post concerning a DEC4280 was on a non-standard firmware version (DEC devices run NVM 4.30), which is not something that's supported.

[pfry]

Does "ifconfig -v" show anything different for the SFP when the DDP is loaded vs. not? It's a reach: looking for SFP vendor lock.

Edit: My e810 doesn't appear to have a vendor lock (I don't have anything within reach to plug it into); the ice driver seems to complain more than ixl (so a silent issue seems unlikely... but I could be misremembering the vendor lock behavior).

[MajorP93]

I tested this on all the Deciso appliance variants running the E810 with 10G-SR and 25G-SR fiber optics. All link and communicate just fine with the DDP package loaded. The link may take a few cycles to come up, but this shouldn't be a problem.

Reading all of the reports in here, as well as https://forum.opnsense.org/index.php?topic=52405 I'm seeing quite some variablity in reported firmware version and indeed https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296989 also reports versions >= 4.60. The only report in the forum post concerning a DEC4280 was on a non-standard firmware version (DEC devices run NVM 4.30), which is not something that's supported.

Maybe this is something worth mentioning in documentation (E810 NVM firmware versions that are officially supported / verified).

[bugacha]

Long time lurker, first time caller.

After upgrading from 26.1.x to 26.7, my WAN interface (ice0, Intel E810-XXV for SFP, 25GBase-LR to my ISP) stopped establishing a link entirely. It appears that link-negotiation fails when the ice driver loads its DDP package. On 26.7 there is no fallback when that negotiation fails, unlike on 26.1. Forcing "Safe Mode" (DDP package not loaded, 1 queue; ice_ddp_load="YES" forced via /boot/loader.conf.local) gives a reliable link on both branches. Updating to 26.7.3_8 gave me multiple retry/re-attach cycles instead of giving up after one but did not fully fix it. Without DDP, no 25Gbit fun :(

Components involved:

• NIC: Intel Ethernet Controller E810-XXV for SFP, genuine Intel reference card (not an OEM-customized part)
• Driver: iflib ice, driver_version 1.43.3-k (identical across all OPNsense versions tested)
• Firmware: fw 7.8.2, api 1.7, nvm 4.80 (identical across all tests)
• DDP: ICE OS Default Package version 1.3.41.0, track id 0xc0000001

I am wondering if...

• anyone else has seen E810 fail to negotiate link in full-DDP mode specifically after upgrading to 26.7, where Safe Mode links fine?
• the queue-reconfiguration-triggered link flap on the DDP-loaded attach is a known iflib/ice interaction, or worth its own upstream report?
• anyone has any recommendation on forcing requested_fec at boot to test the theory above?

Happy to pull more diagnostics.

read up here

https://forum.opnsense.org/index.php?topic=52405.0


FreeBSD driver's bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296989 is being fixed and hopefully can be mainlined into 15

[homelabber]

Update to 26.7.6 fixed it for me, DDP is working fine. No more safe mode :)
I did have to add a tunable for it to work properly with init7:

• dev.ice.0.requested_fec
• Value: FC-FEC/BASE-R

[franco]

Ok great. As mention in the other thread we're considering changing the upgrade path to fix this for people coming directly from 26.1.11 but there's a Hyper-V issue in 26.7.6 that's an odd tradeoff at the moment so we're trying to get that resolved too.


Cheers,
Franco

[franco]

PS: confirmation here https://forum.opnsense.org/index.php?topic=52881.msg275444#msg275444

[HenrikS]

26.7.6 also seems to have fixed it for me.
I also had to specify dev.ice.x.requested_fec explicit as RS-FEC, as it did not auto-negotiate successfull after DDP load without it set.

[1] ice0: <Intel(R) Ethernet Controller E810-XXV for SFP - 1.43.3-k> mem 0xc2000000-0xc3ffffff,0xc4010000-0xc401ffff at device 0.0 numa-domain 0 on pci6
[1] ice0: Loading the iflib ice driver
[1] ice0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice0: fw 7.6.2 api 1.7 nvm 4.60 etid 8001e8d4 netlist 4.3.4000-4.f.0.493a8aab oem 1.3682.0
[1] ice0: Using 4 Tx and Rx queues
[1] ice0: Reserving 4 MSI-X interrupts for iRDMA
[1] ice0: Using MSI-X interrupts with 9 vectors
[1] ice0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
[1] ice0: PCI Express Bus: Speed 16.0GT/s Width x8
[1] ice0: pci_iov_attach failed (error=ENOENT)
[1] ice0: link state changed to UP
[1] ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
[1] ice0: netmap queues/slots: TX 4/1024, RX 4/1024
[1] ice0: link state changed to DOWN
[4] ice0: All configured link modes were attempted but failed to establish link.
[4] ice0: The device will restart the process to establish link.
[4] ice0: Possible Solution: Check link partner connection and configuration.
[25] ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
[25] ice0: link state changed to UP

[1] ice2: <Intel(R) Ethernet Connection E823-C for SFP - 1.43.3-k> mem 0xd0000000-0xd7ffffff,0xda010000-0xda01ffff at device 0.0 numa-domain 0 on pci8
[1] ice2: Loading the iflib ice driver
[1] ice2: The DDP package was successfully loaded: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice2: fw 7.1.6 api 1.7 nvm 3.26 etid 8001bc8c netlist 0.0.2768-1.14.1.59461500 oem 1.3429.0
[1] ice2: Using 4 Tx and Rx queues
[1] ice2: Reserving 4 MSI-X interrupts for iRDMA
[1] ice2: Using MSI-X interrupts with 9 vectors
[1] ice2: Using 1024 TX descriptors and 1024 RX descriptors
[1] ice2: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
[1] ice2: link state changed to UP
[1] ice2: Link is up, 25 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
[1] ice2: netmap queues/slots: TX 4/1024, RX 4/1024
[5] ice2: All configured link modes were attempted but failed to establish link.
[5] ice2: The device will restart the process to establish link.
[5] ice2: Possible Solution: Check link partner connection and configuration.
[5] ice2: link state changed to DOWN
[25] ice2: Link is up, 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
[25] ice2: link state changed to UP