No link with Intel E810 on Opnsense 26.7 / FreeBSD 15 when DDP loaded

[bugacha]

Upgraded to 26.7 today and lost links on E810-XXV-2 , failed to establish any carrier.

I had this in loader.conf.local on 26.1

Code Select Expand

# cat /boot/loader.conf.local
if_ice_load="YES"
ice_ddp_load="YES"


Here what I see in dmesg :

Code Select Expand

[1] ice0: <Intel(R) Ethernet Network Adapter E810-XXV-2 - 1.43.3-k> mem 0x380000000000-0x380001ffffff,0x380002000000-0x38000200ffff irq 16 at device 0.0 on pci1
[1] ice0: Loading the iflib ice driver
[1] ice0: DDP package already present on device: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice0: fw 7.5.4 api 1.7 nvm 4.50 etid 8001d8ba netlist 4.3.5000-1.14.0.99840ef4 oem 1.3597.0
[1] ice0: Using 8 Tx and Rx queues
[1] ice0: Reserving 8 MSI-X interrupts for iRDMA
[1] ice0: Using MSI-X interrupts with 17 vectors
[1] ice0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ice0: Ethernet address: 50:7c:6f:79:ca:e8
[1] ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
[1] ice0: PCI Express Bus: Speed 16.0GT/s Width x8
[1] ice0: pci_iov_attach failed (error=ENOENT)
[1] ice0: Firmware LLDP agent disabled
[1] ice0: netmap queues/slots: TX 8/1024, RX 8/1024
[1] ice0: link state changed to DOWN



[1] ice0: All configured link modes were attempted but failed to establish link.
[1] ice0: The device will restart the process to establish link.
[1] ice0: Possible Solution: Check link partner connection and configuration.


[pfry]

I don't have 15.1 on a machine with an e810 yet, so I can't compare directly (version, at least). I'd update the NVM and see if that helps. Do you have SR-IOV enabled in BIOS?

Just booting the 15.1 installer, my e810 links fine... but doesn't load the DDP. I can mess with the installer, but it'll take some time.

[Thomas Niedermeier]

Hi, I upgraded my big firewall test machine today to 26.7 with DDP enabled on the E810 and it seems fine.

Module is loaded:

Code Select Expand

root@Firewall-1:~ # dmesg | grep -i ddp
[1] ice0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice1: DDP package already present on device: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.


Code Select Expand

root@Firewall-1:~ # kldstat | grep -i ddp
 2    1 0xffffffff8277f000   14ce38 ice_ddp.ko


Code Select Expand

root@Firewall-1:~ # dmesg | grep -i ice0
[1] ice0: <Intel(R) Ethernet Network Adapter E810-C-Q2 - 1.43.3-k> mem 0x380ae000000-0x380afffffff,0x380b0010000-0x380b001ffff irq 120 at device 0.0 on pci10
[1] ice0: Loading the iflib ice driver
[1] ice0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice0: fw 7.8.2 api 1.7 nvm 4.80 etid 80020543 netlist 4.4.2000-3.27.0.597b7167 oem 1.3805.0
[1] ice0: Using 8 Tx and Rx queues
[1] ice0: Reserving 8 MSI-X interrupts for iRDMA
[1] ice0: Using MSI-X interrupts with 17 vectors
[1] ice0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ice0: Ethernet address: 6c:fe:54:c3:3a:80
[1] ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
[1] ice0: PCI Express Bus: Speed 16.0GT/s Width x16
[1] ice0: Firmware LLDP agent disabled
[1] ice0: link state changed to DOWN
[1] ice0: netmap queues/slots: TX 8/1024, RX 8/1024
[8] ice0: All configured link modes were attempted but failed to establish link.
[8] ice0: The device will restart the process to establish link.
[8] ice0: Possible Solution: Check link partner connection and configuration.
[20] ice0: All configured link modes were attempted but failed to establish link.
[20] ice0: The device will restart the process to establish link.
[20] ice0: Possible Solution: Check link partner connection and configuration.
[5789] ice0: Link is up, 100 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: RS-FEC, Autoneg: True, Flow Control: None
[5789] ice0: link state changed to UP

I booted a machine connected to this firewall just recently and it connected, at least a ping to this machine succeeded.

Best regards,
Thomas

[bugacha]

Hi, I upgraded my big firewall test machine today to 26.7 with DDP enabled on the E810 and it seems fine.

Module is loaded:

Code Select Expand

root@Firewall-1:~ # dmesg | grep -i ddp
[1] ice0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice1: DDP package already present on device: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.


Code Select Expand

root@Firewall-1:~ # kldstat | grep -i ddp
 2    1 0xffffffff8277f000   14ce38 ice_ddp.ko


Code Select Expand

root@Firewall-1:~ # dmesg | grep -i ice0
[1] ice0: <Intel(R) Ethernet Network Adapter E810-C-Q2 - 1.43.3-k> mem 0x380ae000000-0x380afffffff,0x380b0010000-0x380b001ffff irq 120 at device 0.0 on pci10
[1] ice0: Loading the iflib ice driver
[1] ice0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice0: fw 7.8.2 api 1.7 nvm 4.80 etid 80020543 netlist 4.4.2000-3.27.0.597b7167 oem 1.3805.0
[1] ice0: Using 8 Tx and Rx queues
[1] ice0: Reserving 8 MSI-X interrupts for iRDMA
[1] ice0: Using MSI-X interrupts with 17 vectors
[1] ice0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ice0: Ethernet address: 6c:fe:54:c3:3a:80
[1] ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
[1] ice0: PCI Express Bus: Speed 16.0GT/s Width x16
[1] ice0: Firmware LLDP agent disabled
[1] ice0: link state changed to DOWN
[1] ice0: netmap queues/slots: TX 8/1024, RX 8/1024
[8] ice0: All configured link modes were attempted but failed to establish link.
[8] ice0: The device will restart the process to establish link.
[8] ice0: Possible Solution: Check link partner connection and configuration.
[20] ice0: All configured link modes were attempted but failed to establish link.
[20] ice0: The device will restart the process to establish link.
[20] ice0: Possible Solution: Check link partner connection and configuration.
[5789] ice0: Link is up, 100 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: RS-FEC, Autoneg: True, Flow Control: None
[5789] ice0: link state changed to UP

I booted a machine connected to this firewall just recently and it connected, at least a ping to this machine succeeded.

Best regards,
Thomas

you're on nvm 4.80 I'm on 4.50

let me try to flash it

thanks!

[PoMpIs]

I also have a 4-port e810.



Mine works fine, although I just checked and there's an update. Is it a good idea to update it?

Code Select Expand

grep -i ddp
[1] ice0: The driver could not load the DDP package file because a compatible DDP package is already present on the device.  The device has package ICE OS Default Package version 1.3.43.0, track id 0xc0000001.  The ice_ddp module has package: ICE OS Default Package version 1.3.41.0.
[1] ice1: The driver could not load the DDP package file because a compatible DDP package is already present on the device.  The device has package ICE OS Default Package version 1.3.43.0, track id 0xc0000001.  The ice_ddp module has package: ICE OS Default Package version 1.3.41.0.
[1] ice2: The driver could not load the DDP package file because a compatible DDP package is already present on the device.  The device has package ICE OS Default Package version 1.3.43.0, track id 0xc0000001.  The ice_ddp module has package: ICE OS Default Package version 1.3.41.0.
root@opn:~ # kldstat | grep -i ddp
12    1 0xffffffff82f71000   14ce38 ice_ddp.ko
root@opn:~ # dmesg | grep -i ice0
[1] ice0: <Intel(R) Ethernet Network Adapter E810-XXV-4 - 1.43.3-k> mem 0x382000000000-0x382001ffffff,0x382002000000-0x38200200ffff irq 21 at device 0.0 on pci5
[1] ice0: Loading the iflib ice driver
[1] ice0: The driver could not load the DDP package file because a compatible DDP package is already present on the device.  The device has package ICE OS Default Package version 1.3.43.0, track id 0xc0000001.  The ice_ddp module has package: ICE OS Default Package version 1.3.41.0.
[1] ice0: fw 7.6.2 api 1.7 nvm 4.60 etid 8001e8b2 netlist 4.3.2000-1.17.0.2c46f598 oem 1.3682.0
[1] ice0: Using 4 Tx and Rx queues
[1] ice0: Reserving 2 MSI-X interrupts for iRDMA
[1] ice0: Using MSI-X interrupts with 7 vectors
[1] ice0: Using 4096 TX descriptors and 4096 RX descriptors
[1] ice0: Ethernet address: 9c:69:b4:68:65:99
[1] ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
[1] ice0: PCI Express Bus: Speed 8.0GT/s Width x16
[1] ice0: pci_iov_attach failed (error=ENOENT)
[1] ice0: Firmware LLDP agent disabled
[1] ice0: link state changed to UP
[1] ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: RS-FEC, Autoneg: True, Flow Control: None
[1] ice0: netmap queues/slots: TX 4/4096, RX 4/4096
[1] ice0: link state changed to DOWN
[1] ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None
[1] ice0: link state changed to UP
[11768] ice0: link state changed to DOWN
[11787] ice0: All configured link modes were attempted but failed to establish link.
[11787] ice0: The device will restart the process to establish link.
[11787] ice0: Possible Solution: Check link partner connection and configuration.
[60739] ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
[60739] ice0: link state changed to UP

Code Select Expand

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.43.37.1a
Copyright(C) 2013 - 2026 Intel Corporation.


WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.
Inventory in progress. Please wait [***-......]


Num Description                          Ver.(hex)  DevId S:B    Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Converged           N/A(N/A)   1563 00:037 Update not   
    Network Adapter X550-T2                                      available     
02) Intel(R) Ethernet Network Adapter   4.96(4.60)   1593 00:042 Update       
    E810-XXV-4                                                   available     

Options: Adapter Index List (comma-separated), [A]ll, e[X]it
Enter selection:

[bugacha]

Nothing works

Tried NVM 4.80 and 4.60 both result in this :

link goes up briefly :

Code Select Expand

# egrep -i ice0 4_60.log
[1] ice0: <Intel(R) Ethernet Network Adapter E810-XXV-2 - 1.43.3-k> mem 0x380000000000-0x380001ffffff,0x380002000000-0x38000200ffff irq 16 at device 0.0 on pci1
[1] ice0: Loading the iflib ice driver
[1] ice0: Error configuring transmit balancing: ICE_ERR_AQ_ERROR
[1] ice0: An unknown error occurred when loading the DDP package.  Entering Safe Mode.
[1] ice0: fw 7.6.2 api 1.7 nvm 4.60 etid 8001e8b3 netlist 4.3.5000-1.14.0.99840ef4 oem 1.3682.0
[1] ice0: Using 1 Tx and Rx queues
[1] ice0: Using MSI-X interrupts with 2 vectors
[1] ice0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ice0: Ethernet address: 50:7c:6f:79:ca:e8
[1] ice0: PCI Express Bus: Speed 16.0GT/s Width x8
[1] ice0: ice_init_dcb_setup: No DCB support
[1] ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
[1] ice0: link state changed to UP
[1] ice0: netmap queues/slots: TX 1/1024, RX 1/1024

But later something happens , I guess driver trying to read something from FW that not supporting it and it goes into Safe mode

Code Select Expand

[1] ice0: link state changed to DOWN
[9] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[12] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[14] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[16] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[29] ice0: Could not acquire current LLDP persistence status, err ICE_ERR_AQ_TIMEOUT aq_err OK
[30] ice0: Could not acquire current LLDP persistence status, err ICE_ERR_AQ_TIMEOUT aq_err OK
[31] ice0: ice_intersect_phy_types_and_speeds: ice_aq_get_phy_caps (ACTIVE) failed; status ICE_ERR_AQ_TIMEOUT, aq_err OK
[32] ice0: ice_intersect_phy_types_and_speeds: ice_aq_get_phy_caps (ACTIVE) failed; status ICE_ERR_AQ_TIMEOUT, aq_err OK
[33] ice0: ice_sysctl_fc_config: ice_aq_get_phy_caps failed; status ICE_ERR_AQ_TIMEOUT, aq_err OK
[34] ice0: ice_sysctl_fc_config: ice_aq_get_phy_caps failed; status ICE_ERR_AQ_TIMEOUT, aq_err OK
[38] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[46] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[48] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[51] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[53] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[65] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[76] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[78] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[80] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[82] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[85] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[87] ice0: Failed to set LAN Tx queue 0 (TC 0, handle 0) context, err ICE_ERR_AQ_TIMEOUT aq_err OK
[87] ice0: Unable to configure the main VSI for Tx: ENODEV
[88] ice0: Could not add new MAC filters, err ICE_ERR_AQ_TIMEOUT aq_err OK
[88] ice0: Failed to synchronize multicast filter list: EIO
[89] ice0: Could not add new MAC filters, err ICE_ERR_AQ_TIMEOUT aq_err OK
[89] ice0: Failed to synchronize multicast filter list: EIO
[90] ice0: Could not add new MAC filters, err ICE_ERR_AQ_TIMEOUT aq_err OK
[90] ice0: Failed to synchronize multicast filter list: EIO
[91] ice0: Could not add new MAC filters, err ICE_ERR_AQ_TIMEOUT aq_err OK
[91] ice0: Failed to synchronize multicast filter list: EIO
[92] ice0: Could not add new MAC filters, err ICE_ERR_AQ_TIMEOUT aq_err OK
[92] ice0: Failed to synchronize multicast filter list: EIO
[96] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[98] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[100] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[102] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK



[bugacha]

ok after playing with FEC settings via sysctl I managed to get it working

had to use requested_fec=rs on WAN and requested_fec=auto on LAN

[PoMpIs]

Well, I've updated mine to the latest version. I've been testing it and it's still working fine, both at 10Gbps and 25Gbps.

Code Select Expand

dmesg | grep -i ice0
[1] ice0: <Intel(R) Ethernet Network Adapter E810-XXV-4 - 1.43.3-k> mem 0x382000000000-0x382001ffffff,0x382002000000-0x38200200ffff irq 21 at device 0.0 on pci5
[1] ice0: Loading the iflib ice driver
[1] ice0: The driver could not load the DDP package file because a compatible DDP package is already present on the device.  The device has package ICE OS Default Package version 1.3.43.0, track id 0xc0000001.  The ice_ddp module has package: ICE OS Default Package version 1.3.41.0.
[1] ice0: fw 8.0.2 api 1.7 nvm 5.01 etid 80021faa netlist 4.4.2000-1.18.0.bc358136 oem 1.4031.0
[1] ice0: Using 4 Tx and Rx queues
[1] ice0: Reserving 2 MSI-X interrupts for iRDMA
[1] ice0: Using MSI-X interrupts with 7 vectors
[1] ice0: Using 4096 TX descriptors and 4096 RX descriptors
[1] ice0: Ethernet address: 9c:69:b4:68:65:99
[1] ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
[1] ice0: PCI Express Bus: Speed 8.0GT/s Width x16
[1] ice0: pci_iov_attach failed (error=ENOENT)
[1] ice0: Firmware LLDP agent disabled
[1] ice0: link state changed to DOWN
[1] ice0: netmap queues/slots: TX 4/4096, RX 4/4096
[11] ice0: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
[11] ice0: link state changed to UP
[1423] ice0: link state changed to DOWN
[1434] ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None
[1434] ice0: link state changed to UP
[1434] ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None

[pfry]

Hm. I updated my main machine (on a physically isolated subnet, so no captures) to NVM 5.01 and booted FreeBSD 14.2 (driver 1.43.2-k) and it seems to work fine. No odd messages. Loads DDP 1.3.41.0. Trying to parse Intel's Feature Support Matrix when using the OS driver is... well, guesswork. I've never had a problem with an up-version NVM, and Intel's FreeBSD driver (in the 31.2.1 package) is apparently version 2.1.8. I need to reinstall 15.1 on the thing, but it'll take some time.

PoMpls: What does "sysctl dev.ice.0.ddp_version" say?

[PoMpIs]

@pfry

Here you go:

Code Select Expand

sysctl dev.ice.0.ddp_version
dev.ice.0.ddp_version: ICE OS Default Package version 1.3.43.0, track id 0xc0000001
This is my card; I saw it a while ago for 157€ and bought it without thinking twice: 😀

https://es.aliexpress.com/item/1005006344562349.html?spm=a2g0o.order_list.order_list_main.15.bb2a194dELSfTX&gatewayAdapt=glo2esp

[bugacha]

Long story short :

my card E810-XXV-2 - only works on 4.50 NVM in FreeBSD 14 (opnsense 26.1) and 15 (opnsense 26.7)
Any higher NVM fails in initializing TX queues balancing.


But

Higher NVMs work fine in Linux. Driver initializes DDP / TX balancing fine, no issues at all.