Messages

https://reviews.freebsd.org/D60168

Fix was committed to master and MFCed to stable/15.

Fix is in 26.10, separate issue.

hey mate, your FEC thing is not related to what this thread is about. You should really create a new thread about it.

ice driver fix for the original problem is soon going to be in stable/15.

Added https://github.com/opnsense/src/commit/a5f857190 to stable/26.7 and built a test kernel:

# opnsense-update -zkr 26.7.4-next2

A confirmation would be good because then we can ship it in 26.7.6 next week.


Cheers,
Franco

https://reviews.freebsd.org/D60168

Fix was committed to master and MFCed to stable/15.

26.7.8_2 works fine here

well that worked better than expected... after I set both tunables, I could upgrade without issues!

thanks alot for the help! love this community!

So I guess it's better to keep the tunables in the future? is there anything you need from my side?

Yes just keep them.

Something in BSD ICE driver doesn't properly negotiate FEC when network switch is in Auto mode.
I've always had this problem on E810 card.

Linux ICE driver doesn't have this issue.

Hey Franco

sorry I couldn't find time earlier for testing...
I just tried the update command and still have no link with the new kernel after reboot.

BUT! I was able to udpate anyway by disabling the DDP Package before the update...
I'm really sorry for the confusion and the false statement from my side! I guess I shouldn't do updates too late in the night...

Bad news... After I was on 26.7.7 I enabled the DDP Package again and had no link after the reboot.

Here are some logs. let me know if you need anything else.

On Version 26.7.7 with DDP enabled:

Code Select Expand

[1] ice0: <Intel(R) Ethernet Connection E823-L for SFP - 1.43.3-k> mem 0xf0000000-0xf7ffffff,0xfa010000-0xfa01ffff at device 0.0 numa-domain 0 on pci10
[1] ice0: Loading the iflib ice driver
[1] ice0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice0: fw 7.2.14 api 1.7 nvm 3.42 etid 8001ff0f netlist 0.1.7000-1.25.0.f083a9d5 oem 1.3755.0
[1] ice0: Using 10 Tx and Rx queues
[1] ice0: Reserving 10 MSI-X interrupts for iRDMA
[1] ice0: Using MSI-X interrupts with 21 vectors
[1] ice0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ice0: Ethernet address: ██:██:██:██:██:██
[1] ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
[1] ice0: Firmware LLDP agent disabled
[1] ice0: link state changed to DOWN
[1] ice0: netmap queues/slots: TX 10/1024, RX 10/1024
[9] ice0: All configured link modes were attempted but failed to establish link.
[9] ice0: The device will restart the process to establish link.
[9] ice0: Possible Solution: Check link partner connection and configuration.
[9] ice0: All configured link modes were attempted but failed to establish link.
[9] ice0: The device will restart the process to establish link.
[9] ice0: Possible Solution: Check link partner connection and configuration.
[47] ice0: Failed to add VLAN filters:
[47] ice0: - vlan 100, status -14
[47] ice0: Failure adding VLAN 100 to main VSI, err ICE_ERR_ALREADY_EXISTS aq_err OK
[56] ice0: All configured link modes were attempted but failed to establish link.
[56] ice0: The device will restart the process to establish link.
[56] ice0: Possible Solution: Check link partner connection and configuration.


What FEC settings do you have on the switch for this port ?

Try to play with RS or FC modes to see if link comes up.

From your log, driver and DDP are loaded fine, the issue is physical link which on this card is usually FEC related.


You can also try explicit tunable in Opnsense

Code Select Expand

dev.ice.0.requested_fec="RS-FEC"


Added https://github.com/opnsense/src/commit/a5f857190 to stable/26.7 and built a test kernel:

# opnsense-update -zkr 26.7.4-next2

A confirmation would be good because then we can ship it in 26.7.6 next week.


Cheers,
Franco

Thanks a lot Franco, just upgraded to 26.7.6 and everything works fine now out of Opnsense without the need for SRV-IO stuff in Proxmox.

I filed bug with FreeBSD 15.3

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296989

It will take a while until this is fixed tho

Looks like there is some movement on this bug and this can be mainlined into 15 version

[Components involved:]

Long time lurker, first time caller.

After upgrading from 26.1.x to 26.7, my WAN interface (ice0, Intel E810-XXV for SFP, 25GBase-LR to my ISP) stopped establishing a link entirely. It appears that link-negotiation fails when the ice driver loads its DDP package. On 26.7 there is no fallback when that negotiation fails, unlike on 26.1. Forcing "Safe Mode" (DDP package not loaded, 1 queue; ice_ddp_load="YES" forced via /boot/loader.conf.local) gives a reliable link on both branches. Updating to 26.7.3_8 gave me multiple retry/re-attach cycles instead of giving up after one but did not fully fix it. Without DDP, no 25Gbit fun :(

Components involved:

• NIC: Intel Ethernet Controller E810-XXV for SFP, genuine Intel reference card (not an OEM-customized part)
• Driver: iflib ice, driver_version 1.43.3-k (identical across all OPNsense versions tested)
• Firmware: fw 7.8.2, api 1.7, nvm 4.80 (identical across all tests)
• DDP: ICE OS Default Package version 1.3.41.0, track id 0xc0000001

I am wondering if...

• anyone else has seen E810 fail to negotiate link in full-DDP mode specifically after upgrading to 26.7, where Safe Mode links fine?
• the queue-reconfiguration-triggered link flap on the DDP-loaded attach is a known iflib/ice interaction, or worth its own upstream report?
• anyone has any recommendation on forcing requested_fec at boot to test the theory above?

Happy to pull more diagnostics.

read up here

https://forum.opnsense.org/index.php?topic=52405.0


FreeBSD driver's bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296989 is being fixed and hopefully can be mainlined into 15

I filed bug with FreeBSD 15.3

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296989

It will take a while until this is fixed tho

Long story short :

my card E810-XXV-2 - only works on 4.50 NVM in FreeBSD 14 (opnsense 26.1) and 15 (opnsense 26.7)
Any higher NVM fails in initializing TX queues balancing.


But

Higher NVMs work fine in Linux. Driver initializes DDP / TX balancing fine, no issues at all.

ok after playing with FEC settings via sysctl I managed to get it working

had to use requested_fec=rs on WAN and requested_fec=auto on LAN

Nothing works

Tried NVM 4.80 and 4.60 both result in this :

link goes up briefly :

Code Select Expand

# egrep -i ice0 4_60.log
[1] ice0: <Intel(R) Ethernet Network Adapter E810-XXV-2 - 1.43.3-k> mem 0x380000000000-0x380001ffffff,0x380002000000-0x38000200ffff irq 16 at device 0.0 on pci1
[1] ice0: Loading the iflib ice driver
[1] ice0: Error configuring transmit balancing: ICE_ERR_AQ_ERROR
[1] ice0: An unknown error occurred when loading the DDP package.  Entering Safe Mode.
[1] ice0: fw 7.6.2 api 1.7 nvm 4.60 etid 8001e8b3 netlist 4.3.5000-1.14.0.99840ef4 oem 1.3682.0
[1] ice0: Using 1 Tx and Rx queues
[1] ice0: Using MSI-X interrupts with 2 vectors
[1] ice0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ice0: Ethernet address: 50:7c:6f:79:ca:e8
[1] ice0: PCI Express Bus: Speed 16.0GT/s Width x8
[1] ice0: ice_init_dcb_setup: No DCB support
[1] ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
[1] ice0: link state changed to UP
[1] ice0: netmap queues/slots: TX 1/1024, RX 1/1024

But later something happens , I guess driver trying to read something from FW that not supporting it and it goes into Safe mode

Code Select Expand

[1] ice0: link state changed to DOWN
[9] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[12] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[14] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[16] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[29] ice0: Could not acquire current LLDP persistence status, err ICE_ERR_AQ_TIMEOUT aq_err OK
[30] ice0: Could not acquire current LLDP persistence status, err ICE_ERR_AQ_TIMEOUT aq_err OK
[31] ice0: ice_intersect_phy_types_and_speeds: ice_aq_get_phy_caps (ACTIVE) failed; status ICE_ERR_AQ_TIMEOUT, aq_err OK
[32] ice0: ice_intersect_phy_types_and_speeds: ice_aq_get_phy_caps (ACTIVE) failed; status ICE_ERR_AQ_TIMEOUT, aq_err OK
[33] ice0: ice_sysctl_fc_config: ice_aq_get_phy_caps failed; status ICE_ERR_AQ_TIMEOUT, aq_err OK
[34] ice0: ice_sysctl_fc_config: ice_aq_get_phy_caps failed; status ICE_ERR_AQ_TIMEOUT, aq_err OK
[38] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[46] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[48] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[51] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[53] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[65] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[76] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[78] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[80] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[82] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[85] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[87] ice0: Failed to set LAN Tx queue 0 (TC 0, handle 0) context, err ICE_ERR_AQ_TIMEOUT aq_err OK
[87] ice0: Unable to configure the main VSI for Tx: ENODEV
[88] ice0: Could not add new MAC filters, err ICE_ERR_AQ_TIMEOUT aq_err OK
[88] ice0: Failed to synchronize multicast filter list: EIO
[89] ice0: Could not add new MAC filters, err ICE_ERR_AQ_TIMEOUT aq_err OK
[89] ice0: Failed to synchronize multicast filter list: EIO
[90] ice0: Could not add new MAC filters, err ICE_ERR_AQ_TIMEOUT aq_err OK
[90] ice0: Failed to synchronize multicast filter list: EIO
[91] ice0: Could not add new MAC filters, err ICE_ERR_AQ_TIMEOUT aq_err OK
[91] ice0: Failed to synchronize multicast filter list: EIO
[92] ice0: Could not add new MAC filters, err ICE_ERR_AQ_TIMEOUT aq_err OK
[92] ice0: Failed to synchronize multicast filter list: EIO
[96] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[98] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[100] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK
[102] ice0: ice_read_sff_eeprom: Error reading I2C data: err ICE_ERR_AQ_TIMEOUT aq_err OK



Hi, I upgraded my big firewall test machine today to 26.7 with DDP enabled on the E810 and it seems fine.

Module is loaded:

Code Select Expand

root@Firewall-1:~ # dmesg | grep -i ddp
[1] ice0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice1: DDP package already present on device: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.


Code Select Expand

root@Firewall-1:~ # kldstat | grep -i ddp
 2    1 0xffffffff8277f000   14ce38 ice_ddp.ko


Code Select Expand

root@Firewall-1:~ # dmesg | grep -i ice0
[1] ice0: <Intel(R) Ethernet Network Adapter E810-C-Q2 - 1.43.3-k> mem 0x380ae000000-0x380afffffff,0x380b0010000-0x380b001ffff irq 120 at device 0.0 on pci10
[1] ice0: Loading the iflib ice driver
[1] ice0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice0: fw 7.8.2 api 1.7 nvm 4.80 etid 80020543 netlist 4.4.2000-3.27.0.597b7167 oem 1.3805.0
[1] ice0: Using 8 Tx and Rx queues
[1] ice0: Reserving 8 MSI-X interrupts for iRDMA
[1] ice0: Using MSI-X interrupts with 17 vectors
[1] ice0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ice0: Ethernet address: 6c:fe:54:c3:3a:80
[1] ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
[1] ice0: PCI Express Bus: Speed 16.0GT/s Width x16
[1] ice0: Firmware LLDP agent disabled
[1] ice0: link state changed to DOWN
[1] ice0: netmap queues/slots: TX 8/1024, RX 8/1024
[8] ice0: All configured link modes were attempted but failed to establish link.
[8] ice0: The device will restart the process to establish link.
[8] ice0: Possible Solution: Check link partner connection and configuration.
[20] ice0: All configured link modes were attempted but failed to establish link.
[20] ice0: The device will restart the process to establish link.
[20] ice0: Possible Solution: Check link partner connection and configuration.
[5789] ice0: Link is up, 100 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: RS-FEC, Autoneg: True, Flow Control: None
[5789] ice0: link state changed to UP

I booted a machine connected to this firewall just recently and it connected, at least a ping to this machine succeeded.

Best regards,
Thomas

you're on nvm 4.80 I'm on 4.50

let me try to flash it

thanks!

Upgraded to 26.7 today and lost links on E810-XXV-2 , failed to establish any carrier.

I had this in loader.conf.local on 26.1

Code Select Expand

# cat /boot/loader.conf.local
if_ice_load="YES"
ice_ddp_load="YES"


Here what I see in dmesg :

Code Select Expand

[1] ice0: <Intel(R) Ethernet Network Adapter E810-XXV-2 - 1.43.3-k> mem 0x380000000000-0x380001ffffff,0x380002000000-0x38000200ffff irq 16 at device 0.0 on pci1
[1] ice0: Loading the iflib ice driver
[1] ice0: DDP package already present on device: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice0: fw 7.5.4 api 1.7 nvm 4.50 etid 8001d8ba netlist 4.3.5000-1.14.0.99840ef4 oem 1.3597.0
[1] ice0: Using 8 Tx and Rx queues
[1] ice0: Reserving 8 MSI-X interrupts for iRDMA
[1] ice0: Using MSI-X interrupts with 17 vectors
[1] ice0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ice0: Ethernet address: 50:7c:6f:79:ca:e8
[1] ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
[1] ice0: PCI Express Bus: Speed 16.0GT/s Width x8
[1] ice0: pci_iov_attach failed (error=ENOENT)
[1] ice0: Firmware LLDP agent disabled
[1] ice0: netmap queues/slots: TX 8/1024, RX 8/1024
[1] ice0: link state changed to DOWN



[1] ice0: All configured link modes were attempted but failed to establish link.
[1] ice0: The device will restart the process to establish link.
[1] ice0: Possible Solution: Check link partner connection and configuration.


Mirroring Reddit thread here - https://www.reddit.com/r/opnsense/comments/1rmqih8/opnsense_setting_wrong_address_with_native/


Since maybe 1-2 weeks ago (upgraded to 26.1.3) my DDNS client suddenly changed external IP to 104.18.0.0

DDNS was set to use cloudflare-ipv4 "Check ip method"

104.18.0.0 - belongs to Cloudflare

Switching to Akamai fixes the problem