No link with Intel E810 on Opnsense 26.7 / FreeBSD 15 when DDP loaded

[paoba]

thanks for the heads up! here some more details about my interfaces and my switch. I was supprised my switch thinks the opnsense LAN is connected by Medium-type "Fiber", which is incorrect.

I don't know what FEC settings my internet provider has on their switch, but if required I can open up a ticket to find out.

I will try the tunable for both ports and see what happens after a reboot and also try to upgrade again.

testing tunables:

Code Select Expand

dev.ice.0.requested_fec="RS-FEC"
dev.ice.1.requested_fec="RS-FEC"

Switch opnsense LAN Port details:

Code Select Expand

Medium-type is Fiber
Admin duplex mode is AUTO, oper duplex is Full
Admin speed is 25G, oper speed is 25G
Flow control admin status is OFF, flow control oper status is OFF
Admin negotiation mode is OFF, oper negotiation state is OFF
Admin FEC mode is auto, oper FEC mode is rs

LAN Interface (ice0):

Code Select Expand

dev.ice.0.requested_fec: RS-FEC
dev.ice.0.negotiated_fec: RS-FEC
dev.ice.0.current_speed: 25 Gbps
media: Ethernet autoselect (25G-AUI <full-duplex>)
plugged: SFP/SFP+/SFP28 25GBASE-CR CA-25G-S (Copper pigtail)
WAN Interface (ice1):

Code Select Expand

dev.ice.1.requested_fec: RS-FEC
dev.ice.1.negotiated_fec: RS-FEC
dev.ice.1.current_speed: 25 Gbps
media: Ethernet autoselect (25GBase-LR <full-duplex>)
plugged: SFP/SFP+/SFP28 100GBASE-LR4 or 25GBASE-LR (LC)

[paoba]

well that worked better than expected... after I set both tunables, I could upgrade without issues!

thanks alot for the help! love this community!

So I guess it's better to keep the tunables in the future? is there anything you need from my side?

here is the dmesg log from 27.7.7:

Code Select Expand

[1] ice0: <Intel(R) Ethernet Connection E823-L for SFP - 1.43.3-k> mem 0xf0000000-0xf7ffffff,0xfa010000-0xfa01ffff at device 0.0 numa-domain 0 on pci10
[1] ice0: Loading the iflib ice driver
[1] ice0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice0: fw 7.2.14 api 1.7 nvm 3.42 etid 8001ff0f netlist 0.1.7000-1.25.0.f083a9d5 oem 1.3755.0
[1] ice0: Using 10 Tx and Rx queues
[1] ice0: Reserving 10 MSI-X interrupts for iRDMA
[1] ice0: Using MSI-X interrupts with 21 vectors
[1] ice0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ice0: Ethernet address: ██:██:██:██:██:██
[1] ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
[1] ice0: Firmware LLDP agent disabled
[1] ice0: link state changed to UP
[1] ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
[1] ice0: netmap queues/slots: TX 10/1024, RX 10/1024
[1] ice0: link state changed to DOWN
[9] ice0: All configured link modes were attempted but failed to establish link.
[9] ice0: The device will restart the process to establish link.
[9] ice0: Possible Solution: Check link partner connection and configuration.
[47] ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
[47] ice0: link state changed to UP
[48] ice0: Failed to add VLAN filters:
[48] ice0: - vlan 100, status -14
[48] ice0: Failure adding VLAN 100 to main VSI, err ICE_ERR_ALREADY_EXISTS aq_err OK


Code Select Expand

[1] ice1: <Intel(R) Ethernet Connection E823-L for SFP - 1.43.3-k> mem 0xe8000000-0xefffffff,0xfa000000-0xfa00ffff at device 0.2 numa-domain 0 on pci10
[1] ice1: Loading the iflib ice driver
[1] ice1: DDP package already present on device: ICE OS Default Package version 1.3.41.0, track id 0xc0000001.
[1] ice1: fw 7.2.14 api 1.7 nvm 3.42 etid 8001ff0f netlist 0.1.7000-1.25.0.f083a9d5 oem 1.3755.0
[1] ice1: Using 10 Tx and Rx queues
[1] ice1: Reserving 10 MSI-X interrupts for iRDMA
[1] ice1: Using MSI-X interrupts with 21 vectors
[1] ice1: Using 1024 TX descriptors and 1024 RX descriptors
[1] ice1: Ethernet address: ██:██:██:██:██:██
[1] ice1: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
[1] ice1: Firmware LLDP agent disabled
[1] ice1: link state changed to UP
[1] ice1: Link is up, 25 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
[1] ice1: netmap queues/slots: TX 10/1024, RX 10/1024
[6] ice1: All configured link modes were attempted but failed to establish link.
[6] ice1: The device will restart the process to establish link.
[6] ice1: Possible Solution: Check link partner connection and configuration.
[6] ice1: link state changed to DOWN
[46] ice1: Link is up, 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
[46] ice1: link state changed to UP


[franco]

Nope, that's great news.

I've worked today on enabling the upgrade path to 26.1.11 -> 26.7.7. It looks good from my perspective so expecting this to be live later today.


Cheers,
Franco

[bugacha]

well that worked better than expected... after I set both tunables, I could upgrade without issues!

thanks alot for the help! love this community!

So I guess it's better to keep the tunables in the future? is there anything you need from my side?

Yes just keep them.

Something in BSD ICE driver doesn't properly negotiate FEC when network switch is in Auto mode.
I've always had this problem on E810 card.

Linux ICE driver doesn't have this issue.

[raft]

# Problem Report: OPNsense services fail to start after reboot

**Subject:** Unbound and WebGUI (lighttpd) fail to start automatically after reboot. Reload all services (console option 11) restores functionality.

## 1. Environment

- OPNsense version: 26.7.7
- FreeBSD: 15.1
- Virtualization: QEMU/KVM with libvirt
- Host operating system: Lubuntu
- Network adapters: VirtIO
- WAN: libvirt NAT network
- LAN: isolated libvirt network

## 2. Network configuration

OPNsense WAN:
- IP address: 192.168.122.23
- Gateway: 192.168.122.1

OPNsense LAN:
- IP address: 192.168.100.1

Debian 13 client:
- IP address: 192.168.100.184
- Default gateway: 192.168.100.1

Both OPNsense interfaces receive their expected IP addresses.

## 3. Problem description

After a normal OPNsense reboot:

- The WAN and LAN interfaces appear to be configured.
- The routing table contains the expected routes.
- configd is running.
- Unbound is not running.
- lighttpd is not running.
- The WebGUI is unavailable.
- The Debian client cannot access the Internet.

The OPNsense console boot log displays:

"Starting web GUI......done."

However, lighttpd is subsequently found not running.

## 4. Reproduction and recovery

The problem occurs after rebooting the OPNsense virtual machine.

Using console option 11, "Reload all services", restores functionality.

After this operation:
- The WebGUI becomes accessible.
- Unbound starts working.
- Debian regains Internet connectivity.

This behaviour is reproducible after reboot.

## 5. Additional observations

Attempting to start Unbound manually with:

`service unbound start`

returns a message indicating that unbound_enable is not set to YES in /etc/rc.conf.

Using:

`service unbound onestart`

produces a socket-buffer warning:

`SO_SNDBUF was not granted: No buffer space available`

Attempting to start lighttpd manually with onestart fails.

The exact lighttpd error still needs to be collected.

The Listen Interfaces setting is already set to "All (recommended)".

Available OPNsense updates have been installed, but the problem remains.

No manual changes have been made to /etc/rc.conf or kernel network-buffer settings.

## 6. Questions for the OPNsense developers

1. Is this a known issue with OPNsense 26.7.7 running under QEMU/KVM?
2. Could this be related to the boot sequence or service dependencies?
3. Why do Unbound and lighttpd fail to remain running after boot?
4. Is the SO_SNDBUF warning relevant to the Unbound startup failure?
5. Is there a recommended diagnostic procedure for this situation?
6. Is there a supported way to execute a service reload after the network initialization phase?
7. Could this be a regression in the current release?

## 7. Requested assistance

We would appreciate guidance on identifying the underlying cause.

We would prefer a permanent correction in the normal startup process rather than relying on a manual reload of all services after every reboot.

Thank you for your assistance.

[Patrick M. Hausen]

`service unbound start`

returns a message indicating that unbound_enable is not set to YES in /etc/rc.conf.

OPNsense does not use the FreeBSD rc system. So you should not manually force it by using "onestart", either. The way to start/stop/restart/... a service is e.g.

Code Select Expand

configctl unbound start

HTH,
Patrick

[mehrmann]

This issue appears to persist in OPNsense Business 26.10, which is based on Community 26.7.7.
On a DEC4280 with Intel E810-XXV-4, the 25G LAN link did not establish after upgrading from Business 26.1 to 26.10. 

ice0 remained administratively UP but reported status: no carrier.
ifconfig -m ice0 still showed 25GBase-SR support. 

Driver/firmware details:
ice 1.43.3-k
DDP 1.3.41.0
FW 7.3.4
NVM 4.30 

dmesg repeatedly showed:
All configured link modes were attempted but failed to establish link. 

Setting
sysctl dev.ice.0.requested_fec=RS-FEC
immediately restored the link. Afterwards:
requested_fec: RS-FEC
negotiated_fec: RS-FEC
current_speed: 25 Gbps 

Link partner is an Aruba 8100. The link worked correctly before the Business 26.10 upgrade.

[nero355]

Business 26.1 to 26.10.

Business Editions are always .4 and .10 while Community Editions are .1 and .7 so I am guessing you were using 26.4 ?!

[redge]

Essentially the same experience as @mehrmann in #36 on a DEC3960 after upgrading from BE 26.4 -> 26.10.

Code Select Expand

dev.ice.0.fw_version: fw 7.3.4 api 1.7 nvm 4.30 etid 8001af27 netlist 4.2.5000-1.12.0.87fd3c77 oem 1.3429.0
dev.ice.0.ddp_version: ICE OS Default Package version 1.3.41.0, track id 0xc0000001

Link partner is a Juniper EX4400-24X with the EX4400-EM-4Y expansion module installed, connected via SFP28 DAC. The DACs are coded for Intel on one end, Juniper on the other, and worked until the update.

As with the other cases mentioned here, disabling DDP brought the links back up. Requesting RS-FEC wasn't successful, but FC-FEC/BASE-R works with DDP enabled.

[tuto2]

The DEC devices carrying E810 25G ports were tested extensively with multiple modules going from 26.4 -> 26.10, tests passed just fine. Seeing the reports makes it likely the link partner plays a role in combination with the updated DDP package. We've seen at least one case where setting the mentioned tunable will cause a link down on 26.4, this is why it can't be set generically.

The options are:

dev.ice.X.requested_fec=auto
dev.ice.X.requested_fec=none
dev.ice.X.requested_fec=rs
dev.ice.X.requested_fec=fc

When using short range (SR) fiber, "fc" seems to be the right value, when using long range (LR), it's likely "rs".

[redge]

The DEC devices carrying E810 25G ports were tested extensively with multiple modules going from 26.4 -> 26.10, tests passed just fine. Seeing the reports makes it likely the link partner plays a role in combination with the updated DDP package. We've seen at least one case where setting the mentioned tunable will cause a link down on 26.4, this is why it can't be set generically.

The options are:

dev.ice.X.requested_fec=auto
dev.ice.X.requested_fec=none
dev.ice.X.requested_fec=rs
dev.ice.X.requested_fec=fc

When using short range (SR) fiber, "fc" seems to be the right value, when using long range (LR), it's likely "rs".

I don't have any further insight or experience to contribute at this stage, but what module coding and link partners have been tested? We don't have a non-production DEC appliance, but we do have DACs, AOCs, and SR/LR SFP28 modules on hand that we can recode at will, so if there's a specific combination that hasn't been tested (or needs validation) I might be able to provide another test case.

[tuto2]

I don't have any further insight or experience to contribute at this stage, but what module coding and link partners have been tested? We don't have a non-production DEC appliance, but we do have DACs, AOCs, and SR/LR SFP28 modules on hand that we can recode at will, so if there's a specific combination that hasn't been tested (or needs validation) I might be able to provide another test case.

This was tested with a connection to a separate desktop machine (debian) also running an E810 expansion card. Modules tested were the ones as listed in the documentation: https://docs.opnsense.org/hardware/sfp_compatibility.html#ice.

Testing the combinations isn't going to be as useful as the amount of products in the wild are going to vary quite a bit, and 25G connectivity gets expensive, fast.

Since you are able to reproduce it, it would be great if you could re-test once we have something to offer.