Intel E610 / ix: link drops at idle, and eee_state sysctl hangs (PR 300670)

[-pletch-]

I recently moved my OPNsense box (26.7, VM on Proxmox with the NIC passed through) to an Intel E610-XT2. I was seeing sporadic link drops and during investigation identified an upstream bug associated with setting EEE via sysctl when attempting to turn disable.

I noticed setting the EEE sysctl on a running ix interface never returns. The sysctl process hangs in an uninterruptible state, and the interface's broadcast address pointer gets corrupted (broadcasts on that interface go to ff:ff:00:00:00:00 until a reboot). The cause is a wrong argument in the driver's EEE handler, if_init(ifp, ifp).

• It's not only E610-specific: any ix adapter that exposes eee_state goes through the same code. Besides the E610, that's the ports built into Xeon D-1500 (10G KR) and Atom C3000 (1G copper, common in small firewall boxes) chips. Plain X550-T1/T2 cards, X540 and 82599 don't have the setting, so they're not affected. If running "sysctl dev.ix.0.eee_state" shows a value on your box, you're exposed. I've only tested on the E610 but the driver code suggests this will happen in these other configurations as well.
• Putting dev.ix.N.eee_state=0 under System > Settings > Tunables runs the same code at boot. I haven't tried it, but expect it to hang there too.
• EEE is on by default on the E610 and the driver has no loader tunable for it, so on a stock kernel this sysctl is the only way to turn it off from the FreeBSD side. Otherwise you have to turn it off on the switch.

I filed an upstream bug report with a patch: FreeBSD PR 300670. I'm running a kernel with the patch, and the sysctl now returns immediately and EEE really turns off.

Drop issues

• WAN port (to my Fiber ONT) dropped twice for a few seconds while linked at 10G. I locked it to 2.5G (my service is 2 Gbit anyway) and it stopped, while EEE was still on, so on that link it seems to be the 10G connection rather than EEE but I haven't investigated further to confirm. It hasn't dropped since.
• LAN port (to a Netgear MS305E 2.5G switch) dropped twice in a day for a few seconds, both times when the network was quiet. Each one was a link down/up with the NIC counting "local faults", and no CRC or other errors.
• The firmware reports EEE as enabled and active on the LAN link (both ends agreed to use it). I read that from the driver with DTrace, since nothing displays it. The dev.ix.N.eee.*_lpi_count counters stayed at 0 even while EEE was active, so don't rely on them to tell whether EEE is in use.
• With EEE off, the LAN port has gone a little over two days without a drop. Fingers crossed that this fixes it and it is reflective of some incompatibility between the E610 and my switch and how they handle EEE transitions.

Figured I'd share here for the benefit of others that might be playing with the Intel E610 cards.  There doesn't seem to be too many reports out there as they are pretty new, a little $$, and not many in the wild thus far.