"DEVD: Ethernet detached event for" for igbx intefaces

[wstemb]

Found a lot of unconlusive topics in older forums (22.7 and older), so i post again there.

In System:Logs:General I had a quantity of "DEVD: Ethernet detached event for ..." and "DEVD: Ethernet attached event for"  few second after for igb interfaces (igb0 and igb1) - from one to three times a day.

The problem is that on one of intefaces (igb1)  we use an application which is sensible to connection lost - so users experience the app crash  every time it happens and call me.

This interface igb1 was activated after the upgrade to 23.1, so I can't say if it is applicable on 22.x or older versions. Later updates are not changing the behavior.

On the firewall I have ix interfaces also, no problem on them.

The Tunable parameters for the igb interfaces are:

Code: [Select]

sysctl -a | grep eee
...
hw.igc.eee_setting: 1
hw.em.eee_setting: 1
dev.igb.1.eee_control: 1
dev.igb.0.eee_control: 1

sysctl -a | grep fc
...
dev.igb.1.fc_low_water: 71664
dev.igb.1.fc_high_water: 71680
dev.igb.1.fc: 0
dev.igb.0.fc_low_water: 71664
dev.igb.0.fc_high_water: 71680
dev.igb.0.fc: 0
...

The problem is imputable to sensei/Zenarmor or the netmap driver , since apparently disappeared on both igb interfaces after I changed the Zenarmor configuration from:

Routed Mode (L3 Mode, Reporting + Blocking) with native netmap driver

to

Passive Mode (Reporting Only).

Only the igb0 interface was Protected by Zenarmor, the other (igb1) not, but in was not important for the behavior of the interfaces, both were involved in restarts. After  the "passivization" of Zenarmor, there are no new detached/attached errors for at least two days in log, and no calls from users. 

[franco]

Hi,

I think this is more or less expected behaviour from netmap mode (also in Suricata IPS) when the interface drops out it goes through a cycle or when Zenarmor needs to reconfigure something and one should aim to minimise reload tasks during critical hours therefore.

It could also be an external factor like the driver dropping out after a network flood or switch glitch/reboot, which is harder to avoid, but still possible with better network design.


Cheers,
Franco

[wstemb]

Nobody worked on OpnSense at some moments of detach/attach, I am sure. It is more or less a stable environment now.  I will reconfigure the network, to see what is causing the interface detach, the external equipment (not under my control) or the interface itself, if it will be possible to find it in the switch log. 

[chemlud]

Log has events from time to time, without any user interference (no changes in config at these times, no nothing)

Code: [Select]

2023-03-21T09:18:20 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static opt3(igb1)
2023-03-21T08:24:40 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for static opt3(igb1)
2023-03-21T08:24:38 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static opt3(igb1)
2023-03-21T08:23:46 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for static opt3(igb1)
2023-03-21T08:23:44 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static opt3(igb1)
2023-03-21T03:46:35 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for static opt3(igb1)
2023-03-20T18:32:38 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static opt3(igb1)
2023-03-20T18:24:02 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for static opt3(igb1)
2023-03-12T11:41:06 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for static lan(em1)
2023-03-12T11:41:03 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for static opt1(em0)
2023-03-12T11:41:01 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for static opt2(igb0)
2023-03-12T11:40:59 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static lan(em1)
2023-03-12T11:40:58 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static opt1(em0)
2023-03-12T11:40:57 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static opt2(igb0)
2023-03-12T11:38:39 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for static opt1(em0)
2023-03-12T11:38:38 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for static lan(em1)
2023-03-12T11:38:37 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static opt2(igb0)
2023-03-12T11:38:36 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static opt1(em0)
2023-03-12T11:38:36 Error opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for static lan(em1)
on various interfaces. Suricata IPS running. This is a bug in netmap, not expected behaviour, but to rare/not reproducible to debug, imho...

[Wiesel]

I had a similar problem some month ago and as you said there are confusing posts and solutions out there from bugs which were already solved to very mystical things.

For me it started to happen with a new Modem from my ISP so I'm not able to tell for sure if the modem or my opnsense nic was causing it. However, what helped was to put an unmanaged switch in between both devices. After that I had connections hanging sometimes (as you said, 2, 3 times a day) for about 2-3 seconds but it didn't drop the connection thanks to the switch.

Since I switched again to another modem, the issue disappeared and I was able to remove the intermediate switch.

My guess is those two NICs where just somehow disliking each other.

Since networking is just a hobby for me and not my job, I haven't traced it down further but maybe it is worth to test this workaround since I read it also several times on reddit with people succeeding by putting a switch in the connection.