Intel ixl interface flapping

[JDA]

I replaced a virtualized opnsense with hardware and now we get a flapping 10GbE (on SFP+ with DAC).

Has anyone a clue on how to debug this?

View from the switch:

Code: [Select]

<190> May 27 11:53:18 sw002-1 DOT1S[dot1s_task]: dot1s_sm.c(314) 1056 %% [INFO] Port (26) inst(0) role changing from ROLE_DESIGNATED to ROLE_DISABLED
<189> May 27 11:53:18 sw002-1 TRAPMGR[trapTask]: traputil.c(708) 1055 %% [NOTE] Link Down: 0/26
<190> May 27 11:40:14 sw002-1 DOT1S[dot1s_task]: dot1s_sm.c(314) 1006 %% [INFO] Port (26) inst(0) role changing from ROLE_DISABLED to ROLE_DESIGNATED
<189> May 27 11:40:14 sw002-1 TRAPMGR[trapTask]: traputil.c(708) 1005 %% [NOTE] Link Up: 0/26
<190> May 27 11:39:51 sw002-1 DOT1S[dot1s_task]: dot1s_sm.c(314) 1001 %% [INFO] Port (26) inst(0) role changing from ROLE_DESIGNATED to ROLE_DISABLED
<189> May 27 11:39:51 sw002-1 TRAPMGR[trapTask]: traputil.c(708) 1000 %% [NOTE] Link Down: 0/26
<190> May 27 11:39:28 sw002-1 DOT1S[dot1s_task]: dot1s_sm.c(314) 989 %% [INFO] Port (26) inst(0) role changing from ROLE_DISABLED to ROLE_DESIGNATED
<189> May 27 11:39:28 sw002-1 TRAPMGR[trapTask]: traputil.c(708) 988 %% [NOTE] Link Up: 0/26
<190> May 27 11:39:06 sw002-1 DOT1S[dot1s_task]: dot1s_sm.c(314) 984 %% [INFO] Port (26) inst(0) role changing from ROLE_DESIGNATED to ROLE_DISABLED
<189> May 27 11:39:06 sw002-1 TRAPMGR[trapTask]: traputil.c(708) 983 %% [NOTE] Link Down: 0/26
<190> May 27 11:15:56 sw002-1 DOT1S[dot1s_task]: dot1s_sm.c(314) 938 %% [INFO] Port (26) inst(0) role changing from ROLE_DISABLED to ROLE_DESIGNATED
<189> May 27 11:15:56 sw002-1 TRAPMGR[trapTask]: traputil.c(708) 937 %% [NOTE] Link Up: 0/26
<190> May 27 11:15:33 sw002-1 DOT1S[dot1s_task]: dot1s_sm.c(314) 933 %% [INFO] Port (26) inst(0) role changing from ROLE_DESIGNATED to ROLE_DISABLED
<189> May 27 11:15:33 sw002-1 TRAPMGR[trapTask]: traputil.c(708) 932 %% [NOTE] Link Down: 0/26
<190> May 27 11:03:36 sw002-1 DOT1S[dot1s_task]: dot1s_sm.c(314) 896 %% [INFO] Port (26) inst(0) role changing from ROLE_DISABLED to ROLE_DESIGNATED
<189> May 27 11:03:36 sw002-1 TRAPMGR[trapTask]: traputil.c(708) 895 %% [NOTE] Link Up: 0/26
<190> May 27 11:03:13 sw002-1 DOT1S[dot1s_task]: dot1s_sm.c(314) 891 %% [INFO] Port (26) inst(0) role changing from ROLE_DESIGNATED to ROLE_DISABLED
<189> May 27 11:03:13 sw002-1 TRAPMGR[trapTask]: traputil.c(708) 890 %% [NOTE] Link Down: 0/26
<189> May 27 10:49:20 sw002-1 TRAPMGR[trapTask]: traputil.c(708) 885 %% [NOTE] Link Up: 0/26

View from the opnsense (not the same events):

Code: [Select]

<13>1 2024-05-27T12:00:01+02:00 fw flowd_aggregate.py 90284 - [meta sequenceId="1"] startup, check database.
<13>1 2024-05-27T12:00:01+02:00 fw flowd_aggregate.py 90284 - [meta sequenceId="2"] start watching flowd
<13>1 2024-05-27T12:00:01+02:00 fw flowd_aggregate.py 90284 - [meta sequenceId="3"] vacuum done
<13>1 2024-05-27T12:03:41+02:00 fw kernel - - [meta sequenceId="1"] <6>ixl0: link state changed to DOWN
<13>1 2024-05-27T12:03:41+02:00 fw kernel - - [meta sequenceId="2"] <6>vlan0.50: link state changed to DOWN
<13>1 2024-05-27T12:03:41+02:00 fw kernel - - [meta sequenceId="3"] <6>vlan0.55: link state changed to DOWN
<13>1 2024-05-27T12:03:41+02:00 fw kernel - - [meta sequenceId="4"] <6>vlan0.54: link state changed to DOWN
<13>1 2024-05-27T12:03:41+02:00 fw kernel - - [meta sequenceId="5"] <6>vlan0.250: link state changed to DOWN
<13>1 2024-05-27T12:03:41+02:00 fw kernel - - [meta sequenceId="6"] <6>vlan0.20: link state changed to DOWN
<13>1 2024-05-27T12:03:41+02:00 fw kernel - - [meta sequenceId="7"] <6>vlan0.10: link state changed to DOWN
<13>1 2024-05-27T12:03:41+02:00 fw kernel - - [meta sequenceId="8"] <6>vlan0.29: link state changed to DOWN
<13>1 2024-05-27T12:03:41+02:00 fw kernel - - [meta sequenceId="9"] <6>vlan0.28: link state changed to DOWN
<13>1 2024-05-27T12:03:41+02:00 fw kernel - - [meta sequenceId="10"] <6>vlan0.30: link state changed to DOWN
<13>1 2024-05-27T12:03:41+02:00 fw opnsense 96488 - [meta sequenceId="11"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for
opt6(vlan0.50)
<13>1 2024-05-27T12:03:41+02:00 fw opnsense 97480 - [meta sequenceId="12"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for
opt8(vlan0.55)
<13>1 2024-05-27T12:03:42+02:00 fw opnsense 98548 - [meta sequenceId="13"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for
opt7(vlan0.54)
<13>1 2024-05-27T12:03:42+02:00 fw opnsense 99510 - [meta sequenceId="14"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for
opt3(vlan0.20)
<13>1 2024-05-27T12:03:42+02:00 fw opnsense 1044 - [meta sequenceId="15"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for o
pt2(vlan0.10)
<13>1 2024-05-27T12:03:42+02:00 fw opnsense 2101 - [meta sequenceId="16"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for o
pt4(vlan0.29)
<13>1 2024-05-27T12:03:43+02:00 fw opnsense 3410 - [meta sequenceId="17"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for o
pt5(vlan0.30)
<28>1 2024-05-27T12:03:43+02:00 fw lldpd 85765 - [meta sequenceId="18"] unable to send packet on real device for ixl0: Network is down
<28>1 2024-05-27T12:03:49+02:00 fw lldpd 85765 - [meta sequenceId="19"] unable to send packet on real device for ixl0: Network is down
<13>1 2024-05-27T12:03:49+02:00 fw kernel - - [meta sequenceId="20"] <5>ixl0: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Neg
otiated FEC: None, Autoneg: False, Flow Control: None
<13>1 2024-05-27T12:03:49+02:00 fw kernel - - [meta sequenceId="21"] <6>ixl0: link state changed to UP
<13>1 2024-05-27T12:03:49+02:00 fw kernel - - [meta sequenceId="22"] <6>vlan0.50: link state changed to UP
<13>1 2024-05-27T12:03:49+02:00 fw kernel - - [meta sequenceId="23"] <6>vlan0.55: link state changed to UP
<13>1 2024-05-27T12:03:49+02:00 fw kernel - - [meta sequenceId="24"] <6>vlan0.54: link state changed to UP
<13>1 2024-05-27T12:03:49+02:00 fw kernel - - [meta sequenceId="25"] <6>vlan0.250: link state changed to UP
<13>1 2024-05-27T12:03:49+02:00 fw kernel - - [meta sequenceId="26"] <6>vlan0.20: link state changed to UP
<13>1 2024-05-27T12:03:49+02:00 fw kernel - - [meta sequenceId="27"] <6>vlan0.10: link state changed to UP
<13>1 2024-05-27T12:03:49+02:00 fw kernel - - [meta sequenceId="28"] <6>vlan0.29: link state changed to UP
<13>1 2024-05-27T12:03:49+02:00 fw kernel - - [meta sequenceId="29"] <6>vlan0.28: link state changed to UP
<13>1 2024-05-27T12:03:49+02:00 fw kernel - - [meta sequenceId="30"] <6>vlan0.30: link state changed to UP
<13>1 2024-05-27T12:03:49+02:00 fw opnsense 15545 - [meta sequenceId="31"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for
opt6(vlan0.50)

I already added a tunable for hw.ixl.core_debug_mask to be 0x00000010 (hope the syntax is correct), but need to wait till I can reboot.

[netnut]

I replaced a virtualized opnsense with hardware and now we get a flapping 10GbE (on SFP+ with DAC).

Has anyone a clue on how to debug this?

* Configure NTP on both switch and OPNsense so you're able to compare correct timestamps

* Post a detailed topology picture and Switch + OPNsense config

Without that, wild guess, you've created a loop (physical ? / incorrect configured bridge ?) and STP is kicking in at your switch port.

[Greg_E]

My first thought is cheap optical modules and a cable, I've had issues with DAC and often fix them or prove them with a fiber cable. Could be a card driver, could be a bad cable, could be a cable that is somehow not compatible. Moving back to known things can save a little time in the troubleshoot process.