23.1.8: axgbe driver unstable, link flapping

[feld]

The changelog says:

Code Select Expand


src: axgbe: fix link issues for gigabit external SFP PHYs and 100/1000 fiber modules
src: axgbe: apply RRC to miibus attached PHYs and add support for variable bitrate 25G SFP+ DACs
src: axgbe: properly release resource in error case


My dmesg with this kernel shows one of my links constantly flapping:

Code Select Expand


ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN
ax1: Link is UP - 10Gbps/Full - flow control off
ax1: link state changed to UP
ax1: Link is DOWN
ax1: link state changed to DOWN


Booting into the previous kernel fixes it.

My hardware is the official DEC840

[franco]

Which module (vendor) are you using?

Any netmap involved? VLANs? LAGGs? etc. that could be relevant.


Cheers,
Franco

[franco]

One more here: https://www.reddit.com/r/opnsense/comments/13rgxcs/comment/jlm3p5w/?utm_source=reddit&utm_medium=web2x&context=3

Updated this afternoon, seems to have caused a link flap with a 10Gtek ASF-10G-T SFP. Using a StarTech SFP1000TXST for the time being.
EDIT: I'm using an OPNsense (Deciso) DEC740.

[feld]

No netmap, no VLANs on that interface. That's my A&TT upstream interface and I know I had to turn off hardware vlan filtering because I'm bypassing their CPE hardware and there may be some VLAN 0 voodoo going on? I may have to capture some packets and see what they look like.

I will get you the exact model of the SFP+ module tomorrow

[franco]

To limit the exposure of the previous patches perhaps this can help? https://github.com/opnsense/src/commit/fb81510bd

To install...

# opnsense-update -zkr 23.1.8_1

(keep in mind this will delete the old kernel, if you want to retain an older kernel try to revert to that first)

# opnsense-update -kr 23.1.6
(reboot)
# opnsense-update -zkr 23.1.8_1


Cheers,
Franco

[feld]

The SFP+ module is SFP-10G-T from FS.com that is Juniper coded

https://www.fs.com/products/73107.html?attribute=27&id=425

[feld]

To limit the exposure of the previous patches perhaps this can help? https://github.com/opnsense/src/commit/fb81510bd

To install...

# opnsense-update -zkr 23.1.8_1

This kernel is working. The interface is not flapping now.

[feld]

there may be some VLAN 0 voodoo going on?

I have confirmed there is vlan0, if anyone is curious

Code Select Expand


13:41:09.525982 a0:f3:e4:63:0b:7b > f4:90:ea:00:62:2e, ethertype 802.1Q (0x8100), length 70: vlan 0, p 0, ethertype IPv4, 78.192.134.61.13103 > 75.13.68.65.13000: Flags [R.], seq 1, ack 59, win 509, options [nop,nop,TS val 3436954555 ecr 1177160148], length 0
13:41:09.553622 a0:f3:e4:63:0b:7b > f4:90:ea:00:62:2e, ethertype 802.1Q (0x8100), length 60: vlan 0, p 0, ethertype IPv4, 94.102.61.38.50380 > 75.13.68.70.5004: Flags [S], seq 794064754, win 65535, length 0
13:41:09.605744 a0:f3:e4:63:0b:7b > f4:90:ea:00:62:2e, ethertype 802.1Q (0x8100), length 180: vlan 0, p 0, ethertype IPv4, 205.251.197.161.53 > 172.13.126.189.32944: 2930 NXDomain*-$ 0/1/1 (134)


[franco]

Thanks we may check the mentioned modules as time permits but if the fix works we will replace the kernel accordingly now.

VLAN 0 doesn't appear to be an issue at first glance, but thanks for mentioning that.


Cheers,
Franco