DEC840 gets stuck on boot if ax1 is linked

[feld]

OPNsense 26.1.8_5-amd64

I have Quantum Fiber for my internet service and my CPE equipment is a C5500XK. Unfortunately I don't have a GPON SFP that will let me bypass this equipment at this time.

I have a 10GBASE-T SFP+ 30m https://www.fs.com/products/111919.html

If I attempt to boot my DEC840 with this physical link up it gets stuck and fails to complete booting. I have tried replacing this SFP as I have many on hand, but they all produce the same behavior. Changing this out for an 1G transceiver works fine and it boots noticeably faster than the 10G.

I am attaching the boot log captured over serial console at boot via script+screen. I had to clean this up a bit as it had all the terminal control characters and color sequences littered throughout but I think it's fully intact.

What I have observed is that there are some additional i2c devices detected when the 10G module is used, and it seems to fail to configure/communicate with the SFP based on the timeouts it displays in the boot log.

If I boot the firewall without the physical link up (unplug ethernet), and then plug in after fully booted it works fine -- or at least I don't notice any issues with the link or its stability.


edit: I want to mention that this hardware configuration worked fine for a couple years, but I can't be certain when this problem started happening. I believe it started somewhere within the 25.x series.

[tuto2]

Hi,

I took the liberty of testing with an older DEC850 model (same platform, 8 instead of 4 cores). Since I had both the FS 10GSR-85 and the FS 10G-T lying around, plugged these in as well but can't seem to reproduce your issue.

It can be helpful to set both dev.ax.0.axgbe_debug_level and dev.ax.1.axgbe_debug_level to "1" in system -> settings -> tunables, and reboot. This will produce a lot of noise over serial so make sure you can access via ssh to reset these tunables by hand (# sysctl dev.ax.0|1.axgbe_debug_level=0) or make sure you're able to access the GUI to reset the tunables there.

Instead of directly fetching from the serial console you can also pipe "# dmesg" into a file, but make sure it has the ---<<BOOT>>--- in there.

Possibly unrelated, but is one of the ax ports configured for DHCP as a client?

[feld]

Instead of directly fetching from the serial console you can also pipe "# dmesg" into a file, but make sure it has the ---<<BOOT>>--- in there.

I can't, because it doesn't finish booting to the point where I can get a shell and look at dmesg :)

> Possibly unrelated, but is one of the ax ports configured for DHCP as a client?

that is correct, ax1 is a DHCP client to my ISP

[feld]

oh this went very, very badly.

I set the sysctl in the webif as you suggested and instantly it killed the entire firewall. No traffic can go through, serial console spewing out debug data at max rate. I'm guessing 115200 is still a bottleneck, this clogs up the kernel from doing anything useful. Even if I wanted to SSH from one of the igb interfaces, it wouldn't work.

Not good.

Eventually manage to get a single user mode shell before those sysctls get applied on boot. I manually edit the config.xml to remove the sysctl entries. Reboot again.

Now I get a kernel panic every boot when it loads the driver. Removing the SFPs and leaving it powered off for a while doesn't help, every boot is a kernel panic.

Luckily I have a spare DEC 2700 I could install/restore my config onto and swap the hardware in place to get my network back online, but this DEC840 is pretty much a brick right now.

In the debugger after panic, I get this output:

Code Select Expand

db> bt
Tracing pid 0 tid 100000 td 0xffffffff81d81d60
kdb_enter() at kdb_enter+0x33/frame 0xffffffff82e6d6b0
panic() at panic+0x43/frame 0xffffffff82e6d710
trap_pfault() at trap_pfault+0x3da/frame 0xffffffff82e6d760
calltrap() at calltrap+0x8/frame 0xffffffff82e6d760
--- trap 0xc, rip = 0xffffffff810f9dc0, rsp = 0xffffffff82e6d830, rbp = 0xffffffff82e6d850 ---
xgbe_dump_dma_registers() at xgbe_dump_dma_registers+0x3c0/frame 0xffffffff82e6d850
axgbe_pci_init() at axgbe_pci_init+0xe0/frame 0xffffffff82e6d870
axgbe_if_attach_post() at axgbe_if_attach_post+0x395/frame 0xffffffff82e6d8b0
iflib_device_register() at iflib_device_register+0x2655/frame 0xffffffff82e6d9e0
iflib_device_attach() at iflib_device_attach+0xaa/frame 0xffffffff82e6da10
device_attach() at device_attach+0x43d/frame 0xffffffff82e6da60
bus_generic_attach() at bus_generic_attach+0x2d/frame 0xffffffff82e6da80
pci_attach() at pci_attach+0xc7/frame 0xffffffff82e6dab0
acpi_pci_attach() at acpi_pci_attach+0x15/frame 0xffffffff82e6daf0
device_attach() at device_attach+0x43d/frame 0xffffffff82e6db40
bus_generic_attach() at bus_generic_attach+0x2d/frame 0xffffffff82e6db60
acpi_pcib_pci_attach() at acpi_pcib_pci_attach+0x95/frame 0xffffffff82e6db90
device_attach() at device_attach+0x43d/frame 0xffffffff82e6dbe0
bus_generic_attach() at bus_generic_attach+0x2d/frame 0xffffffff82e6dc00
pci_attach() at pci_attach+0xc7/frame 0xffffffff82e6dc30
acpi_pci_attach() at acpi_pci_attach+0x15/frame 0xffffffff82e6dc70
device_attach() at device_attach+0x43d/frame 0xffffffff82e6dcc0
bus_generic_attach() at bus_generic_attach+0x2d/frame 0xffffffff82e6dce0
acpi_pcib_acpi_attach() at acpi_pcib_acpi_attach+0x42b/frame 0xffffffff82e6dd50
device_attach() at device_attach+0x43d/frame 0xffffffff82e6dda0
bus_generic_attach() at bus_generic_attach+0x2d/frame 0xffffffff82e6ddc0
acpi_probe_children() at acpi_probe_children+0x6f/frame 0xffffffff82e6de20
acpi_attach() at acpi_attach+0x9dc/frame 0xffffffff82e6deb0
device_attach() at device_attach+0x43d/frame 0xffffffff82e6df00
bus_generic_attach() at bus_generic_attach+0x2d/frame 0xffffffff82e6df20
device_attach() at device_attach+0x43d/frame 0xffffffff82e6df70
bus_generic_new_pass() at bus_generic_new_pass+0x109/frame 0xffffffff82e6dfa0
root_bus_configure() at root_bus_configure+0x26/frame 0xffffffff82e6dfc0
configure() at configure+0x9/frame 0xffffffff82e6dfd0
mi_startup() at mi_startup+0xb5/frame 0xffffffff82e6dff0


Attempting a full reinstall on this DEC840 now.

edit: the reinstall was successful