Router Dead After Latest Update

[pholt]

I just applied the latest update through the gui. The router appears to have crashed and won't restart. I'd reinstall, but I need access to the internet for that. Suggestions?

[newsense]

Without seeing where it dies it's hard to diagnose anything.

Try to connect a monitor or serial cable if applicable and post the details here

[nero355]

I'd reinstall, but I need access to the internet for that.

You don't have any USB Stick laying around from which you can Live Boot OPNsense ?!

I always keep a copy of the latest OPNsense ISO and my config.xml for emergency purposes on both my PC and Laptop and also one of my USB Sticks.

And if everything goes seriously "down south" then there is always the Router provided by my ISP and my old xDSL Router which I could use to get back up and running again :)

[pholt]

I was able to connect to the machine via Meshcommander, and I could see that the boot just fails. A few lines flash by too fast to read, then it hangs. Using  the same workaround that nero355 suggested, get a fresh copy of the software and reinstalled it on the machine.

The remaining question is why the update failed so catastrophically and if this has happened to anyone else. Otherwise, I might've been some kind of weird local corruption that finally bit back.

[franco]

Sounds like a kernel panic that caused a boot loop, but without looking at the stack trace it's not easy to tell. There were little kernel patches in 26.7.7 compared to 26.7.6 and nobody else reported something so far so it's unclear what the cause was/is.


Cheers,
Franco

[(MARLOO)]

Positive report: 26.7.6 -> 26.7.7 update completed successfully via the GUI.

After reboot:

- opnsense-version reports OPNsense 26.7.7 (amd64)
- No kernel panic, segfault or Hyper-V-related errors in dmesg
- pf is running and has active states
- DNS and WAN connectivity are working
- configd, Unbound, dhcpd, watchfrr, mgmtd and zebra are running
- FRR: watchfrr is running

The only matching line in the system log was the expected controlled
termination of ppp during the service restart:

ppp ... caught fatal signal TERM

No issues observed so far after the update.

[(MARLOO)]

Update to my previous positive report.

The 26.7.6 -> 26.7.7 update completed successfully and the system ran normally afterwards.

Later, while the firewall was running normally, the LAN interface briefly went down and came back up after a few seconds. This caused a temporary loss of connectivity.

After that, a scheduled reboot was triggered. The shutdown started normally, but the firewall never completed the reboot.

From Grafana I could see that CPU usage went to about 99%, but the firewall remained hung and unreachable. I had to power-cycle it manually.

After the manual power cycle, OPNsense booted normally.

I then tested a manual reboot from the GUI. It completed successfully and the system came back online.

No kernel panic, crash dump or watchdog message was found. CPU temperature was normal, and there was plenty of free RAM and no swap usage.

I have disabled the scheduled reboot for now. Has anyone else seen a brief LAN link loss followed by a scheduled reboot leaving OPNsense hung with high CPU usage?