Router Dead After Latest Update

[pholt]

I just applied the latest update through the gui. The router appears to have crashed and won't restart. I'd reinstall, but I need access to the internet for that. Suggestions?

[newsense]

Without seeing where it dies it's hard to diagnose anything.

Try to connect a monitor or serial cable if applicable and post the details here

[nero355]

I'd reinstall, but I need access to the internet for that.

You don't have any USB Stick laying around from which you can Live Boot OPNsense ?!

I always keep a copy of the latest OPNsense ISO and my config.xml for emergency purposes on both my PC and Laptop and also one of my USB Sticks.

And if everything goes seriously "down south" then there is always the Router provided by my ISP and my old xDSL Router which I could use to get back up and running again :)

[pholt]

I was able to connect to the machine via Meshcommander, and I could see that the boot just fails. A few lines flash by too fast to read, then it hangs. Using  the same workaround that nero355 suggested, get a fresh copy of the software and reinstalled it on the machine.

The remaining question is why the update failed so catastrophically and if this has happened to anyone else. Otherwise, I might've been some kind of weird local corruption that finally bit back.

[franco]

Sounds like a kernel panic that caused a boot loop, but without looking at the stack trace it's not easy to tell. There were little kernel patches in 26.7.7 compared to 26.7.6 and nobody else reported something so far so it's unclear what the cause was/is.


Cheers,
Franco

[(MARLOO)]

Positive report: 26.7.6 -> 26.7.7 update completed successfully via the GUI.

After reboot:

- opnsense-version reports OPNsense 26.7.7 (amd64)
- No kernel panic, segfault or Hyper-V-related errors in dmesg
- pf is running and has active states
- DNS and WAN connectivity are working
- configd, Unbound, dhcpd, watchfrr, mgmtd and zebra are running
- FRR: watchfrr is running

The only matching line in the system log was the expected controlled
termination of ppp during the service restart:

ppp ... caught fatal signal TERM

No issues observed so far after the update.

[(MARLOO)]

Update to my previous positive report.

The 26.7.6 -> 26.7.7 update completed successfully and the system ran normally afterwards.

Later, while the firewall was running normally, the LAN interface briefly went down and came back up after a few seconds. This caused a temporary loss of connectivity.

After that, a scheduled reboot was triggered. The shutdown started normally, but the firewall never completed the reboot.

From Grafana I could see that CPU usage went to about 99%, but the firewall remained hung and unreachable. I had to power-cycle it manually.

After the manual power cycle, OPNsense booted normally.

I then tested a manual reboot from the GUI. It completed successfully and the system came back online.

No kernel panic, crash dump or watchdog message was found. CPU temperature was normal, and there was plenty of free RAM and no swap usage.

I have disabled the scheduled reboot for now. Has anyone else seen a brief LAN link loss followed by a scheduled reboot leaving OPNsense hung with high CPU usage?

[(MARLOO)]

At the moment I cannot provide a stack trace or crash dump.

The two unexpected boots at 07:46 and 08:23 left no panic, trap, watchdog or OOM message in dmesg, and no dump in /var/crash. They also had no preceding shutdown/Uptime lines, unlike the controlled reboots at 06:42 and 08:13.

Crash dumps are now fully enabled (dump device nda0p3, 8 GB swap, savecore enabled, dumpdir /var/crash), so if the issue recurs as a kernel panic I should be able to provide a backtrace.

For now the system has been stable for several hours. I will monitor and report back if it happens again, either with a vmcore or with confirmation that no dump was produced.

[Nullman]

Uptated to 26.7.7 without any issues. In fact, its been a year of updates with zero issues.

[pholt]

Reinstalled yesterday. Ran the update. Everything worked fine. I ran the update on a backup server. Everything worked fine. The only detail that might be of any use is the last thing that appeared before the crash was the refresh.sh script. That's where the system appeared to hang and never came back. I even waited about ten minutes thinking that something complicated might be going on.

Things appear to be back to normal.

[(MARLOO)]

Reinstalled yesterday. Ran the update. Everything worked fine. I ran the update on a backup server. Everything worked fine. The only detail that might be of any use is the last thing that appeared before the crash was the refresh.sh script. That's where the system appeared to hang and never came back. I even waited about ten minutes thinking that something complicated might be going on.

Things appear to be back to normal.

In my case the update itself completed normally and the system ran fine afterwards. The problem appeared later: a brief LAN link loss, followed by a scheduled reboot that hung with very high CPU usage. After a manual power cycle it booted normally, and a manual GUI reboot also completed successfully.