Opnsense randomly (?) crashes

Started by meikel, June 02, 2026, 09:06:59 AM

Previous topic - Next topic
So I've been using Opnsense for quite some time now and am very pleased with it.

However just Yesterday morning I went into my home office and noticed that I have no internet. After a short troubleshoot I found out that OPNsense is powered up and running but I get no IP or anything from it, I can't ping, ssh into it or get to the Web UI. I just quickly hard rebooted it and the issue was solved. Until today where this issue appeared again. I solved it quickly the same way as Yesterday however the issue just reappeared just about an hour later after the first reboot.

I'm unable to diagnose this issue. The logs give no information about what could have happened:

<13>1 2026-06-02T07:39:05+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="70"] /usr/local/etc/rc.linkup: plugins_configure ipsec (,lan)
<13>1 2026-06-02T07:39:05+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="71"] /usr/local/etc/rc.linkup: plugins_configure ipsec (execute task : ipsec_configure_do(,lan))
<13>1 2026-06-02T07:39:05+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="72"] /usr/local/etc/rc.linkup: plugins_configure dhcp ()
<13>1 2026-06-02T07:39:05+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="73"] /usr/local/etc/rc.linkup: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2026-06-02T07:39:06+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="74"] /usr/local/etc/rc.linkup: plugins_configure dhcp (execute task : radvd_configure_dhcp())
<13>1 2026-06-02T07:39:06+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="75"] /usr/local/etc/rc.linkup: plugins_configure dns ()
<13>1 2026-06-02T07:39:06+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="76"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2026-06-02T07:39:06+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="77"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : unbound_configure_do())
<12>1 2026-06-02T07:39:06+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="78"] /usr/local/etc/rc.linkup: warning: ignoring missing default tunable request: vm.pmap.pti
<12>1 2026-06-02T07:39:06+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="79"] /usr/local/etc/rc.linkup: warning: ignoring missing default tunable request: hw.ibrs_disable
<13>1 2026-06-02T07:39:07+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="80"] /usr/local/etc/rc.linkup: plugins_configure newwanip:rfc2136 (,[lan])
<13>1 2026-06-02T07:39:29+02:00 OPNsense.intern kernel - - [meta sequenceId="81"] <6>[102] igb1: promiscuous mode enabled
<45>1 2026-06-02T08:39:09+02:00 OPNsense.intern syslog-ng 20363 - [meta sequenceId="1"] syslog-ng starting up; version='4.11.0'
<13>1 2026-06-02T08:39:09+02:00 OPNsense.intern kernel - - [meta sequenceId="2"] ---<<BOOT>>---
<13>1 2026-06-02T08:39:09+02:00 OPNsense.intern kernel - - [meta sequenceId="3"] Copyright (c) 1992-2023 The FreeBSD Project.
<13>1 2026-06-02T08:39:09+02:00 OPNsense.intern kernel - - [meta sequenceId="4"] Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
<13>1 2026-06-02T08:39:09+02:00 OPNsense.intern kernel - - [meta sequenceId="5"]        The Regents of the University of California. All rights reserved.
<13>1 2026-06-02T08:39:09+02:00 OPNsense.intern kernel - - [meta sequenceId="6"] FreeBSD is a registered trademark of The FreeBSD Foundation.
<13>1 2026-06-02T08:39:09+02:00 OPNsense.intern kernel - - [meta sequenceId="7"] FreeBSD 14.3-RELEASE-p12 stable/26.1-n272089-81f87c4d694c SMP amd64

I rebooted the device at about 07:39 and 08:39 so the logs from 07:xx are just the boot logs, no more logs after that.

Are there any other logs I can look into? I used opnsense-log to look at these logs.

I also was in the room once the device became faulty (It threw me out of my remote connection to work) and I noticed that Opnsense also did a beep (I don't know which kind of beep it is).

I saw some users suggesting the DIMM could be overheating but before I replace that I'd like to verify that these are in fact the issue. Even though it's somewhat summer here in Germany right now it's not really as hot in the room where the hardware is and it hasn't been a problem in the last year.

My Opnsense Version: OPNsense 26.1.8_5-amd64
My Hardware: Sophos SG 310 Rev.1

Everything you described points to a hardware issue. It could be ram, it could be power supply or motherboard. There is no easy way to diagnose this.

... or a failing SSD.

I'd probably try to keep a serial console connection open all the time, and look there for clues when it locks up.

I doubt it's the SSD(s) as it's a raid1 (zfs)

Quote from: meikel on June 02, 2026, 01:56:57 PMI doubt it's the SSD(s) as it's a raid1 (zfs)

I would start with full 24h memtest86, followed by Prime95 for few hours and then end it with dedicated diagnostic software for your storage just in case. I would also check thermal pads/thermal paste and dust build up. Check the fans too. Sophos appliances are known to be very reliable hardware wise. Every time i saw issues with with them, it was always faulty power supply or faulty fans because the units were not attached to PSU and they were operating in very harsh and dirty environments.

and before starting, a reminder that if you have more than one memory stick, memtest will register errors if there are any but won't point which stick/slot they are coming from. Hence the need to do one stick at the time.

Quote from: cookiemonster on June 02, 2026, 02:39:11 PMand before starting, a reminder that if you have more than one memory stick, memtest will register errors if there are any but won't point which stick/slot they are coming from. Hence the need to do one stick at the time.
Thats correct. Good point.

So I conneted a display and withnessed a crash today. It actually crashes. When the crash happens opnsense beeps first and floods the UI with log messages (maybe kernel messages). After that the device reboots itself and is stuck in:

"Reboot and Select proper Boot device or Insert Boot Media in selected Boot device and press a key"

Is this an indication that the drive is faulty or doesn't this proof anything?
As this is my gateway to the internet a longly test should be my last resort right now.

Quote from: meikel on Today at 04:09:07 PMSo I conneted a display and withnessed a crash today. It actually crashes. When the crash happens opnsense beeps first and floods the UI with log messages (maybe kernel messages). After that the device reboots itself and is stuck in:

"Reboot and Select proper Boot device or Insert Boot Media in selected Boot device and press a key"

Is this an indication that the drive is faulty or doesn't this proof anything?
As this is my gateway to the internet a longly test should be my last resort right now.

According to what you just described, yes. That indicates drive issues. But unfortunately, that is still a wild guess. The only way to be sure is by running hardware tests. As i said in previous post, there is no easy way to diagnose this problem. If you have spare storage, you can save your opnsense config, reinstall opnsense on spare drive and restore your config and see how system behaves.

Does your device have a serial console? If yes, connect a PC with a terminal program and let that run until the next crash. The serial console output will not be cleared on reboot unlike VGA/HDMI.
Deciso DEC750
People who think they know everything are a great annoyance to those of us who do. (Isaac Asimov)