So I've been using Opnsense for quite some time now and am very pleased with it.
However just Yesterday morning I went into my home office and noticed that I have no internet. After a short troubleshoot I found out that OPNsense is powered up and running but I get no IP or anything from it, I can't ping, ssh into it or get to the Web UI. I just quickly hard rebooted it and the issue was solved. Until today where this issue appeared again. I solved it quickly the same way as Yesterday however the issue just reappeared just about an hour later after the first reboot.
I'm unable to diagnose this issue. The logs give no information about what could have happened:
<13>1 2026-06-02T07:39:05+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="70"] /usr/local/etc/rc.linkup: plugins_configure ipsec (,lan)
<13>1 2026-06-02T07:39:05+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="71"] /usr/local/etc/rc.linkup: plugins_configure ipsec (execute task : ipsec_configure_do(,lan))
<13>1 2026-06-02T07:39:05+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="72"] /usr/local/etc/rc.linkup: plugins_configure dhcp ()
<13>1 2026-06-02T07:39:05+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="73"] /usr/local/etc/rc.linkup: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2026-06-02T07:39:06+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="74"] /usr/local/etc/rc.linkup: plugins_configure dhcp (execute task : radvd_configure_dhcp())
<13>1 2026-06-02T07:39:06+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="75"] /usr/local/etc/rc.linkup: plugins_configure dns ()
<13>1 2026-06-02T07:39:06+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="76"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2026-06-02T07:39:06+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="77"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : unbound_configure_do())
<12>1 2026-06-02T07:39:06+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="78"] /usr/local/etc/rc.linkup: warning: ignoring missing default tunable request: vm.pmap.pti
<12>1 2026-06-02T07:39:06+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="79"] /usr/local/etc/rc.linkup: warning: ignoring missing default tunable request: hw.ibrs_disable
<13>1 2026-06-02T07:39:07+02:00 OPNsense.intern opnsense 61182 - [meta sequenceId="80"] /usr/local/etc/rc.linkup: plugins_configure newwanip:rfc2136 (,[lan])
<13>1 2026-06-02T07:39:29+02:00 OPNsense.intern kernel - - [meta sequenceId="81"] <6>[102] igb1: promiscuous mode enabled
<45>1 2026-06-02T08:39:09+02:00 OPNsense.intern syslog-ng 20363 - [meta sequenceId="1"] syslog-ng starting up; version='4.11.0'
<13>1 2026-06-02T08:39:09+02:00 OPNsense.intern kernel - - [meta sequenceId="2"] ---<<BOOT>>---
<13>1 2026-06-02T08:39:09+02:00 OPNsense.intern kernel - - [meta sequenceId="3"] Copyright (c) 1992-2023 The FreeBSD Project.
<13>1 2026-06-02T08:39:09+02:00 OPNsense.intern kernel - - [meta sequenceId="4"] Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
<13>1 2026-06-02T08:39:09+02:00 OPNsense.intern kernel - - [meta sequenceId="5"] The Regents of the University of California. All rights reserved.
<13>1 2026-06-02T08:39:09+02:00 OPNsense.intern kernel - - [meta sequenceId="6"] FreeBSD is a registered trademark of The FreeBSD Foundation.
<13>1 2026-06-02T08:39:09+02:00 OPNsense.intern kernel - - [meta sequenceId="7"] FreeBSD 14.3-RELEASE-p12 stable/26.1-n272089-81f87c4d694c SMP amd64
I rebooted the device at about 07:39 and 08:39 so the logs from 07:xx are just the boot logs, no more logs after that.
Are there any other logs I can look into? I used opnsense-log to look at these logs.
I also was in the room once the device became faulty (It threw me out of my remote connection to work) and I noticed that Opnsense also did a beep (I don't know which kind of beep it is).
I saw some users suggesting the DIMM could be overheating but before I replace that I'd like to verify that these are in fact the issue. Even though it's somewhat summer here in Germany right now it's not really as hot in the room where the hardware is and it hasn't been a problem in the last year.
My Opnsense Version: OPNsense 26.1.8_5-amd64
My Hardware: Sophos SG 310 Rev.1
Everything you described points to a hardware issue. It could be ram, it could be power supply or motherboard. There is no easy way to diagnose this.
... or a failing SSD.
I'd probably try to keep a serial console connection open all the time, and look there for clues when it locks up.
I doubt it's the SSD(s) as it's a raid1 (zfs)
Quote from: meikel on June 02, 2026, 01:56:57 PMI doubt it's the SSD(s) as it's a raid1 (zfs)
I would start with full 24h memtest86, followed by Prime95 for few hours and then end it with dedicated diagnostic software for your storage just in case. I would also check thermal pads/thermal paste and dust build up. Check the fans too. Sophos appliances are known to be very reliable hardware wise. Every time i saw issues with with them, it was always faulty power supply or faulty fans because the units were not attached to PSU and they were operating in very harsh and dirty environments.
and before starting, a reminder that if you have more than one memory stick, memtest will register errors if there are any but won't point which stick/slot they are coming from. Hence the need to do one stick at the time.
Quote from: cookiemonster on June 02, 2026, 02:39:11 PMand before starting, a reminder that if you have more than one memory stick, memtest will register errors if there are any but won't point which stick/slot they are coming from. Hence the need to do one stick at the time.
Thats correct. Good point.
So I conneted a display and withnessed a crash today. It actually crashes. When the crash happens opnsense beeps first and floods the UI with log messages (maybe kernel messages). After that the device reboots itself and is stuck in:
"Reboot and Select proper Boot device or Insert Boot Media in selected Boot device and press a key"
Is this an indication that the drive is faulty or doesn't this proof anything?
As this is my gateway to the internet a longly test should be my last resort right now.
Quote from: meikel on June 03, 2026, 04:09:07 PMSo I conneted a display and withnessed a crash today. It actually crashes. When the crash happens opnsense beeps first and floods the UI with log messages (maybe kernel messages). After that the device reboots itself and is stuck in:
"Reboot and Select proper Boot device or Insert Boot Media in selected Boot device and press a key"
Is this an indication that the drive is faulty or doesn't this proof anything?
As this is my gateway to the internet a longly test should be my last resort right now.
According to what you just described, yes. That indicates drive issues. But unfortunately, that is still a wild guess. The only way to be sure is by running hardware tests. As i said in previous post, there is no easy way to diagnose this problem. If you have spare storage, you can save your opnsense config, reinstall opnsense on spare drive and restore your config and see how system behaves.
Does your device have a serial console? If yes, connect a PC with a terminal program and let that run until the next crash. The serial console output will not be cleared on reboot unlike VGA/HDMI.
Quote from: meikel on June 03, 2026, 04:09:07 PMSo I conneted a display and withnessed a crash today. It actually crashes. When the crash happens opnsense beeps first and floods the UI with log messages (maybe kernel messages). After that the device reboots itself and is stuck in:
"Reboot and Select proper Boot device or Insert Boot Media in selected Boot device and press a key"
Is this an indication that the drive is faulty or doesn't this proof anything?
As this is my gateway to the internet a longly test should be my last resort right now.
Sounds 100% like unstable Storage (HDD/SSD) or if you are seriously unlucky the SATA or NVMe Port it's attached to...
Tell us which model it is and if you are lucky someone will say a firmware update might fix it.
If not then simply replace it and be happy :)
Quote from: nero355 on June 03, 2026, 06:45:55 PMTell us which model it is and if you are lucky someone will say a firmware update might fix it.
If not then simply replace it and be happy :)
Its right there in his signature.
Quote from: Patrick M. Hausen on June 03, 2026, 04:36:36 PMDoes your device have a serial console? If yes, connect a PC with a terminal program and let that run until the next crash. The serial console output will not be cleared on reboot unlike VGA/HDMI.
I just ordered a serial cable and will try that. I noticed in the flashy screen that there were a lot of hex codes so I ran me test.
Sadly both ram sticks had no issues after 30min.
I tried running a live debian13 and the device hung after some seconds on the desktop.
I improved cooling (airflow) even though I don't think this is the issue (the issue persists even after the cooling body feels way cooler now).
I also replaced the CMOS battery as a friend suggested this can be the root cause of such issues but no luck here either.
I will wait for the serial cable and try to debug that way. In the meantime I might try to update the bios of the mainboard.
I also have to correct myself: the SSD is not running in raid1. It's a single SSD but as a live OS is also crashing I assume it's not the SSD. I also doubt that a faulty SSD crashes the whole machine.
Quote from: meikel on June 03, 2026, 08:10:22 PMSadly both ram sticks had no issues after 30min.
30 minutes is not enough.
Quote from: meikel on June 03, 2026, 08:10:22 PMI tried running a live debian13 and the device hung after some seconds on the desktop.
That rules out SSD issue.
Quote from: meikel on June 03, 2026, 08:10:22 PMI improved cooling (airflow) even though I don't think this is the issue (the issue persists even after the cooling body feels way cooler now).
Improving the airflow wont fix bad thermal contact between heat sink and the cpu. Also, bad thermal contact can cause overheating even if your case is cold to the touch.
Quote from: meikel on June 03, 2026, 08:10:22 PMI also replaced the CMOS battery as a friend suggested this can be the root cause of such issues but no luck here either.
CMOS is active only when you turn on your system. Once POST is completed, it does nothing.
Quote from: meikel on June 03, 2026, 08:10:22 PMI will wait for the serial cable and try to debug that way. In the meantime I might try to update the bios of the mainboard.
You should not update anything until you find what the problem is.
Quote from: meikel on June 03, 2026, 08:10:22 PMI also have to correct myself: the SSD is not running in raid1. It's a single SSD but as a live OS is also crashing I assume it's not the SSD. I also doubt that a faulty SSD crashes the whole machine.
Yeah SSD is not the issue here. You are probably dealing with either faulty memory, faulty motherboard, or faulty power supply.
Quote from: Nullman on June 03, 2026, 08:22:11 PMYou are probably dealing with either faulty memory, faulty motherboard, or faulty power supply.
For a hardware related issue this sums it up, though overheating should also be considered, as mentioned earlier regarding fans.
Run
sudo sysctl -a | grep 'temperature' to obtain temperature sensor information.
Power supply faults can create strange and bewildering problems.
Memory exercising should be run for many hours with many passes.
Logging the serial console output to a file via your terminal program is a good idea.
Good luck.
Quote from: Nullman on June 03, 2026, 08:02:35 PMIts right there in his signature.
I see signatures of users but his is empty ?!
Quote from: meikel on June 03, 2026, 08:10:22 PMI also have to correct myself: the SSD is not running in raid1.
It's a single SSD but as a live OS is also crashing I assume it's not the SSD.
I also doubt that a faulty SSD crashes the whole machine.
Sometimes Live CD/ISO Images try to mount local storage automatically and if that fails in a certain way it can cause issues too so I would check that again just to be sure !!
- Which model is it ?
- Are there firmware updates available ?
- What does
smartctl/nvmecontrol say ?
- Are there known issues with it in certain scenarios ?
And indeed : Memory testing should be done for at least 24 hours !!
The memtest was fine:
(https://i.ibb.co/4ZBv2ZBD/IMG-20260604-213731.jpg)
(https://i.ibb.co/Q3QsyMyB/IMG-20260604-064837.jpg)
So I think I found the culprit:
(https://i.ibb.co/sJ1KN6y5/Screenshot-2026-06-04-223259.png)
Well I guess it was the SSD all along. I didn't think a system drive failing would have such symptoms.
Sadly the story didn't end here:
* I tried to clone the drive onto a new drive which took way to long and didn't work at the end. The OS booted but couldn't find some files (don't ask me what exactly).
* So I took another road: Backup opnsense via UI and import in a freshly installed version. Should be straightforward, right? Wrong
* I didn't ever bothering migrating from "deprecated" features as long as they worked or I get the message "This will no longer work in the next version". It's just my homelab after all, if it's working it's good enough.
* Today I learned ISC DCHP is so hard deprecated that it is not even backed up when you create a backup. It backups the config values but does not backup (now a plugin) ISC from what I could tell.
* I booted the original drive (and hoped it would survive as long as necessary (spoiler: it did not - had to restart multiple times) and migrated everything to Kea according to this guide (https://homenetworkguy.com/how-to/migrate-from-isc-dhcp-to-dnsmasq-or-kea-dhcp-in-opnsense/)
* Now everything is working as before and I hope the device survives longer this time
Afterthought: Shouldn't opnsense warn me about bad smart values? Is there any way to enable this?
Thanks for everyone helping me, bringing ideas to the table and for your valuable time.
There is a SMARTS plugin you can install and run tests from OPNsense directly.
It comes with a widget to show the health of the latest test.
Regards,
S.
Hah! Called the failing SSD!
Quote from: meikel on June 05, 2026, 08:31:06 AM* Today I learned ISC DCHP is so hard deprecated that it is not even backed up when you create a backup. It backups the config values but does not backup (now a plugin) ISC from what I could tell.
I just checked a backup that I made last week, when I still had old(disabled) ISC DHCP config left behind after transitioning to Kea. That config is included in the backup XML, under sections "dhcpd" and "dhcpdv6".
AFAIK, you do need to install the plugin (on a fresh installation) before you can restore the config for it, though.
That Sophos UTM firewall appliance is at least 10 years old, where did you buy it?
I'll assume the Intel SSD was new, as it only ran for 479 days and was turned on 69 times.
What's the state of the other Intel SSD in the RAID1 configuration?
Quote from: vpx on June 05, 2026, 03:05:18 PMThat Sophos UTM firewall appliance is at least 10 years old, where did you buy it?
I see no problem with that as long as it works. Its a very solid machine built with hq components.
Quote from: vpx on June 05, 2026, 03:05:18 PMWhat's the state of the other Intel SSD in the RAID1 configuration?
There is no other drive and there is no RAID. He has single SSD.
Quote from: meikel on June 05, 2026, 08:31:06 AMThe memtest was fine
I am a fan of 24 hour tests, but OK :)
QuoteSo I think I found the culprit:
(https://i.ibb.co/sJ1KN6y5/Screenshot-2026-06-04-223259.png)
Well I guess it was the SSD all along. I didn't think a system drive failing would have such symptoms.
Told you! :)
It seems to be one of those models that I call "Fake Intel SSD" : https://duckduckgo.com/?q=ssdsc2bw180h6&ia=web
No wonder it failed despite being a MLC model : They had no additional spare space like the Intel Enterprise models had at the time :)
(Don't get me started on everything else that's wrong with them... LOL!)QuoteSadly the story didn't end here:
* I tried to clone the drive onto a new drive which took way to long and didn't work at the end. The OS booted but couldn't find some files (don't ask me what exactly).
That's to be expected : Defect drives are very often impossible to clone!
Quote* So I took another road: Backup opnsense via UI and import in a freshly installed version. Should be straightforward, right? Wrong
I am surprised you got that far considering the issues!
You would also risk having a corrupt config.xml that way IMO.
Quote* Today I learned ISC DCHP is so hard deprecated that it is not even backed up when you create a backup.
It backups the config values but does not backup (now a plugin) ISC from what I could tell.
If you are talking about
'Static DHCP Mappings based on the MAC Address' then you need to Export those into .CSV files via the webGUI and Import them into KEA or DNSmasqd ;)
Quote* Now everything is working as before and I hope the device survives longer this time
What's the storage this time ?
So I think I found the culprit:
(https://i.ibb.co/sJ1KN6y5/Screenshot-2026-06-04-223259.png)[/quote]
These results are quite confusing. S.M.A.R.T parameters are clearly indicating that SSD is pretty much dead. Confusing part is the fact that machine still crashed running Debian live. This indicates another issue beside dead drive.
@nero355
It's an SAMSUNG EVO 850 250gb - it's used but when this issue happens again I'm fine with downtime. I may move to a raid1 at some point but it's not worth the hustle right now as there is no dedicated space for another ssd.
Once I knew what the issue was (very easy to find out if you don't blindfold yourself) it's quite easy to fix with a backup. Except the dhcp migration part.
Quote from: meikel on June 05, 2026, 06:31:57 PMIt's an SAMSUNG EVO 850 250gb - it's used but when this issue happens again I'm fine with downtime. I may move to a raid1 at some point but it's not worth the hustle right now as there is no dedicated space for another ssd.
I have seen the Evo 830 or 840 being written to death way past what the specifications officially stated so I hope you never have to replace it ever again :)
It's not the greatest thing, but I think it should last a long time!
Quote from: Nullman on June 05, 2026, 05:58:58 PMThese results are quite confusing. S.M.A.R.T parameters are clearly indicating that SSD is pretty much dead. Confusing part is the fact that machine still crashed running Debian live. This indicates another issue beside dead drive.
IMO it's 100% to be expected and probably the cause of all the issues...