Unusual Behavior: Hardware Issue or Intrusion?

[eno2001]

Starting in this section of the forum, since I'm not sure where this question fit.I have been running OPNSense as a home router for an at&t gigabit connection for a about 10 months, starting with the 19.x series. I did not keep up with updates until about August just after the the 19.x series was deprecated. Within the first week of August (still on 19x), I woke up to no internet access in my house. I'd recently built a separate DHCP/DNS/NTP server on Debian 10 and assumed the issue was there. After checking and seeing that name resolution was failing against the internal caching server, I tried the Google public DNS server and was getting timeouts there too. That's when I turned my attention to the OPNSense router.

When I tried to hit it via the web UI and SSH it failed, and was not pingable. I went to the basement and saw that the OPNSense "server" (It's a 10 year old minitower PC) was powered off. I turned it back on and everything started working. Since I'm not that familiar with BSD Unix, I probed around a bit to figure out how to read syslog and noticed there was nothing in my syslog for a few days before the shutdown until the entries that started up when I powered the system on. There was no power outage, so I was suspicious. I don't have the web UI enabled for external access, so no one should be able to get in from the WAN interface in any fashion.

I ran the audit and the updates and got the system up to 20.7.x after bringing it back up just in case there were any exploits (I didn't find any for 19.x in my searches). Hoping this shutdown was just a fluke, and that the missing syslog data was just some artifact of the fluke, I went about business as usual. About two or three days later, it was shutdown again overnight. In both cases internet access was good until about midnight (based on my usage) and dead in the morning. This time I was a bit more concerned. Once again, syslog data and the logs in the UI seemed to be missing time frames. Since I was concerned that the box might be getting some kind of attack, I set up a GNU screen session from another box to watch syslog live. That way, if the box is shutdown again, I might see some info in the log before it's removed. I set that up on the box on 9/16 and there wasn't a single new event in syslog (I was following the log with clog) before the next thing happened.

The system stayed up for about 13 days with no shutdown. This morning I reattached to the screen session and noticed the machine rebooted about three days ago. This happened some time in the morning on 9/29. Looking with clog just now, this is where the transition happened:

Sep 16 09:34:54 ginger kernel: GEOM_MIRROR: Device OPNsense: rebuilding provider ada0 finished.
Sep 29 09:20:22 ginger syslogd: kernel boot file is /boot/kernel/kernel
Sep 29 09:20:22 ginger kernel: ---<<BOOT>>---
Sep 29 09:20:22 ginger kernel: Copyright (c) 2013-2019 The HardenedBSD Project.
Sep 29 09:20:22 ginger kernel: Copyright (c) 1992-2019 The FreeBSD Project.
Sep 29 09:20:22 ginger kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994

I am about 50/50 on this being a hardware issue rather than a compromised box. I know I did not reboot the machine on the 29th, so I have to conclude that it's more likely to be hardware failure or some bug in the CPU that is causing issues. Here are my hardware specs:

AMD Athlon(tm) 64 X2 Dual Core Processor 4000+ (2 cores)
4GB of RAM

Marvell Yukon 88E8056 Gigabit Ethernet (on mobo)
Intel Gigabit Ethernet

Current Version after today's update (was on 20.7.2 up until today):

OPNsense 20.7.3-amd64
FreeBSD 12.1-RELEASE-p10-HBSD
OpenSSL 1.1.1g 21 Apr 2020

I searched the forums for people experiencing random shutdowns and reboots and came up empty. Has anyone else experienced these issues? I selected OPNSense since it's BSD and built with security in mind which is why I'm leaning towards this being hardware failure rather than a compromised machine. Any recommendations? Outside of these shutdowns and reboots, OPNSense has been solid as a rock and the increase in bandwidth for my gigabit connection (got it last year) vs. my old WRTG54 router with ddwrt is awesome. Keep in mind, I'm using old hardware because this is just for home use. I'm considering a fanless PC in the $200-300 range if this turns out to be hardware failure since it would save me a bit on electricity. If I need to provide more info, please let me know.

Thanks,
eno2001

[eno2001]

After discussing this issue with a number of friends and *nix users, I'm thinking this is a potential hardware bug in the CPU vs. the code in the FreeBSD kernel that gets tripped from time to time. One item supporting this view is that when I originally tried to install PFSense on this machine, it failed at kernel load. It would just hang. That is the reason I went with OPNSense, it would actually boot and allow me to install.  I've had similar issues in the past with AMD CPUs vs. Intel. Usually highly odd, hard to reproduce and unresolvable. At this point, I think I will try to move to other hardware. Since I like the OPNSense experience so far, I will stick with OPNSense. Hopefully this information might be helpful to someone else who encounters similar issues running OPNSense on older AMD CPUs.

[nycaleksey]

I had similar symptoms and it turned out my firewall was turning itself off due to a cooling issue and CPU overheating. There were log entries in the BIOS log, but nothing at the OS level. Once I corrected the airflow and the CPU stopped overheating the issue went away.

[curto]

I am getting something similar happening.

Full Story

Purchased 3 x Identical QOTOM mini PC - Intel i5 - added brand new identical Crucial RAM to each machine and identical Samsung 840 Pro 128 GB SSDs.

One machine was installed and configured for home use running untangle - this machine has been absolutely rock solid and stable - not a single issue. This is my home system.

the 2nd machine was due to go to a friends house with a gigabit internet connection running OpnSense 20.7 - we initially experienced what appeared to be random hangs/sleep issues where the box on a fairly predictable weekly cycle would power itself down (or at the very least not respond to anything and not allow any access, in/out or admin. Only a complete power off would resolve it.

I thought it might be a BIOS sleep issue so changed many settings (based on our firend google) for this hardware and then swapped our 3rd hot spare box in with all the BIOS changes.

Took the apparently failing box and left it on a test bench - powered on - but not connected to anything and it ran for two weeks - so did not appear to be a BIOS sleep issue for low load etc.

Ran the replacement box with a brand new install of OpnSense and restored settings and BIOS settings updated to remove all sleep options at my friends place and then same thing happened - this box once again shutdown after approximately 2 weeks.

Made a few more BIOS changes and also enabled remote syslogging - and have just had a restart over the weekend after 14 days - i am about to go through the logs to try and see what the syslog box saw at the time of the reboot.

Will report back

Craig

[curto]

Have just gone through the full firewall logs and there is nothing in there - here is a screenshot from on the box itself - the remote syslog host sees the same thing i.e. not entries logged of not prior to the kernel restarting.

I am thinking this might be something to do with a failing UPS (or a UPS that goes into test mode each week) as i noticed the interface on the Linux Syslog host lost connectivity at almost the same time (it is in a different cabinet and on a different UPS - but the main network switch is in that cabinet.)

Craig