"
After discussing this issue with a number of friends and *nix users, I'm thinking this is a potential hardware bug in the CPU vs. the code in the FreeBSD kernel that gets tripped from time to time. One item supporting this view is that when I originally tried to install PFSense on this machine, it failed at kernel load. It would just hang. That is the reason I went with OPNSense, it would actually boot and allow me to install. I've had similar issues in the past with AMD CPUs vs. Intel. Usually highly odd, hard to reproduce and unresolvable. At this point, I think I will try to move to other hardware. Since I like the OPNSense experience so far, I will stick with OPNSense. Hopefully this information might be helpful to someone else who encounters similar issues running OPNSense on older AMD CPUs.
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
Pages1
#1
General Discussion / Re: Unusual Behavior: Hardware Issue or Intrusion?
October 13, 2020, 06:56:24 PM #2
General Discussion / Unusual Behavior: Hardware Issue or Intrusion?
October 02, 2020, 07:21:38 PM
Starting in this section of the forum, since I'm not sure where this question fit.I have been running OPNSense as a home router for an at&t gigabit connection for a about 10 months, starting with the 19.x series. I did not keep up with updates until about August just after the the 19.x series was deprecated. Within the first week of August (still on 19x), I woke up to no internet access in my house. I'd recently built a separate DHCP/DNS/NTP server on Debian 10 and assumed the issue was there. After checking and seeing that name resolution was failing against the internal caching server, I tried the Google public DNS server and was getting timeouts there too. That's when I turned my attention to the OPNSense router.
When I tried to hit it via the web UI and SSH it failed, and was not pingable. I went to the basement and saw that the OPNSense "server" (It's a 10 year old minitower PC) was powered off. I turned it back on and everything started working. Since I'm not that familiar with BSD Unix, I probed around a bit to figure out how to read syslog and noticed there was nothing in my syslog for a few days before the shutdown until the entries that started up when I powered the system on. There was no power outage, so I was suspicious. I don't have the web UI enabled for external access, so no one should be able to get in from the WAN interface in any fashion.
I ran the audit and the updates and got the system up to 20.7.x after bringing it back up just in case there were any exploits (I didn't find any for 19.x in my searches). Hoping this shutdown was just a fluke, and that the missing syslog data was just some artifact of the fluke, I went about business as usual. About two or three days later, it was shutdown again overnight. In both cases internet access was good until about midnight (based on my usage) and dead in the morning. This time I was a bit more concerned. Once again, syslog data and the logs in the UI seemed to be missing time frames. Since I was concerned that the box might be getting some kind of attack, I set up a GNU screen session from another box to watch syslog live. That way, if the box is shutdown again, I might see some info in the log before it's removed. I set that up on the box on 9/16 and there wasn't a single new event in syslog (I was following the log with clog) before the next thing happened.
The system stayed up for about 13 days with no shutdown. This morning I reattached to the screen session and noticed the machine rebooted about three days ago. This happened some time in the morning on 9/29. Looking with clog just now, this is where the transition happened:
Sep 16 09:34:54 ginger kernel: GEOM_MIRROR: Device OPNsense: rebuilding provider ada0 finished.
Sep 29 09:20:22 ginger syslogd: kernel boot file is /boot/kernel/kernel
Sep 29 09:20:22 ginger kernel: ---<<BOOT>>---
Sep 29 09:20:22 ginger kernel: Copyright (c) 2013-2019 The HardenedBSD Project.
Sep 29 09:20:22 ginger kernel: Copyright (c) 1992-2019 The FreeBSD Project.
Sep 29 09:20:22 ginger kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
I am about 50/50 on this being a hardware issue rather than a compromised box. I know I did not reboot the machine on the 29th, so I have to conclude that it's more likely to be hardware failure or some bug in the CPU that is causing issues. Here are my hardware specs:
AMD Athlon(tm) 64 X2 Dual Core Processor 4000+ (2 cores)
4GB of RAM
Marvell Yukon 88E8056 Gigabit Ethernet (on mobo)
Intel Gigabit Ethernet
Current Version after today's update (was on 20.7.2 up until today):
OPNsense 20.7.3-amd64
FreeBSD 12.1-RELEASE-p10-HBSD
OpenSSL 1.1.1g 21 Apr 2020
I searched the forums for people experiencing random shutdowns and reboots and came up empty. Has anyone else experienced these issues? I selected OPNSense since it's BSD and built with security in mind which is why I'm leaning towards this being hardware failure rather than a compromised machine. Any recommendations? Outside of these shutdowns and reboots, OPNSense has been solid as a rock and the increase in bandwidth for my gigabit connection (got it last year) vs. my old WRTG54 router with ddwrt is awesome. Keep in mind, I'm using old hardware because this is just for home use. I'm considering a fanless PC in the $200-300 range if this turns out to be hardware failure since it would save me a bit on electricity. If I need to provide more info, please let me know.
Thanks,
eno2001
When I tried to hit it via the web UI and SSH it failed, and was not pingable. I went to the basement and saw that the OPNSense "server" (It's a 10 year old minitower PC) was powered off. I turned it back on and everything started working. Since I'm not that familiar with BSD Unix, I probed around a bit to figure out how to read syslog and noticed there was nothing in my syslog for a few days before the shutdown until the entries that started up when I powered the system on. There was no power outage, so I was suspicious. I don't have the web UI enabled for external access, so no one should be able to get in from the WAN interface in any fashion.
I ran the audit and the updates and got the system up to 20.7.x after bringing it back up just in case there were any exploits (I didn't find any for 19.x in my searches). Hoping this shutdown was just a fluke, and that the missing syslog data was just some artifact of the fluke, I went about business as usual. About two or three days later, it was shutdown again overnight. In both cases internet access was good until about midnight (based on my usage) and dead in the morning. This time I was a bit more concerned. Once again, syslog data and the logs in the UI seemed to be missing time frames. Since I was concerned that the box might be getting some kind of attack, I set up a GNU screen session from another box to watch syslog live. That way, if the box is shutdown again, I might see some info in the log before it's removed. I set that up on the box on 9/16 and there wasn't a single new event in syslog (I was following the log with clog) before the next thing happened.
The system stayed up for about 13 days with no shutdown. This morning I reattached to the screen session and noticed the machine rebooted about three days ago. This happened some time in the morning on 9/29. Looking with clog just now, this is where the transition happened:
Sep 16 09:34:54 ginger kernel: GEOM_MIRROR: Device OPNsense: rebuilding provider ada0 finished.
Sep 29 09:20:22 ginger syslogd: kernel boot file is /boot/kernel/kernel
Sep 29 09:20:22 ginger kernel: ---<<BOOT>>---
Sep 29 09:20:22 ginger kernel: Copyright (c) 2013-2019 The HardenedBSD Project.
Sep 29 09:20:22 ginger kernel: Copyright (c) 1992-2019 The FreeBSD Project.
Sep 29 09:20:22 ginger kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
I am about 50/50 on this being a hardware issue rather than a compromised box. I know I did not reboot the machine on the 29th, so I have to conclude that it's more likely to be hardware failure or some bug in the CPU that is causing issues. Here are my hardware specs:
AMD Athlon(tm) 64 X2 Dual Core Processor 4000+ (2 cores)
4GB of RAM
Marvell Yukon 88E8056 Gigabit Ethernet (on mobo)
Intel Gigabit Ethernet
Current Version after today's update (was on 20.7.2 up until today):
OPNsense 20.7.3-amd64
FreeBSD 12.1-RELEASE-p10-HBSD
OpenSSL 1.1.1g 21 Apr 2020
I searched the forums for people experiencing random shutdowns and reboots and came up empty. Has anyone else experienced these issues? I selected OPNSense since it's BSD and built with security in mind which is why I'm leaning towards this being hardware failure rather than a compromised machine. Any recommendations? Outside of these shutdowns and reboots, OPNSense has been solid as a rock and the increase in bandwidth for my gigabit connection (got it last year) vs. my old WRTG54 router with ddwrt is awesome. Keep in mind, I'm using old hardware because this is just for home use. I'm considering a fanless PC in the $200-300 range if this turns out to be hardware failure since it would save me a bit on electricity. If I need to provide more info, please let me know.
Thanks,
eno2001
Pages1
