OPNsense frequent crashes

[restrepo]

Hello,

I installed OPNsense in a mini PC (AWOW Model AK34 - Intel Celeron N3450, 6Gb DDR4, 128GB SSD) recently.

I've noticed that the firewall continuously crashes, at least once or twice daily, sometimes even more.

I have no clue as to what may be happening, reason for me to reach out to more hardcore geeks.  I wonder if it is a hardware incompatibility.

Attached is the log of the latest crash.

[opnfwb]

Have any diagnostics been performed on the hardware yet? If not, start with the basics like updating to the latest BIOS firmware, then do a full memtest (this will take a long time) and run some CPU stress tests. The memtest and CPU stress tests can be done with a bootable linux ISO such as Ubuntu or Fedora.

Other things to check would be the power supply and the CPU temperature. Both can be a common source of instability.

[restrepo]

Thank you opnfwb.  Yes, agreed. I should have done that to begin with. I normally use the bootable Knoppix for those cases, which I will do in my case. Sorry for popping up as a shoemaker that fails to wear shoes.   

[opnfwb]

No worries, let us know how you get on with the diagnostics.

Something else to add, I find that with these integrated SoC based X86 systems they tend to have components that aren't needed at all on a firewall. While you're in there checking the BIOS and potentially updating it, I'd also go through and disable all the onboard devices that you don't need. The sound card, LPT ports, serial ports (if unused), etc. can easily be disabled if you're just using VGA or HDMI output to a monitor.

Also, if the BIOS has a power management setting, make sure that it is set to "performance" or if possible, disable it and just let the system run at maximum speed.

[restrepo]

Hello opnfwb,
Thanks for your tips.  I went into BIOS and there is nothing to be turned off.  Cheap little mini PC. 
I have been having unusual difficulties to get this mini PC to boot under Knoppix.  Tried with StressLinux as well with no success.       
I am getting pretty annoyed by this darn little thing.  Not giving up as of yet. 

[opnfwb]

It may be UEFI only. Try GPT volumes and UEFI booting and see if you can get the diagnostics to fire up?

[restrepo]

Yes, I believe it is UEFI only, and I have been using UEFI since the beginning.  Can't get Knoppix to boot. It hangs during boot.  I was able to get MemTest86 to start, I left the PC overnight.  This morning, all tests passed. I have started a second round of tests which may take longer, as it uses one of the 4 CPUs on a round-robin basis.  Last night's test was with the 4 CPUs simultaneously and it took about 10hs to complete.

Coincidentally, MemTest86 did not crash during the testing.   

[franco]

It does look like a BIOS/hardware incompatibility vs. FreeBSD 12.1... nothing specific to operation of OPNsense itself.

Here might be clues or similar reports regarding the hardware: https://bugs.freebsd.org/bugzilla/


Sorry,
Franco

[franco]

PS: Here is one diagnosing a very old crash with a similar conclusion https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=1171

[restrepo]

Thanks Franco. I ran the memory test multiple times,  all PASSed... Still pending other HW tests are pending.
One thing I did not like is that I had to disable a disk test during initial install because the cheap Chinese SSD did not support it.
I am seriously considering to return this machine to Amazon and replacing it with a Protectli Vault or Qotom Vault. Any thoughts are appreciated.

[franco]

Memtest might be trivial enough to not trigger the context switch amount causing this problem or that simply FreeBSD reacts badly to something the BIOS does in the particular "error" case.

All the same in the end, although FreeBSD 13 might be better, but that's still 8 months away.


Cheers,
Franco

[restrepo]

Agreed. Since the next version is still so far out, without any guarantees of resolving the problem, it reinforces the idea of replacing the hardware.

[franco]

I concur if that is possible. There is a lot of good hardware out there.


Cheers,
Franco

[restrepo]

Hi Franco, yes, I finally returned that darn unit yesterday.  Life is too short to deal with cheap, unreliable hardware. I am currently looking at a few Sophos firewall appliances on eBay to replace that mini PC.  Is there some others that I should be looking for?  It will be used in my own Home/Office production environment. I am open and appreciate suggestions.