OPNsense Forum

Archive => 20.1 Legacy Series => Topic started by: ooker on May 27, 2020, 11:32:28 pm

Title: RESOLVED: Periodic Reboot--Fatal trap 12-**only with traffic shaper enabled**
Post by: ooker on May 27, 2020, 11:32:28 pm
I'd appreciate some help with this please.
I recently started using traffic shaping and I'm getting a periodic fatal trap and reboot of my OPNSense box (see below).  It's running on a Protectli box with Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz (4 cores)

Does anyone have any suggestions for tracking this down and resolving it?

This seems to occur only when traffic shaper rules are enabled.  As a test, I disabled the traffic shaper rules and the firewall has run for 14 hours without rebooting. Enabling traffic shaper rules results in a reboot within 30m-3 hours.

Here is the version info off the dashboard:
OPNsense 20.1.7-amd64
FreeBSD 11.2-RELEASE-p20-HBSD
OpenSSL 1.1.1g 21 Apr 2020
-----------------------------------
Fatal trap 12: page fault while in kernel mode

cpuid = 2; apic id = 02
fault virtual address   = 0xffffffff00040061
fault code      = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80d9c2e3
stack pointer           = 0x28:0xfffffe0232b352a0
frame pointer           = 0x28:0xfffffe0232b352a0

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xffffffff00040061
fault code      = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80daa0b3
stack pointer           = 0x28:0xfffffe0232dee7a0
frame pointer           = 0x28:0xfffffe0232dee7f0
code segment      = base 0x0, limit 0xfffff, type 0x1b
         = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process      = 0 (dummynet)
version.txt0600007013663554527  7550 ustarrootwheelFreeBSD 11.2-RELEASE-p20-HBSD  07ef86ce9ca(stable/20.1)

UPDATE: Resolved.
After backing up the configuration, then re-installing OPNsense and restoring the configuration, my OPNsense box has not encountered a crash/random reboot with traffic shaping enabled. The re-install installed 20.1 then upgraded to 20.1.7. I've been running for over 14 hours without a crash.

Note: Before re-installing, I had performed a Firmware audit several times and everything checked out, but I was still encountering periodic crashes and reboots only when traffic shaping was enabled. 
I also tried re-installing key packages including opnsense, but I continued to encounter random crashes and reboots only when traffic shaping was enabled.
Conclusion: Successful completion of firmware audit does not provide a comprehensive check of the integrity of the OPNSense install.
Title: Re: Help Wanted: Periodic Reboot--Fatal trap 12: page fault while in kernel mode
Post by: mimugmail on May 28, 2020, 07:11:53 am
Do you use latest 20.1.7?
Title: Re: Help Wanted: Periodic Reboot--Fatal trap 12: page fault while in kernel mode
Post by: ooker on May 28, 2020, 04:21:58 pm
Here is the version info off the dashboard:
OPNsense 20.1.7-amd64
FreeBSD 11.2-RELEASE-p20-HBSD
OpenSSL 1.1.1g 21 Apr 2020

I believe this is related to traffic shaping.  If I disable all the shaper rules, I don't see the periodic reboots (uptime of the firewall is 14hours and counting). If I enable the traffic shaper rules, I'll get the periodic reboots happening again (uptime <2 hours before a reboot).
Title: Re: Help Wanted: Periodic Reboot--Fatal trap 12-**only with traffic shaper enabled**
Post by: ooker on May 29, 2020, 12:00:54 am
Is anyone actively using traffic shaping in OPNsense 20.1.7-amd64, and NOT getting random reboots?
Title: Re: Help Wanted: Periodic Reboot--Fatal trap 12-**only with traffic shaper enabled**
Post by: gpb on May 29, 2020, 02:53:31 am
I use traffic shaper but only as a throttle to help with buffer bloat.  Nothing fancy.  No reboots.
Title: Re: Help Wanted: Periodic Reboot--Fatal trap 12-**only with traffic shaper enabled**
Post by: mimugmail on May 29, 2020, 08:54:11 am
My shaper also runs fine since update to 20.1.7 (5 days ago) ...
Title: Re: Help Wanted: Periodic Reboot--Fatal trap 12-**only with traffic shaper enabled**
Post by: ooker on May 31, 2020, 05:20:45 pm
For people who are running traffic shaping without periodic crashes/reboots, what hardware are you running OPNsense on?  I am seeing the issues on an I5-7200U
Title: Re: Help Wanted: Periodic Reboot--Fatal trap 12-**only with traffic shaper enabled**
Post by: crd on June 01, 2020, 01:10:14 am
Do you have PIE active queue management enabled on any of your pipes or queues? I have experienced stability problems (seemed to be a complete lockup, but not a reboot) with PIE enabled before, but everything works fine when I disable PIE and use CoDel.
Title: Re: Help Wanted: Periodic Reboot--Fatal trap 12-**only with traffic shaper enabled**
Post by: ooker on June 01, 2020, 06:36:25 pm
I am only using FlowQueue-CoDel.
On my i5-7200U hardware, I performed a health audit and even reinstalled key packages (like the core OPNsense package), but I'm still getting random reboots.

As a test, I installed OPNsense on a box with a Celeron J3455 (also with Intel NICs), and restored the config from the i5-7200U box.  So far, it has run for 14 hours without a random reboot. 

I am currently running memtest on the i5-7200U box--so far no errors.  Then I'll try a fresh install of OPNSense on that box & restore the config to that box as a test.