RESOLVED: Periodic Reboot--Fatal trap 12-**only with traffic shaper enabled**

[ooker]

I'd appreciate some help with this please.
I recently started using traffic shaping and I'm getting a periodic fatal trap and reboot of my OPNSense box (see below).  It's running on a Protectli box with Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz (4 cores)

Does anyone have any suggestions for tracking this down and resolving it?

This seems to occur only when traffic shaper rules are enabled.  As a test, I disabled the traffic shaper rules and the firewall has run for 14 hours without rebooting. Enabling traffic shaper rules results in a reboot within 30m-3 hours.

Here is the version info off the dashboard:
OPNsense 20.1.7-amd64
FreeBSD 11.2-RELEASE-p20-HBSD
OpenSSL 1.1.1g 21 Apr 2020
-----------------------------------
Fatal trap 12: page fault while in kernel mode

cpuid = 2; apic id = 02
fault virtual address   = 0xffffffff00040061
fault code      = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80d9c2e3
stack pointer           = 0x28:0xfffffe0232b352a0
frame pointer           = 0x28:0xfffffe0232b352a0

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xffffffff00040061
fault code      = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80daa0b3
stack pointer           = 0x28:0xfffffe0232dee7a0
frame pointer           = 0x28:0xfffffe0232dee7f0
code segment      = base 0x0, limit 0xfffff, type 0x1b
         = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process      = 0 (dummynet)
version.txt0600007013663554527  7550 ustarrootwheelFreeBSD 11.2-RELEASE-p20-HBSD  07ef86ce9ca(stable/20.1)

UPDATE: Resolved.
After backing up the configuration, then re-installing OPNsense and restoring the configuration, my OPNsense box has not encountered a crash/random reboot with traffic shaping enabled. The re-install installed 20.1 then upgraded to 20.1.7. I've been running for over 14 hours without a crash.

Note: Before re-installing, I had performed a Firmware audit several times and everything checked out, but I was still encountering periodic crashes and reboots only when traffic shaping was enabled. 
I also tried re-installing key packages including opnsense, but I continued to encounter random crashes and reboots only when traffic shaping was enabled.
Conclusion: Successful completion of firmware audit does not provide a comprehensive check of the integrity of the OPNSense install.

[mimugmail]

Do you use latest 20.1.7?

[ooker]

Here is the version info off the dashboard:
OPNsense 20.1.7-amd64
FreeBSD 11.2-RELEASE-p20-HBSD
OpenSSL 1.1.1g 21 Apr 2020

I believe this is related to traffic shaping.  If I disable all the shaper rules, I don't see the periodic reboots (uptime of the firewall is 14hours and counting). If I enable the traffic shaper rules, I'll get the periodic reboots happening again (uptime <2 hours before a reboot).

[ooker]

Is anyone actively using traffic shaping in OPNsense 20.1.7-amd64, and NOT getting random reboots?

[gpb]

I use traffic shaper but only as a throttle to help with buffer bloat.  Nothing fancy.  No reboots.

[mimugmail]

My shaper also runs fine since update to 20.1.7 (5 days ago) ...

[ooker]

For people who are running traffic shaping without periodic crashes/reboots, what hardware are you running OPNsense on?  I am seeing the issues on an I5-7200U

[crd]

Do you have PIE active queue management enabled on any of your pipes or queues? I have experienced stability problems (seemed to be a complete lockup, but not a reboot) with PIE enabled before, but everything works fine when I disable PIE and use CoDel.

[ooker]

I am only using FlowQueue-CoDel.
On my i5-7200U hardware, I performed a health audit and even reinstalled key packages (like the core OPNsense package), but I'm still getting random reboots.

As a test, I installed OPNsense on a box with a Celeron J3455 (also with Intel NICs), and restored the config from the i5-7200U box.  So far, it has run for 14 hours without a random reboot. 

I am currently running memtest on the i5-7200U box--so far no errors.  Then I'll try a fresh install of OPNSense on that box & restore the config to that box as a test.