Opnsense 24.1.7_4 GUI hangs after some time

[DocGonzo74]

I have an issue where, after anywhere from 5 to 25 minutes, my Opnsense GUI will hang and become unresponsive.  When I try to issue a reboot from the CLI, the reboot hangs after it shuts down services.  When I reboot, the GUI is responsive again for some time. 

This is a baremetal server, not VM.   Currently running a memtest (2/4 passes so far came back clear). 

I'm running out of ideas on what to try to fix this.

System Firmware Reporter shows the crash logs.  At the end of /var/crash/textdump.tar.1, I get this output:

Fatal trap 12: page fault while in kernel mode
cpuid = 10; apic id = 0a
fault virtual address   = 0x0
fault code      = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff8239957f
stack pointer           = 0x28:0xfffffe00e1abc600
frame pointer           = 0x28:0xfffffe00e1abc660
code segment      = base 0x0, limit 0xfffff, type 0x1b
         = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process      = 0 (if_io_tqg_10)
trap number      = 12
panic: page fault
cpuid = 10
time = 1711851180
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00e1abc3c0
vpanic() at vpanic+0x151/frame 0xfffffe00e1abc410
panic() at panic+0x43/frame 0xfffffe00e1abc470
trap_fatal() at trap_fatal+0x387/frame 0xfffffe00e1abc4d0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00e1abc530
calltrap() at calltrap+0x8/frame 0xfffffe00e1abc530
--- trap 0xc, rip = 0xffffffff8239957f, rsp = 0xfffffe00e1abc600, rbp = 0xfffffe00e1abc660 ---
pf_test_state_udp() at pf_test_state_udp+0x28f/frame 0xfffffe00e1abc660
pf_test() at pf_test+0xc57/frame 0xfffffe00e1abc7d0
pf_check_in() at pf_check_in+0x25/frame 0xfffffe00e1abc7f0
pfil_run_hooks() at pfil_run_hooks+0x97/frame 0xfffffe00e1abc830
ip_tryforward() at ip_tryforward+0x181/frame 0xfffffe00e1abc8f0
ip_input() at ip_input+0x724/frame 0xfffffe00e1abc980
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00e1abc9d0
ether_demux() at ether_demux+0x159/frame 0xfffffe00e1abca00
ng_ether_rcv_upper() at ng_ether_rcv_upper+0x8c/frame 0xfffffe00e1abca20
ng_apply_item() at ng_apply_item+0x2bf/frame 0xfffffe00e1abcab0
ng_snd_item() at ng_snd_item+0x28e/frame 0xfffffe00e1abcaf0
ng_apply_item() at ng_apply_item+0x2bf/frame 0xfffffe00e1abcb80
ng_snd_item() at ng_snd_item+0x28e/frame 0xfffffe00e1abcbc0
ng_ether_input() at ng_ether_input+0x4c/frame 0xfffffe00e1abcbf0
ether_nh_input() at ether_nh_input+0x1f2/frame 0xfffffe00e1abcc50
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00e1abcca0
ether_input() at ether_input+0x69/frame 0xfffffe00e1abcd00
iflib_rxeof() at iflib_rxeof+0xbcb/frame 0xfffffe00e1abce00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe00e1abce40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x15d/frame 0xfffffe00e1abcec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc3/frame 0xfffffe00e1abcef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00e1abcf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e1abcf30
--- trap 0x7eb4a043, rip = 0x9566c7f262db2889, rsp = 0xc14793d336fa7ca8, rbp = 0xb3244499abe27515 ---
KDB: enter: panic
panic.txt0600001214602143254  7132 ustarrootwheelpage faultversion.txt0600007414602143254  7535 ustarrootwheelFreeBSD 13.2-RELEASE-p7 stable/23.7-n254871-d5ec322cffc SMP

[meyergru]

Damn, my crystal ball is broken...

Care to share what hardware this is on? Just because it sure looks like a hardware problem (i.e. memory problem, CPU microcodes not current, defective or problematic NIC, abysmal CPU cooling or something to that extent).

[DocGonzo74]

I ran a memtest and all 4 passes came back clean.

Memory is 64GB (4 x 16) of Patriot DDR4 memory 2400. 

CPU is an i9-9900K with a passive cooling tower cooler.  I ran this as a gaming rig for a while and repurposed it. Never had an overheating problem on this system.

Found SMBIOS entry point in EFI, reading table from /dev/mem.
SMBIOS 3.2 present.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
        Manufacturer: ASUSTeK COMPUTER INC.
        Product Name: ROG STRIX Z390-E GAMING
        Version: Rev 1.xx
        Serial Number: 190449688100832
        Asset Tag: Default string


NIC info:
root@PRIMARY:/home/RitchieHome #  sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.em.0.%desc: Intel(R) I219-V CNP(7)
dev.igb.7.%desc: Intel(R) PRO/1000 ET 82576 (Quad Copper)
dev.igb.6.%desc: Intel(R) PRO/1000 ET 82576 (Quad Copper)
dev.igb.5.%desc: Intel(R) PRO/1000 ET 82576 (Quad Copper)
dev.igb.4.%desc: Intel(R) PRO/1000 ET 82576 (Quad Copper)
dev.igb.3.%desc: Intel(R) PRO/1000 ET 82576 (Quad Copper)
dev.igb.2.%desc: Intel(R) PRO/1000 ET 82576 (Quad Copper)
dev.igb.1.%desc: Intel(R) PRO/1000 ET 82576 (Quad Copper)
dev.igb.0.%desc: Intel(R) PRO/1000 ET 82576 (Quad Copper)


The system works fine save for the webgui.  Once it crashes, I have to reboot to get it to come back. 
configctl webgui restart renew doesn't work when it crashes.

[meyergru]

Guessing from your dmidecode output, your BIOS seems quite old (Rev. 1.xx). There is a new BIOS 2101, which:

"1, Improve system stability"

Being from march of 2024, it most probably also has more current CPU microcode updates, too. There are also newer Intel ME updates for that board as well. I would try to apply those.

Also, in the tutorials section, there is a posting about how to update CPU microcodes even in the absence of available vendor updates. BTW: Windows has these updates built in, FreeBSD has not per default, and you used that rig before as a gaming platform, so go figure.

That being said, CPU thermal paste does dry out, too...

[DocGonzo74]

I updated the BIOS to 2801 a few weeks back trying to solve the issue.

Weirdness, though.

I did the latest update after a few tries, to 24.1.8.  After that update, my system has been responsive for 24 hours.  I re-installed the rest of my RAM and have restarted to see if the system remains stable. 

I can't imagine something in that new build fixed my issue, but I've seen much more weirdness in the past.

Fingers crossed.