Firewall Frequently Locking Up, Requiring Hard Reboot

Started by milkywaygoodfellas, August 14, 2022, 01:45:45 AM

Previous topic - Next topic
Past few days have just had random crashing still... seems like it might just be that these changwang PCs are unreliable. I decided to try a Qotom box instead, so I'll transfer config to that when it gets here and we'll see.

Another crash dump for pfctl... seems to be most consistently the thing causing the kernel panics.

Have you swapped hardware yet?
OPNsense 24.7.7 running on:
Dell Optiplex 3050
Intel I5-7600 @ 3.5Ghz (4 Cores)
Intel I350-T4 Nic
8G DDR4
256G SSD

Quote from: axsdenied on August 27, 2022, 08:09:31 PM
Have you swapped hardware yet?
Not yet, new box gets here in a couple days. Just figured I'd keep posting/submitting the dumps in the meantime just in case it is a software issue.

September 04, 2022, 07:27:09 PM #19 Last Edit: September 04, 2022, 08:08:23 PM by milkywaygoodfellas
Well, I got the new box set up. This one actually still outputs via the HDMI port when it crashes, so I could finally figure out it was an SSD issue... seems like from time to time the SSD disappears and the system boots into the UEFI shell. If the SSD is overheating, could that explain the odd kernel panics from pfctl as well?

Spoke too soon - another freeze and this time it didn't output any video again. Reseated everything and verified temps all look okay.

I tried to see if 22.1 would have the issue, but the new hardware has Intel I226-V NICs and 22.1 would not detect them, so I just did a fresh re-install of 22.7, re-installed plugins, restored config, and updated to latest again. We'll see if that helps at all...

Nope... had another crash dump not too long ago. At least this time it didn't freeze entirely and require a power cycle...

Hopefully I'm not spreading fud but didn't the I226-V NIC's have stability issues with FreeBSD?
OPNsense 24.7.7 running on:
Dell Optiplex 3050
Intel I5-7600 @ 3.5Ghz (4 Cores)
Intel I350-T4 Nic
8G DDR4
256G SSD

Quote from: axsdenied on September 06, 2022, 09:49:52 PM
Hopefully I'm not spreading fud but didn't the I226-V NIC's have stability issues with FreeBSD?
I'm not aware of anything BSD-specific, but then I don't exactly follow networking development in BSD... I know the I225's first two revisions had issues in general, and have seen some reports, albeit far lesser in number, about the third revision, too, but nothing I226-specific so far.

Another one from pf.
/var/crash/textdump.tar.0:

ddb.txt06000014000014306365744  7106 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo>  show alllocks
No such command; use "help" to list available commands
db:1:lockinfo>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 2
dynamic pcpu = 0xfffffe009ea2bf40
curthread    = 0xfffffe00dba04020: pid 6 tid 100106 critnest 1 "pf purge"
curpcb       = 0xfffffe00dba04530
fpcurthread  = none
idlethread   = 0xfffffe0020765560: tid 100005 "idle: cpu2"
self         = 0xffffffff82c12000
curpmap      = 0xffffffff81ea0d38
tssp         = 0xffffffff82c12384
rsp0         = 0xfffffe00d9eb5000
kcr3         = 0x68567000
ucr3         = 0xffffffffffffffff
scr3         = 0x1ddb7c000
gs32p        = 0xffffffff82c12404
ldt          = 0xffffffff82c12444
tss          = 0xffffffff82c12434
curvnet      = 0xfffff800011ffc80
db:0:kdb.enter.default>  bt
Tracing pid 6 tid 100106 td 0xfffffe00dba04020
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00d9eb4c60
vpanic() at vpanic+0x1b0/frame 0xfffffe00d9eb4cb0
panic() at panic+0x43/frame 0xfffffe00d9eb4d10
trap_fatal() at trap_fatal+0x385/frame 0xfffffe00d9eb4d70
calltrap() at calltrap+0x8/frame 0xfffffe00d9eb4d70
--- trap 0x9, rip = 0xffffffff80ceca6c, rsp = 0xfffffe00d9eb4e40, rbp = 0xfffffe00d9eb4e40 ---
counter_u64_fetch() at counter_u64_fetch+0x4c/frame 0xfffffe00d9eb4e40
pf_state_expires() at pf_state_expires+0x90/frame 0xfffffe00d9eb4e70
pf_purge_expired_states() at pf_purge_expired_states+0xe5/frame 0xfffffe00d9eb4ec0
pf_purge_thread() at pf_purge_thread+0x13b/frame 0xfffffe00d9eb4ef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00d9eb4f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d9eb4f30
--- trap 0x2b4a000, rip = 0xffffffff80c313cf, rsp = 0, rbp = 0xffffffff81ea04e0 ---
mi_startup() at mi_startup+0xdf/frame 0xffffffff81ea04e0
proc0() at proc0/frame 0xffffffff81ef7dc8
_binary_elf_vdso_so_1_size() at 0x30000/frame 0xffffffff81354b2a

So that's a fairly new chipset.  Even Intel itself only has windows drivers:
https://www.intel.com/content/www/us/en/products/sku/210599/intel-ethernet-controller-i226v/downloads.html

So if BSD does support them, it's on it's own within the kernel.  I would question this greatly.
OPNsense 24.7.7 running on:
Dell Optiplex 3050
Intel I5-7600 @ 3.5Ghz (4 Cores)
Intel I350-T4 Nic
8G DDR4
256G SSD

Quote from: axsdenied on September 10, 2022, 02:07:50 AM
So that's a fairly new chipset.  Even Intel itself only has windows drivers:
https://www.intel.com/content/www/us/en/products/sku/210599/intel-ethernet-controller-i226v/downloads.html

So if BSD does support them, it's on it's own within the kernel.  I would question this greatly.
Question what? BSD 13 has the igc driver for the I226.

BSD also has the realtek driver but it's not stable.
OPNsense 24.7.7 running on:
Dell Optiplex 3050
Intel I5-7600 @ 3.5Ghz (4 Cores)
Intel I350-T4 Nic
8G DDR4
256G SSD

September 11, 2022, 05:09:47 PM #28 Last Edit: September 11, 2022, 09:20:09 PM by milkywaygoodfellas
Ok.

Another set of hardware, this time some different things are crashing. Luckily so far, the services have recovered on their own and none have required a hard power cycle.
Got this crash dump most recently.