Messages

Another one... odd if it were hardware issues across three separate sets of hardware, but maybe I'm just extraordinarily unlucky this time. I'll try swapping out the RAM and/or SSD and see if either of those resolves it.

Another set of hardware, this time some different things are crashing. Luckily so far, the services have recovered on their own and none have required a hard power cycle.
Got this crash dump most recently.

Ok.

So that's a fairly new chipset.  Even Intel itself only has windows drivers:
https://www.intel.com/content/www/us/en/products/sku/210599/intel-ethernet-controller-i226v/downloads.html

So if BSD does support them, it's on it's own within the kernel.  I would question this greatly.

Question what? BSD 13 has the igc driver for the I226.

Another one from pf.

Code Select Expand

/var/crash/textdump.tar.0:

ddb.txt06000014000014306365744  7106 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo>  show alllocks
No such command; use "help" to list available commands
db:1:lockinfo>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 2
dynamic pcpu = 0xfffffe009ea2bf40
curthread    = 0xfffffe00dba04020: pid 6 tid 100106 critnest 1 "pf purge"
curpcb       = 0xfffffe00dba04530
fpcurthread  = none
idlethread   = 0xfffffe0020765560: tid 100005 "idle: cpu2"
self         = 0xffffffff82c12000
curpmap      = 0xffffffff81ea0d38
tssp         = 0xffffffff82c12384
rsp0         = 0xfffffe00d9eb5000
kcr3         = 0x68567000
ucr3         = 0xffffffffffffffff
scr3         = 0x1ddb7c000
gs32p        = 0xffffffff82c12404
ldt          = 0xffffffff82c12444
tss          = 0xffffffff82c12434
curvnet      = 0xfffff800011ffc80
db:0:kdb.enter.default>  bt
Tracing pid 6 tid 100106 td 0xfffffe00dba04020
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00d9eb4c60
vpanic() at vpanic+0x1b0/frame 0xfffffe00d9eb4cb0
panic() at panic+0x43/frame 0xfffffe00d9eb4d10
trap_fatal() at trap_fatal+0x385/frame 0xfffffe00d9eb4d70
calltrap() at calltrap+0x8/frame 0xfffffe00d9eb4d70
--- trap 0x9, rip = 0xffffffff80ceca6c, rsp = 0xfffffe00d9eb4e40, rbp = 0xfffffe00d9eb4e40 ---
counter_u64_fetch() at counter_u64_fetch+0x4c/frame 0xfffffe00d9eb4e40
pf_state_expires() at pf_state_expires+0x90/frame 0xfffffe00d9eb4e70
pf_purge_expired_states() at pf_purge_expired_states+0xe5/frame 0xfffffe00d9eb4ec0
pf_purge_thread() at pf_purge_thread+0x13b/frame 0xfffffe00d9eb4ef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00d9eb4f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d9eb4f30
--- trap 0x2b4a000, rip = 0xffffffff80c313cf, rsp = 0, rbp = 0xffffffff81ea04e0 ---
mi_startup() at mi_startup+0xdf/frame 0xffffffff81ea04e0
proc0() at proc0/frame 0xffffffff81ef7dc8
_binary_elf_vdso_so_1_size() at 0x30000/frame 0xffffffff81354b2a

Hopefully I'm not spreading fud but didn't the I226-V NIC's have stability issues with FreeBSD?

I'm not aware of anything BSD-specific, but then I don't exactly follow networking development in BSD... I know the I225's first two revisions had issues in general, and have seen some reports, albeit far lesser in number, about the third revision, too, but nothing I226-specific so far.

Nope... had another crash dump not too long ago. At least this time it didn't freeze entirely and require a power cycle...

Spoke too soon - another freeze and this time it didn't output any video again. Reseated everything and verified temps all look okay.

I tried to see if 22.1 would have the issue, but the new hardware has Intel I226-V NICs and 22.1 would not detect them, so I just did a fresh re-install of 22.7, re-installed plugins, restored config, and updated to latest again. We'll see if that helps at all...

Well, I got the new box set up. This one actually still outputs via the HDMI port when it crashes, so I could finally figure out it was an SSD issue... seems like from time to time the SSD disappears and the system boots into the UEFI shell. If the SSD is overheating, could that explain the odd kernel panics from pfctl as well?

Have you swapped hardware yet?

Not yet, new box gets here in a couple days. Just figured I'd keep posting/submitting the dumps in the meantime just in case it is a software issue.

Another crash dump for pfctl... seems to be most consistently the thing causing the kernel panics.

Past few days have just had random crashing still... seems like it might just be that these changwang PCs are unreliable. I decided to try a Qotom box instead, so I'll transfer config to that when it gets here and we'll see.

Given the change in behavior, this is feeling more like potentially a hardware issue, but it's still not remotely clear.

To rule that out, are you able to go back to 22.1 and test?

Otherwise potentially check CPU temps, or setup alerts.
You could also, just for good measure, run a memtest on the box?

Historically, for me, it's rarely been memory issues however it WAS 1 out of the 99 times.  And that 1 time, drove me nuts in troubleshooting before I discovered the issue ;)

I can try a live disk of 22.1 to see, but I made some tweaks and it was running stable again so I turned IDS/IPS back on and it almost immediately locked up with no crash dump, same as before. Turned it back off and so far so good, but it's only been a couple of hours.

Did you have any hardware offloading enabled?  i.e. CRC, TSO, LRO or VLAN?

Nope, all disabled.

And I spoke too soon... another crash dump some time yesterday apparently. This time, however, the firewall rebooted itself instead of staying locked up until I power cycled it.

Caused by PHP this time, apparently?

You uploaded one of my crash dumps, not yours.