I'd love to, but I can't even SSH into it when it happens.
Looks like the panic was caused by "pfctl". You doing packet inspection of any kind? Perhaps chocking session states?
db:0:kdb.enter.default> show pcpucpuid = 0dynamic pcpu = 0xfc0f40curthread = 0xfffffe0138c28720: pid 3489 tid 102014 critnest 1 "pfctl"curpcb = 0xfffffe0138c28c30fpcurthread = 0xfffffe0138c28720: pid 3489 "pfctl"idlethread = 0xfffffe00207933a0: tid 100003 "idle: cpu0"self = 0xffffffff82c10000curpmap = 0xfffffe011668f518tssp = 0xffffffff82c10384rsp0 = 0xfffffe0118fea000kcr3 = 0x351ae2000ucr3 = 0x16fe6d000scr3 = 0x16fe6d000gs32p = 0xffffffff82c10404ldt = 0xffffffff82c10444tss = 0xffffffff82c10434curvnet = 0xfffff80001202dc0db:0:kdb.enter.default> btTracing pid 3489 tid 102014 td 0xfffffe0138c28720kdb_enter() at kdb_enter+0x37/frame 0xfffffe0118fe93c0vpanic() at vpanic+0x1b0/frame 0xfffffe0118fe9410panic() at panic+0x43/frame 0xfffffe0118fe9470trap_fatal() at trap_fatal+0x385/frame 0xfffffe0118fe94d0trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0118fe9530calltrap() at calltrap+0x8/frame 0xfffffe0118fe9530--- trap 0xc, rip = 0xffffffff80debe14, rsp = 0xfffffe0118fe9600, rbp = 0xfffffe0118fe9620 ---rn_walktree() at rn_walktree+0x64/frame 0xfffffe0118fe9620pfr_get_addrs() at pfr_get_addrs+0x219/frame 0xfffffe0118fe9680pfioctl() at pfioctl+0x23be/frame 0xfffffe0118fe9b50devfs_ioctl() at devfs_ioctl+0xc6/frame 0xfffffe0118fe9ba0vn_ioctl() at vn_ioctl+0x1a4/frame 0xfffffe0118fe9cb0devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe0118fe9cd0kern_ioctl() at kern_ioctl+0x25b/frame 0xfffffe0118fe9d40sys_ioctl() at sys_ioctl+0xf1/frame 0xfffffe0118fe9e00amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe0118fe9f30fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0118fe9f30--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8012446da, rsp = 0x7fffffffdc38, rbp = 0x7fffffffe0d0 ---
For readability:Code: [Select]db:0:kdb.enter.default> show pcpucpuid = 0dynamic pcpu = 0xfc0f40curthread = 0xfffffe0138c28720: pid 3489 tid 102014 critnest 1 "pfctl"curpcb = 0xfffffe0138c28c30fpcurthread = 0xfffffe0138c28720: pid 3489 "pfctl"idlethread = 0xfffffe00207933a0: tid 100003 "idle: cpu0"self = 0xffffffff82c10000curpmap = 0xfffffe011668f518tssp = 0xffffffff82c10384rsp0 = 0xfffffe0118fea000kcr3 = 0x351ae2000ucr3 = 0x16fe6d000scr3 = 0x16fe6d000gs32p = 0xffffffff82c10404ldt = 0xffffffff82c10444tss = 0xffffffff82c10434curvnet = 0xfffff80001202dc0db:0:kdb.enter.default> btTracing pid 3489 tid 102014 td 0xfffffe0138c28720kdb_enter() at kdb_enter+0x37/frame 0xfffffe0118fe93c0vpanic() at vpanic+0x1b0/frame 0xfffffe0118fe9410panic() at panic+0x43/frame 0xfffffe0118fe9470trap_fatal() at trap_fatal+0x385/frame 0xfffffe0118fe94d0trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0118fe9530calltrap() at calltrap+0x8/frame 0xfffffe0118fe9530--- trap 0xc, rip = 0xffffffff80debe14, rsp = 0xfffffe0118fe9600, rbp = 0xfffffe0118fe9620 ---rn_walktree() at rn_walktree+0x64/frame 0xfffffe0118fe9620pfr_get_addrs() at pfr_get_addrs+0x219/frame 0xfffffe0118fe9680pfioctl() at pfioctl+0x23be/frame 0xfffffe0118fe9b50devfs_ioctl() at devfs_ioctl+0xc6/frame 0xfffffe0118fe9ba0vn_ioctl() at vn_ioctl+0x1a4/frame 0xfffffe0118fe9cb0devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe0118fe9cd0kern_ioctl() at kern_ioctl+0x25b/frame 0xfffffe0118fe9d40sys_ioctl() at sys_ioctl+0xf1/frame 0xfffffe0118fe9e00amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe0118fe9f30fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0118fe9f30--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8012446da, rsp = 0x7fffffffdc38, rbp = 0x7fffffffe0d0 ---I haven't seen this before but if it doesn't happen on 22.1 it should be easy to find the bad commit.This is new for 22.7, right?Cheers,Franco
Did you have any hardware offloading enabled? i.e. CRC, TSO, LRO or VLAN?
Given the change in behavior, this is feeling more like potentially a hardware issue, but it's still not remotely clear.To rule that out, are you able to go back to 22.1 and test?Otherwise potentially check CPU temps, or setup alerts.You could also, just for good measure, run a memtest on the box?Historically, for me, it's rarely been memory issues however it WAS 1 out of the 99 times. And that 1 time, drove me nuts in troubleshooting before I discovered the issue