OPNsense Forum
Archive => 22.7 Legacy Series => Topic started by: milkywaygoodfellas on August 14, 2022, 01:45:45 am
-
Every so often, up to multiple times per day, my firewall appliance locks up and requires a hard reboot to restore services and internet connectivity.
So far, I have been unable to find any logs or crash dumps that would help me isolate the issue outside of one time, which I did submit via the web interface.
I have no idea where to start. Can someone point me in the right direction to troubleshoot this issue? At this point I'm not sure if it's hardware or software.
I'm running it on a KingNovy fanless PC with 6x Intel I225-V, a Celeron N5105, 16 GB of RAM, and a 256 GB NVMe drive.
-
The start would be connecting to the console when it's locked up and seeing what it says.
-
I'd love to, but I can't even SSH into it when it happens.
-
I think he means locally on the device. Not remoting into it ;)
-
I'd love to, but I can't even SSH into it when it happens.
Key word, "console"
-
I managed to retrieve these crash dumps. Briefly going through them, I'm starting to suspect overheating or other hardware issues?
-
Looks like the panic was caused by "pfctl". You doing packet inspection of any kind? Perhaps chocking session states?
-
Looks like the panic was caused by "pfctl". You doing packet inspection of any kind? Perhaps chocking session states?
Just the defaults... IDS was enabled in IPS mode but with no rules downloaded. I did not modify any of those settings from the base install.
-
For readability:
db:0:kdb.enter.default> show pcpu
cpuid = 0
dynamic pcpu = 0xfc0f40
curthread = 0xfffffe0138c28720: pid 3489 tid 102014 critnest 1 "pfctl"
curpcb = 0xfffffe0138c28c30
fpcurthread = 0xfffffe0138c28720: pid 3489 "pfctl"
idlethread = 0xfffffe00207933a0: tid 100003 "idle: cpu0"
self = 0xffffffff82c10000
curpmap = 0xfffffe011668f518
tssp = 0xffffffff82c10384
rsp0 = 0xfffffe0118fea000
kcr3 = 0x351ae2000
ucr3 = 0x16fe6d000
scr3 = 0x16fe6d000
gs32p = 0xffffffff82c10404
ldt = 0xffffffff82c10444
tss = 0xffffffff82c10434
curvnet = 0xfffff80001202dc0
db:0:kdb.enter.default> bt
Tracing pid 3489 tid 102014 td 0xfffffe0138c28720
kdb_enter() at kdb_enter+0x37/frame 0xfffffe0118fe93c0
vpanic() at vpanic+0x1b0/frame 0xfffffe0118fe9410
panic() at panic+0x43/frame 0xfffffe0118fe9470
trap_fatal() at trap_fatal+0x385/frame 0xfffffe0118fe94d0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0118fe9530
calltrap() at calltrap+0x8/frame 0xfffffe0118fe9530
--- trap 0xc, rip = 0xffffffff80debe14, rsp = 0xfffffe0118fe9600, rbp = 0xfffffe0118fe9620 ---
rn_walktree() at rn_walktree+0x64/frame 0xfffffe0118fe9620
pfr_get_addrs() at pfr_get_addrs+0x219/frame 0xfffffe0118fe9680
pfioctl() at pfioctl+0x23be/frame 0xfffffe0118fe9b50
devfs_ioctl() at devfs_ioctl+0xc6/frame 0xfffffe0118fe9ba0
vn_ioctl() at vn_ioctl+0x1a4/frame 0xfffffe0118fe9cb0
devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe0118fe9cd0
kern_ioctl() at kern_ioctl+0x25b/frame 0xfffffe0118fe9d40
sys_ioctl() at sys_ioctl+0xf1/frame 0xfffffe0118fe9e00
amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe0118fe9f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0118fe9f30
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8012446da, rsp = 0x7fffffffdc38, rbp = 0x7fffffffe0d0 ---
I haven't seen this before but if it doesn't happen on 22.1 it should be easy to find the bad commit.
This is new for 22.7, right?
Cheers,
Franco
-
For readability:
db:0:kdb.enter.default> show pcpu
cpuid = 0
dynamic pcpu = 0xfc0f40
curthread = 0xfffffe0138c28720: pid 3489 tid 102014 critnest 1 "pfctl"
curpcb = 0xfffffe0138c28c30
fpcurthread = 0xfffffe0138c28720: pid 3489 "pfctl"
idlethread = 0xfffffe00207933a0: tid 100003 "idle: cpu0"
self = 0xffffffff82c10000
curpmap = 0xfffffe011668f518
tssp = 0xffffffff82c10384
rsp0 = 0xfffffe0118fea000
kcr3 = 0x351ae2000
ucr3 = 0x16fe6d000
scr3 = 0x16fe6d000
gs32p = 0xffffffff82c10404
ldt = 0xffffffff82c10444
tss = 0xffffffff82c10434
curvnet = 0xfffff80001202dc0
db:0:kdb.enter.default> bt
Tracing pid 3489 tid 102014 td 0xfffffe0138c28720
kdb_enter() at kdb_enter+0x37/frame 0xfffffe0118fe93c0
vpanic() at vpanic+0x1b0/frame 0xfffffe0118fe9410
panic() at panic+0x43/frame 0xfffffe0118fe9470
trap_fatal() at trap_fatal+0x385/frame 0xfffffe0118fe94d0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0118fe9530
calltrap() at calltrap+0x8/frame 0xfffffe0118fe9530
--- trap 0xc, rip = 0xffffffff80debe14, rsp = 0xfffffe0118fe9600, rbp = 0xfffffe0118fe9620 ---
rn_walktree() at rn_walktree+0x64/frame 0xfffffe0118fe9620
pfr_get_addrs() at pfr_get_addrs+0x219/frame 0xfffffe0118fe9680
pfioctl() at pfioctl+0x23be/frame 0xfffffe0118fe9b50
devfs_ioctl() at devfs_ioctl+0xc6/frame 0xfffffe0118fe9ba0
vn_ioctl() at vn_ioctl+0x1a4/frame 0xfffffe0118fe9cb0
devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe0118fe9cd0
kern_ioctl() at kern_ioctl+0x25b/frame 0xfffffe0118fe9d40
sys_ioctl() at sys_ioctl+0xf1/frame 0xfffffe0118fe9e00
amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe0118fe9f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0118fe9f30
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8012446da, rsp = 0x7fffffffdc38, rbp = 0x7fffffffe0d0 ---
I haven't seen this before but if it doesn't happen on 22.1 it should be easy to find the bad commit.
This is new for 22.7, right?
Cheers,
Franco
Yeah, never had this problem on 22.1 before. I disabled IPS/IDS entirely and it seems to have greatly helped the stability - it was crashing multiple times a day today and yesterday and since turning off Intrustion Detection in services, it hasn't crashed again (yet).
-
Just a quick update - since disabling IDS/IPS in my last post, the firewall has not crashed again as of this reply.
-
Did you have any hardware offloading enabled? i.e. CRC, TSO, LRO or VLAN?
-
Did you have any hardware offloading enabled? i.e. CRC, TSO, LRO or VLAN?
Nope, all disabled.
And I spoke too soon... another crash dump some time yesterday apparently. This time, however, the firewall rebooted itself instead of staying locked up until I power cycled it.
Caused by PHP this time, apparently?
-
Given the change in behavior, this is feeling more like potentially a hardware issue, but it's still not remotely clear.
To rule that out, are you able to go back to 22.1 and test?
Otherwise potentially check CPU temps, or setup alerts.
You could also, just for good measure, run a memtest on the box?
Historically, for me, it's rarely been memory issues however it WAS 1 out of the 99 times. And that 1 time, drove me nuts in troubleshooting before I discovered the issue ;)
-
Given the change in behavior, this is feeling more like potentially a hardware issue, but it's still not remotely clear.
To rule that out, are you able to go back to 22.1 and test?
Otherwise potentially check CPU temps, or setup alerts.
You could also, just for good measure, run a memtest on the box?
Historically, for me, it's rarely been memory issues however it WAS 1 out of the 99 times. And that 1 time, drove me nuts in troubleshooting before I discovered the issue ;)
I can try a live disk of 22.1 to see, but I made some tweaks and it was running stable again so I turned IDS/IPS back on and it almost immediately locked up with no crash dump, same as before. Turned it back off and so far so good, but it's only been a couple of hours.
-
Past few days have just had random crashing still... seems like it might just be that these changwang PCs are unreliable. I decided to try a Qotom box instead, so I'll transfer config to that when it gets here and we'll see.
-
Another crash dump for pfctl... seems to be most consistently the thing causing the kernel panics.
-
Have you swapped hardware yet?
-
Have you swapped hardware yet?
Not yet, new box gets here in a couple days. Just figured I'd keep posting/submitting the dumps in the meantime just in case it is a software issue.
-
Well, I got the new box set up. This one actually still outputs via the HDMI port when it crashes, so I could finally figure out it was an SSD issue... seems like from time to time the SSD disappears and the system boots into the UEFI shell. If the SSD is overheating, could that explain the odd kernel panics from pfctl as well?
-
Spoke too soon - another freeze and this time it didn't output any video again. Reseated everything and verified temps all look okay.
I tried to see if 22.1 would have the issue, but the new hardware has Intel I226-V NICs and 22.1 would not detect them, so I just did a fresh re-install of 22.7, re-installed plugins, restored config, and updated to latest again. We'll see if that helps at all...
-
Nope... had another crash dump not too long ago. At least this time it didn't freeze entirely and require a power cycle...
-
Hopefully I'm not spreading fud but didn't the I226-V NIC's have stability issues with FreeBSD?
-
Hopefully I'm not spreading fud but didn't the I226-V NIC's have stability issues with FreeBSD?
I'm not aware of anything BSD-specific, but then I don't exactly follow networking development in BSD... I know the I225's first two revisions had issues in general, and have seen some reports, albeit far lesser in number, about the third revision, too, but nothing I226-specific so far.
-
Another one from pf.
/var/crash/textdump.tar.0:
ddb.txt06000014000014306365744 7106 ustarrootwheeldb:0:kdb.enter.default> run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo> show alllocks
No such command; use "help" to list available commands
db:1:lockinfo> show lockedvnods
Locked vnodes
db:0:kdb.enter.default> show pcpu
cpuid = 2
dynamic pcpu = 0xfffffe009ea2bf40
curthread = 0xfffffe00dba04020: pid 6 tid 100106 critnest 1 "pf purge"
curpcb = 0xfffffe00dba04530
fpcurthread = none
idlethread = 0xfffffe0020765560: tid 100005 "idle: cpu2"
self = 0xffffffff82c12000
curpmap = 0xffffffff81ea0d38
tssp = 0xffffffff82c12384
rsp0 = 0xfffffe00d9eb5000
kcr3 = 0x68567000
ucr3 = 0xffffffffffffffff
scr3 = 0x1ddb7c000
gs32p = 0xffffffff82c12404
ldt = 0xffffffff82c12444
tss = 0xffffffff82c12434
curvnet = 0xfffff800011ffc80
db:0:kdb.enter.default> bt
Tracing pid 6 tid 100106 td 0xfffffe00dba04020
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00d9eb4c60
vpanic() at vpanic+0x1b0/frame 0xfffffe00d9eb4cb0
panic() at panic+0x43/frame 0xfffffe00d9eb4d10
trap_fatal() at trap_fatal+0x385/frame 0xfffffe00d9eb4d70
calltrap() at calltrap+0x8/frame 0xfffffe00d9eb4d70
--- trap 0x9, rip = 0xffffffff80ceca6c, rsp = 0xfffffe00d9eb4e40, rbp = 0xfffffe00d9eb4e40 ---
counter_u64_fetch() at counter_u64_fetch+0x4c/frame 0xfffffe00d9eb4e40
pf_state_expires() at pf_state_expires+0x90/frame 0xfffffe00d9eb4e70
pf_purge_expired_states() at pf_purge_expired_states+0xe5/frame 0xfffffe00d9eb4ec0
pf_purge_thread() at pf_purge_thread+0x13b/frame 0xfffffe00d9eb4ef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00d9eb4f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d9eb4f30
--- trap 0x2b4a000, rip = 0xffffffff80c313cf, rsp = 0, rbp = 0xffffffff81ea04e0 ---
mi_startup() at mi_startup+0xdf/frame 0xffffffff81ea04e0
proc0() at proc0/frame 0xffffffff81ef7dc8
_binary_elf_vdso_so_1_size() at 0x30000/frame 0xffffffff81354b2a
-
So that's a fairly new chipset. Even Intel itself only has windows drivers:
https://www.intel.com/content/www/us/en/products/sku/210599/intel-ethernet-controller-i226v/downloads.html
So if BSD does support them, it's on it's own within the kernel. I would question this greatly.
-
So that's a fairly new chipset. Even Intel itself only has windows drivers:
https://www.intel.com/content/www/us/en/products/sku/210599/intel-ethernet-controller-i226v/downloads.html
So if BSD does support them, it's on it's own within the kernel. I would question this greatly.
Question what? BSD 13 has the igc driver for the I226.
-
BSD also has the realtek driver but it's not stable.
-
Ok.
-
Another set of hardware, this time some different things are crashing. Luckily so far, the services have recovered on their own and none have required a hard power cycle.
Got this crash dump most recently.
-
Your crash dumps seem erratic and not related to network drivers... I would suspect a general hardware issue that manifests itself under system strain in unpredictable ways.
Cheers,
Franco
-
Another one... odd if it were hardware issues across three separate sets of hardware, but maybe I'm just extraordinarily unlucky this time. I'll try swapping out the RAM and/or SSD and see if either of those resolves it.
-
I figured it out - bad RAM. It tested fine on the first device but just to be sure I tested it again... this time it threw so many errors memtest86 couldn't even complete a full pass.
D'oh.
-
That would explain it. Maybe not all hardware is affected like this... would be best to keep track of traces separately for each machine. Maybe the other two have a single panic to trace.
Cheers,
Franco
-
That would explain it. Maybe not all hardware is affected like this... would be best to keep track of traces separately for each machine. Maybe the other two have a single panic to trace.
Cheers,
Franco
It was the same RAM modules in all three of them in this case as I just swapped them into each box, but you are right that keeping track of which machine generated which trace is a good idea in the future. Though hopefully with new RAM it won't be necessary...