FYI - there is a pf regression in the 24.7.10 kernel that causes OPNsense to crash and reboot -- twice for me on two different FWs in the first hour.
I've been working on this with Franco and there's a test kernel that is stable for me for 3.5+ hours and counting.
https://github.com/opnsense/src/commit/83757627 (https://github.com/opnsense/src/commit/83757627)
# opnsense-update -zkr 24.7.10-state
# opnsense-shell reboot
I have had the same issues. Three panics in one hour. I have implemented the fix as suggested.
Update: 2.5 hours later, all is well in the world again.
Potential similar issues here as well. I had my system crash twice today after upgrading to 24.7.10_1 when normally my system is very stable.
Running the test kernel here as well to see if that clears up my crashes too.
For reference here's one of the stack traces courtesy of newsense:
db:0:kdb.enter.default> show pcpu
cpuid = 2
dynamic pcpu = 0xfffffe0090458080
curthread = 0xfffff800019ad000: pid 7 tid 100104 critnest 1 "pf purge"
curpcb = 0xfffff800019ad520
fpcurthread = none
idlethread = 0xfffff800016d0740: tid 100005 "idle: cpu2"
self = 0xffffffff83a12000
curpmap = 0xffffffff81b81670
tssp = 0xffffffff83a12384
rsp0 = 0xfffffe00853e9000
kcr3 = 0x74480000
ucr3 = 0xffffffffffffffff
scr3 = 0x2286f0000
gs32p = 0xffffffff83a12404
ldt = 0xffffffff83a12444
tss = 0xffffffff83a12434
curvnet = 0xfffff800011afd80
db:0:kdb.enter.default> bt
Tracing pid 7 tid 100104 td 0xfffff800019ad000
kdb_enter() at kdb_enter+0x33/frame 0xfffffe00853e8c20
panic() at panic+0x43/frame 0xfffffe00853e8c80
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe00853e8ce0
trap_pfault() at trap_pfault+0x46/frame 0xfffffe00853e8d30
calltrap() at calltrap+0x8/frame 0xfffffe00853e8d30
--- trap 0xc, rip = 0xffffffff82197d9c, rsp = 0xfffffe00853e8e00, rbp = 0xfffffe00853e8e30 ---
pf_detach_state() at pf_detach_state+0x5fc/frame 0xfffffe00853e8e30
pf_unlink_state() at pf_unlink_state+0x290/frame 0xfffffe00853e8e70
pf_purge_expired_states() at pf_purge_expired_states+0x188/frame 0xfffffe00853e8ec0
pf_purge_thread() at pf_purge_thread+0x13b/frame 0xfffffe00853e8ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00853e8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00853e8f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
This'll explain why I didn't have any network this morning. I've applied the patch, will reboot once the chance presents itself (or the system crashes, whichever comes first).
Make sure to check the kernel version and act accordingly:
Please note we had to hotfix the kernel which will not reinstall
automatically if you caught the bad version. If you experience
panics on 24.7.10 relating to pf(4) please reinstall from the GUI
(which includes an automatic reboot) or run "opnsense-update -fk"
from the shell followed by a manual reboot. The correct kernel
identifies itself as "stable/24.7-n267981-8375762712f" using
"uname -v".
Someone brave enough to help get a vmcore for debugging purposes / reporting upstream?
# opnsense-update -zkr dbg-24.7.10
just reboot, let it crash. vmcore will be under /var/crash/vmcore.X -- move back to safe kernel
# opnsense-update -k
I'm making an effort to report this in an acceptable upstream manner.
Thanks,
Franco
Thanks a lot for the fix!
I thought I was going crazy this morning when I got no Internet anymore... :)
I have also the same issues. "opnsense-update -zkr 24.7.10-state" in shell fixed it, but after that i saw the available hotfix 24.7.10_2. After the update in the GUI i had 2 panics again. After running "opnsense-update -zkr 24.7.10-state" again, it seems stable.
uname -v shows FreeBSD 14.1-RELEASE-p6 route_del_fix-n267981-8375762712f SMP
Quote from: franco on December 04, 2024, 12:12:56 PM
Make sure to check the kernel version and act accordingly:
Please note we had to hotfix the kernel which will not reinstall
automatically if you caught the bad version. If you experience
panics on 24.7.10 relating to pf(4) please reinstall from the GUI
(which includes an automatic reboot) or run "opnsense-update -fk"
from the shell followed by a manual reboot. The correct kernel
identifies itself as "stable/24.7-n267981-8375762712f" using
"uname -v".
martin87?????? I mean I said it here and everywhere else. Your kernel availability will be subject to the mirror you are using. I cannot speak for anything but the default mirror.
I had a hard crash and reboot this morning too. How do we know this is the cause is there a particular log or detail to reference?
Check your kernel with "uname -v" and/or post the crash report here if your box offers it. Just the "--- trap" enclosed things.
Cheers,
Franco
--- trap 0x9, rip = 0xffffffff8277de0d, rsp = 0xfffffe00c498be00, rbp = 0xfffffe00c498be30 ---
pf_detach_state() at pf_detach_state+0x66d/frame 0xfffffe00c498be30
pf_unlink_state() at pf_unlink_state+0x290/frame 0xfffffe00c498be70
pf_purge_expired_states() at pf_purge_expired_states+0x188/frame 0xfffffe00c498bec0
pf_purge_thread() at pf_purge_thread+0x13b/frame 0xfffffe00c498bef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00c498bf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c498bf30
--- trap 0xc8e6f912, rip = 0x233a94f6b6accd0a, rsp = 0x373380ffa2a5d903, rbp = 0x72eb97b310c02134 ---
FreeBSD 14.1-RELEASE-p6 stable/24.7-n267979-0d692990122 SMP
Just learned how to view the crash report, guess crashes don't happen enough for me to even know how :)
But based on that log def looks like the same issue.
I'm on the default mirror but running "opnsense-update -fk" produces the same kernel I pasted above instead of the desired kernel Franco posted.
I'm dumb I need to reboot before checking kernel version... ::)
Yes, same same, not different.
My backtrace is slightly different if this helps.
db:0:kdb.enter.default> bt
Tracing pid 0 tid 100041 td 0xfffff80103950740
kdb_enter() at kdb_enter+0x33/frame 0xfffffe00d3bf22b0
panic() at panic+0x43/frame 0xfffffe00d3bf2310
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe00d3bf2370
trap_pfault() at trap_pfault+0x46/frame 0xfffffe00d3bf23c0
calltrap() at calltrap+0x8/frame 0xfffffe00d3bf23c0
--- trap 0xc, rip = 0xffffffff8109fa60, rsp = 0xfffffe00d3bf2490, rbp = 0xfffffe00d3bf2490 ---
memcmp() at memcmp+0x110/frame 0xfffffe00d3bf2490
pf_find_state() at pf_find_state+0xc0/frame 0xfffffe00d3bf24e0
pf_test_state_tcp() at pf_test_state_tcp+0x1c4/frame 0xfffffe00d3bf2650
pf_test() at pf_test+0x131e/frame 0xfffffe00d3bf2800
pf_check_in() at pf_check_in+0x27/frame 0xfffffe00d3bf2820
pfil_mbuf_in() at pfil_mbuf_in+0x38/frame 0xfffffe00d3bf2850
ip_tryforward() at ip_tryforward+0x17f/frame 0xfffffe00d3bf2910
ip_input() at ip_input+0x56c/frame 0xfffffe00d3bf2970
netisr_dispatch_src() at netisr_dispatch_src+0x9e/frame 0xfffffe00d3bf29c0
ether_demux() at ether_demux+0x149/frame 0xfffffe00d3bf29f0
ng_ether_rcv_upper() at ng_ether_rcv_upper+0x8c/frame 0xfffffe00d3bf2a10
ng_apply_item() at ng_apply_item+0x13e/frame 0xfffffe00d3bf2ab0
ng_snd_item() at ng_snd_item+0x274/frame 0xfffffe00d3bf2af0
ng_apply_item() at ng_apply_item+0x13e/frame 0xfffffe00d3bf2b90
ng_snd_item() at ng_snd_item+0x274/frame 0xfffffe00d3bf2bd0
ng_ether_input() at ng_ether_input+0x4c/frame 0xfffffe00d3bf2c00
ether_nh_input() at ether_nh_input+0x1dc/frame 0xfffffe00d3bf2c60
netisr_dispatch_src() at netisr_dispatch_src+0x9e/frame 0xfffffe00d3bf2cb0
ether_input() at ether_input+0x56/frame 0xfffffe00d3bf2d00
iflib_rxeof() at iflib_rxeof+0xc0e/frame 0xfffffe00d3bf2e00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe00d3bf2e40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x14e/frame 0xfffffe00d3bf2ec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe00d3bf2ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00d3bf2f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d3bf2f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
> My backtrace is slightly different if this helps.
Hmm, juicy... a memcpy()? Are you sure yours is fixed with the amended kernel? It might just keep happening if we assume the revert does nothing for your case.
If it keeps happening try the 24.7.8 kernel to see if that fixes it.
Cheers,
Franco
I reverted to the 24.7.8 kernel yesterday. I have 3 panics from the original 24.7.10 kernel, 2 from pf_find_state() and 1 from pf_detach_state().
Could be related, got a GitHub issue with it now too: https://github.com/opnsense/src/issues/230
It would further point to the commit in question corrupting the states to set up a panic in another code path later on. That is if the new 24.7.10 kernel fixes this issue, too.
Cheers,
Franco
Updated to the latest kernel now, will report issues here.
After "opnsense-update -fk" uname -v shows me "stable/24.7-n267981-8375762712f"
With this kernel the system crashes too.
After "opnsense-update -zkr 24.7.10-state" uname -v shows me "route_del_fix-n267981-8375762712f".
This kernel runs stable. I'm using the default mirror.
Quote from: martin87 on December 04, 2024, 06:57:36 PM
After "opnsense-update -fk" uname -v shows me "stable/24.7-n267981-8375762712f"
With this kernel the system crashes too.
After "opnsense-update -zkr 24.7.10-state" uname -v shows me "route_del_fix-n267981-8375762712f".
This kernel runs stable. I'm using the default mirror.
I highlighted the commit hashes to emphasise that the builds are in fact the same.
Cheers,
Franco
Quote from: franco on December 04, 2024, 07:07:48 PM
Quote from: martin87 on December 04, 2024, 06:57:36 PM
After "opnsense-update -fk" uname -v shows me "stable/24.7-n267981-8375762712f"
With this kernel the system crashes too.
After "opnsense-update -zkr 24.7.10-state" uname -v shows me "route_del_fix-n267981-8375762712f".
This kernel runs stable. I'm using the default mirror.
I highlighted the commit hashes to emphasise that the builds are in fact the same.
Cheers,
Franco
Ok thank you, but with the "stable/24.7-n267981-8375762712f" it crashes with
--- trap 0xc, rip = 0xffffffff80f6be42, rsp = 0xfffffe00b2899c40, rbp = 0xfffffe00b2899c50 ---
vm_radix_lookup_unlocked() at vm_radix_lookup_unlocked+0x62/frame 0xfffffe00b2899c50
vm_fault() at vm_fault+0x85d/frame 0xfffffe00b2899d70
vm_fault_trap() at vm_fault_trap+0x4d/frame 0xfffffe00b2899dc0
trap_pfault() at trap_pfault+0x1be/frame 0xfffffe00b2899e10
trap() at trap+0x4ab/frame 0xfffffe00b2899f30
calltrap() at calltrap+0x8/frame 0xfffffe00b2899f30
--- trap 0xc, rip = 0x82213a86e, rsp = 0x820f06990, rbp = 0x820f069a0 ---
Hmm, that's not the panic we're looking for (Star Wars giggles)
In all seriousness it may be coincidental. Is this recurring or a one off?
The panic occurs after ~1 hour after the update to "stable/24.7-n267981-8375762712f"
After that i changed to "route_del_fix-n267981-8375762712f". This is for me stable. I tried it one more time, but after ~ 1 hour it crashed again.
Yesterday with "stable/24.7-n267979-0d692990122" i get this:
--- trap 0x9, rip = 0xffffffff80d053f7, rsp = 0xfffffe00b2e4c690, rbp = 0xfffffe00b2e4c6b0 ---
rn_walktree() at rn_walktree+0x77/frame 0xfffffe00b2e4c6b0
pfr_get_addrs() at pfr_get_addrs+0x122/frame 0xfffffe00b2e4c710
pfioctl() at pfioctl+0x221e/frame 0xfffffe00b2e4cbf0
devfs_ioctl() at devfs_ioctl+0xcb/frame 0xfffffe00b2e4cc40
vn_ioctl() at vn_ioctl+0xce/frame 0xfffffe00b2e4ccb0
devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe00b2e4ccd0
kern_ioctl() at kern_ioctl+0x255/frame 0xfffffe00b2e4cd40
sys_ioctl() at sys_ioctl+0xff/frame 0xfffffe00b2e4ce00
amd64_syscall() at amd64_syscall+0x100/frame 0xfffffe00b2e4cf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00b2e4cf30
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0xc46b237d5fa, rsp = 0xc46af1d9bf8, rbp = 0xc46af1da090 ---
> rn_walktree() at rn_walktree+0x77/frame 0xfffffe00b2e4c6b0
> pfr_get_addrs() at pfr_get_addrs+0x122/frame 0xfffffe00b2e4c710
I'm aware of this panic, apparently some pfctl invoke is causing this, but we have no further debug info.
It's certainly interesting that a significant number panics point to pf(4) code.
Cheers,
Franco
I sent the crash report via GUI yesterday. Or should I post it here?
Now it would be interesting to know why the system crashes with "stable/24.7-n267981-8375762712f" and not with "route_del_fix-n267981-8375762712f". Otherwise I'll just do a clean install tomorrow.
No need to reinstall, nothing to gain doing that.
If none of the .10 kernels work for you simply go back to .8 until everything is sorted out.
# opnsense-update -kr 24.7.8
# opnsense-shell reboot
Don't know, it seems circumstantial. We have a report on the business edition which means this issue is inherent to 24.7.6 or 24.7.8 kernels anyway.
Cheers,
Franco
Quote from: newsense on December 04, 2024, 08:30:35 PM
No need to reinstall, nothing to gain doing that.
If none of the .10 kernels work for you simply go back to .8 until everything is sorted out.
# opnsense-update -kr 24.7.8
# opnsense-shell reboot
Ok, then I'll wait until everything is sorted out. The "route_del_fix-n267981-8375762712f" kernel works stable.
ok
this is why my FW are crashing (no reboot juste gone for FW2)
I am seeing issues as well, what is the last stable release prior to this pf regression?
I am getting massive latency after 24h going from >1ms to 4000ms on my WAN interface.
For now reverted back to the 24.7.8 kernel
# opnsense-update -kr 24.7.8
# opnsense-shell reboot
I am still seeing kernel panics on a DEC3860 even 24 hours after an upgrade. The initial upgrade caused kernel panics every hour on the hour. What's interesting is my kernel version is not the same as the fixed version:
QuoteFreeBSD 14.1-RELEASE-p6 stable/24.7-n267979-0d692990122 SMP
I'm using the default mirror for my box. If I roll back to a previous snapshot that was using 24.7.9_2 prior to the upgrade the box is stable again.
That's because you are on the wrong kernel
opnsense-update -fk
opnsense-shell reboot
Quote from: ezhik on December 08, 2024, 03:05:50 PM
I am seeing issues as well, what is the last stable release prior to this pf regression?
I am getting massive latency after 24h going from >1ms to 4000ms on my WAN interface.
For now reverted back to the 24.7.8 kernel
# opnsense-update -kr 24.7.8
# opnsense-shell reboot
That's some wan there operation above the speed of light - in the olden days of 24.7.8
Things are absolutely fine on the fixed 24.7.10 kernel,
reverting forward using the official guidance in the Announcements section for 24.7.10 is the only thing needed for those affected by the bug in the original 24.7.10, everyone else simply update to it.
Conflating ISP issues with FreeBSD bugs and ad-hoc reverts is not helpful guidance in this thread.
Quote from: newsense on December 08, 2024, 09:35:01 PM
That's because you are on the wrong kernel
opnsense-update -fk
opnsense-shell reboot
I'm aware that it's on the wrong kernel version. I should have included in my original post that I already did the reinstall from the CLI.
Quote from: newsense on December 08, 2024, 09:45:05 PM
Quote from: ezhik on December 08, 2024, 03:05:50 PM
I am seeing issues as well, what is the last stable release prior to this pf regression?
I am getting massive latency after 24h going from >1ms to 4000ms on my WAN interface.
For now reverted back to the 24.7.8 kernel
# opnsense-update -kr 24.7.8
# opnsense-shell reboot
That's some wan there operation above the speed of light - in the olden days of 24.7.8
Things are absolutely fine on the fixed 24.7.10 kernel, reverting forward using the official guidance in the Announcements section for 24.7.10 is the only thing needed for those affected by the bug in the original 24.7.10, everyone else simply update to it.
Conflating ISP issues with FreeBSD bugs and ad-hoc reverts is not helpful guidance in this thread.
The unfortunate part this started happening after 24.7.10 update. So what do you advise here ?
64 bytes from 8.8.8.8: icmp_seq=13325 ttl=117 time=509 ms
64 bytes from 8.8.8.8: icmp_seq=13326 ttl=117 time=511 ms
64 bytes from 8.8.8.8: icmp_seq=13327 ttl=117 time=998 ms
64 bytes from 8.8.8.8: icmp_seq=13328 ttl=117 time=18.6 ms
64 bytes from 8.8.8.8: icmp_seq=13329 ttl=117 time=772 ms
64 bytes from 8.8.8.8: icmp_seq=13330 ttl=117 time=19.0 ms
64 bytes from 8.8.8.8: icmp_seq=13331 ttl=117 time=1022 ms
64 bytes from 8.8.8.8: icmp_seq=13332 ttl=117 time=1133 ms
64 bytes from 8.8.8.8: icmp_seq=13333 ttl=117 time=803 ms
64 bytes from 8.8.8.8: icmp_seq=13334 ttl=117 time=1062 ms
64 bytes from 8.8.8.8: icmp_seq=13335 ttl=117 time=1073 ms
64 bytes from 8.8.8.8: icmp_seq=13336 ttl=117 time=573 ms
64 bytes from 8.8.8.8: icmp_seq=13337 ttl=117 time=589 ms
64 bytes from 8.8.8.8: icmp_seq=13338 ttl=117 time=1141 ms
64 bytes from 8.8.8.8: icmp_seq=13339 ttl=117 time=626 ms
64 bytes from 8.8.8.8: icmp_seq=13340 ttl=117 time=566 ms
64 bytes from 8.8.8.8: icmp_seq=13341 ttl=117 time=680 ms
64 bytes from 8.8.8.8: icmp_seq=13342 ttl=117 time=588 ms
64 bytes from 8.8.8.8: icmp_seq=13343 ttl=117 time=90.0 ms
64 bytes from 8.8.8.8: icmp_seq=13344 ttl=117 time=598 ms
64 bytes from 8.8.8.8: icmp_seq=13345 ttl=117 time=622 ms
64 bytes from 8.8.8.8: icmp_seq=13346 ttl=117 time=117 ms
64 bytes from 8.8.8.8: icmp_seq=13347 ttl=117 time=142 ms
64 bytes from 8.8.8.8: icmp_seq=13348 ttl=117 time=211 ms
64 bytes from 8.8.8.8: icmp_seq=13349 ttl=117 time=157 ms
64 bytes from 8.8.8.8: icmp_seq=13350 ttl=117 time=1193 ms
64 bytes from 8.8.8.8: icmp_seq=13351 ttl=117 time=1670 ms
64 bytes from 8.8.8.8: icmp_seq=13352 ttl=117 time=2189 ms
64 bytes from 8.8.8.8: icmp_seq=13353 ttl=117 time=2224 ms
64 bytes from 8.8.8.8: icmp_seq=13354 ttl=117 time=3230 ms
64 bytes from 8.8.8.8: icmp_seq=13355 ttl=117 time=4263 ms
64 bytes from 8.8.8.8: icmp_seq=13356 ttl=117 time=4256 ms
64 bytes from 8.8.8.8: icmp_seq=13357 ttl=117 time=3892 ms
64 bytes from 8.8.8.8: icmp_seq=13358 ttl=117 time=3260 ms
64 bytes from 8.8.8.8: icmp_seq=13359 ttl=117 time=3201 ms
64 bytes from 8.8.8.8: icmp_seq=13360 ttl=117 time=2463 ms
64 bytes from 8.8.8.8: icmp_seq=13361 ttl=117 time=2314 ms
64 bytes from 8.8.8.8: icmp_seq=13362 ttl=117 time=3036 ms
64 bytes from 8.8.8.8: icmp_seq=13364 ttl=117 time=5457 ms
64 bytes from 8.8.8.8: icmp_seq=13365 ttl=117 time=5399 ms
64 bytes from 8.8.8.8: icmp_seq=13367 ttl=117 time=5527 ms
For those values I can think of 3 possibilities: ISP issues, VPN issues if the pings go over VPN, ISP + VPN issues in aggregate.
For reference, I get 6-20 msec pinging 1.1.1.1 over the wan, between 120-180ms pinging 8.8.8.8 and 8.8.4.4 over various VPNs and 50-70ms between OPN FWs on the latest kernel a few thousand miles/km apart.
If you have a modem - power cycle it. If the results are consistent - talk to your ISP.
Quote from: newsense on December 09, 2024, 12:44:32 AM
For those values I can think of 3 possibilities: ISP issues, VPN issues if the pings go over VPN, ISP + VPN issues in aggregate.
For reference, I get 6-20 msec pinging 1.1.1.1 over the wan, between 120-180ms pinging 8.8.8.8 and 8.8.4.4 over various VPNs and 50-70ms between OPN FWs on the latest kernel a few thousand miles/km apart.
If you have a modem - power cycle it. If the results are consistent - talk to your ISP.
That's a sound advise, I confirmed the issue is with ISP model. Cheers mate.