PSA: PF regression in 24.7.10 kernel and fix

[stanthewizzard]

ok
this is why my FW are crashing (no reboot juste gone for FW2)

[ezhik]

I am seeing issues as well, what is the last stable release prior to this pf regression?

I am getting massive latency after 24h going from >1ms to 4000ms on my WAN interface.

For now reverted back to the 24.7.8 kernel

# opnsense-update -kr 24.7.8
# opnsense-shell reboot

[infinisourcekc]

I am still seeing kernel panics on a DEC3860 even 24 hours after an upgrade.  The initial upgrade caused kernel panics every hour on the hour.  What's interesting is my kernel version is not the same as the fixed version:

FreeBSD 14.1-RELEASE-p6 stable/24.7-n267979-0d692990122 SMP

I'm using the default mirror for my box.  If I roll back to a previous snapshot that was using 24.7.9_2 prior to the upgrade the box is stable again.

[newsense]

That's because you are on the wrong kernel

Code Select Expand


opnsense-update -fk

opnsense-shell reboot


[newsense]

I am seeing issues as well, what is the last stable release prior to this pf regression?

I am getting massive latency after 24h going from >1ms to 4000ms on my WAN interface.

For now reverted back to the 24.7.8 kernel

# opnsense-update -kr 24.7.8
# opnsense-shell reboot

That's some wan there operation above the speed of light - in the olden days of 24.7.8

Things are absolutely fine on the fixed 24.7.10 kernel, reverting forward using the official guidance in the Announcements section for 24.7.10 is the only thing needed for those affected by the bug in the original 24.7.10, everyone else simply update to it.

Conflating ISP issues with FreeBSD bugs and ad-hoc reverts is not helpful guidance in this thread.

[infinisourcekc]

That's because you are on the wrong kernel

Code Select Expand


opnsense-update -fk

opnsense-shell reboot


I'm aware that it's on the wrong kernel version.  I should have included in my original post that I already did the reinstall from the CLI.

[ezhik]

I am seeing issues as well, what is the last stable release prior to this pf regression?

I am getting massive latency after 24h going from >1ms to 4000ms on my WAN interface.

For now reverted back to the 24.7.8 kernel

# opnsense-update -kr 24.7.8
# opnsense-shell reboot

That's some wan there operation above the speed of light - in the olden days of 24.7.8

Things are absolutely fine on the fixed 24.7.10 kernel, reverting forward using the official guidance in the Announcements section for 24.7.10 is the only thing needed for those affected by the bug in the original 24.7.10, everyone else simply update to it.

Conflating ISP issues with FreeBSD bugs and ad-hoc reverts is not helpful guidance in this thread.

The unfortunate part this started happening after 24.7.10 update. So what do you advise here ?

Code Select Expand


64 bytes from 8.8.8.8: icmp_seq=13325 ttl=117 time=509 ms
64 bytes from 8.8.8.8: icmp_seq=13326 ttl=117 time=511 ms
64 bytes from 8.8.8.8: icmp_seq=13327 ttl=117 time=998 ms
64 bytes from 8.8.8.8: icmp_seq=13328 ttl=117 time=18.6 ms
64 bytes from 8.8.8.8: icmp_seq=13329 ttl=117 time=772 ms
64 bytes from 8.8.8.8: icmp_seq=13330 ttl=117 time=19.0 ms
64 bytes from 8.8.8.8: icmp_seq=13331 ttl=117 time=1022 ms
64 bytes from 8.8.8.8: icmp_seq=13332 ttl=117 time=1133 ms
64 bytes from 8.8.8.8: icmp_seq=13333 ttl=117 time=803 ms
64 bytes from 8.8.8.8: icmp_seq=13334 ttl=117 time=1062 ms
64 bytes from 8.8.8.8: icmp_seq=13335 ttl=117 time=1073 ms
64 bytes from 8.8.8.8: icmp_seq=13336 ttl=117 time=573 ms
64 bytes from 8.8.8.8: icmp_seq=13337 ttl=117 time=589 ms
64 bytes from 8.8.8.8: icmp_seq=13338 ttl=117 time=1141 ms
64 bytes from 8.8.8.8: icmp_seq=13339 ttl=117 time=626 ms
64 bytes from 8.8.8.8: icmp_seq=13340 ttl=117 time=566 ms
64 bytes from 8.8.8.8: icmp_seq=13341 ttl=117 time=680 ms
64 bytes from 8.8.8.8: icmp_seq=13342 ttl=117 time=588 ms
64 bytes from 8.8.8.8: icmp_seq=13343 ttl=117 time=90.0 ms
64 bytes from 8.8.8.8: icmp_seq=13344 ttl=117 time=598 ms
64 bytes from 8.8.8.8: icmp_seq=13345 ttl=117 time=622 ms
64 bytes from 8.8.8.8: icmp_seq=13346 ttl=117 time=117 ms
64 bytes from 8.8.8.8: icmp_seq=13347 ttl=117 time=142 ms
64 bytes from 8.8.8.8: icmp_seq=13348 ttl=117 time=211 ms
64 bytes from 8.8.8.8: icmp_seq=13349 ttl=117 time=157 ms
64 bytes from 8.8.8.8: icmp_seq=13350 ttl=117 time=1193 ms
64 bytes from 8.8.8.8: icmp_seq=13351 ttl=117 time=1670 ms
64 bytes from 8.8.8.8: icmp_seq=13352 ttl=117 time=2189 ms
64 bytes from 8.8.8.8: icmp_seq=13353 ttl=117 time=2224 ms
64 bytes from 8.8.8.8: icmp_seq=13354 ttl=117 time=3230 ms
64 bytes from 8.8.8.8: icmp_seq=13355 ttl=117 time=4263 ms
64 bytes from 8.8.8.8: icmp_seq=13356 ttl=117 time=4256 ms
64 bytes from 8.8.8.8: icmp_seq=13357 ttl=117 time=3892 ms
64 bytes from 8.8.8.8: icmp_seq=13358 ttl=117 time=3260 ms
64 bytes from 8.8.8.8: icmp_seq=13359 ttl=117 time=3201 ms
64 bytes from 8.8.8.8: icmp_seq=13360 ttl=117 time=2463 ms
64 bytes from 8.8.8.8: icmp_seq=13361 ttl=117 time=2314 ms
64 bytes from 8.8.8.8: icmp_seq=13362 ttl=117 time=3036 ms
64 bytes from 8.8.8.8: icmp_seq=13364 ttl=117 time=5457 ms
64 bytes from 8.8.8.8: icmp_seq=13365 ttl=117 time=5399 ms
64 bytes from 8.8.8.8: icmp_seq=13367 ttl=117 time=5527 ms


[newsense]

For those values I can think of 3 possibilities: ISP issues, VPN issues if the pings go over VPN, ISP + VPN issues in aggregate.

For reference, I get 6-20 msec pinging 1.1.1.1 over the wan, between 120-180ms pinging 8.8.8.8 and 8.8.4.4 over various VPNs and 50-70ms between OPN FWs on the latest kernel a few thousand miles/km apart.


If you have a modem - power cycle it. If the results are consistent - talk to your ISP.

[ezhik]

For those values I can think of 3 possibilities: ISP issues, VPN issues if the pings go over VPN, ISP + VPN issues in aggregate.

For reference, I get 6-20 msec pinging 1.1.1.1 over the wan, between 120-180ms pinging 8.8.8.8 and 8.8.4.4 over various VPNs and 50-70ms between OPN FWs on the latest kernel a few thousand miles/km apart.


If you have a modem - power cycle it. If the results are consistent - talk to your ISP.

That's a sound advise, I confirmed the issue is with ISP model. Cheers mate.