Updated to 24.7.1 earlier today. All went well, except I can no longer run traceroutes from any Windows machines, on any VLAN.
Traceroutes from a Linux machine (my Raspberry Pi for example) work just fine, and traceroutes from the OPNsense Web GUI are working properly as well.
My ISP has routing / peering issues with some server providers sometimes, so I use WinMTR often to diagnose issues and report them so they can get resolved.
However, after the 24.7.1 update, it seems something funky is happening with ICMP and anything after the first hop gets dropped and I just see 'Request timed out'.
I haven't made added any new rules recently, and my existing firewall rules are exactly the same as they were before updating.
As I understand, Windows traceroutes use ICMP whereas on Linux they use UDP.
Any tips on how to go about diagnosing this or any insight on what changed with 24.7.1 that suddenly started causing this? It was fine on all previous versions, including 24.7_9.
Screenshots attached (Linux vs Windows).
https://imgur.com/a/yhDp4Jo
Experiencing the same behaviour here.
Interesting. Maybe this also explains this:
mtr -rzw opnsense.org ─╯
Start: 2024-08-08T19:53:20+0200
HOST: mpp Loss% Snt Last Avg Best Wrst StDev
1. AS??? opnsense 0.0% 10 0.4 0.4 0.3 0.4 0.0
2. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
3. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
4. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
5. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
6. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
7. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
8. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
9. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
10. AS60781 178.162.131.118 0.0% 10 20.2 20.3 20.2 20.6 0.1
As soon as I switch to ICMP or TCP it's working again.
Quote from: Chaosphere64 on August 08, 2024, 07:55:02 PMAs soon as I switch to ICMP or TCP it's working again.
I believe mtr uses ICMP by default, and if I use
mtr dns.googleI get a result just like yours. However, if I use the -T or -u flags (for TCP or UDP) then the trace works normally.
I can confirm this. Probably, the default ICMP rules have changed... UDP or TCP work fine.
My bet is on https://www.freebsd.org/security/advisories/FreeBSD-SA-24:05.pf.asc which pulled in hundreds of lines of changes in the pf ICMP handling code. I've seen it previously pass by on stable/14 and I wasn't planing to merge it right away, but the SA tipped the scale in favour of including it.
# opnsense-update -kr 24.7
If the old kernel works it's probably that.
Cheers,
Franco
You guys are right, I was under the impression that mtr on Unix/Linux would use UDP as a default like the traceroute implementation on these platforms does. Which is clearly not the case.
So, it's ICMP that's affected here.
I allow ICMP echo and IPV6-ICMP echo from anywhere to anywhere, because I do believe in troubleshooting tools and not quite as much in security by obscurity.
To me it looks like the previous version of pf treated an ICMP time exceeded in reply to an ICMP echo as part of the same state/connection and hence permitted the reply in. Now it doesn't. I wonder if this was intended.
Also simply permitting ICMP time exceeded in addition to echo does not help, because the NAT state is missing in case of IPv4.
Back to the drawing board, FreeBSD ;)
Quote from: franco on August 08, 2024, 09:58:22 PM
My bet is on https://www.freebsd.org/security/advisories/FreeBSD-SA-24:05.pf.asc which pulled in hundreds of lines of changes in the pf ICMP handling code. I've seen it previously pass by on stable/14 and I wasn't planing to merge it right away, but the SA tipped the scale in favour of including it.
# opnsense-update -kr 24.7
If the old kernel works it's probably that.
Cheers,
Franco
With the old kernel it works again as it should.
Quote from: Patrick M. Hausen on August 08, 2024, 10:35:03 PM
Back to the drawing board, FreeBSD ;)
Indeed.
"Crafted Echo Request packet after a Neighbor Solicitation (NS) can trigger an Echo Reply" - oh noes, shields up, Captain! We are doomed! ☠️😱
Best to revert this broken fix, IMO.
Same issue here after a two hops upgrade 24.1.10 -> 24.7 -> 24.7.1.
mtr is not working on Ubuntu Linux and macOS clients. The macOS traceroute is working though.
Can see the ICMP Echo Reply being blocked on WAN in firewall log for state violation. Opening all incoming ICMP on WAN does not help.
My first bet is also that it is related to the ICMPv6 security fix.
So it's the new kernel? Anybody confirmed it? Might also be possible to confirm with pfctl -d / test traceroute / pfctl -e as a quick test that pf is doing it.
To be frank we're doomed when we ship security updates too late according to some.
And now we're doomed because we ship security issues in a timely manner because the same corner that said we don't ship them soon enough feeds suboptimal patches to FreeBSD.
Isn't it ironic...
Jokes aside this should probably be reported to https://bugs.freebsd.org but at this point I have no hopes somebody even cares giving the number of past and pending issues in that general direction.
Cheers,
Franco
Quote from: franco on August 09, 2024, 08:10:55 AM
So it's the new kernel? Anybody confirmed it? Might also be possible to confirm with pfctl -d / test traceroute / pfctl -e as a quick test that pf is doing it.
To be frank we're doomed when we ship security updates too late according to some.
And now we're doomed because we ship security issues in a timely manner because the same corner that said we don't ship them soon enough feeds suboptimal patches to FreeBSD.
Isn't it ironic...
Jokes aside this should probably be reported to https://bugs.freebsd.org but at this point I have no hopes somebody even cares giving the number of past and pending issues in that general direction.
Cheers,
Franco
The bug was not present on 24.7 and was definitely introduced with 24.7.1.
Any chance this particular change can be reverted?
> Any chance this particular change can be reverted?
No plans for today. You can revert the kernel as suggested. This needs attention in FreeBSD either way.
Cheers,
Franco
With pfctl -d, the problem is gone (at least with IPv6, as it also turns off NAT, I cannot test IPv4).
Quote from: franco on August 09, 2024, 08:10:55 AM
Might also be possible to confirm with pfctl -d / test traceroute / pfctl -e as a quick test that pf is doing it.
Well yes, disabling the firewall fixes the problem, feel much safer now compared to leaking replies to ping. ;D :P
Quote from: franco on August 09, 2024, 08:57:09 AM
You can revert the kernel as suggested.
That brings back the kernel that panics with IPS, doesn't it? Just as a warning for people. Might rather live with broken ICMP for the moment.
Now, this crafted ping packets nonsense reminds me of this rant I wrote almost 20 years ago. (https://www.wilderssecurity.com/threads/rant-grcs-shields-up-and-true-stealth-firewall-test-or-harmful-fud.216892/)
Ugh.
@doktornotor Are you using Xen?
Quote from: franco on August 09, 2024, 09:50:07 AM
@doktornotor Are you using Xen?
Just for some testing.
Quote from: doktornotor on August 09, 2024, 09:47:42 AM
Now, this crafted ping packets nonsense reminds me of this rant I wrote almost 20 years ago. (https://www.wilderssecurity.com/threads/rant-grcs-shields-up-and-true-stealth-firewall-test-or-harmful-fud.216892/)
http://www.ranum.com/security/computer_security/papers/a1-firewall/index.html
8)
@doktornotor
# opnsense-update -zkr 24.7-xen3
I'll leave it there for a while longer then.
Cheers,
Franco
Quote from: Patrick M. Hausen on August 09, 2024, 09:57:05 AM
@doktornotor
http://www.ranum.com/security/computer_security/papers/a1-firewall/index.html
8)
LMAO! Printed that to PDF. 8) ;D
Quote from: franco on August 09, 2024, 09:58:27 AM
@doktornotor
# opnsense-update -zkr 24.7-xen3
I'll leave it there for a while longer then.
Thanks, will be potentially useful for others as well.
Quote from: franco on August 09, 2024, 08:10:55 AM
Jokes aside this should probably be reported to https://bugs.freebsd.org but at this point I have no hopes somebody even cares giving the number of past and pending issues in that general direction.
Done if someone here wants to chime in there - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701, I most likely won't have time to follow the usual requested steps to reproduce an apparent bug just because it manifests on OPNsense instead of "vanilla" FreeBSD.
Quote from: franco on August 09, 2024, 09:58:27 AM
# opnsense-update -zkr 24.7-xen3
I'll leave it there for a while longer then.
What are the differences and benefits of that "xen Kernel"?
This was the 24.7.1 kernel state before the FreeBSD security advisories hit yesterday with all pressing user reported things fixed.
Cheers,
Franco
@franco - added so far requested info (seems to go well as usual ::)) to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701; if people want to take over from there, I really don't have days to debug this.
Summary:
- 24.7 or 24.7-xen3 -> working traceroute
- 24.7.1 - traceroute broken.
- reproduced on a box with default OPNsense firewall rules, DHCP WAN, default LAN.
Wondering who's testing these patches on "stable" really. Sigh.
So it is not a special kernel which should be generally used with opnsense in a Xen hypervisor scenario?
xen, xen2 and xen3 were test patch iterations while working on IPS/netmap crashes within Xen since the problem appeared with FreeBSD 14.1. It was fixed. It's included in 24.7.1 too.
Cheers,
Franco
Thx Franco for the clarification and your relentless commitment
I'm on a bare metal install and also seeing the ICMP issue on traceroutes. Screenshot of my liveview log attempt to run mtr to one of quad9's IPV6 servers.
Quote from: franco on August 09, 2024, 09:58:27 AM
@doktornotor
# opnsense-update -zkr 24.7-xen3
I'll leave it there for a while longer then.
Cheers,
Franco
Is this supposed to revert to a kernel that has the expected ICMP behavior?
Quote from: motoridersd on August 11, 2024, 07:16:28 PM
Is this supposed to revert to a kernel that has the expected ICMP behavior?
Yes.
I'll publish a couple of revert kernels tomorrow to see which one is the culprit:
https://github.com/opnsense/src/commit/b34fe955
https://github.com/opnsense/src/commit/38384a54
https://github.com/opnsense/src/commit/f924c2e1f
https://github.com/opnsense/src/commit/9ceb7fda9
It's only a couple of hundreds of lines changed overall ;)
Cheers,
Franco
At such a late stage in the game, we might want to wait for FreeBSD 15.1-RELEASE to be out first ? Heard there's a couple extraordinary patches in the queue for FreeBSD 18.0-RELEASE written by a bunch of 13yo kids that they just committed.
Let's be fair. From the looks of it it doesn't get better if we don't help out. That being said:
# opnsense-update -zkr 24.7.1-pf1 reverts b34fe955
# opnsense-update -zkr 24.7.1-pf2 also reverts 38384a54
# opnsense-update -zkr 24.7.1-pf3 also reverts f924c2e1f
# opnsense-update -zkr 24.7.1-pf4 reverts all (including 9ceb7fda9)
Cheers,
Franco
opnsense-update -zkr 24.7.1-pf4 restores traceroute functionality.
Thanks, that means somewhere in the biggest patch ;)
https://github.com/opnsense/src/commit/9ceb7fda9
2 changed files with 317 additions and 83 deletions.
Cheers,
Franco
Great!
Can't help you with that.
But if more testing is needed today i can help.
No worries, that helps a lot already. More confirmations of which kernel starts are welcome too so we can say "independently verified" :D
For some reason I cannot reproduce this at first glance on my end.
Cheers,
Franco
Quote from: franco on August 12, 2024, 10:22:14 AM
Thanks, that means somewhere in the biggest patch ;)
https://github.com/opnsense/src/commit/9ceb7fda9
2 changed files with 317 additions and 83 deletions.
Cheers,
Franco
Hmmm, so that's basically a 2009 OpenBSD patch, pretty much verbatim (on a very quick look)? Not even sure what to think about all of this... How many of the subsequent 15 years of fixes have been missed? ??? ::)
https://github.com/openbsd/src/commit/70bf7555ef4c33faa35582dadab7c01bcf61b3ac
I'm not even sure which one the security fix is per se, but it's correct that a lot has been pulled in to make the security fix happen. That's exactly why we are in this situation now and why I've advocated for better stable branch management and better bug report responses in FreeBSD to no avail. I'm not even making this up, but it's being ignored because I'm the annoying one helping run an adjacent project that gets responses such as this for raising concern and bug reports and fixes:
"FreeBSD is a volunteer project. If you don't like what you get, contribute."
Quite the hot take. I mean a cherry-pick only costs 10 seconds of real work if you ask me that's been dragging on for weeks. But that's more of a general rant.
Cheers,
Franco
I can confirm that -pf3 is NOT sufficient to fix the regression. (Did not test -pf[12], trying to test on some real deployment and people at home are getting kinda angry of the reboots, seems pointless anyway.)
While browsing the OpenBSD github, I've seen some 15+ later commits regarding ICMP/ICMPv6 and states in pf.c alone.
Considering this pressing security issue apparently has been there for 15+ years unnoticed by anyone, let alone anyone exploiting echo replies via "crafted packets", eh... WTH really. #SMH
Probably relevant and as expected @franco - https://marc.info/?l=openbsd-misc&m=128218328308200&w=2 so that narrows the fix down to between the broken forward-ported patch from 2009 and OpenBSD 4.8 release.
This one then? https://github.com/openbsd/src/commit/ef4bccd7509e
Likely... can you do a quick -pf5? ;D Willing to annoy people at home once again.
Doesn't apply at first glance. Don't want to spend my time on this just yet.
Cheers,
Franco
Quote from: franco on August 12, 2024, 11:51:43 AM
Doesn't apply at first glance. Don't want to spend my time on this just yet.
Hah. That's probably why it's been omitted. :D :P
8)
Here's another independent review :)
Started mtr on a pi behind the FW and installed/rebooted the kernels starting with PF1.
PF4 is working as expected, the others don't.
New sponsored patch committed. :P
# opnsense-update -zkr 24.7.1-icmp
It boots. Promising start?
Cheers,
Franco
Quote from: franco on August 13, 2024, 02:24:26 PM
It boots. Promising start?
8) Will test tonight ;D
I just tested opnsense-update -zkr 24.7.1-icmp; results:
- traceroute is working but feels slower than usual
- mtr is working only in tcp/udp mode, it is not working in regular (icmp) mode
For me, traceroute is completely broken with ICMPv6 - even from the firewall itself, with the latest patch. Also as noted above, mtr does not work at all with ICMP, even with IPv4.
Ok, will likely revert (similar to pf4) for 24.7.2 until this has been completely addressed in FreeBSD 14.1. The MFC for the commit is said to be 1 week so I don't think we'll see this progress soon enough either way.
Cheers,
Franco
There is an upstream patch out today that I believe addresses this issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701#c14
I understand that was exactly the patch contained in the last kernel by Franco.
However, the first part of the patch only addresses test cases and the second part says something about icmp-in-icmp lookups, which I do not understand and which obviously did not solve the problem.
Yep, 24.7.1-icmp was from this branch containing the two commits: https://github.com/opnsense/src/commits/pf_icmp/
It's safe to say the test case was fixed. The new test cases being the first traceroute test, however, means there is no coverage for further cases which then still need a fix. From early user feedback here that seems to be the case.
The size of the fix is also in no means equivalent to the OpenBSD pf patch fixing ICMP:
https://github.com/opnsense/src/commit/5c2b2da661
vs.
https://github.com/openbsd/src/commit/ef4bccd7509e
While it doesn't have to be it would suggest the same thing as the test case situation and early user feedback.
We'll be reverting this for 24.7.2 as mentioned because I don't think this will (or can?) move as urgently as it should.
Cheers,
Franco
Re-tested on another box. IPv6 is definitely still broken. (Now, even the first hop is not shown when you try ICMPv6 traceroute/mtr or similar from machines behind)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701#c15
Indeed seems best to revert the commits until fixed properly.
Thanks for following up in the FreeBSD bug tracker. <3
Quote from: franco on August 14, 2024, 10:00:56 AM
Thanks for following up in the FreeBSD bug tracker. <3
No problem. Did another reboot while applying the new PPPoE patch (https://forum.opnsense.org/index.php?topic=42081.msg207971#msg207971) as well - that one works just fine. ;D
Ok, I've reverted on stable/24.7 now, the 24.7.1-pf4 is still there but holds the code that is going to be 24.7.2 next week and remove the other ones. 24.7.4-icmp is still up if more datapoints need to be collected. With 24.7.2 I can hopefully clear the bulk of the test kernel snapshots. ;)
Cheers,
Franco
Any idea why I can't get my hand on 24.7.1-pf4 ?
# opnsense-update -zkr 24.7.1-pf4
Fetching kernel-24.7.1-pf4-amd64.txz: ..[fetch: https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/sets/kernel-24.7.1-pf4-amd64.txz.sig: Not Found] failed, no signature found
Uploaded it, but didn't publish. Fixed now. Sorry.
Cheers,
Franco
Quote from: franco on August 14, 2024, 02:38:04 PM
Uploaded it, but didn't publish. Fixed now. Sorry.
On that note... Wasted some time yesterday before I realized I forgot to plug WAN cable in. Had nothing to do with signatures missing. 🤦♂️ ;D
Can't fix that, sorry ;)
I actually wrote a tool to ease mirror maintenance because I was constantly double and triple-checking if the manual cp/mv/rm was a good idea and working as intended. Now the tool is easy to use but also easy to forget.
Cheers,
Franco
Hmm, -icmp2? :P
https://cgit.freebsd.org/src/commit/?id=89f6723288b0d27d3f14f93e6e83f672fa2b8aca
In the name of science, why not? It's online.
Cheers,
Franco
Quote from: franco on August 14, 2024, 05:19:40 PM
In the name of science, why not? It's online.
Seems like it's finally working with IPv4 and IPv6... :o :o :o Waiting for others here to chime in:
opnsense-update -zkr 24.7.1-icmp2and reboot.
And a Confirmation from my side as well!
Windows 11 pro traceroute via IPv6 AND IPv4 is working fine!
-icmp2 working for me too.
traceroute command works but mtr doesnt on icmp2 for me. It's the same as before.
Quote from: Sasquatch5177 on August 14, 2024, 06:36:04 PM
traceroute command works but mtr doesnt on icmp2 for me. It's the same as before.
Hmmm, works here both from the firewall and local networks.
# mtr -wrn -c 10 -4 www.google.com
Start: 2024-08-14T18:39:28+0200
HOST: unifi Loss% Snt Last Avg Best Wrst StDev
1.|-- 192.168.0.254 0.0% 10 0.5 0.7 0.4 1.6 0.3
2.|-- 89.24.145.107 0.0% 10 6.4 6.6 6.1 6.9 0.3
3.|-- 213.29.94.202 90.0% 10 6.5 6.5 6.5 6.5 0.0
4.|-- 213.29.94.201 0.0% 10 7.7 7.7 7.2 9.3 0.6
5.|-- 192.178.68.76 0.0% 10 7.3 7.1 6.9 7.7 0.3
6.|-- 192.178.98.175 0.0% 10 7.1 7.3 6.7 8.1 0.5
7.|-- 209.85.245.247 0.0% 10 6.9 7.0 6.4 8.0 0.5
8.|-- 142.251.36.68 0.0% 10 6.9 7.2 6.7 7.8 0.4
# mtr -wrn -c 10 -6 www.google.com
Start: 2024-08-14T18:39:54+0200
HOST: unifi Loss% Snt Last Avg Best Wrst StDev
1.|-- 2001:1ae9::xxxx 0.0% 10 0.8 0.8 0.7 1.0 0.1
2.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
4.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
5.|-- 2001:4860:1:1::1d50 0.0% 10 8.0 7.4 7.1 8.0 0.3
6.|-- 2001:4860:0:1::7e69 0.0% 10 8.4 16.5 7.8 57.4 17.7
7.|-- 2001:4860:0:1::3b6f 0.0% 10 8.5 7.1 6.7 8.5 0.5
8.|-- 2a00:1450:4014:80a::2004 0.0% 10 7.6 7.1 6.7 7.6 0.3
mtr works for me too on icmp2. ipv4 & ipv6.
Kernel icmp2 works for me, too, for both IPv4 and IPv6.
Quote from: doktornotor on August 14, 2024, 05:39:41 PM
Quote from: franco on August 14, 2024, 05:19:40 PM
In the name of science, why not? It's online.
Seems like it's finally working with IPv4 and IPv6... :o :o :o Waiting for others here to chime in:
opnsense-update -zkr 24.7.1-icmp2
and reboot.
Can confirm, works here too.
In the name of even more science :), how is the pf4 that was rebuilt today as well different from icmp2 ? Do we need to test that one too ?
Well, icmp2 has all the upstream FreeBSD patches from the bug linked in the thread.
The pf4 is the one with all reverted.
Thanks guys, what I meant was the time stamp changed on the pf4 as it would have been rebuilt today, but that's ok. Running on icmp2 now, looks good so far.
kernel-24.7.1-pf4-amd64.txz 2024-08-14 12:39 33M
kernel-24.7.1-pf4-amd64.txz.sig 2024-08-14 12:39 1.3K
kernel-24.7.1-icmp2-amd64.txz 2024-08-14 15:20 33M
kernel-24.7.1-icmp2-amd64.txz.sig 2024-08-14 15:20 1.3K
Quote from: franco on August 14, 2024, 01:36:24 PM
Ok, I've reverted on stable/24.7 now, the 24.7.1-pf4 is still there but holds the code that is going to be 24.7.2 next week
;)
Well, it's not official until we get people panicking about the revised SA and when it will be in OPNsense :D
TBH, I'm still waiting for that video saying we ship incomplete code and that's why we are not being used. 8)
No worries, that was just the pilot episode. Spacetime will be bent again by the power of his mind - right after the next unaffiliated payment is received.
How should we proceed? Ship the fixes or stay with the reverts?
Cheers,
Franco
GIven the security nature of it my vote is to ship. It is unlikely FreeBSD will make other changes if there aren't any other tests and the confirmed regressions have been addressed.
Agree with newsense FWIW. Ship the fixes gets my vote.
+1: Ship the fixes.
Ship the fix!
gotta love you all <3
Upstream, upstream, upstream! Ship, ship, ship! ;D
Tested -icmp2 with ipv4 and ipv6 traffic, both forwarded traffic as well as traffic originating from the firewall itself, and everything seems to be working now. I'd say ship this one.
Hmm, icmp2 is not working for me. Am I missing something?
opnsense-update -zkr 24.7.1-icmp2
and rebooted. Can see that I am running "pf_icmp-n267786-b4771b598e90" but pings and traceroute don't work from the firewall or behind it.
Well, the obvious question would be did any kernel on 24.7 ever work on your end?
Cheers,
Franco
Just as another data point, icmp2 works for me, IPv4+6, local + forwarded ICMP traceroutes tested. Thanks to all involved!
Quote from: franco on August 16, 2024, 08:23:38 AM
Well, the obvious question would be did any kernel on 24.7 ever work on your end?
Cheers,
Franco
No, I upgraded from 24.1.10 directly to 24.7.1 and that's when I noticed pings weren't working. I don't use them daily on my network, but often do and had never noticed they didn't work. My first indication they were broken was seeing the Gateway as "down" in the new dashboard.
Quote from: motoridersd on August 16, 2024, 04:33:19 PM
My first indication they were broken was seeing the Gateway as "down" in the new dashboard.
I don"t think we are discussing the same issue at all. I'd suggest to start a new topic.
I have the same issue
tracert doesn't work as it suppose to work
If I do it in the opnsense GUI it works fine.
Quote from: doktornotor on August 14, 2024, 05:39:41 PM
Quote from: franco on August 14, 2024, 05:19:40 PM
In the name of science, why not? It's online.
Seems like it's finally working with IPv4 and IPv6... :o :o :o Waiting for others here to chime in:
opnsense-update -zkr 24.7.1-icmp2
and reboot.
Confirm this works as well
Hi there,
I can also confirm that the -icmp2 kernel works fine here, both ipv4 and ipv6
br br
Ran into gateway monitoring nonsense and I thought I was going insane ;D
opnsense-update -zkr 24.7.1-icmp2
-> fixes things beautifully :)
With 24.7.1-icmp2 i still have problems on the ipv6 link.
sometime packetloss on all ipv6 addresses ... ipv4 is working.
16 bytes from 2a00:1450:4001:81c::2003, icmp_seq=60 hlim=117 time=5.803 ms
16 bytes from 2a00:1450:4001:81c::2003, icmp_seq=61 hlim=117 time=5.750 ms
16 bytes from 2a00:1450:4001:81c::2003, icmp_seq=62 hlim=117 time=5.698 ms
16 bytes from 2a00:1450:4001:81c::2003, icmp_seq=63 hlim=117 time=5.712 ms
16 bytes from 2a00:1450:4001:81c::2003, icmp_seq=79 hlim=117 time=2081.821 ms
16 bytes from 2a00:1450:4001:81c::2003, icmp_seq=80 hlim=117 time=1028.673 ms
16 bytes from 2a00:1450:4001:81c::2003, icmp_seq=81 hlim=117 time=10.623 ms
16 bytes from 2a00:1450:4001:81c::2003, icmp_seq=82 hlim=117 time=5.689 ms
16 bytes from 2a00:1450:4001:81c::2003, icmp_seq=83 hlim=117 time=5.617 ms
16 bytes from 2a00:1450:4001:81c::2003, icmp_seq=84 hlim=117 time=5.603 ms
16 bytes from 2a01:xx icmp_seq=50 hlim=58 time=13.888 ms
16 bytes from 2a01:xx, icmp_seq=51 hlim=58 time=13.758 ms
16 bytes from 2a01:xx, icmp_seq=52 hlim=58 time=13.216 ms
16 bytes from 2a01:xx, icmp_seq=53 hlim=58 time=13.322 ms
16 bytes from 2a01:xx, icmp_seq=70 hlim=58 time=1412.567 ms
16 bytes from 2a01:xx, icmp_seq=71 hlim=58 time=367.940 ms
16 bytes from 2a01:xx, icmp_seq=72 hlim=58 time=13.363 ms
With 24.7.1-pf4 and 24.7 Kernel no issues. Switch back to 24.7.1-icmp2 and the issue are back.
QuoteWith 24.7.1-pf4 and 24.7 Kernel no issues. Switch back to 24.7.1-icmp2 and the issue are back.
I'm experiencing the same issue. For example, all my Android devices frequently think they are offline for a few seconds at a time—this happens a couple of times an hour that I've noticed.
The issue seems to occur inconsistently, which makes it challenging to pinpoint.
If you need any logs or additional information, please let me know. I'm happy to help.
With icmp2 kernel i see multiple reconnects of a google chromecast "google tv" in my unifi dashboard.
Running kernel pf4 for a while now. And it looks like the reconnects are gone.
Maybe the reverts are the best for 24.7.2
Quote from: staticznld on August 18, 2024, 07:59:02 PM
Maybe the reverts are the best for 24.7.2
I get your sentiment but it looks better to move ahead and follow-up in the FreeBSD ticket. Who knows how long it will take to fix all use cases?
Cheers,
Franco
Quote from: franco on August 19, 2024, 11:47:26 AM
Quote from: staticznld on August 18, 2024, 07:59:02 PM
Maybe the reverts are the best for 24.7.2
I get your sentiment but it looks better to move ahead and follow-up in the FreeBSD ticket. Who knows how long it will take to fix all use cases?
Cheers,
Franco
Hi Franco,
Is it safe to expect the 24.7.2 release this week?
Yes, Wednesday is a safe bet.
Cheers,
Franco
Running ICMP2 kernel for 22 hours now and not seeing any reconnect from the Chromecast!
I think my Chromecast is the problem.
It could be the patches. I'm pretty sure they don't cover all edge cases yet. But as noted it's not good to revert to the old state because I don't think we'll see fixes sooner then if we're the only ones noticing.
Cheers,
Franco
Just installed 24.7.2
Traceroutes / ICMP behaviour seems back to normal (for IPv4 at least). I don't use IPv6 so can't test that.
Huge thanks to Franco for getting an update out so quick to fix it and to doktornotor and the others for testing and submitting the bug report to FreeBSD.
I would also like to thank everyone who helped test this! :)
I think this isn't over yet, but at least we are one step further:
https://forum.opnsense.org/index.php?topic=42270.0
https://github.com/opnsense/src/issues/217
Cheers,
Franco
updated to 24.7.2, but i'm still losing some random packets on ipv6 ping (going out, only a jump after my firewall)
--- ipv6.abc.xyz ping statistics ---
1566 packets transmitted, 1547 received, 1.21328% packet loss, time 1567560ms
rtt min/avg/max/mdev = 0.185/2.232/1482.222/43.446 ms, pipe 2
I was more stable on "24.7.1-pf4" (no packet loss)
Yep.
It seems like shipping the FreeBSD fixes was not the right decicion after all as there are still problems with neighbor discovery and upstream does not care all that much:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701
Not surprising, but it is what it has been for a while now. The bug was closed mainly to wrap up the revised SA and ship it in 14.1-RELEASE-p4. These types of issues often have a long round trip time WRT to the remaining issues.
Shipping the initial SA in all supported versions of FreeBSD with the scope of hundreds of lines of code changed was a release engineering mistake. This is clear and simple.
That being said I appreciate that someone actually went ahead and fixed the main mistakes with it without making an immediate scene about it.
Cheers,
Franco
Ok, if someone wants to file a new bug with FreeBSD for the remaining regressions caused by patching a security non-issue, feel free. I've had it with upstream for some time, it may be harmful to mental health apparently.
Could not resist commenting on the whole SA - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701#c40
@Uwe I'm sure we can test on FreeBSD, but they need to give a direction of what they actually want instead of saying that some FreeBSD commit actually is solely a downstream issue... It actually doesn't matter if we test on FreeBSD or OPNsense kernel because we talk about the same code change. This is mind boggling to me. :)
@doktornotor appreciate what you did there from the start
The exercise in patience and restraint that this one of many sagas is commendable.
That discussion on the FreeBSD list is wild. People report issues in good faith, and show that reverting the offending code fixes things. At the same time, FreeBSD maintainers are deaf and point to others and just close the topic :o
Probably the easiest is to just not ship kernels with the offending "fixes" and when pfsense hits the same issue, people will probably believe it :-X :o
Quote from: franco on August 23, 2024, 11:32:50 AM
It actually doesn't matter if we test on FreeBSD or OPNsense kernel because we talk about the same code change. This is mind boggling to me. :)
Yeah, it doesn't matter - except that the obvious fact that guy who committed all of this code without any test coverage causing the regressions here uses the FreeBSD code on the "other project" and does not care about breaking stable FreeBSD releases at all, since that other project happens to run on -CURRENT.
For future interaction with upstream - how much of a trouble / overhead would shipping a matching vanilla FreeBSD kernel for the regression debugging purposes be? I mean, identical config, just no patches. With the boot environments (snapshots) in place, shouldn't be much of an issue even if it fails to boot altogether.
I know that it doesn't matter at all (and I indicated it in the bug report), all that would do is to provide proof that it is not a downstream issue, so they are forced to come out of their corner and admit that their fix cannot be accepted as final.
When you think about it, you can:
1. Prove that it is an upstream issue and hope for an upstream fix.
2. Wait for the "other project" to stumble over this as well.
If you opt for 1, 24.7.3 could potentially fix the problem going forward and be done with it. If you go with 2, you should revert the patches for OpnSense if you want an intermediate fix. In the latter case, you would have to touch it again if/after a real upstream fix. On the other hand, choosing option 1 would help the "other project" - maybe without them even ever knowing.
P.S.: This is only a discussion of how to proceed on OpnSense's behalf, as clearly I also do not like how upstream shrugs this off as an "OPP" (other people's problem, or in german: "PAL" (Problem anderer Leute)). 8)
Ever since that awkward exchange for https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273207#c18 and making up FreeBSD policies on the spot I think Kristof never followed up on any of my patch submissions and bug reports and even his own bugs in FreeBSD ports for example. This wasn't limited to my reports either as other mishaps record...
I've talked to multiple distinguished FreeBSD authorities and the same message is repaeted over and over:
* FreeBSD is a volunteer project
* FreeBSD committers are volunteers
* Mistakes happen, everyone is human
I agree with these things, but the reality is nothing gets done to the point of valid patches for actual release regressions being ignored and FreeBSD release versions "surprisingly" turning up broken with a repeating and problem-scope-increasing committer pattern.
I don't see a reason to build something on a vanilla FreeBSD unless someone there outgrows their current lethargy that allows everyone to do whatever they want with the FreeBSD code. As far as code correctness is concerned I draw the line. FreeBSD currently draws the line at the user/downstream relationships. I have no clue what is happening there and what the goal of this is.
I plan to work on the code next week to see if we can find it. If anyone will bother to review and accept a potential patch in FreeBSD is the big money question. :)
Cheers,
Franco
Quote from: franco on August 23, 2024, 12:14:43 PM
I don't see a reason to build something on a vanilla FreeBSD unless someone there outgrows their current lethargy that allows everyone to do whatever they want with the FreeBSD code. As far as code correctness is concerned I draw the line. FreeBSD currently draws the line at the user/downstream relationships. I have no clue what is happening there and what the goal of this is.
FWIW, I haven't seen similar amount of hostility towards others actually making use of open-source code for their projects pretty much anywhere else. The whole thing is indeed absurd.
Additionally, while fixes for many well-known issues never make it back to "stable", no matter how minor and how much time was there to backport them - now all of a sudden someone forward-ports hundreds of lines of OpenBSD code, rushes it
all the way back to 13.x, presents this as security issue and causes severe regressions that they subsequently cannot be bothered to fix completely and properly.
I happen to have the impression that even if I took vanilla FreeBSD and the only modification done was changing
/etc/motd to say OPNsense, everything reported would be a downstream issue.
Nice waste of time of the people involved.
Quote from: franco on August 23, 2024, 12:14:43 PM
I plan to work on the code next week to see if we can find it. If anyone will bother to review and accept a potential patch in FreeBSD is the big money question. :)
If you need any testers, I can reproduce the situation here. Feel free to reach out in the topic here.
Quote from: doktornotor on August 23, 2024, 12:34:55 PM
FWIW, I haven't seen similar amount of hostility towards others actually making use of open-source code for their projects pretty much anywhere else. The whole thing is indeed absurd.
The same has been true for OpenBSD for many years when you present them a non standard kernel build. Everyone will ignore you. In this day and age and open source collaboration focusing on patches and code changes is crucial and effective since git entered the scene making this much much much much much much easier (and practically fool-proof). Some of the developers don't like that I think, but to be honest I'd rather trust my random OPNsense user testing something for me than a OS developer who I know will lie to my face because it makes his day easier. This is based on real world evidence with OpenBSD. Patch submissions are similarly derailed because "FreeBSD kernel code is stupid". If you want that funny exchange I can find it for you.
Quote from: doktornotor on August 23, 2024, 12:34:55 PM
Additionally, while fixes for many well-known issues never make it back to "stable", no matter how minor and how much time was there to backport them - now all of a sudden someone forward-ports hundreds of lines of OpenBSD code, rushes it all the way back to 13.x, presents this as security issue and causes severe regressions that they subsequently cannot be bothered to fix completely and properly.
No lines drawn by release engineering. This had to backfire. For me it's nice to have this general issue on record with FreeBSD release engineering who assured me all things we see are honest mistakes before this escalated in the SA. I can only assume it was green-lit by RE despite the actual code related and general concerns voiced over process. So: concerns not heard, no policies to prevent this, escalating problems in releases. We haven't seen the end of it even.
And for the record FreeBSD 14.1 was a very smooth jump overall. I'm happy with the quality level for everyone involved. I even don't mind these things slipping through (even if you argue 2022 is a long time for something to go unnoticed). The real deal is how defects and regressions are being handled and perhaps prevented. I'm glad we have this discussion out in the open at the moment at least.
Quote from: Wirehead on August 23, 2024, 12:58:52 PM
If you need any testers, I can reproduce the situation here. Feel free to reach out in the topic here.
Thanks, mate. I will publish two test kernels to discern which exact FreeBSD commit this is and dump it over there for all that it's worth. In the name of progress and cooperation. :)
Cheers,
Franco
Quote from: franco on August 23, 2024, 01:00:40 PM
In this day and age and open source collaboration focusing on patches and code changes is crucial and effective since git entered the scene making this much much much much much much easier (and practically fool-proof). Some of the developers don't like that I think, but to be honest I'd rather trust my random OPNsense user testing something for me than a OS developer who I know will lie to my face because it makes his day easier.
I mean, BSD does not exactly have a huge user base, does it. Where it's being used significantly - those are (were) downstream projects such as routers/firewalls, or NAS due to ZFS. In the latter field, due to ZFS making it to Linux, the major guys apparently already decided it's not worth the trouble any more - so there's just XigmaNAS left (which is barely alive).
If you discourage the remaining projects and contributors/users they bring, then there's nothing left, pretty much. "The other project" which will be undownloadeable soon won't cut it. Congrats, you now have a system that noone is using, beyond the bunch of committers and hardcore enthusiasts.
:(
I'm not going to comment on what I think here overall but I will say that I believe downstream projects are a healthy part of upstream projects. Unfortunately a number of important people do not believe that to be so.
That being said we've built all these helper tools in OPNsense over the years to allow for rapid test and fix deployment for the sake of bringing the code base forward which includes all upstream projects. You can see where that stops being useful when we start sitting on patches which is also the only way we can go forward because otherwise we don't go forward at all if we need to wait for upstream to move but upstream won't move because we have "modifications". For the most part this boils down to opinions on necessity and trust. But opinions are just that. Fixes are nicer to have than not IMO.
Expect an EuroBSDCon 2024 presentation about this exact topic. :)
Cheers,
Franco
Because we talked about this internally this official statement came to memory again. ;)
https://www.netgate.com/blog/pfsense-software-is-moving-ahead
If anyone wants to participate in the identification of the ND-ICMPv6 stall commit cause we started another hunt:
https://github.com/opnsense/src/issues/218#issuecomment-2307051831
Thanks,
Franco
There is a test kernel out now that disables the newly-added state tracking for neighbour discovery packets:
https://github.com/opnsense/src/issues/218#issuecomment-2308039278
This can be a baseline for further investigation or maybe even a viable workaround for the time being. Let's see what happens.
Cheers,
Franco
Franco, I got completely confused by the chain of patches and reverts and patches of patches... So, did just have a look at the relevant parts of the current code in pf.c in OpenBSD vs. FreeBSD. Yeah, there's something amiss.
pass in quick inet6 proto ipv6-icmp from {any} to {any} icmp6-type {1,2,135,136} keep state label "9dff917e83b570f19343d5e2941a545e" # IPv6 RFC4890 requirements (ICMP)
pass out quick inet6 proto ipv6-icmp from {(self)} to {fe80::/10,ff02::/16} icmp6-type {128,129,133,134,135,136} keep state label "fb0cc70ad35caa7bea0138f49c30623d" # IPv6 RFC4890 requirements (ICMP)
pass in quick inet6 proto ipv6-icmp from {fe80::/10} to {fe80::/10,ff02::/16} icmp6-type {128,133,134,135,136} keep state label "d147534c4012c8dd65eda59292c0ab90" # IPv6 RFC4890 requirements (ICMP)
pass in quick inet6 proto ipv6-icmp from {ff02::/16} to {fe80::/10} icmp6-type {128,133,134,135,136} keep state label "df042096359aa49094a20b3ac111f4b7" # IPv6 RFC4890 requirements (ICMP)
pass in quick inet6 proto ipv6-icmp from {::} to {ff02::/16} icmp6-type {128,133,134,135,136} keep state label "d8fdc41aeac05a86adfb74e6052317d8" # IPv6 RFC4890 requirements (ICMP)
Wondering what'd be the effect of using sloppy state here.
Definitely something about state. The 24.7.2-nd patch is really the latest FreeBSD state shipped with the 24.7.2 release and that single line change to deactivate state tracking for ND.
I think you can graft something with state none or sloppy in the ruleset, but nothing says eternally stuck with it like a short hack to avoid this as a workaround in a production release.
Cheers,
Franco
Yeah. For ICMP, it even lacks sloppy states documentation in FreeBSD (or I am blind). For OpenBSD:
https://man.openbsd.org/pf.conf#sloppy
Quote
For ICMP, this option allows states to be created from replies, not just requests.
The ICMP sloppy looks like an amendment loosely based on the 2009/2010 patches. So it's pretty natural this doesn't show in FreeBSD (and probably doesn't work yet, possibly also missing code).
I'd say FreeBSD would strongly benefit from merging as much as possible from the pf implementation in OpenBSD instead of porting random cherrypicks in incomplete and broken ways. Pretty sure that's not going to happen though.
Anyway, 24.7.2-nd looks perfectly fine here. Not sure what to do with that regarding telling that bad news to the maintainer though - since it's of course a downstream issue. After all, they've already been hinted (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701#c11) that the code porting was incomplete.
Kristof's reply is priceless:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701#c49
Quote(In reply to Gordon Tetlow from comment #48)
> kp, does the analysis in comment 46 indicate an issue that needs further review?
What issue? There's been a lot of conspiracy theorising, but no actual bug report beyond "opnsense is broken".
I genuinely don't know what's supposed to be broken.
I wish he would comment on things like these ever:
https://github.com/freebsd/freebsd-src/pull/1390
https://github.com/freebsd/freebsd-src/pull/1391
:)
Quote from: franco on August 24, 2024, 10:16:14 AM
Kristof's reply is priceless:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701#c49
Not even sure what to say. When things get to this state, in normally managed open-source projects, developer relations / project leadership (in whatever form) intervenes and take the offending developers to the task.
If that does not work with FreeBSD project, there's not much hope.
Anybody else seeing the irony in discussing applicability of downstream bugs during a bug report session regarding a series of loosely applied patches from OpenBSD on FreeBSD?
Cheers,
Franco
I am still giving them the benefit of the doubt. In my 45 years of experience in this field, I have found that developer qualifications can differ by powers of ten.
Some type of developers tend to react in unexpected and justifying ways if they feel that they are being critized instead of just assuming everyone only wants to make things (i.e. software) better. Guess if those are the better or the worse ones? Maybe that is hard to grasp for someone more gifted... ;)
Hence why I tend to deescalate and put less stress on them if I want to achieve something. Works most of the time, see the last reaction.
Quote from: franco on August 24, 2024, 12:17:37 PM
Anybody else seeing the irony in discussing applicability of downstream bugs during a bug report session regarding a series of loosely applied patches from OpenBSD on FreeBSD?
Meh... As I noted on the bug, it's gonna be broken exactly in the same way in their upcoming delayed release. (Unless they maintain separate code which either does not have the SA applied or has fixes they've "forgotten" to merge back upstream - wouldn't be the first time either, IIRC).
Instead of making use of the downstream projects and their users to improve and fix the code, lets dismiss all those bugs reports. Cannot see how that helps anyone.
Most users certainly lack the skills to be selectively reverting parts of a huge port of code from another OS and recompiling kernels over and over again.
I wouldn't consider insisting that an issue exists is escalating. This is a FreeBSD scope discussion. Eventually FreeBSD can do what it wants. From what we know this issue won't go away either. It doesn't hurt stressing this point.
However, over the last 12 months we have seen a developer who has almost a dozen recorded cases of discarding bug reports from users ("non" FreeBSD and plain FreeBSD users), not following up on review questions, discarding external patch submissions by not responding to them, suboptimal upstream patching in the case of OpenVPN, leaving FreeBSD ports broken ('member miniupnpd) while sitting on a patch for weeks, leaving FreeBSD release versions broken due to not adhering to MFC policies and now derailing a bug report on a security-relevant patch set that nobody has the slightest idea about in terms of impact.
Where will we go next?
I got unverified information just then that this IPv6 weirdness we're seeing is wider than just ND solicit and advertise. Good times.
Cheers,
Franco
Quote from: franco on August 24, 2024, 12:43:48 PM
I got unverified information just then that this IPv6 weirdness we're seeing is wider than just ND solicit and advertise. Good times.
Well, that would not be surprising. Requires test cases for each and every ICMP type/code that all of a sudden attained "stateful" handling in
pf. Those mostly don't exist, and I'm seriously discouraged from spending time experimenting with all that stuff to find out what else got broken on the way to report it only to be told "that's downstream problem".
> Requires test cases for each and every ICMP type/code that all of a sudden attained "stateful" handling in pf.
Here is the last thought of the day:
I think OpenBSD made the right choice to pull this off in 2009 when IPv6 was still being laughed at for being almost nonexistent. There just wasn't a lot of real world evidence and much was repaired and aligned over the span of 15 years judging by the code.
Pulling this off in 2024, however, with IPv6 being a daily driver messing with its lifeline ICMPv6 with experiments from 2009 and then saying there is no issue (except downstream of course) is controversial for a multitude of reasons. Sure, the effects are barely visible, but so many people have noticed already here in such a short timeframe contrary to involved developer's belief. And this isn't 15.0 introducing it... it was shortcut into all supported releases with such conviction and against all basic release engineering principles and lack of understanding of IPv6.
I sincerely hope pfSense users are going to be spared from this whole situation. But only time will tell. :)
Good night,
Franco
My thoughts exactly. You cannot fast-forward 15 years of technical debt in this way without breaking lots of things. Ok, that is normal. The denial is not *insert Gordon Ramsay angry face here*. And yes, that's something that should have been targeted for 15 release.
Quote from: franco on August 23, 2024, 11:32:50 AM
It actually doesn't matter if we test on FreeBSD or OPNsense kernel because we talk about the same code change.
I'm late to the party, but I got the impression that upstream thinks we did this to ourselves by choosing to diverge from their kernel. Would validating it on a vanilla FreeBSD kernel as @Uwe suggested remove that argument? It is a slippery slope.
Quote from: doktornotor on August 24, 2024, 12:49:34 PM
I'm seriously discouraged from spending time experimenting with all that stuff to find out what else got broken on the way to report it only to be told "that's downstream problem".
Can a quick A-B test with their kernel help here as well? Now, this assumes our kernels are not bifurcated to the extent we lose functionality or significantly affect production. I also do not know how feasible this is, or how much work is involved to do this. It was just a thought after reading @Uwe's comment.
I think my intervention already sufficed to give the FreeBSD maintainer reasonable hints to investigate without any further sidesteps - or, in his words: "an actionable item (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701#c54)".
Nevertheless, we will have a corrected 24.7.3 in the meantime (https://github.com/opnsense/src/issues/218). I would have liked to have seen this fixed upstream before Franco needed to step in, because if upstream had reacted earlier, there was no need to rollback / merge any downstream fixes when / if FreeBSD fixes it. I guess that ship has sailed, however...
Quote from: allan on August 25, 2024, 11:14:35 PM
I'm late to the party, but I got the impression that upstream thinks we did this to ourselves by choosing to diverge from their kernel. Would validating it on a vanilla FreeBSD kernel as @Uwe suggested remove that argument? It is a slippery slope.
The problem is given a single unmodified FreeBSD commit breaking this I have no reason to believe we did something elsewhere that would break OPNsense but not FreeBSD. Mind you, this is while applying an OpenBSD patch to FreeBSD assuming that it will go fine and OpenBSD kept refining the ICMPv6 state tracking behaviour over the span of 2009 (where FreeBSD is at now with the patch minus strange inconsistencies) - 2023 (where OpenBSD ended up doing the actual work necessary).
Quote from: allan on August 25, 2024, 11:14:35 PM
Can a quick A-B test with their kernel help here as well? Now, this assumes our kernels are not bifurcated to the extent we lose functionality or significantly affect production. I also do not know how feasible this is, or how much work is involved to do this. It was just a thought after reading @Uwe's comment.
It will not stop the same arguments from happening (modifed base system, modified RC system, modified cheesecake recipe). This is a policy issue, not a technical issue. In the discussion I also mention why downstreams are forced into diverging even on kernel issues. It's not because we want. It's because FreeBSD doesn't care to take patches that it thinks not applicable or there is just nobody who will merge work done by others as frequently as needed. One could argue this would make FreeBSD better, but FreeBSD doesn't necessarily think that.
Cheers,
Franco
/me snickers...
https://github.com/openbsd/src/commit/2633ae8c4c8a64
https://cgit.freebsd.org/src/commit/?id=5ab1e5f7e5585558a73b723f07528977a82cee82
How many of those OpenBSD patches are still missing to fix the downstream issues? :P
Yeah...somehow it looks like 13.4-RC2 was build ahead of the commit being added to the bugzilla...so it's unclear whether it landed there or not.
At least we know what to expect tomorrow morning:
- somebody wakes up, morning light is too bright to read through the commit changes
-builds test kernel(s)
-breakfast
-team meeting(s)
-review new kernel feedback
-finally gets out of bed since it's 7AM :D
You will need to be more specific. The commit has 4 additional commits on master. One is a ND test and one is a cleanup. The other two don't inspire confidence in the previous work. And the OpenBSD 2023 commit is still missing.
Quick, someone else, go test this. I'm pretty sure nobody is going to ask us to test again?
The period to hit stable branches is 1 week anyway. We still have time to wait. 13.4 too it seems.
Cheers,
Franco
Quote from: SA-translatorHear ye, hear ye
In an effort to secure your IPvSixes we've dumped some very old code into our stack from a neighboring project.
Downstream complainers should appreciate the utmost security provided by the fixes that made parts/all of their IPv6 stack unusable, have some vanilla vanilla and check back in another 12-15 years when the next code import is scheduled to be sponsored
End of SecurityAdvisory
P.S. We're working hard so you can boot a machine in miliseconds if you promise not you use TCP/IP
I don't know. It doesn't feel like a course correction patching the issue that allegedly doesn't exist without further comment and a single test case.
No "Tested by:", no "Reviewed by:" on all of these including the latest. I'm going to contact the researchers for clarification on the testing process. Maybe someone can answer that for me.
Cheers,
Franco
Probably two of the bigger questions they need to have an answer for are:
1) Is this SA still a valid concern ? If not will it be retracted ?
2) IF the SA still stands, is it safe for all the machines on the internet running 13 or 14 releases to be passing IPv6 traffic - if this problematic SA hasn't beed applied yet ?
Quote from: franco on September 02, 2024, 06:45:52 AM
You will need to be more specific. The commit has 4 additional commits on master. One is a ND test and one is a cleanup. The other two don't inspire confidence in the previous work. And the OpenBSD 2023 commit is still missing.
Hey, this one (https://cgit.freebsd.org/src/commit/?id=3da3eb6081a2e2f6ea2fed1728d5dd7f9e8786e5) looks like a very universal workaround for all the regressions introduced. ;D (Though, those ICMP sloppy states remain completely undocumented in FreeBSD as discussed previously, so not sure that the work is complete at all...)
> 1) Is this SA still a valid concern ? If not will it be retracted ?
Given the amount of work and grief sunk into this I think not. There was no one to dispute this in FreeBSD or nobody cares either way.
> 2) IF the SA still stands, is it safe for all the machines on the internet running 13 or 14 releases to be passing IPv6 traffic - if this problematic SA hasn't beed applied yet
I will not comment on "safe" but I will say it works however wonky that may be.
As seen in the other thread if you have a lot of devices you will have a lot of ND drops and spurious state creation for no apparent benefit. Unsolicited advertise are likely still broken. IPv6 will silently drop and cause intermittent packet loss for up to roughly 20-40 seconds depending on the behaviour of the devices around the FreeBSD router.
https://github.com/openbsd/src/commit/49f39043a02d
But I mentioned all of that to the relevant authors.
Cheers,
Franco
Quote from: newsense on September 02, 2024, 08:36:13 AM
1) Is this SA still a valid concern ?
My personal take on this is that responding to ICMP echo requests is not a valid security concern, since attempting to block that traffic does introduce any additional security in the first place. (See that longest paragraph in my rant (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701#c40)).
🤷♂️