Traceroute / ICMP issue after 24.7.1 update

[doktornotor]

Kristof's reply is priceless:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701#c49

Not even sure what to say. When things get to this state, in normally managed open-source projects, developer relations / project leadership (in whatever form) intervenes and take the offending developers to the task.

If that does not work with FreeBSD project, there's not much hope.

[franco]

Anybody else seeing the irony in discussing applicability of downstream bugs during a bug report session regarding a series of loosely applied patches from OpenBSD on FreeBSD?


Cheers,
Franco

[meyergru]

I am still giving them the benefit of the doubt. In my 45 years of experience in this field, I have found that developer qualifications can differ by powers of ten.

Some type of developers tend to react in unexpected and justifying ways if they feel that they are being critized instead of just assuming everyone only wants to make things (i.e. software) better. Guess if those are the better or the worse ones? Maybe that is hard to grasp for someone more gifted...  ;)

Hence why I tend to deescalate and put less stress on them if I want to achieve something. Works most of the time, see the last reaction.

[doktornotor]

Anybody else seeing the irony in discussing applicability of downstream bugs during a bug report session regarding a series of loosely applied patches from OpenBSD on FreeBSD?

Meh... As I noted on the bug, it's gonna be broken exactly in the same way in their upcoming delayed release. (Unless they maintain separate code which either does not have the SA applied or has fixes they've "forgotten" to merge back upstream - wouldn't be the first time either, IIRC).

Instead of making use of the downstream projects and their users to improve and fix the code, lets dismiss all those bugs reports. Cannot see how that helps anyone.

Most users certainly lack the skills to be selectively reverting parts of a huge port of code from another OS and recompiling kernels over and over again.

[franco]

I wouldn't consider insisting that an issue exists is escalating. This is a FreeBSD scope discussion. Eventually FreeBSD can do what it wants. From what we know this issue won't go away either. It doesn't hurt stressing this point.

However, over the last 12 months we have seen a developer who has almost a dozen recorded cases of discarding bug reports from users ("non" FreeBSD and plain FreeBSD users), not following up on review questions, discarding external patch submissions by not responding to them, suboptimal upstream patching in the case of OpenVPN, leaving FreeBSD ports broken ('member miniupnpd) while sitting on a patch for weeks, leaving FreeBSD release versions broken due to not adhering to MFC policies and now derailing a bug report on a security-relevant patch set that nobody has the slightest idea about in terms of impact.

Where will we go next?

I got unverified information just then that this IPv6 weirdness we're seeing is wider than just ND solicit and advertise. Good times.


Cheers,
Franco

[doktornotor]

I got unverified information just then that this IPv6 weirdness we're seeing is wider than just ND solicit and advertise. Good times.

Well, that would not be surprising. Requires test cases for each and every ICMP type/code that all of a sudden attained "stateful" handling in pf. Those mostly don't exist, and I'm seriously discouraged from spending time experimenting with all that stuff to find out what else got broken on the way to report it only to be told "that's downstream problem".

[franco]

> Requires test cases for each and every ICMP type/code that all of a sudden attained "stateful" handling in pf.

Here is the last thought of the day:

I think OpenBSD made the right choice to pull this off in 2009 when IPv6 was still being laughed at for being almost nonexistent. There just wasn't a lot of real world evidence and much was repaired and aligned over the span of 15 years judging by the code.

Pulling this off in 2024, however, with IPv6 being a daily driver messing with its lifeline ICMPv6 with experiments from 2009 and then saying there is no issue (except downstream of course) is controversial for a multitude of reasons. Sure, the effects are barely visible, but so many people have noticed already here in such a short timeframe contrary to involved developer's belief. And this isn't 15.0 introducing it... it was shortcut into all supported releases with such conviction and against all basic release engineering principles and lack of understanding of IPv6.

I sincerely hope pfSense users are going to be spared from this whole situation. But only time will tell. :)


Good night,
Franco

[doktornotor]

My thoughts exactly. You cannot fast-forward 15 years of technical debt in this way without breaking lots of things. Ok, that is normal. The denial is not *insert Gordon Ramsay angry face here*. And yes, that's something that should have been targeted for 15 release.

[allan]

It actually doesn't matter if we test on FreeBSD or OPNsense kernel because we talk about the same code change.

I'm late to the party, but I got the impression that upstream thinks we did this to ourselves by choosing to diverge from their kernel. Would validating it on a vanilla FreeBSD kernel as @Uwe suggested remove that argument? It is a slippery slope.

I'm seriously discouraged from spending time experimenting with all that stuff to find out what else got broken on the way to report it only to be told "that's downstream problem".

Can a quick A-B test with their kernel help here as well? Now, this assumes our kernels are not bifurcated to the extent we lose functionality or significantly affect production. I also do not know how feasible this is, or how much work is involved to do this. It was just a thought after reading @Uwe's comment.

[meyergru]

I think my intervention already sufficed to give the FreeBSD maintainer reasonable hints to investigate without any further sidesteps - or, in his words: "an actionable item".

Nevertheless, we will have a corrected 24.7.3 in the meantime. I would have liked to have seen this fixed upstream  before Franco needed to step in, because if upstream had reacted earlier, there was no need to rollback / merge any downstream fixes when / if FreeBSD fixes it. I guess that ship has sailed, however...

[franco]

I'm late to the party, but I got the impression that upstream thinks we did this to ourselves by choosing to diverge from their kernel. Would validating it on a vanilla FreeBSD kernel as @Uwe suggested remove that argument? It is a slippery slope.

The problem is given a single unmodified FreeBSD commit breaking this I have no reason to believe we did something elsewhere that would break OPNsense but not FreeBSD. Mind you, this is while applying an OpenBSD patch to FreeBSD assuming that it will go fine and OpenBSD kept refining the ICMPv6 state tracking behaviour over the span of 2009 (where FreeBSD is at now with the patch minus strange inconsistencies) - 2023 (where OpenBSD ended up doing the actual work necessary).

Can a quick A-B test with their kernel help here as well? Now, this assumes our kernels are not bifurcated to the extent we lose functionality or significantly affect production. I also do not know how feasible this is, or how much work is involved to do this. It was just a thought after reading @Uwe's comment.

It will not stop the same arguments from happening (modifed base system, modified RC system, modified cheesecake recipe). This is a policy issue, not a technical issue. In the discussion I also mention why downstreams are forced into diverging even on kernel issues. It's not because we want. It's because FreeBSD doesn't care to take patches that it thinks not applicable or there is just nobody who will merge work done by others as frequently as needed. One could argue this would make FreeBSD better, but FreeBSD doesn't necessarily think that.


Cheers,
Franco

[doktornotor]

/me snickers...

https://github.com/openbsd/src/commit/2633ae8c4c8a64
https://cgit.freebsd.org/src/commit/?id=5ab1e5f7e5585558a73b723f07528977a82cee82

How many of those OpenBSD patches are still missing to fix the downstream issues?  :P

[newsense]

Yeah...somehow it looks like 13.4-RC2 was build ahead of the commit being added to the bugzilla...so it's unclear whether it landed there or not.



At least we know what to expect tomorrow morning:

- somebody wakes up, morning light is too bright to read through the commit changes

-builds test kernel(s)

-breakfast

-team meeting(s)

-review new kernel feedback

-finally gets out of bed since it's 7AM :D

[franco]

You will need to be more specific. The commit has 4 additional commits on master. One is a ND test and one is a cleanup. The other two don't inspire confidence in the previous work. And the OpenBSD 2023 commit is still missing.

Quick, someone else, go test this. I'm pretty sure nobody is going to ask us to test again?

The period to hit stable branches is 1 week anyway. We still have time to wait. 13.4 too it seems.


Cheers,
Franco

[newsense]

Hear ye, hear ye

In an effort to secure your IPvSixes we've dumped some very old code into our stack from a neighboring project.

Downstream complainers should appreciate the utmost security  provided by the fixes that made parts/all of their IPv6 stack unusable, have some vanilla vanilla and check back in another 12-15 years when the next code import is scheduled to be sponsored

End of SecurityAdvisory


P.S. We're working hard so you can boot a machine in miliseconds if you promise not you use TCP/IP