Traceroute / ICMP issue after 24.7.1 update

[franco]

Ever since that awkward exchange for https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273207#c18 and making up FreeBSD policies on the spot I think Kristof never followed up on any of my patch submissions and bug reports and even his own bugs in FreeBSD ports for example. This wasn't limited to my reports either as other mishaps record...

I've talked to multiple distinguished FreeBSD authorities and the same message is repaeted over and over:

* FreeBSD is a volunteer project
* FreeBSD committers are volunteers
* Mistakes happen, everyone is human

I agree with these things, but the reality is nothing gets done to the point of valid patches for actual release regressions being ignored and FreeBSD release versions "surprisingly" turning up broken with a repeating and problem-scope-increasing committer pattern.

I don't see a reason to build something on a vanilla FreeBSD unless someone there outgrows their current lethargy that allows everyone to do whatever they want with the FreeBSD code. As far as code correctness is concerned I draw the line. FreeBSD currently draws the line at the user/downstream relationships. I have no clue what is happening there and what the goal of this is.

I plan to work on the code next week to see if we can find it. If anyone will bother to review and accept a potential patch in FreeBSD is the big money question. 


Cheers,
Franco

[doktornotor]

I don't see a reason to build something on a vanilla FreeBSD unless someone there outgrows their current lethargy that allows everyone to do whatever they want with the FreeBSD code. As far as code correctness is concerned I draw the line. FreeBSD currently draws the line at the user/downstream relationships. I have no clue what is happening there and what the goal of this is.

FWIW, I haven't seen similar amount of hostility towards others actually making use of open-source code for their projects pretty much anywhere else. The whole thing is indeed absurd.

Additionally, while fixes for many well-known issues never make it back to "stable", no matter how minor and how much time was there to backport them - now all of a sudden someone forward-ports hundreds of lines of OpenBSD code, rushes it all the way back to 13.x, presents this as security issue and causes severe regressions that they subsequently cannot be bothered to fix completely and properly.

I happen to have the impression that even if I took vanilla FreeBSD and the only modification done was changing /etc/motd to say OPNsense, everything reported would be a downstream issue.

Nice waste of time of the people involved.

[Wirehead]

I plan to work on the code next week to see if we can find it. If anyone will bother to review and accept a potential patch in FreeBSD is the big money question. 

If you need any testers, I can reproduce the situation here. Feel free to reach out in the topic here.

[franco]

FWIW, I haven't seen similar amount of hostility towards others actually making use of open-source code for their projects pretty much anywhere else. The whole thing is indeed absurd.

The same has been true for OpenBSD for many years when you present them a non standard kernel build. Everyone will ignore you. In this day and age and open source collaboration focusing on patches and code changes is crucial and effective since git entered the scene making this much much much much much much easier (and practically fool-proof). Some of the developers don't like that I think, but to be honest I'd rather trust my random OPNsense user testing something for me than a OS developer who I know will lie to my face because it makes his day easier. This is based on real world evidence with OpenBSD. Patch submissions are similarly derailed because "FreeBSD kernel code is stupid". If you want that funny exchange I can find it for you.

Additionally, while fixes for many well-known issues never make it back to "stable", no matter how minor and how much time was there to backport them - now all of a sudden someone forward-ports hundreds of lines of OpenBSD code, rushes it all the way back to 13.x, presents this as security issue and causes severe regressions that they subsequently cannot be bothered to fix completely and properly.

No lines drawn by release engineering. This had to backfire. For me it's nice to have this general issue on record with FreeBSD release engineering who assured me all things we see are honest mistakes before this escalated in the SA. I can only assume it was green-lit by RE despite the actual code related and general concerns voiced over process. So: concerns not heard, no policies to prevent this, escalating problems in releases. We haven't seen the end of it even.

And for the record FreeBSD 14.1 was a very smooth jump overall. I'm happy with the quality level for everyone involved. I even don't mind these things slipping through (even if you argue 2022 is a long time for something to go unnoticed). The real deal is how defects and regressions are being handled and perhaps prevented. I'm glad we have this discussion out in the open at the moment at least.

If you need any testers, I can reproduce the situation here. Feel free to reach out in the topic here.

Thanks, mate. I will publish two test kernels to discern which exact FreeBSD commit this is and dump it over there for all that it's worth. In the name of progress and cooperation.


Cheers,
Franco

[doktornotor]

In this day and age and open source collaboration focusing on patches and code changes is crucial and effective since git entered the scene making this much much much much much much easier (and practically fool-proof). Some of the developers don't like that I think, but to be honest I'd rather trust my random OPNsense user testing something for me than a OS developer who I know will lie to my face because it makes his day easier.

I mean, BSD does not exactly have a huge user base, does it. Where it's being used significantly - those are (were) downstream projects such as routers/firewalls, or NAS due to ZFS. In the latter field, due to ZFS making it to Linux, the major guys apparently already decided it's not worth the trouble any more - so there's just XigmaNAS left (which is barely alive).

If you discourage the remaining projects and contributors/users they bring, then there's nothing left, pretty much. "The other project" which will be undownloadeable soon won't cut it. Congrats, you now have a system that noone is using, beyond the bunch of committers and hardcore enthusiasts.

 

[franco]

I'm not going to comment on what I think here overall but I will say that I believe downstream projects are a healthy part of upstream projects. Unfortunately a number of important people do not believe that to be so.

That being said we've built all these helper tools in OPNsense over the years to allow for rapid test and fix deployment for the sake of bringing the code base forward which includes all upstream projects. You can see where that stops being useful when we start sitting on patches which is also the only way we can go forward because otherwise we don't go forward at all if we need to wait for upstream to move but upstream won't move because we have "modifications". For the most part this boils down to opinions on necessity and trust. But opinions are just that. Fixes are nicer to have than not IMO.

Expect an EuroBSDCon 2024 presentation about this exact topic.


Cheers,
Franco

[franco]

Because we talked about this internally this official statement came to memory again.

https://www.netgate.com/blog/pfsense-software-is-moving-ahead

[franco]

If anyone wants to participate in the identification of the ND-ICMPv6 stall commit cause we started another hunt:

https://github.com/opnsense/src/issues/218#issuecomment-2307051831



Thanks,
Franco

[franco]

There is a test kernel out now that disables the newly-added state tracking for neighbour discovery packets:

https://github.com/opnsense/src/issues/218#issuecomment-2308039278

This can be a baseline for further investigation or maybe even a viable workaround for the time being. Let's see what happens.


Cheers,
Franco

[doktornotor]

Franco, I got completely confused by the chain of patches and reverts and patches of patches... So, did just have a look at the relevant parts of the current code in pf.c in OpenBSD vs. FreeBSD. Yeah, there's something amiss.

Code: [Select]

pass in quick inet6 proto ipv6-icmp from {any} to {any} icmp6-type {1,2,135,136} keep state label "9dff917e83b570f19343d5e2941a545e" # IPv6 RFC4890 requirements (ICMP)
pass out quick inet6 proto ipv6-icmp from {(self)} to {fe80::/10,ff02::/16} icmp6-type {128,129,133,134,135,136} keep state label "fb0cc70ad35caa7bea0138f49c30623d" # IPv6 RFC4890 requirements (ICMP)
pass in quick inet6 proto ipv6-icmp from {fe80::/10} to {fe80::/10,ff02::/16} icmp6-type {128,133,134,135,136} keep state label "d147534c4012c8dd65eda59292c0ab90" # IPv6 RFC4890 requirements (ICMP)
pass in quick inet6 proto ipv6-icmp from {ff02::/16} to {fe80::/10} icmp6-type {128,133,134,135,136} keep state label "df042096359aa49094a20b3ac111f4b7" # IPv6 RFC4890 requirements (ICMP)
pass in quick inet6 proto ipv6-icmp from {::} to {ff02::/16} icmp6-type {128,133,134,135,136} keep state label "d8fdc41aeac05a86adfb74e6052317d8" # IPv6 RFC4890 requirements (ICMP)

Wondering what'd be the effect of using sloppy state here.

[franco]

Definitely something about state. The 24.7.2-nd patch is really the latest FreeBSD state shipped with the 24.7.2 release and that single line change to deactivate state tracking for ND.

I think you can graft something with state none or sloppy in the ruleset, but nothing says eternally stuck with it like a short hack to avoid this as a workaround in a production release.


Cheers,
Franco

[doktornotor]

Yeah. For ICMP, it even lacks sloppy states documentation in FreeBSD (or I am blind). For OpenBSD:

https://man.openbsd.org/pf.conf#sloppy

For ICMP, this option allows states to be created from replies, not just requests.

[franco]

The ICMP sloppy looks like an amendment loosely based on the 2009/2010 patches. So it's pretty natural this doesn't show in FreeBSD (and probably doesn't work yet, possibly also missing code).

[doktornotor]

I'd say FreeBSD would strongly benefit from merging as much as possible from the pf implementation in OpenBSD instead of porting random cherrypicks in incomplete and broken ways. Pretty sure that's not going to happen though.

Anyway, 24.7.2-nd looks perfectly fine here. Not sure what to do with that regarding telling that bad news to the maintainer though - since it's of course a downstream issue. After all, they've already been hinted that the code porting was incomplete.

[franco]

Kristof's reply is priceless:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280701#c49

(In reply to Gordon Tetlow from comment #48)
> kp, does the analysis in comment 46 indicate an issue that needs further review?

What issue? There's been a lot of conspiracy theorising, but no actual bug report beyond "opnsense is broken".

I genuinely don't know what's supposed to be broken.

I wish he would comment on things like these ever:

https://github.com/freebsd/freebsd-src/pull/1390
https://github.com/freebsd/freebsd-src/pull/1391