[solved] Log flooded with "pf: ICMP error message too short (ip6)"

[doktornotor]

Well yes, if you create explicit one with drop and do NOT log, maybe it gets muted. Or not.

[rkube]

You can easily check if the SA is the culprit by trying the kernel with the SA completely removed via

Code Select Expand

opnsense-update -zkr 24.7.3-no_sa

and reboot, see this.

After switching kernel to version 14.1-RELEASE-p3 no_sa-n267804-164bfe67604) the debug message 'pf: ICMP error message too short (ip6)' disappeared.

It also looks to me that the no_SA kernel solves this (one of several?) [ironic]downstream[/ironic] problems.

Other, but perhaps unproblematic debug messages such as 'pf: loose state match...' and 'pf: dropping packet with ip options' are still often logged if debug logging is enabled.


Br
Reza

[doktornotor]

I really would be only concerned about flooding logs with things that are unexpected and appear to be caused by something being obviously broken. Otherwise, with debug logs comes a lot of noise, kinda normal.

[cloudz]

Are these also known to -hog- a device like the DEC740? I really saw lag spikes / interrupt spikes when it was turned on.

[TheDJ]

Is it possible that the 24.7.3-no_sa kernel is not online anymore? Fetching it times out for me (tried it earlier today and now again.

Anyways: as I said before, I already tried the kernel a few days ago and the TCP state losses (on v4) were still very much a thing even with the -no_sa kernel. I don't have any logs from that run, but the behavior described by Reza sounds like it is still having those problems even with the -no_sa kernel:

Other, but perhaps unproblematic debug messages such as 'pf: loose state match...' and 'pf: dropping packet with ip options' are still often logged if debug logging is enabled.

So I am really not sure what is going on and if it is really the upstream bug.

It's one thing that there are debug messages but a completely different thing to actually experience legitimate packets hitting the FW rule due to the state losses.
If the state loss problems should be treated differently from the ICMP ip6 log messages, I am very open to discuss them in this thread: https://forum.opnsense.org/index.php?topic=42657.0

[doktornotor]

Fetching the kernel works fine for me. As for the rest, it's for the upstream to ask for debugging all of their broken junk.Noone can be bothered there apparently.

Works fine without those awesome "security" fixes.

[TheDJ]

Fetching the kernel works fine for me.

It's very weird, but fetching the kernel does not work for me anymore: after the status "Fetching kernel-24.7.3-no_sa-amd64.txz" the loading dots just keep on running for a long time. No error message or anything.
The same happens for the other test kernel for 24.7.3 in the snapshot directory (https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/sets/)

Trying to fetch a non-existing kernel times out immediately with an error message.

Code Select Expand

Fetching kernel-24.7.3-test-amd64.txz: ..[fetch: https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/sets/kernel-24.7.3-test-amd64.txz.sig: Not Found] failed, no signature found
I already reverted to the zfs snapshot that I set up before my kernel testing earlier this week.
Maybe, my opnsense installation is more broken than I thought.

This means, I can't currently verify anything regarding the TCP timeouts with the -no_sa kernel.

EDIT: Scratch that - changing the mirror works. I will now also test the -no_sa deeper than before.

[doktornotor]

The no_sa kernel is definitely still there:

https://pkg.opnsense.org/FreeBSD:14:amd64/snapshots/sets/

[TheDJ]

Yeah, I switched the mirror and then the download worked.

However, even with multiple more hours on the no_sa kernel, the TCP state losses (and FW rule hits) are still there and completely the same.
The "ICMP error message too short (ip6)" (which were the initial starting point for this thread) are gone (like the others described), but the TCP behavior did not change.

[rkube]

Hi,

To briefly conclude this topic: Using the "no_SA" kernel prevents unwanted effects when processing ICMPv6 packets. The (debug) message 'pf: ICMP error message too short (ip6)' first observed by cloudz has disappeared; I think we all agree that other ICMPv6/ND6 issues are also fixed with the patched kernel.

Code Select Expand


opnsense-update -zkr 24.7.3-no_sa
fixes the problem.


If cloudz wants, he can mark this topic as [Solved] again ;-)

However, the observed symptom of TCP (IPv4) states being "lost" and thus blocking traffic still persists. I think that the (debug) message 'pf: loose state match...' actually points to the underlying problem.

TheDJ has already created a separate topic about this: Topic: Regular LAN Traffic hits Default deny / state violation rule since 24.7 (https://forum.opnsense.org/index.php?topic=42657.0).

Br Reza