crashing opnsense 23.1.7_3-amd64 with ping6

[9axqe]

I can reliably (100% of the time so far) crash opnsense 23.1.7_3-amd64 with a specific ping command from my computer (connected to LAN intf). No traffic can go through then, and when GUI is finally available again, I see "The system is currently booting. Not all services have been started yet."

The command I use (running macOS):

Code: [Select]

sudo ping6 -G 1508,1410 -D 2001:4860:4860::8844

The effect is immediate for me, it takes a second, then the router crashes.

[9axqe]

Forgot to mention: I reported the crash via the GUI.

[franco]

I think I remember this one... IPv6 fragmentation across a PBR. The matching rule on the firewall has a gateway set, right?


Cheers,
Franco

[9axqe]

you're good, I indeed have put a top level firewall rule to "protect" pinging from the other deny rules (for troubleshooting purposes mostly) and it's sending it to a gateway indeed, which sends it to the Wireguard intf – or should send it.

I was trying to troubleshoot some MTU issues on my WAN interface and actually that makes me realise that I need to avoid the gateway if I want to troubleshoot this.

[franco]

I'll try the command later today. The main issue was how to reproduce this quickly so you may have helped out here a lot.

Two things that I'd be interested in if you can help further:

1. What happens if you disable "shared forwarding" on Firewall: Settings: Advanced with the gateway in the rule set?
2. What happens when you don't have a gateway set in the rule? (both shared and non-shared would be interesting)


Thanks,
Franco

[9axqe]

1. "shared forwarding":

I disabled and reenabled twice, testing in between each time and the issue only happens if shared forwarding is enabled (checkbox is checked). Interestingly, even if "shared forwarding" is disabled, I have 100% packet loss somehow (but opnsense does not crash).

2. gateways:

I simply disabled the gateway to the wireguard intf (the one the ICMPv6 fw rules points to), forcing traffic onto the default gateway to the WAN intf and there is no issue anymore then. "shared forwarding" was enabled. In this case the pings are successful (at least below a certain packet size).

Let me know if that's sufficient for you.

[franco]

Thanks a lot. Based on code not available in FreeBSD 13 (where the issue first appeared) I'm inclined to test this shortcut dropping the bad traffic for the time being:

https://github.com/opnsense/src/commit/5d8cfe7c1eb

# opnsense-update -zkr 23.1.6-refragment

I couldn't reproduce this today due to fighting with prefix delegation inception (not wanting to crash my main box) so I'm not 100% it prevents the panic. Traffic should still be dropped though.


Cheers,
Franco

[9axqe]

Looks good, I can't get it to crash anymore with this command, even though " Shared forwarding" is enabled. I rebooted the router a couple of times to be sure.

On the dashboard it still reports the version as being "OPNsense 23.1.7_3-amd64" though, is that expected?

[franco]

Yes, only "uname -a" should report something along the lines of "5d8cfe7" which is the commit hash for the build since this only replaced the kernel.

Good enough for me. I will add this to stable/23.1 branch then.


Thanks again,
Franco

[franco]

PS: The "bad" upstream change appears to be https://github.com/opnsense/src/commit/53a4886d5d which broke the pf(4) end assuming ip6_forward() was safe in all cases -- well, at least it was before FreeBSD 13.

https://github.com/freebsd/freebsd-src/commit/b52b61c0b6b appears to fix this, but was never added to stable/13 and doesn't apply cleanly there as well so that's why I went with the commit mentioned earlier.

[9axqe]

This means that at some point in the future, such packets will work again over opnsense and not be dropped anymore, but at this point in time, it seems there is no clear date or target version as to when this will be possible, is my understanding correct?

[franco]

I might revisit this sooner than later, but I'm unable to use upstream work easily here ETA is in a more or less long-term, correct.

The next fixed version would be the one using FreeBSD 14.1, but 14.0 isn't even out yet so that might be one year away for 24.7 at the earliest. 23.7 plans to move to FreeBSD 13.2 but the problem is the same there (also stable/13 branch without any such fixes).


Cheers,
Franco

[9axqe]

Thanks, it's not a big deal (now that it does not crash anymore), just wanted to check my understanding.

[franco]

Fair enough, happy to share that information from the release side.


Cheers,
Franco

[franco]

Actually I do have an idea on how to fix this without overcomplicating it. It might not be 100% correct but we could still try to forward/output those packets now being dropped. Want to try a patch?