Crashes when reconnecting PPPoE repeatedly

[craig]

Sometimes I need to repeatedly reconnect my PPPoE connection as my ISP doesn't properly weight their gateways and I end up on one the other side of the country.

Recently (I think since 23.7), when I do this, after a few times OPNsense completely locks up and restarts, I've submitted a few crash reports but wanted to check if anyone else here is able to reproduce?

I have the crash log which I can upload if it'd help anyone (and is there anything other than IPs to remove from the logs?)

[thatso]

Welcome to the club. 
I stayed at 22.7 until recently exactly because of this dreaded crash and reboot on PPPoE reconnect bug. Was running OPNsense for several years without any problem ever. Right after I dared to finally upgrade to 23.7.1 because problem reports ceased to show up, I was promptly hit by this bug I managed so long to avoid.
Reading past reports, I understand that the developers have a hard time fixing this as none of them has an ISP with PPPoE.
Like you, I've sent lots of bug reports lately.
The weird thing is that sometimes my daily PPPoE reconnect by cron is successful for several days in a row while out of the blue it crashes and reboots for the next few days. Naturally, nothing changed in my environment meanwhile.
I found a FreeBSD kernel bug which seems to be the culprit. Not sure though, if OPNsense can do something about it regarding the MPD5 daemon or if we have to patiently wait until this gets fixed upstream.

[Monviech (Cedrik)]

To test PPPoE you don't really need an ISP with PPPoE. You can create a fully functional PPPoE Server on linux or freebsd and configure it like an ISP one. Then you can connect your PPPoE client to it on the Opnsense and do all kinds of tests. I already did that to prove something different though.

https://github.com/opnsense/core/issues/6650#issuecomment-1635663267

[thatso]

@Monviech: You are right. I was merely quoting @franco's statement. 

[Patrick M. Hausen]

@thatso if I read that FreeBSD issue correctly it should only concern users of FreeBSD 14?

I also found this mailing list discussion that might relate to your problem:
https://lists.freebsd.org/archives/freebsd-net/2023-October/004104.html

Are you using IPv6 with PPPoE?

Kind regards,
Patrick

[craig]

Yes, I am using IPv6 

edit: I've uploaded the textdump file to my original post

[Monviech (Cedrik)]

In the Kernel panic I can see that it's caused by a page fault of the CPU.

The processes responsible seem to be:
- ether_demux (demultiplexes ethernet packets, looks into them, sees that they're IPv6 for example, and passes them to ip6_input)
- ip6_input (receives IPv6 packets and handles them, passing them to ip6_tryforward for example)
- ip6_tryforward (forwards IPv6 packets to the best path to its destination)

I could also see a lot of "fq-codel" messages, which show that you use traffic shaping. Maybe try to deactivate traffic shaping pipes for a while and see how it goes.

On first glance it doesn't look like PPPoE crashed the kernel. Its one of the above things that crashes, so maybe PPPoE calls them wrongly and that crashes the kernel. But somebody else might know better.

---------------------------------------------------------

(I use pppoe at home with a hardware opnsense and didn't experience crashes yet, also ipv6, but I have static prefixes so I can't be compared. I will try to reconnect it a few times later to see if I can make it crash 

[Patrick M. Hausen]

Read the mailing list. 

If an IPv6 interface goes away - like when PPPoE disconnects - a certain data structure is deallocated, while occasionally another thread tries to use it to forward a queued packet. Which of course causes a crash.

Looks like this is precisely the bug hitting our OP.

Kristof Provost and friends are currently discussing how to best fix it.

[thatso]

@Patrick: your explanation seems to be right on the spot.
BTW: like @craig, I use PPPoE with IPv4 and an additional IPv6 /56 prefix.
The FreeBSD bug tracker says that kernels 12.0 - 13.2 (the current OPNsense kernel version) are affected and the main cause is the MPD5 daemon.
There was a lengthy discussion about the same problem in the past, unfortunately it stopped without any final solution besides @schnipp preventing the crash with a modified script of his own.

[franco]

I'm unsure the current discussion on the mailing list is the bug (half hoping this was only FreeBSD 14) but ip6_tryforward() is at least suspicious enough to take a closer look. Let me prepare a debug kernel tomorrow so we can get a core dump.

Backtrace for easier reference:

Tracing pid 0 tid 100025 td 0xfffffe00387fc020
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00e003f5f0
vpanic() at vpanic+0x182/frame 0xfffffe00e003f640
panic() at panic+0x43/frame 0xfffffe00e003f6a0
trap_fatal() at trap_fatal+0x387/frame 0xfffffe00e003f700
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00e003f760
calltrap() at calltrap+0x8/frame 0xfffffe00e003f760
--- trap 0xc, rip = 0xffffffff80ea3574, rsp = 0xfffffe00e003f830, rbp = 0xfffffe00e003f8a0 ---
ip6_tryforward() at ip6_tryforward+0x274/frame 0xfffffe00e003f8a0
ip6_input() at ip6_input+0x5e4/frame 0xfffffe00e003f980
netisr_dispatch_src() at netisr_dispatch_src+0x295/frame 0xfffffe00e003f9d0
ether_demux() at ether_demux+0x159/frame 0xfffffe00e003fa00
ng_ether_rcv_upper() at ng_ether_rcv_upper+0x8c/frame 0xfffffe00e003fa20
ng_apply_item() at ng_apply_item+0x2bf/frame 0xfffffe00e003fab0
ng_snd_item() at ng_snd_item+0x28e/frame 0xfffffe00e003faf0
ng_apply_item() at ng_apply_item+0x2bf/frame 0xfffffe00e003fb80
ng_snd_item() at ng_snd_item+0x28e/frame 0xfffffe00e003fbc0
ng_ether_input() at ng_ether_input+0x4c/frame 0xfffffe00e003fbf0
ether_nh_input() at ether_nh_input+0x1f2/frame 0xfffffe00e003fc50
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00e003fca0
ether_input() at ether_input+0x69/frame 0xfffffe00e003fd00
iflib_rxeof() at iflib_rxeof+0xbcb/frame 0xfffffe00e003fe00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe00e003fe40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x15d/frame 0xfffffe00e003fec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc3/frame 0xfffffe00e003fef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00e003ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e003ff30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---


Cheers,
Franco

[franco]

Promised kernel (make sure you are on 23.7.7 beforehand):

# opnsense-update -zkr dbg-23.7.7
# opnsense-shell reboot

After crash and automatic reboot there are vmcore.* files under /var/crash that I'd need to look at.

The debug kernel is a little more trigger-happy for panics, like opening system: tunables will crash the system. Try to avoid operating the GUI as much until the crash happens and then go back to regular kernel:

# opnsense-update -kf


Cheers,
Franco

[thatso]

Installed and waiting for the next crash. Naturally, today's reconnect did not crash. 

[franco]

Fingers crossed. Thanks a lot for the help!


Cheers,
Franco

[franco]

PS: @thatso can't directly confirm your panic is the same as the OP one so just wanted to mention that to level expectation

[craig]

Sorry I've been away for a few days.

I installed the debug kernel last night, but after doing OPNSense panics on boot.

I had to get things back up and running, so used the console port to select the previous kernel - I'll try and get the panic this evening to see if we can work around it.