Home
Help
Search
Login
Register
OPNsense Forum
»
Archive
»
23.7 Legacy Series
»
Crashes when reconnecting PPPoE repeatedly
« previous
next »
Print
Pages: [
1
]
2
3
Author
Topic: Crashes when reconnecting PPPoE repeatedly (Read 6617 times)
craig
Newbie
Posts: 12
Karma: 0
Crashes when reconnecting PPPoE repeatedly
«
on:
October 23, 2023, 12:00:33 pm »
Sometimes I need to repeatedly reconnect my PPPoE connection as my ISP doesn't properly weight their gateways and I end up on one the other side of the country.
Recently (I think since 23.7), when I do this, after a few times OPNsense completely locks up and restarts, I've submitted a few crash reports but wanted to check if anyone else here is able to reproduce?
I have the crash log which I can upload if it'd help anyone (and is there anything other than IPs to remove from the logs?)
«
Last Edit: October 26, 2023, 03:23:46 pm by craig
»
Logged
thatso
Newbie
Posts: 22
Karma: 3
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #1 on:
October 24, 2023, 08:19:55 pm »
Welcome to the club.
I stayed at 22.7 until recently exactly because of this dreaded crash and reboot on PPPoE reconnect bug. Was running OPNsense for several years without any problem ever. Right after I dared to finally upgrade to 23.7.1 because problem reports ceased to show up, I was promptly hit by this bug I managed so long to avoid.
Reading past reports, I understand that the developers have a hard time fixing this as none of them has an ISP with PPPoE.
Like you, I've sent lots of bug reports lately.
The weird thing is that sometimes my daily PPPoE reconnect by cron is successful for several days in a row while out of the blue it crashes and reboots for the next few days. Naturally, nothing changed in my environment meanwhile.
I found a
FreeBSD kernel bug
which seems to be the culprit. Not sure though, if OPNsense can do something about it regarding the MPD5 daemon or if we have to patiently wait until this gets fixed upstream.
«
Last Edit: October 26, 2023, 08:33:23 pm by thatso
»
Logged
Monviech (Cedrik)
Global Moderator
Hero Member
Posts: 1522
Karma: 172
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #2 on:
October 24, 2023, 08:46:30 pm »
To test PPPoE you don't really need an ISP with PPPoE. You can create a fully functional PPPoE Server on linux or freebsd and configure it like an ISP one. Then you can connect your PPPoE client to it on the Opnsense and do all kinds of tests. I already did that to prove something different though.
https://github.com/opnsense/core/issues/6650#issuecomment-1635663267
Logged
Hardware:
DEC740
thatso
Newbie
Posts: 22
Karma: 3
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #3 on:
October 24, 2023, 10:12:14 pm »
@Monviech: You are right. I was merely quoting @franco's
statement
.
Logged
Patrick M. Hausen
Hero Member
Posts: 6709
Karma: 565
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #4 on:
October 26, 2023, 11:01:07 am »
@thatso if I read that FreeBSD issue correctly it should only concern users of FreeBSD 14?
I also found this mailing list discussion that might relate to your problem:
https://lists.freebsd.org/archives/freebsd-net/2023-October/004104.html
Are you using IPv6 with PPPoE?
Kind regards,
Patrick
Logged
Deciso DEC750
People who think they know everything are a great annoyance to those of us who do.
(Isaac Asimov)
craig
Newbie
Posts: 12
Karma: 0
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #5 on:
October 26, 2023, 03:12:35 pm »
Yes, I am using IPv6
edit: I've uploaded the textdump file to my original post
«
Last Edit: October 26, 2023, 03:24:14 pm by craig
»
Logged
Monviech (Cedrik)
Global Moderator
Hero Member
Posts: 1522
Karma: 172
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #6 on:
October 26, 2023, 04:14:28 pm »
In the Kernel panic I can see that it's caused by a page fault of the CPU.
The processes responsible seem to be:
- ether_demux (demultiplexes ethernet packets, looks into them, sees that they're IPv6 for example, and passes them to ip6_input)
- ip6_input (receives IPv6 packets and handles them, passing them to ip6_tryforward for example)
- ip6_tryforward (forwards IPv6 packets to the best path to its destination)
I could also see a lot of "fq-codel" messages, which show that you use traffic shaping. Maybe try to deactivate traffic shaping pipes for a while and see how it goes.
On first glance it doesn't look like PPPoE crashed the kernel. Its one of the above things that crashes, so maybe PPPoE calls them wrongly and that crashes the kernel. But somebody else might know better.
---------------------------------------------------------
(I use pppoe at home with a hardware opnsense and didn't experience crashes yet, also ipv6, but I have static prefixes so I can't be compared. I will try to reconnect it a few times later to see if I can make it crash
«
Last Edit: October 26, 2023, 04:19:41 pm by Monviech
»
Logged
Hardware:
DEC740
Patrick M. Hausen
Hero Member
Posts: 6709
Karma: 565
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #7 on:
October 26, 2023, 05:26:27 pm »
Read the mailing list.
If an IPv6 interface goes away - like when PPPoE disconnects - a certain data structure is deallocated, while occasionally another thread tries to use it to forward a queued packet. Which of course causes a crash.
Looks like this is precisely the bug hitting our OP.
Kristof Provost and friends are currently discussing how to best fix it.
Logged
Deciso DEC750
People who think they know everything are a great annoyance to those of us who do.
(Isaac Asimov)
thatso
Newbie
Posts: 22
Karma: 3
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #8 on:
October 26, 2023, 08:31:30 pm »
@Patrick: your explanation seems to be right on the spot.
BTW: like @craig, I use PPPoE with IPv4 and an additional IPv6 /56 prefix.
The FreeBSD bug tracker says that kernels 12.0 - 13.2 (the current OPNsense kernel version) are affected and the main cause is the MPD5 daemon.
There was a
lengthy discussion
about the same problem in the past, unfortunately it stopped without any final solution besides @schnipp preventing the crash with
a modified script
of his own.
«
Last Edit: October 26, 2023, 08:41:48 pm by thatso
»
Logged
franco
Administrator
Hero Member
Posts: 17619
Karma: 1605
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #9 on:
October 26, 2023, 09:00:59 pm »
I'm unsure the current discussion on the mailing list is the bug (half hoping this was only FreeBSD 14) but ip6_tryforward() is at least suspicious enough to take a closer look. Let me prepare a debug kernel tomorrow so we can get a core dump.
Backtrace for easier reference:
Tracing pid 0 tid 100025 td 0xfffffe00387fc020
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00e003f5f0
vpanic() at vpanic+0x182/frame 0xfffffe00e003f640
panic() at panic+0x43/frame 0xfffffe00e003f6a0
trap_fatal() at trap_fatal+0x387/frame 0xfffffe00e003f700
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00e003f760
calltrap() at calltrap+0x8/frame 0xfffffe00e003f760
--- trap 0xc, rip = 0xffffffff80ea3574, rsp = 0xfffffe00e003f830, rbp = 0xfffffe00e003f8a0 ---
ip6_tryforward() at ip6_tryforward+0x274/frame 0xfffffe00e003f8a0
ip6_input() at ip6_input+0x5e4/frame 0xfffffe00e003f980
netisr_dispatch_src() at netisr_dispatch_src+0x295/frame 0xfffffe00e003f9d0
ether_demux() at ether_demux+0x159/frame 0xfffffe00e003fa00
ng_ether_rcv_upper() at ng_ether_rcv_upper+0x8c/frame 0xfffffe00e003fa20
ng_apply_item() at ng_apply_item+0x2bf/frame 0xfffffe00e003fab0
ng_snd_item() at ng_snd_item+0x28e/frame 0xfffffe00e003faf0
ng_apply_item() at ng_apply_item+0x2bf/frame 0xfffffe00e003fb80
ng_snd_item() at ng_snd_item+0x28e/frame 0xfffffe00e003fbc0
ng_ether_input() at ng_ether_input+0x4c/frame 0xfffffe00e003fbf0
ether_nh_input() at ether_nh_input+0x1f2/frame 0xfffffe00e003fc50
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00e003fca0
ether_input() at ether_input+0x69/frame 0xfffffe00e003fd00
iflib_rxeof() at iflib_rxeof+0xbcb/frame 0xfffffe00e003fe00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe00e003fe40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x15d/frame 0xfffffe00e003fec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc3/frame 0xfffffe00e003fef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00e003ff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e003ff30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Cheers,
Franco
Logged
franco
Administrator
Hero Member
Posts: 17619
Karma: 1605
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #10 on:
October 27, 2023, 11:32:30 am »
Promised kernel (make sure you are on 23.7.7 beforehand):
# opnsense-update -zkr dbg-23.7.7
# opnsense-shell reboot
After crash and automatic reboot there are vmcore.* files under /var/crash that I'd need to look at.
The debug kernel is a little more trigger-happy for panics, like opening system: tunables will crash the system. Try to avoid operating the GUI as much until the crash happens and then go back to regular kernel:
# opnsense-update -kf
Cheers,
Franco
Logged
thatso
Newbie
Posts: 22
Karma: 3
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #11 on:
October 28, 2023, 07:28:11 pm »
Installed and waiting for the next crash. Naturally, today's reconnect did
not
crash.
Logged
franco
Administrator
Hero Member
Posts: 17619
Karma: 1605
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #12 on:
October 28, 2023, 08:10:28 pm »
Fingers crossed. Thanks a lot for the help!
Cheers,
Franco
Logged
franco
Administrator
Hero Member
Posts: 17619
Karma: 1605
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #13 on:
October 28, 2023, 08:11:43 pm »
PS: @thatso can't directly confirm your panic is the same as the OP one so just wanted to mention that to level expectation
Logged
craig
Newbie
Posts: 12
Karma: 0
Re: Crashes when reconnecting PPPoE repeatedly
«
Reply #14 on:
October 31, 2023, 11:49:34 am »
Sorry I've been away for a few days.
I installed the debug kernel last night, but after doing OPNSense panics on boot.
I had to get things back up and running, so used the console port to select the previous kernel - I'll try and get the panic this evening to see if we can work around it.
Logged
Print
Pages: [
1
]
2
3
« previous
next »
OPNsense Forum
»
Archive
»
23.7 Legacy Series
»
Crashes when reconnecting PPPoE repeatedly