Fatal trap 12: page fault while in kernel mode - Please help?

[franco]

It was added to upcoming 22.1.8 as well. We are not sure this is the right commit, but it doesn't look like it would hurt either.

As for STABLE and RELEASE... we wanted to aim for 13.1-RELEASE but that wasn't available for 22.1 yet so we decided to use 13-STABLE instead of 13.0-RELEASE since there were a number of fixes in it. It was worked out ok and we can safely land 13.1 in 22.7 now that it's out.


Cheers,
Franco

[magnust]

Do I get back to the current 22.1 track by simply accepting the normal upgrade in the dashboard?

[fraenki]

I'm also experiencing crashes and my trace looks pretty similar (but it's not the same):

I'be upgraded to 22.7.b. Unfortunately, in my case the firewall still crashes with pretty much the same panic:

Code Select Expand


Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address = 0x18
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80d3d9ed
stack pointer         = 0x28:0xfffffe013339d500
frame pointer         = 0x28:0xfffffe013339d570
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (if_io_tqg_2)
trap number = 12
panic: page fault
cpuid = 2
time = 1655658689

KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe013339d2c0
vpanic() at vpanic+0x17f/frame 0xfffffe013339d310
panic() at panic+0x43/frame 0xfffffe013339d370
trap_fatal() at trap_fatal+0x385/frame 0xfffffe013339d3d0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe013339d430
calltrap() at calltrap+0x8/frame 0xfffffe013339d430
--- trap 0xc, rip = 0xffffffff80d3d9ed, rsp = 0xfffffe013339d500, rbp = 0xfffffe013339d570 ---
m_copydata() at m_copydata+0x4d/frame 0xfffffe013339d570
tcp_output() at tcp_output+0x1339/frame 0xfffffe013339d750
tcp_do_segment() at tcp_do_segment+0x2cfd/frame 0xfffffe013339d830
tcp_input_with_port() at tcp_input_with_port+0xafb/frame 0xfffffe013339d990
tcp_input() at tcp_input+0xb/frame 0xfffffe013339d9a0
ip_input() at ip_input+0x15f/frame 0xfffffe013339da30
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe013339da80
ether_demux() at ether_demux+0x138/frame 0xfffffe013339dab0
ether_nh_input() at ether_nh_input+0x355/frame 0xfffffe013339db10
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe013339db60
ether_input() at ether_input+0x69/frame 0xfffffe013339dbc0
ether_demux() at ether_demux+0x121/frame 0xfffffe013339dbf0
ether_nh_input() at ether_nh_input+0x355/frame 0xfffffe013339dc50
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe013339dca0
ether_input() at ether_input+0x69/frame 0xfffffe013339dd00
iflib_rxeof() at iflib_rxeof+0xc27/frame 0xfffffe013339de00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe013339de40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x15d/frame 0xfffffe013339dec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe013339def0
fork_exit() at fork_exit+0x7e/frame 0xfffffe013339df30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe013339df30
--- trap 0, rip = 0xffffffff80c3137f, rsp = 0, rbp = 0x6 ---
mi_startup() at mi_startup+0xdf/frame 0x6


[franco]

Do I get back to the current 22.1 track by simply accepting the normal upgrade in the dashboard?

Yes.

I'be upgraded to 22.7.b. Unfortunately, in my case the firewall still crashes with pretty much the same panic:

Looks to be the same as https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264257


Cheers,
Franco

[fraenki]

The crashes are gone after enabling the workaround that was suggested in the FreeBSD bug ticket:
add tunable net.inet.tcp.sack.enable with value "0" to disable SACK support.


Regards
- Frank

[magnust]

The crashes are gone after enabling the workaround that was suggested in the FreeBSD bug ticket:
add tunable net.inet.tcp.sack.enable with value "0" to disable SACK support.


Regards
- Frank

Thank you!!!

:D  :D :D :D

[fraenki]

A fix has finally arrived in FreeBSD:
https://cgit.freebsd.org/src/commit/?id=5ae83e0d871bc7cbe4dcc9a33d37eb689e631efe

Maybe @franco could incooparate it into the next (minor) release of OPNsense  :)

[franco]

How about testing it first? :P

# opnsense-update -kzr 22.7.4-tcp
# yes | opnsense-shell reboot


Cheers,
Franco

[fraenki]

Thanks for providing a test build!

Unfortunately I will not be able to test it in the upcoming weeks. Maybe someone else could test it?
Be sure to change the tunable net.inet.tcp.sack.enable back to "1" in order to see if the new build solves this issue.

[franco]

Let's try to ask on Twitter for more testers then :)

[weust]

Just updated to the test kernel.

Not sure if this matters:

root@unicron:~ # uname -a
FreeBSD unicron.tweust.nl 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 stable/22.7-n250237-022b49bd54f SMP amd64

[franco]

Yup, thanks! We have no extended feedback but this looks fine in testing so we are going to ship this rather sooner than later.


Cheers,
Franco

[magnust]

So with the new 22.7.5 release it's time to remove the disabling of net.inet.tcp.sack.enable in tunables?

[franco]

From what I heard panics are gone, yes.