Fatal trap 12: page fault while in kernel mode

[magnust]

After upgrading to v22.1 my QOTOM i5 7200U box started crashing/rebooting randomly. I tried some different bios settings but nothing helped. So I installed OPNsense in Hyper-V on my server instead and restored a backup and I'm running it that way now instead. But it still randomly crashes and reboots (once every few days)

Not much added to the basic setup:
os-acme-client-3.8
os-dyndns-1.27_3
os-haproxy-3.10

I get the "a problem was detected" message in the GUI dashboard and have submitted reports (three times so far) but I'm getting a bit worried.

A pretty standard setup on Hyper-V crashing, have anyone else had problems?




a short bit from the report:

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address   = 0x18
fault code      = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80d37edd
stack pointer           = 0x28:0xfffffe00ad3922a0
frame pointer           = 0x28:0xfffffe00ad392310
code segment      = base 0x0, limit 0xfffff, type 0x1b
         = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process      = 20269 (W#01-hn3)
trap number      = 12
panic: page fault
cpuid = 2
time = 1645948400
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00ad392060
vpanic() at vpanic+0x17f/frame 0xfffffe00ad3920b0
panic() at panic+0x43/frame 0xfffffe00ad392110
trap_fatal() at trap_fatal+0x385/frame 0xfffffe00ad392170
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00ad3921d0
calltrap() at calltrap+0x8/frame 0xfffffe00ad3921d0
--- trap 0xc, rip = 0xffffffff80d37edd, rsp = 0xfffffe00ad3922a0, rbp = 0xfffffe00ad392310 ---
m_copydata() at m_copydata+0x4d/frame 0xfffffe00ad392310
tcp_output() at tcp_output+0x1339/frame 0xfffffe00ad3924f0
tcp_do_segment() at tcp_do_segment+0x2cd5/frame 0xfffffe00ad3925d0
tcp_input_with_port() at tcp_input_with_port+0xafb/frame 0xfffffe00ad392730
tcp_input() at tcp_input+0xb/frame 0xfffffe00ad392740
ip_input() at ip_input+0x15f/frame 0xfffffe00ad3927d0
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00ad392820
ether_demux() at ether_demux+0x138/frame 0xfffffe00ad392850
ether_nh_input() at ether_nh_input+0x355/frame 0xfffffe00ad3928b0
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00ad392900
ether_input() at ether_input+0x69/frame 0xfffffe00ad392960
nm_os_send_up() at nm_os_send_up+0x11/frame 0xfffffe00ad392970
netmap_send_up() at netmap_send_up+0x4e/frame 0xfffffe00ad3929c0
netmap_txsync_to_host() at netmap_txsync_to_host+0x74/frame 0xfffffe00ad392a40
netmap_ioctl() at netmap_ioctl+0x1b4/frame 0xfffffe00ad392b10
freebsd_netmap_ioctl() at freebsd_netmap_ioctl+0x74/frame 0xfffffe00ad392b50
devfs_ioctl() at devfs_ioctl+0xc6/frame 0xfffffe00ad392ba0
vn_ioctl() at vn_ioctl+0x1a4/frame 0xfffffe00ad392cb0
devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe00ad392cd0
kern_ioctl() at kern_ioctl+0x25b/frame 0xfffffe00ad392d40
sys_ioctl() at sys_ioctl+0xf1/frame 0xfffffe00ad392e00
amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe00ad392f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00ad392f30
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x801b410ca, rsp = 0x7fffdfffddf8, rbp = 0x7fffdfffde70 ---
KDB: enter: panic
panic.txt0600001214206626760  7142 ustarrootwheelpage faultversion.txt0600007014206626760  7541 ustarrootwheelFreeBSD 13.0-STABLE stable/22.1-n248057-239b52c9023 SMP

[mimugmail]

freebsd_netmap_ioctl() at freebsd_netmap_ioctl+0x74/frame 0xfffffe00ad392b50


IPS or Sensei/Zenarmor?

[magnust]

freebsd_netmap_ioctl() at freebsd_netmap_ioctl+0x74/frame 0xfffffe00ad392b50


IPS or Sensei/Zenarmor?

Eh what?

 

The default Intrusion prevention is turned on, yes

[franco]

Looking at the stack can't do much about it in the short-term, mid-term we will have FreeBSD 13.1 with more patches in the area but it remains doubtful this will ensure fixing it: some network buffer gets corrupted in the kernel for no visible reason. Could be software or hardware issue, either intermittent or permanent.

Funny part is this isn't tied to a network card driver problem since there is no driver-specific routine.


Cheers,
Franco

[magnust]

Looking at the stack can't do much about it in the short-term, mid-term we will have FreeBSD 13.1 with more patches in the area but it remains doubtful this will ensure fixing it: some network buffer gets corrupted in the kernel for no visible reason. Could be software or hardware issue, either intermittent or permanent.

Funny part is this isn't tied to a network card driver problem since there is no driver-specific routine.


Cheers,
Franco

Since it's running virtually in Hyper-V the NIC is also virtualized (no pass through). Can different NIC hardware result in different behavior (crash or stable) within the virtual machine like here with FreeBSD? Or is it completely isolated and virtualized so changing the NIC hardware can't really change anything?



TBH I have a slight slight feeling there could be a number of more users than me with the firewall restarting from time to time without them noticing, now after the upgrade.

[franco]

Unlikely since the network card (even if emulated) is not involved, but if you can you could try.

If this is a guest system it could also indicate the host actually screwed up the memory handling for some reason, but it would be rather rare. I'd still rather suspect a memory corruption in HW or an actual issue with the kernel processing of the packet in question.


Cheers,
Franco

[magnust]

Looking through the host for any hints of problems....

[magnust]

Installed it again from scratch and used a restore file, now as a Hyper-V Gen 1 guest instead of the Gen 2 I was currently running. Let's see if this makes any difference.

[magnust]

Looks promising running a Gen1 instead of Gen2 Hyper-V, no problems so far. Odd.

[franco]

Hmm rather strange, but wouldn't be the first issue with Hyper-V in FreeBSD 13 we've seen so far.


Cheers,
Franco

[magnust]

Hmm rather strange, but wouldn't be the first issue with Hyper-V in FreeBSD 13 we've seen so far.


Cheers,
Franco

Let's put it in the 'That's odd. But good to know' category 

[magnust]

Nooooo, I jinxed it 


EDIT: This was right after a small change in HAproxy, hmmmm


But this is really confusing, The Web GUI Dashboard says this:
Current date/time   Thu Mar 3 22:07:20 CET 2022

But the error report that popped up in the web GUI (that I sent in also) is from the future?

Code: [Select]

User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36
FreeBSD 13.0-STABLE stable/22.1-n248059-267a40127d5 SMP amd64
OPNsense 22.1.2 92fcdd63e
Plugins os-acme-client-3.8 os-dyndns-1.27_3 os-haproxy-3.10

Time Sat, 05 Mar 2022 06:02:04 +0100

OpenSSL 1.1.1m  14 Dec 2021
PHP 7.4.28
dmesg.boot:
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-STABLE stable/22.1-n248059-267a40127d5 SMP amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
SRAT: Ignoring memory at addr 0x108200000
SRAT: Ignoring memory at addr 0x1000000000
VT(vga): text 80x25
Hyper-V Version: 10.0.14393 [SP4]
  Features=0x2e7f
  PM Features=0x0 [C2]
  Features3=0xed7b2
Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (3192.01-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906ea  Family=0x6  Model=0x9e  Stepping=10
  Features=0x1f83fbff
  Features2=0xfeda3203
  AMD Features=0x2c100800
  AMD Features2=0x121
  Structured Extended Features=0x9c6fb9
  Structured Extended Features3=0x9c000000
  XSAVE Features=0xb
Hypervisor: Origin = "Microsoft Hv"
real memory  = 4294967296 (4096 MB)
avail memory = 4125024256 (3933 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table:
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 cache groups x 1 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0: MADT APIC ID 0 != hw id 4
ioapic0  irqs 0-23
Launching APs: 3 2 1
wlan: mac acl policy registered
Timecounter "Hyper-V-TSC" frequency 10000000 Hz quality 3000
random: entropy device external interface
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
vtvga0:
aesni0:
acpi0:
acpi0: Power Button (fixed)
cpu0:  on acpi0
attimer0:  port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0:  port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0



Anyway, the last bit of the message:

Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address   = 0x18
fault code      = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80d37eed
stack pointer           = 0x28:0xfffffe00ad91c2a0
frame pointer           = 0x28:0xfffffe00ad91c310
code segment      = base 0x0, limit 0xfffff, type 0x1b
         = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process      = 8632 (W#01-hn3)
trap number      = 12
panic: page fault
cpuid = 3
time = 1646341271
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00ad91c060
vpanic() at vpanic+0x17f/frame 0xfffffe00ad91c0b0
panic() at panic+0x43/frame 0xfffffe00ad91c110
trap_fatal() at trap_fatal+0x385/frame 0xfffffe00ad91c170
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00ad91c1d0
calltrap() at calltrap+0x8/frame 0xfffffe00ad91c1d0
--- trap 0xc, rip = 0xffffffff80d37eed, rsp = 0xfffffe00ad91c2a0, rbp = 0xfffffe00ad91c310 ---
m_copydata() at m_copydata+0x4d/frame 0xfffffe00ad91c310
tcp_output() at tcp_output+0x1339/frame 0xfffffe00ad91c4f0
tcp_do_segment() at tcp_do_segment+0x2cd5/frame 0xfffffe00ad91c5d0
tcp_input_with_port() at tcp_input_with_port+0xafb/frame 0xfffffe00ad91c730
tcp_input() at tcp_input+0xb/frame 0xfffffe00ad91c740
ip_input() at ip_input+0x15f/frame 0xfffffe00ad91c7d0
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00ad91c820
ether_demux() at ether_demux+0x138/frame 0xfffffe00ad91c850
ether_nh_input() at ether_nh_input+0x355/frame 0xfffffe00ad91c8b0
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00ad91c900
ether_input() at ether_input+0x69/frame 0xfffffe00ad91c960
nm_os_send_up() at nm_os_send_up+0x11/frame 0xfffffe00ad91c970
netmap_send_up() at netmap_send_up+0x4e/frame 0xfffffe00ad91c9c0
netmap_txsync_to_host() at netmap_txsync_to_host+0x74/frame 0xfffffe00ad91ca40
netmap_ioctl() at netmap_ioctl+0x1b4/frame 0xfffffe00ad91cb10
freebsd_netmap_ioctl() at freebsd_netmap_ioctl+0x74/frame 0xfffffe00ad91cb50
devfs_ioctl() at devfs_ioctl+0xc6/frame 0xfffffe00ad91cba0
vn_ioctl() at vn_ioctl+0x1a4/frame 0xfffffe00ad91ccb0
devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe00ad91ccd0
kern_ioctl() at kern_ioctl+0x25b/frame 0xfffffe00ad91cd40
sys_ioctl() at sys_ioctl+0xf1/frame 0xfffffe00ad91ce00
amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe00ad91cf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00ad91cf30
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x801b410ca, rsp = 0x7fffdfffddf8, rbp = 0x7fffdfffde70 ---
KDB: enter: panic
panic.txt0600001214210226227  7127 ustarrootwheelpage faultversion.txt0600007014210226227  7526 ustarrootwheelFreeBSD 13.0-STABLE stable/22.1-n248059-267a40127d5 SMP

[magnust]

And ten minutes later


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x18
fault code      = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80d37eed
stack pointer           = 0x28:0xfffffe00ad90e2a0
frame pointer           = 0x28:0xfffffe00ad90e310
code segment      = base 0x0, limit 0xfffff, type 0x1b
         = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process      = 26283 (W#01-hn3)
trap number      = 12
panic: page fault
cpuid = 0
time = 1646341980
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00ad90e060
vpanic() at vpanic+0x17f/frame 0xfffffe00ad90e0b0
panic() at panic+0x43/frame 0xfffffe00ad90e110
trap_fatal() at trap_fatal+0x385/frame 0xfffffe00ad90e170
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00ad90e1d0
calltrap() at calltrap+0x8/frame 0xfffffe00ad90e1d0
--- trap 0xc, rip = 0xffffffff80d37eed, rsp = 0xfffffe00ad90e2a0, rbp = 0xfffffe00ad90e310 ---
m_copydata() at m_copydata+0x4d/frame 0xfffffe00ad90e310
tcp_output() at tcp_output+0x1339/frame 0xfffffe00ad90e4f0
tcp_do_segment() at tcp_do_segment+0x2cd5/frame 0xfffffe00ad90e5d0
tcp_input_with_port() at tcp_input_with_port+0xafb/frame 0xfffffe00ad90e730
tcp_input() at tcp_input+0xb/frame 0xfffffe00ad90e740
ip_input() at ip_input+0x15f/frame 0xfffffe00ad90e7d0
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00ad90e820
ether_demux() at ether_demux+0x138/frame 0xfffffe00ad90e850
ether_nh_input() at ether_nh_input+0x355/frame 0xfffffe00ad90e8b0
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00ad90e900
ether_input() at ether_input+0x69/frame 0xfffffe00ad90e960
nm_os_send_up() at nm_os_send_up+0x11/frame 0xfffffe00ad90e970
netmap_send_up() at netmap_send_up+0x4e/frame 0xfffffe00ad90e9c0
netmap_txsync_to_host() at netmap_txsync_to_host+0x74/frame 0xfffffe00ad90ea40
netmap_ioctl() at netmap_ioctl+0x1b4/frame 0xfffffe00ad90eb10
freebsd_netmap_ioctl() at freebsd_netmap_ioctl+0x74/frame 0xfffffe00ad90eb50
devfs_ioctl() at devfs_ioctl+0xc6/frame 0xfffffe00ad90eba0
vn_ioctl() at vn_ioctl+0x1a4/frame 0xfffffe00ad90ecb0
devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe00ad90ecd0
kern_ioctl() at kern_ioctl+0x25b/frame 0xfffffe00ad90ed40
sys_ioctl() at sys_ioctl+0xf1/frame 0xfffffe00ad90ee00
amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe00ad90ef30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00ad90ef30
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x801b410ca, rsp = 0x7fffdfffddf8, rbp = 0x7fffdfffde70 ---
KDB: enter: panic
panic.txt0600001214210227534  7131 ustarrootwheelpage faultversion.txt0600007014210227534  7530 ustarrootwheelFreeBSD 13.0-STABLE stable/22.1-n248059-267a40127d5 SMP

[mimugmail]

I'd reinstall and If it still happens its a hardware error

[magnust]

I've now done a install from scratch. Manually setting up HAproxy, fw rules, ACME and everything else. Let's see if this works. I've also run a simple base install without HAproxy and so on and that one has run 100% stable.

It might be IDS or HAproxy that makes it crash (on both bare metal and virtual). Just a wild guess grasping at straws.