17.1.r1 Kernel Panic

[Dean E. Weimer]

I updated my System on Friday from 16.7.13 to 17.1.r1, so far both Saturday and Sunday it has had a Kernel Panic at just after 7:00am Local time. I was able to capture the console output on today's crash.

Code: [Select]

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address   = 0x30
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80bfcd0c
stack pointer           = 0x28:0xfffffe0119c09430
frame pointer           = 0x28:0xfffffe0119c09460
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 87062 (sh)
[ thread pid 87062 tid 100087 ]
Stopped at      turnstile_broadcast+0x9c:       movq    0x20(%rbx,%rax,1),%rcx
db>

I looked through the crontab -l output and the /etc/crontab file I can't see anything set to run at about that time. But the fact that It did apparently at the same time, make me really suspicious and of course being that the process was sh, it could be any shell script I haven't been able to find a log message on the system indicating what happened.
Unfortunately later today I am going to have to roll it back until next weekend, this is installed at my house, for my home network and mail server, on weekdays I am in the office by 7, so if the system crashes just afterwards my mail server will be offline all day.
Anyone have any ideas on what I could do to try to narrow this down today as to the cause.

[weust]

I think some information on what hardware you use would be helpful here.
Or a virtual machine, which hypervisor.

[franco]

This is what I could find. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214344

Can you post the output of "bt" as well?

[Dean E. Weimer]

Hardware is a Jetway Intel Celeron N2930 Quad Core Dual Intel LAN Fanless - HBJC311U93W-2930-B https://www.amazon.com/gp/product/B00OY8Q0QC/ref=oh_aui_search_detailpage?ie=UTF8&psc=1 with 4G of ram and a 30G MSATA

The only plugin installed is the os-smart, here is the model information from smart on the MSATA.
Model Family:     Intel 525 Series SSDs
Device Model:     INTEL SSDMCEAC030B3)

If it crashes again today I will get the output of bt under the ddb, I wasn't familiar with it and had no way to look up the information online until I rebooted it. I will have to roll it back to the 16.7 release though before the end of the day. As I won't be able to work around it to test during the weekdays.

[Dean E. Weimer]

I reinstalled Friday, it did make it through Saturday morning ok, but crashed again on Sunday.

Code: [Select]

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address   = 0x30
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80bfcd0c
stack pointer           = 0x28:0xfffffe0119d5e430
frame pointer           = 0x28:0xfffffe0119d5e460
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 56207 (sh)
[ thread pid 56207 tid 100142 ]
Stopped at      turnstile_broadcast+0x9c:       movq    0x20(%rbx,%rax,1),%rcx
db> where
Tracing pid 56207 tid 100142 td 0xfffff8003b47aa00
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119d5e460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119d5e490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119d5e510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119d5e540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119d5e5d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119d5e6e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119d5ea50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119d5ead0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119d5ebf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119d5ebf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x2eb86e92dfa, rsp = 0x617648f9ee48, rbp = 0x617648f9ef90 ---
db> where /u
Tracing pid 56207 tid 100142 td 0xfffff8003b47aa00
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119d5e460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119d5e490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119d5e510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119d5e540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119d5e5d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119d5e6e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119d5ea50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119d5ead0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119d5ebf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119d5ebf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x2eb86e92dfa, rsp = 0x617648f9ee48, rbp = 0x617648f9ef90 ---


[franco]

Can you try this without os-smart?


Thanks,
Franco

[Dean E. Weimer]

I went ahead and removed the os-smart plugin. it didn't actually delete the smartmontools pkg so I went ahead and removed that from the command line with pkg remove command.

I have managed to setup a back door into my network using SSH on a second interface to a FreeBSD server with locally installed ip filter firewall only routing one external IP from one of our proxy servers at work through that interface. So that I can test on weekdays as well, and if it crashes after I leave I can ssh to that host and access the serial console to reboot it.

[franco]

Ok, thank you.

[Dean E. Weimer]

This time it crashed much earlier, appears to be about 3am.

Code: [Select]

Tracing pid 67294 tid 100272 td 0xfffff800a1756000
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119e8b460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119e8b490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119e8b510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119e8b540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119e8b5d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119e8b6e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119e8ba50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119e8bad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119e8bbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119e8bbf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x57b76a57dfa, rsp = 0x7cea88a94bf8, rbp = 0x7cea88a94d40 ---


[lattera]

Are you running 32-bit or 64-bit?

[Dean E. Weimer]

64 bit, it did crash again just after 7 unfortunately my back door failed. I used my wireless adapter to create the second network connection. I forgot by Ubiquiti UniFi APs go into isolation mode and shutdown wireless when they lose access to their default gateway. Which in turn shutdown the new SSID I setup connected to my Internet VLAN. I will swing by my house and grab the output from db during my lunch hour.

[Dean E. Weimer]

In case its helpful, here's the uname -a output.

# uname -a
FreeBSD opnsense.dweimer.local 11.0-RELEASE-p7 FreeBSD 11.0-RELEASE-p7 #0 175886459(master): Mon Jan 16 02:00:58 CET 2017     root@sensey64:/usr/obj/usr/src/sys/SMP  amd64

Code: [Select]

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x30
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80bfcd0c
stack pointer           = 0x28:0xfffffe0119dc2430
frame pointer           = 0x28:0xfffffe0119dc2460
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 62022 (sh)
[ thread pid 62022 tid 100164 ]
Stopped at      turnstile_broadcast+0x9c:       movq    0x20(%rbx,%rax,1),%rcx
db> where
Tracing pid 62022 tid 100164 td 0xfffff8003ef07000
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119dc2460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119dc2490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119dc2510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119dc2540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119dc25d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119dc26e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119dc2a50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119dc2ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119dc2bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119dc2bf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x4c20ce58dfa, rsp = 0x7520364f0aa8, rbp = 0x7520364f0bf0 ---


[lattera]

Do you have Suricata enabled in IPS mode?

[Dean E. Weimer]

It was enabled, but not in IPS mode. I went ahead and disabled it, the attachment shows the settings only thing changed with the disable was the unchecking of the enabled option.

[Dean E. Weimer]

No change, crashed just after midnight.

Code: [Select]

Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address   = 0x30
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80bfcd0c
stack pointer           = 0x28:0xfffffe0119c77430
frame pointer           = 0x28:0xfffffe0119c77460
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 70014 (sh)
[ thread pid 70014 tid 100087 ]
Stopped at      turnstile_broadcast+0x9c:       movq    0x20(%rbx,%rax,1),%rcx
db> where
Tracing pid 70014 tid 100087 td 0xfffff80004ca0000
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119c77460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119c77490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119c77510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119c77540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119c775d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119c776e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119c77a50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119c77ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119c77bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119c77bf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x6025cfe3dfa, rsp = 0x6676bfa9b668, rbp = 0x6676bfa9b7b0 ---