17.1.r1 Kernel Panic

[Dean E. Weimer]

I went ahead and upgraded to the 17.1 production release after the reboot, so far it has survived its just after 7 crash time that it normally did. I am waiting until 8 local time to restart my failed backups from overnight. Bacula doesn't work so well when it loses access to the database due to DNS resolution failures.

[silent_mastodon]

I have the same hardware you do, with a similar config, but haven't updated it yet to the 17.x releases.

Definitely hoping the production software works better for you, or I'll be using 16.x for a long while yet.

[Dean E. Weimer]

Probably should also note, I switched over to the LibreSSL option after the update to production. As I was running that in the 16.7 branch prior to the upgrade. I did however switch to OpenSSL and reinstall necessary packages prior to the upgrade to 17.1.rc1.

I re-enabled os-smart plugin as well since disabling it didn't help. I left the Suricata disabled for now.

[Dean E. Weimer]

So far so good with production release, made it over 24 hours.

Code: [Select]

# uptime
 7:46AM  up 1 day,  1:37, 1 users, load averages: 0.38, 0.39, 0.34
Re-Enabling Suricata with the same settings I had before hopefully it continues to run another day.

[Dean E. Weimer]

Crashed Sometime between noon and 1pm today, removed os-smart plugin and disabled Suricata again.

Code: [Select]

racing pid 54070 tid 100149 td 0xfffff8003abe4a00
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119d81460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119d81490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119d81510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119d81540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119d815d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119d816e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119d81a50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119d81ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119d81bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119d81bf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x58f49f5cdfa, rsp = 0x6188be6ae848, rbp = 0x6188be6ae990 ---

[Dean E. Weimer]

I am starting to wonder if its more of coincidence that it was at first happening around the same time of day. It crashed again this afternoon.

Code: [Select]

Tracing pid 24954 tid 100149 td 0xfffff80123e46500
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119d81460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119d81490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119d81510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119d81540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119d815d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119d816e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119d81a50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119d81ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119d81bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119d81bf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x4701264adfa, rsp = 0x63d138394508, rbp = 0x63d138394650 ---

[franco]

This is starting to be suspiciously precise and matches this one:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

Sadly, no progress there since October. If anyone has a clue that would be great.

[lattera]

This is starting to be suspiciously precise and matches this one:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

Sadly, no progress there since October. If anyone has a clue that would be great.

At first, I suspected ASLR, but I think you hit the nail right on the head. I'm glad to see that no one has had a single issue with HardenedBSD's ASLR. Shows how robust our implementation is.

I'll ping upstream FreeBSD about it and see if there are any updates regarding that issue.

[franco]

Hi Dean,

A FreeBSD developer provided a test kernel:

# opnsense-update -kr 17.1.1-rwdebug
# /usr/local/etc/rc.reboot

It still panics, but will print vital debug information when doing so.

Running it and providing the output will hopefully help resolve this.


Cheers,
Franco

[franco]

PS: Please direct further info to the main thread: https://forum.opnsense.org/index.php?topic=4414.0

[franco]

Dean,

A patch from FreeBSD was reverted in 17.1.2. If you can, report back even on positive results.


Thanks,
Franco