Messages

I don't think this is a nut issue, but something more specific on the OPNsense pkg. I am running mine in net client mode with a FreeBSD actually connected to the UPS. That FreeBSD server is running NUT installed from a local Poudriere build with version 2.8.5_1. The server is working without issue.
root@supermicro:/ # pkg info nut-2.8.5_1
nut-2.8.5_1
Name           : nut
Version        : 2.8.5_1
Installed on   : Wed May 27 06:48:15 2026 CDT
Origin         : sysutils/nut
Architecture   : FreeBSD:15:amd64
Prefix         : /usr/local
Categories     : sysutils
Licenses       : GPLv3+, GPLv2+, ART10, GPLv1+
Maintainer     : cy@FreeBSD.org
WWW            : https://www.networkupstools.org/
Comment        : Network UPS Tools

I am back checking in again on OPNSense, I have one feature that I have never been able to get to work correctly that has kept me from making the transition. And that one thing is DNS conditional forwarding to multiple servers. I support multiple remote tunnels with various internal DNS suffixes. Most of these DNS suffix's are managed by multiple redundant servers for example Active Directory Domain Controllers, and a few Samba servers running samba domains. I have been unable to get Unbound or DNSMasq to handle a secondary server. So if remote end reboots domain controller the lookup's start failing even though I have a secondary path available and even after primary DNS name server is back up appear to cache negative result lookup for a while until a restart of service.
The only successful solution I have found that handles this correctly is using Forward Only DNS zones (Conditional Forwarders) in Bind DNS. Though the OPNSense Bind DNS plugin has made a lot of improvements since I last checked in, it still lacks the option to create forward only zones.
Am I missing someway to do this with DNSmasq or Unbound?
In the case of the Samba DNS there is no support for slave zones so a forward zone is the only possibility, and I don't always get permissions granted to slave some zones even if I wanted to use extra overhead to run a slave zone instead of a forward only zone.

I am starting to wonder if its more of coincidence that it was at first happening around the same time of day. It crashed again this afternoon.

Code Select Expand

Tracing pid 24954 tid 100149 td 0xfffff80123e46500
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119d81460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119d81490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119d81510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119d81540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119d815d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119d816e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119d81a50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119d81ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119d81bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119d81bf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x4701264adfa, rsp = 0x63d138394508, rbp = 0x63d138394650 ---

Crashed Sometime between noon and 1pm today, removed os-smart plugin and disabled Suricata again.

Code Select Expand

racing pid 54070 tid 100149 td 0xfffff8003abe4a00
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119d81460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119d81490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119d81510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119d81540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119d815d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119d816e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119d81a50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119d81ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119d81bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119d81bf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x58f49f5cdfa, rsp = 0x6188be6ae848, rbp = 0x6188be6ae990 ---

So far so good with production release, made it over 24 hours.

Code Select Expand

# uptime
7:46AM  up 1 day,  1:37, 1 users, load averages: 0.38, 0.39, 0.34

Re-Enabling Suricata with the same settings I had before hopefully it continues to run another day.

Probably should also note, I switched over to the LibreSSL option after the update to production. As I was running that in the 16.7 branch prior to the upgrade. I did however switch to OpenSSL and reinstall necessary packages prior to the upgrade to 17.1.rc1.

I re-enabled os-smart plugin as well since disabling it didn't help. I left the Suricata disabled for now.

I went ahead and upgraded to the 17.1 production release after the reboot, so far it has survived its just after 7 crash time that it normally did. I am waiting until 8 local time to restart my failed backups from overnight. Bacula doesn't work so well when it loses access to the database due to DNS resolution failures.

No change, crashed just after midnight.

Code Select Expand

Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address   = 0x30
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80bfcd0c
stack pointer           = 0x28:0xfffffe0119c77430
frame pointer           = 0x28:0xfffffe0119c77460
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 70014 (sh)
[ thread pid 70014 tid 100087 ]
Stopped at      turnstile_broadcast+0x9c:       movq    0x20(%rbx,%rax,1),%rcx
db> where
Tracing pid 70014 tid 100087 td 0xfffff80004ca0000
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119c77460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119c77490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119c77510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119c77540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119c775d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119c776e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119c77a50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119c77ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119c77bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119c77bf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x6025cfe3dfa, rsp = 0x6676bfa9b668, rbp = 0x6676bfa9b7b0 ---


It was enabled, but not in IPS mode. I went ahead and disabled it, the attachment shows the settings only thing changed with the disable was the unchecking of the enabled option.

In case its helpful, here's the uname -a output.

# uname -a
FreeBSD opnsense.dweimer.local 11.0-RELEASE-p7 FreeBSD 11.0-RELEASE-p7 #0 175886459(master): Mon Jan 16 02:00:58 CET 2017     root@sensey64:/usr/obj/usr/src/sys/SMP  amd64

Code Select Expand

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x30
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80bfcd0c
stack pointer           = 0x28:0xfffffe0119dc2430
frame pointer           = 0x28:0xfffffe0119dc2460
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 62022 (sh)
[ thread pid 62022 tid 100164 ]
Stopped at      turnstile_broadcast+0x9c:       movq    0x20(%rbx,%rax,1),%rcx
db> where
Tracing pid 62022 tid 100164 td 0xfffff8003ef07000
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119dc2460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119dc2490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119dc2510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119dc2540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119dc25d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119dc26e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119dc2a50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119dc2ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119dc2bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119dc2bf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x4c20ce58dfa, rsp = 0x7520364f0aa8, rbp = 0x7520364f0bf0 ---


64 bit, it did crash again just after 7 unfortunately my back door failed. I used my wireless adapter to create the second network connection. I forgot by Ubiquiti UniFi APs go into isolation mode and shutdown wireless when they lose access to their default gateway. Which in turn shutdown the new SSID I setup connected to my Internet VLAN. I will swing by my house and grab the output from db during my lunch hour.

This time it crashed much earlier, appears to be about 3am.

Code Select Expand

Tracing pid 67294 tid 100272 td 0xfffff800a1756000
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119e8b460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119e8b490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119e8b510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119e8b540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119e8b5d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119e8b6e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119e8ba50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119e8bad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119e8bbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119e8bbf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x57b76a57dfa, rsp = 0x7cea88a94bf8, rbp = 0x7cea88a94d40 ---


I went ahead and removed the os-smart plugin. it didn't actually delete the smartmontools pkg so I went ahead and removed that from the command line with pkg remove command.

I have managed to setup a back door into my network using SSH on a second interface to a FreeBSD server with locally installed ip filter firewall only routing one external IP from one of our proxy servers at work through that interface. So that I can test on weekdays as well, and if it crashes after I leave I can ssh to that host and access the serial console to reboot it.

I reinstalled Friday, it did make it through Saturday morning ok, but crashed again on Sunday.

Code Select Expand

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address   = 0x30
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80bfcd0c
stack pointer           = 0x28:0xfffffe0119d5e430
frame pointer           = 0x28:0xfffffe0119d5e460
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 56207 (sh)
[ thread pid 56207 tid 100142 ]
Stopped at      turnstile_broadcast+0x9c:       movq    0x20(%rbx,%rax,1),%rcx
db> where
Tracing pid 56207 tid 100142 td 0xfffff8003b47aa00
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119d5e460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119d5e490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119d5e510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119d5e540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119d5e5d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119d5e6e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119d5ea50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119d5ead0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119d5ebf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119d5ebf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x2eb86e92dfa, rsp = 0x617648f9ee48, rbp = 0x617648f9ef90 ---
db> where /u
Tracing pid 56207 tid 100142 td 0xfffff8003b47aa00
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0119d5e460
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0119d5e490
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0119d5e510
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0119d5e540
exec_new_vmspace() at exec_new_vmspace+0x225/frame 0xfffffe0119d5e5d0
exec_elf64_imgact() at exec_elf64_imgact+0xa50/frame 0xfffffe0119d5e6e0
kern_execve() at kern_execve+0x7f9/frame 0xfffffe0119d5ea50
sys_execve() at sys_execve+0x4c/frame 0xfffffe0119d5ead0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0119d5ebf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0119d5ebf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x2eb86e92dfa, rsp = 0x617648f9ee48, rbp = 0x617648f9ef90 ---


Hardware is a Jetway Intel Celeron N2930 Quad Core Dual Intel LAN Fanless - HBJC311U93W-2930-B https://www.amazon.com/gp/product/B00OY8Q0QC/ref=oh_aui_search_detailpage?ie=UTF8&psc=1 with 4G of ram and a 30G MSATA

The only plugin installed is the os-smart, here is the model information from smart on the MSATA.
Model Family:     Intel 525 Series SSDs
Device Model:     INTEL SSDMCEAC030B3)

If it crashes again today I will get the output of bt under the ddb, I wasn't familiar with it and had no way to look up the information online until I rebooted it. I will have to roll it back to the 16.7 release though before the end of the day. As I won't be able to work around it to test during the weekdays.