Messages

Hello, I have a genuine question regarding Zenarmor's development. I've noticed there haven't been any updates since February, and the roadmap seems to be on hold. Additionally, the UI hasn't yet been updated to align with the new OPNsense interface. Like many others, I'm really hoping the promised multicore support will be released in July and made accessible beyond just premium users. I sincerely hope the team is doing well and continuing their great work. Thank you for all your efforts!

Code Select Expand

User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36
FreeBSD 14.2-RELEASE stable/25.1-n269614-36155813721 SMP amd64
OPNsense 25.1 da994c043
Plugins os-acme-client-4.8 os-cpu-microcode-intel-1.1 os-igmp-proxy-1.5_4 os-mdns-repeater-1.2 os-sensei-1.18.5 os-sensei-agent-1.18.5 os-sensei-updater-1.17 os-smart-2.3 os-sunnyvalley-1.4_3 os-theme-advanced-1.0 os-theme-vicuna-1.48
Time Wed, 29 Jan 2025 14:35:05 +0100
OpenSSL 3.0.15
Python 3.11.11
PHP 8.3.15

Code Select Expand

[29-Jan-2025 13:35:05 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'mongodb.so' (tried: /usr/local/lib/php/20230831/mongodb.so (Cannot open "/usr/local/lib/php/20230831/mongodb.so"), /usr/local/lib/php/20230831/mongodb.so.so (Cannot open "/usr/local/lib/php/20230831/mongodb.so.so")) in Unknown on line 0

Code Select Expand

Fetching change log information, please wait... done

This will automatically fetch all available updates and apply them.

Proceed with this action? [y/N]: y

Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating SunnyValley repository catalogue...
SunnyValley repository is up to date.
All repositories are up to date.
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating SunnyValley repository catalogue...
SunnyValley repository is up to date.
All repositories are up to date.
Checking for upgrades (2 candidates): .. done
Processing candidates (2 candidates): . done
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking integrity... done (0 conflicting)
Nothing to do.
Checking all packages: .......... done
php82-pecl-mongodb has a missing dependency: php82

>>> Missing package dependencies were detected.
>>> Found 1 issue(s) in the package database.

pkg-static: No packages available to install matching 'php82' have been found in the repositories
>>> Summary of actions performed:

php82 dependency failed to be fixed

>>> There are still missing dependencies.
>>> Try fixing them manually.

>>> Also make sure to check 'pkg updating' for known issues.
Nothing to do.
Nothing to do.
Starting web GUI...done.


Another panic :(

Code Select Expand

Fatal trap 9: general protection fault while in kernel mode
cpuid = 9; apic id = 22
instruction pointer = 0x20:0xffffffff8108f4ee
stack pointer         = 0x28:0xfffffe0158e9bbd0
frame pointer         = 0x28:0xfffffe0158e9bc00
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 26738 (python3.11)
rdi: fffffe001ea8a480 rsi: 000000000000000c rdx: 0000000000000024
rcx: 46ff382abf19b8e7  r8: 000007fffffff000  r9: fffff8001ac55600
rax: fffff80188830168 rbx: fffffe0017e62a28 rbp: fffffe0158e9bc00
r10: 80000003ad429425 r11: fffff80000000000 r12: ffffffff81807940
r13: 0000000000000000 r14: fffff80188830160 r15: fffffe0158e9bc60
trap number = 9
panic: general protection fault
cpuid = 9
time = 1727453132
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0158e9b910
vpanic() at vpanic+0x131/frame 0xfffffe0158e9ba40
panic() at panic+0x43/frame 0xfffffe0158e9baa0
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe0158e9bb00
calltrap() at calltrap+0x8/frame 0xfffffe0158e9bb00
--- trap 0x9, rip = 0xffffffff8108f4ee, rsp = 0xfffffe0158e9bbd0, rbp = 0xfffffe0158e9bc00 ---
pmap_try_insert_pv_entry() at pmap_try_insert_pv_entry+0xbe/frame 0xfffffe0158e9bc00
pmap_copy() at pmap_copy+0x549/frame 0xfffffe0158e9bcb0
vmspace_fork() at vmspace_fork+0xc90/frame 0xfffffe0158e9bd30
fork1() at fork1+0x52e/frame 0xfffffe0158e9bda0
sys_fork() at sys_fork+0x54/frame 0xfffffe0158e9be00
amd64_syscall() at amd64_syscall+0x100/frame 0xfffffe0158e9bf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0158e9bf30
--- syscall (0, FreeBSD ELF64, syscall), rip = 0x8262491fa, rsp = 0x838afb3f8, rbp = 0x838afb450 ---
KDB: enter: panic
panic.txt0600003014675553714  7152 ustarrootwheelgeneral protection faultversion.txt0600007414675553714  7555 ustarrootwheelFreeBSD 14.1-RELEASE-p5 stable/24.7-n267840-e62d514886a SMP

It's beginning to look more and more like a hardware issue.


Sorry,
Franco

I found this:

Code Select Expand

<6>pid 77267 (python3.11), jid 0, uid 0: exited on signal 10 (no core dump - bad address)
<6>pid 77999 (python3.11), jid 0, uid 0: exited on signal 10 (no core dump - bad address)


Also in System>log files>general:

Code Select Expand

2024-09-27T13:43:31 Error opnsense /usr/local/etc/rc.newwanipv6: The command '/bin/kill -'TERM' '73423''(pid:/var/run/unbound.pid) returned exit code '1', the output was 'kill: 73423: No such process'
2024-09-27T13:40:38 Error opnsense /usr/local/sbin/pluginctl: The command '/bin/kill -'TERM' '72153''(pid:/var/run/unbound.pid) returned exit code '1', the output was 'kill: 72153: No such process'

Once family is sleeping, I will boot the machine with memtest86 and run it. Otherwise, I don't know what else to do...

The happyness was short :( After 10 hours of upgraded to 2.7.5, I got another panic:

Should I completed disable Wireguard? Do you think it's a hardware issue?

Code Select Expand

Fatal trap 12: page fault while in kernel mode
cpuid = 6; apic id = 18
fault virtual address = 0x30
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff810909f0
stack pointer         = 0x28:0xfffffe01549b0710
frame pointer         = 0x28:0xfffffe01549b0860
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 77999 (python3.11)
rdi: fffffe001ea27940 rsi: fffff80215ba0740 rdx: 0000000200000000
rcx: 0000000000000001  r8: 000007fffffff000  r9: 0000000000000063
rax: c6083eb6eceb6cea rbx: fffffffc00000000 rbp: fffffe01549b0860
r10: fffff80039fb8ce0 r11: fffff801dace1000 r12: 0000000000000021
r13: fffff80000000000 r14: 0000000000000000 r15: 39f7c14913149315
trap number = 12
panic: page fault
cpuid = 6
time = 1727378741
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01549b0400
vpanic() at vpanic+0x131/frame 0xfffffe01549b0530
panic() at panic+0x43/frame 0xfffffe01549b0590
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe01549b05f0
trap_pfault() at trap_pfault+0x46/frame 0xfffffe01549b0640
calltrap() at calltrap+0x8/frame 0xfffffe01549b0640
--- trap 0xc, rip = 0xffffffff810909f0, rsp = 0xfffffe01549b0710, rbp = 0xfffffe01549b0860 ---
pmap_remove_pages() at pmap_remove_pages+0x5f0/frame 0xfffffe01549b0860
vmspace_exit() at vmspace_exit+0x80/frame 0xfffffe01549b0890
exit1() at exit1+0x53a/frame 0xfffffe01549b08f0
sigexit() at sigexit+0x13d/frame 0xfffffe01549b0d60
postsig() at postsig+0x23a/frame 0xfffffe01549b0e20
ast_sig() at ast_sig+0x1d7/frame 0xfffffe01549b0ed0
ast_handler() at ast_handler+0x88/frame 0xfffffe01549b0f10
ast() at ast+0x20/frame 0xfffffe01549b0f30
doreti_ast() at doreti_ast+0x1c/frame 0x87e205ef0
KDB: enter: panic
panic.txt0600001214675332465  7150 ustarrootwheelpage faultversion.txt0600007414675332465  7553 ustarrootwheelFreeBSD 14.1-RELEASE-p5 stable/24.7-n267840-e62d514886a SMP

Reinstalled. Wish me luck :) Keep you posted if another panic comes to my way.

Another panic today, while using heavily a WG tunnel for a long period of time:

Code Select Expand

panic: Memory modified after free 0xfffff8015ea40800(2048) val=1ce4029760df7eac @ 0xfffff8015ea40dc0

cpuid = 3
time = 1726747522
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0149b4b8b0
vpanic() at vpanic+0x131/frame 0xfffffe0149b4b9e0
panic() at panic+0x43/frame 0xfffffe0149b4ba40
trash_ctor() at trash_ctor+0x53/frame 0xfffffe0149b4ba50
mb_ctor_pack() at mb_ctor_pack+0x3e/frame 0xfffffe0149b4ba90
item_ctor() at item_ctor+0x117/frame 0xfffffe0149b4bae0
m_getm2() at m_getm2+0x1aa/frame 0xfffffe0149b4bb50
m_uiotombuf() at m_uiotombuf+0x6f/frame 0xfffffe0149b4bbe0
uipc_sosend_dgram() at uipc_sosend_dgram+0x170/frame 0xfffffe0149b4bc70
sousrsend() at sousrsend+0x79/frame 0xfffffe0149b4bcd0
kern_sendit() at kern_sendit+0x1bc/frame 0xfffffe0149b4bd60
sendit() at sendit+0x184/frame 0xfffffe0149b4bdb0
sys_sendto() at sys_sendto+0x4d/frame 0xfffffe0149b4be00
amd64_syscall() at amd64_syscall+0x140/frame 0xfffffe0149b4bf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0149b4bf30
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x25520240c57a, rsp = 0x2551ff2eeea8, rbp = 0x2551ff2f30c0 ---
KDB: enter: panic

Going to backup the config, and do a clean re-install, and I will remove PCIe wifi that has no used on this box (just in case).

Otherwise kids and wife will open a sev1 and escalate it :D

Looks like different panic?

db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0149b4b8b0
vpanic() at vpanic+0x131/frame 0xfffffe0149b4b9e0
panic() at panic+0x43/frame 0xfffffe0149b4ba40
trash_ctor() at trash_ctor+0x53/frame 0xfffffe0149b4ba50
mb_ctor_pack() at mb_ctor_pack+0x3e/frame 0xfffffe0149b4ba90
item_ctor() at item_ctor+0x117/frame 0xfffffe0149b4bae0
m_getm2() at m_getm2+0x1aa/frame 0xfffffe0149b4bb50
m_uiotombuf() at m_uiotombuf+0x6f/frame 0xfffffe0149b4bbe0
uipc_sosend_dgram() at uipc_sosend_dgram+0x170/frame 0xfffffe0149b4bc70
sousrsend() at sousrsend+0x79/frame 0xfffffe0149b4bcd0
kern_sendit() at kern_sendit+0x1bc/frame 0xfffffe0149b4bd60
sendit() at sendit+0x184/frame 0xfffffe0149b4bdb0
sys_sendto() at sys_sendto+0x4d/frame 0xfffffe0149b4be00
amd64_syscall() at amd64_syscall+0x140/frame 0xfffffe0149b4bf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0149b4bf30

> panic("Memory modified after free %p(%d) val=%lx @ %p\n", mem, size, *p, p);

Yes, well, this seems to point to a memory corruption that's going on for whatever reason, apparently in UDP (which woul also point to the WireGuard kernel module).

It could still be the same panic, but since the debug kernel has more panics like this one it tries to catch errors earlier, but here also the damage was already done.

The question is if this is caused inherently by hardware and it needs to be replaced or the errors go away without using WireGuard? This doesn't seem to be a prevalent issue, but it could still be a code problem.


Cheers,
Franco

Should I try reinstalling maybe? One of the first panic was about zpool, but never happened again, but per what you are saying, looks like it's never about the same :(


At the end of the log I still see it though:

Code Select Expand

Timecounter "TSC-low" frequency 1593603556 Hz quality 1000
Timecounters tick every 1.000 msec
ugen0.1: <Intel XHCI root HUB> at usbus0
ixl1: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: True, Flow Control: None
ixl1: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from ixl1 (ifp 0xfffff8000326e000), ignoring.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ugen1.1: <Intel XHCI root HUB> at usbus1
uhub1 on usbus1
uhub1: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
nvme0: Allocated 64MB host memory buffer
nda0 at nvme0 bus 0 scbus0 target 0 lun 1
nda0: <CT500P3PSSD8 P9CR413 2417487F0AA6>
nda0: Serial Number X
nda0: nvme version 1.4
nda0: 476940MB (976773168 512 byte sectors)
Trying to mount root from zfs:zroot/ROOT/default []...
uhub0: 4 ports with 4 removable, self powered
uhub1: 16 ports with 16 removable, self powered
ugen1.2: <MediaTek Inc. WirelessDevice> at usbus1
pid 31 (zpool) is attempting to use unsafe AIO requests - not logging anymore

1. Wait for panic
2. Grab /var/crash/vmcore.0 after panic
3. Share it with me privately (it's a bigger file)


Cheers,
Franco

Hi @franco! I've sent you the file. Anything you need, just let me know.

Regards,
ff

Ok, a wireguard panic...

Can you get a vmcore.X file for me from the 24.7.4 debug kernel?

# opnsense-update -zkr dbg-24.7.4

(reboot to activate)


Thanks,
Franco

I've applied opnsense-update -zkr dbg-24.7.4 and reboot, but I will need some help to capture vmcore. Would you mind poiting me what I should run to get it?

Ok fair enough :)

Hi @franco! After applying the update 24.7.4_1, the kernel panics reboot came back :(

24.7.4 was running 4 or 5 days in a row without a single reboot. If you need me to share any specific log, let me know. Running WAN on PPPoE conenction.

Code Select Expand

Fatal trap 9: general protection fault while in kernel mode
cpuid = 11; apic id = 26
instruction pointer = 0x20:0xffffffff80bfd5ad
stack pointer         = 0x28:0xfffffe01d6227930
frame pointer         = 0x28:0xfffffe01d6227940
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 84083 (dpinger)
rdi: 8cafc2d4f55ff2a0 rsi: fffff800075aa078 rdx: 0000000000000001
rcx: 0000000000000078  r8: 0000000000000002  r9: 0000000000000580
rax: fffff800075aa000 rbx: 8cafc2d4f55ff2a0 rbp: fffffe01d6227940
r10: fffff8018ce43000 r11: fffff80206fd7500 r12: fffff802b8b52958
r13: 0000000000000000 r14: fffff800075aa078 r15: fffff8001365fde0
trap number = 9
panic: general protection fault
cpuid = 11
time = 1726434089
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01d6227670
vpanic() at vpanic+0x131/frame 0xfffffe01d62277a0
panic() at panic+0x43/frame 0xfffffe01d6227800
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe01d6227860
calltrap() at calltrap+0x8/frame 0xfffffe01d6227860
--- trap 0x9, rip = 0xffffffff80bfd5ad, rsp = 0xfffffe01d6227930, rbp = 0xfffffe01d6227940 ---
grouptaskqueue_enqueue() at grouptaskqueue_enqueue+0xd/frame 0xfffffe01d6227940
wg_peer_send_staged() at wg_peer_send_staged+0x1a7/frame 0xfffffe01d62279b0
wg_xmit() at wg_xmit+0x198/frame 0xfffffe01d6227a50
ip_output() at ip_output+0x129c/frame 0xfffffe01d6227b40
rip_send() at rip_send+0x40b/frame 0xfffffe01d6227bb0
sosend_generic() at sosend_generic+0x643/frame 0xfffffe01d6227c70
sousrsend() at sousrsend+0x5f/frame 0xfffffe01d6227cd0
kern_sendit() at kern_sendit+0x1be/frame 0xfffffe01d6227d60
sendit() at sendit+0x181/frame 0xfffffe01d6227db0
sys_sendto() at sys_sendto+0x4d/frame 0xfffffe01d6227e00
amd64_syscall() at amd64_syscall+0x100/frame 0xfffffe01d6227f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe01d6227f30
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x3e541904157a, rsp = 0x3e541cc45f28, rbp = 0x3e541cc45f70 ---
KDB: enter: panic
panic.txt0600003014671645451  7146 ustarrootwheelgeneral protection faultversion.txt0600007414671645451  7551 ustarrootwheelFreeBSD 14.1-RELEASE-p4 stable/24.7-n267825-d0d18dbbaba SMP

Code Select Expand

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 24.7.4_1 at Mon Sep 16 08:50:38 CEST 2024
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 24.7.4 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 24.7.4 is correct.
>>> Check for missing or altered base files
Error 2 occurred.
etc/sysctl.conf:
size (299, 333)
sha256digest (0x45f469e7a9b4eef887bab7b55397305043fe101e1d6ce6f7e23d758e72f56dc6, 0x8cc5c942d7c5827a96087f872ae6ef860d1dd42172f226acf925b98181eda850)
>>> Check installed repositories
OPNsense
SunnyValley
>>> Check installed plugins
os-acme-client 4.5
os-cpu-microcode-intel 1.0
os-igmp-proxy 1.5_3
os-mdns-repeater 1.1_1
os-sensei 1.17.6
os-sensei-agent 1.17.5
os-sensei-updater 1.17
os-sunnyvalley 1.4_3
os-theme-advanced 1.0
os-theme-vicuna 1.48
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" has 68 dependencies to check.
Checking packages: ..................................................................... done
***DONE***

Hi All,
What could be causing this? It's not opnsense, but this is how it looks in a fresh windows 11 installation:

https://i.imgur.com/24zu2qJ.mp4

From my mobile or another pc, it's working fine.

ps. I tested Chrome, Edge, Firewall, DuckDuckGo browers. The same with all of them, so looks like there is something wrong with the PC, like missing something? or not having install something that is needed to draw this graph?

Any ideas?

I didn't reinstall or change anything. Just installed the update  ;D

@Franco, just fyi. After upgrading to 24.7.3, I didn't re-apply this patch and so far....Uptime 4 days, 10:45:51, without a single reboot/crash. Seems what it was causing the crash, is gone :)

Code Select Expand

All patches have been applied successfully.  Have a nice day.

I will keep you posted if crashes went away :) Thanks F.