sysctl -A crashing Appliance

[CruxtheNinth]

Hi there,

while i was looking into some tuning guides i noticed that my Appliance is crashing during execute of sysctl -A, its reproducible every time and i am a bit lost on where to start the troubleshooting.


output for sysctl -A until the point it crashes

Code Select Expand



root@opnframe:~ # sysctl -A
kern.ostype: FreeBSD
kern.osrelease: 13.2-RELEASE-p10
kern.osrevision: 199506
kern.version: FreeBSD 13.2-RELEASE-p10 stable/24.1-n254984-f7b006edfa8 SMP

kern.maxvnodes: 345608
kern.maxproc: 21508
kern.maxfiles: 515848
kern.argmax: 524288
kern.securelevel: -1
kern.hostname: opnframe.REMOVED
kern.hostid: 1391663422
kern.clockrate: { hz = 1000, tick = 1000, profhz = 8128, stathz = 127 }
kern.proc.all: Format:S,proc Length:459136 Dump:0x40040000000000000000000000000000...
kern.proc.proc: Format:N Length:94656 Dump:0x40040000000000000000000000000000...
kern.proc.proc_td: Format:N Length:459136 Dump:0x40040000000000000000000000000000...
kern.file: Format:S,xfile Length:110464 Dump:0x80000000000000009ef7000000000000...
kern.posix1version: 200112
kern.ngroups: 1023
kern.job_control: 1
kern.saved_ids: 0
kern.boottime: { sec = 1708607767, usec = 919207 } Thu Feb 22 14:16:07 2024
kern.domainname:
kern.osreldate: 1302001
kern.bootfile: /boot/kernel/kernel
kern.maxfilesperproc: 464256
kern.maxprocperuid: 19357
kern.ipc.maxsockbuf: 4262144
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.max_linkhdr: 16
kern.ipc.max_protohdr: 60
kern.ipc.max_hdr: 76
kern.ipc.max_datalen: 92
kern.ipc.sooverinterval: 60
kern.ipc.maxsockets: 515848
kern.ipc.numopensockets: 207
kern.ipc.soacceptqueue: 128
kern.ipc.tls.toe.chacha20: 0
kern.ipc.tls.toe.gcm: 0
kern.ipc.tls.toe.cbc: 0
kern.ipc.tls.ifnet.permitted: 0
kern.ipc.tls.ifnet.reset_failed: 0
kern.ipc.tls.ifnet.reset_dropped: 0
kern.ipc.tls.ifnet.reset: 0
kern.ipc.tls.ifnet.chacha20: 0
kern.ipc.tls.ifnet.gcm: 0
kern.ipc.tls.ifnet.cbc: 0
kern.ipc.tls.sw.chacha20: 0
kern.ipc.tls.sw.gcm: 0
kern.ipc.tls.sw.cbc: 0
kern.ipc.tls.tasks_active: 0
kern.ipc.tls.cbc_enable: 1
kern.ipc.tls.enable: 0
kern.ipc.tls.maxlen: 16384
kern.ipc.tls.bind_threads: 1
kern.ipc.tls.allow_unload: 0
kern.ipc.tls.stats.switch_failed: 0
kern.ipc.tls.stats.switch_to_sw: 0
kern.ipc.tls.stats.switch_to_ifnet: 0
kern.ipc.tls.stats.failed_crypto: 0
kern.ipc.tls.stats.corrupted_records: 0
kern.ipc.tls.stats.active: 0
kern.ipc.tls.stats.enable_calls: 0
kern.ipc.tls.stats.offload_total: 0
kern.ipc.tls.stats.sw_rx_inqueue: 0
kern.ipc.tls.stats.sw_tx_inqueue: 0
kern.ipc.tls.stats.sw_tx_pending: 0
kern.ipc.tls.stats.threads: 4
kern.ipc.shmsegs: Format: Length:19968 Dump:0x00000000000000000000000000000000...
kern.ipc.shm_allow_removed: 1
kern.ipc.shm_use_phys: 0
kern.ipc.shmall: 131072
kern.ipc.shmseg: 128
kern.ipc.shmmni: 192
kern.ipc.shmmin: 1
kern.ipc.shmmax: 536870912
kern.ipc.sema: Format: Length:4000 Dump:0x00000000000000000000000000000000...
kern.ipc.semaem: 16384
kern.ipc.semvmx: 32767
kern.ipc.semusz: 632
kern.ipc.semume: 50
kern.ipc.semopm: 100
kern.ipc.semmsl: 340
kern.ipc.semmnu: 150
kern.ipc.semmns: 340
kern.ipc.semmni: 50
kern.ipc.msqids: Format: Length:4800 Dump:0x00000000000000000000000000000000...
kern.ipc.msgseg: 2048
kern.ipc.msgssz: 8
kern.ipc.msgtql: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgmni: 40
kern.ipc.msgmax: 16384
kern.ipc.aio.lifetime: 30000
kern.ipc.aio.target_procs: 4
kern.ipc.aio.num_procs: 0
kern.ipc.aio.max_procs: 32
kern.ipc.aio.empty_retries: 0
kern.ipc.aio.empty_results: 0
kern.ipc.piperesizeallowed: 1
kern.ipc.piperesizefail: 0
kern.ipc.pipeallocfail: 0
kern.ipc.pipefragretry: 0
kern.ipc.pipekva: 827392
kern.ipc.maxpipekva: 264114176
kern.ipc.umtx_max_robust: 1000
kern.ipc.umtx_vnode_persistent: 0
kern.ipc.sfstat: Format:I Length:88 Dump:0x00000000000000000000000000000000...
kern.ipc.nmbufs: 6409532
kern.ipc.nmbjumbo16: 83457
kern.ipc.nmbjumbo9: 148368
kern.ipc.nmbjumbop: 500744
kern.ipc.nmbclusters: 1001489
kern.ipc.num_snd_tags: 0
kern.ipc.maxmbufmem: 8204201984
kern.ipc.mb_use_ext_pgs: 1
kern.ps_strings: 49749686519200
kern.usrstack: 49749686521856
kern.logsigexit: 1
kern.iov_max: 1024
kern.hostuuid: 3752709c-7b23-11ec-9ae2-7c2be113920f
kern.maxphys: 1048576
kern.lockf: Format:S,lockf Length:59840 Dump:0x400400000200000001000000ffffffff...
kern.cryptodevallowsoft: 0
kern.userasymcrypto: 1
kern.crypto_workers_num: 4
kern.crypto.stats: Format:S Length:64 Dump:0x63190000000000000000000000000000...
kern.crypto.allow_soft: 0
kern.crypto.asym_enable: 1
kern.crypto.num_workers: 4
kern.minvnodes: 86402
kern.metadelay: 28
kern.dirdelay: 29
kern.filedelay: 30
kern.tty_nout: 2670637
kern.tty_nin: 52880
kern.tty_inq_flush_secure: 1
kern.tty_info_kstacks: 2


/var/log/latest for the same timestamp

Code Select Expand


<45>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED syslog-ng 15995 - [meta sequenceId="1"] syslog-ng starting up; version='4.6.0'
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="2"] frame pointer         = 0x28:0xfffffe0117267bd0
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="3"] code segment          = base 0x0, limit 0xfffff, type 0x1b
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="4"]                       = DPL 0, pres 1, long 1, def32 0, gran 1
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="5"] processor eflags      = interrupt enabled, resume, IOPL = 0
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="6"] current process               = 63390 (sysctl)
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="7"] trap number           = 12
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="8"] panic: page fault
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="9"] cpuid = 0
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="10"] time = 1708607919
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="11"] KDB: stack backtrace:
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="12"] db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0117267950
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="13"] vpanic() at vpanic+0x151/frame 0xfffffe01172679a0
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="14"] panic() at panic+0x43/frame 0xfffffe0117267a00
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="15"] trap_fatal() at trap_fatal+0x387/frame 0xfffffe0117267a60
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="16"] trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0117267ac0
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="17"] calltrap() at calltrap+0x8/frame 0xfffffe0117267ac0
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="18"] --- trap 0xc, rip = 0xffffffff80d13641, rsp = 0xfffffe0117267b90, rbp = 0xfffffe0117267bd0 ---
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="19"] sysctl_kern_ttys() at sysctl_kern_ttys+0xb1/frame 0xfffffe0117267bd0
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="20"] sysctl_root_handler_locked() at sysctl_root_handler_locked+0x90/frame 0xfffffe0117267c20
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="21"] sysctl_root() at sysctl_root+0x299/frame 0xfffffe0117267ca0
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="22"] userland_sysctl() at userland_sysctl+0x177/frame 0xfffffe0117267d50
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="23"] sys___sysctl() at sys___sysctl+0x5c/frame 0xfffffe0117267e00
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="24"] amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe0117267f30
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="25"] fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0117267f30
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="26"] --- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x2d3f42169a6a, rsp = 0x2d3f40646898, rbp = 0x2d3f406468d0 ---
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="27"] KDB: enter: panic
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="28"] ---<<BOOT>>---
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="29"] Copyright (c) 1992-2021 The FreeBSD Project.
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="30"] Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="31"]      The Regents of the University of California. All rights reserved.
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="32"] FreeBSD is a registered trademark of The FreeBSD Foundation.
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="33"] FreeBSD 13.2-RELEASE-p10 stable/24.1-n254984-f7b006edfa8 SMP amd64
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="34"] FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c)
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="35"] VT(vga): resolution 640x480
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="36"] CPU: Intel(R) Pentium(R) Silver N6005 @ 2.00GHz (1996.80-MHz K8-class CPU)
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="37"]   Origin="GenuineIntel"  Id=0x906c0  Family=0x6  Model=0x9c  Stepping=0
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="38"]   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="39"]   Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="40"]   AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="41"]   AMD Features2=0x101<LAHF,Prefetch>
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="42"]   Structured Extended Features=0x2394a2c3<FSGSBASE,TSCADJ,FDPEXC,SMEP,ERMS,NFPUSG,PQE,RDSEED,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,SHA>
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="43"]   Structured Extended Features2=0x18400124<UMIP,WAITPKG,GFNI,RDPID,MOVDIRI,MOVDIR64B>
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="44"]   Structured Extended Features3=0xfc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD>
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="45"]   XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="46"]   IA32_ARCH_CAPS=0x6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="47"]   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="48"]   TSC: P-state invariant, performance statistics
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="49"] real memory  = 17179869184 (16384 MB)
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="50"] avail memory = 16354148352 (15596 MB)
<13>1 2024-02-22T14:19:39+01:00 opnframe.REMOVED kernel - - [meta sequenceId="51"] Event timer "LAPIC" quality 600


The Appliance is a CWWK n6005 Box, OPNsense 24.1.2_1-amd64 on baremetal. No Surricata currently but Zenarmor in with native netmap on LAN.

Any hints/ideas how to narrow this down further?

Thanks a lot

Kind regards
CruxTheNinth

[iMx]

> Any hints/ideas how to narrow this down further?

Remove all 'tuning', make sure it doesn't happen then.

Assuming it doesn't, add 1 at a time, until you find the culprit.

[CruxtheNinth]

sorry maybe i did not express myself correct in the problem description.
I did not apply any tunings yet, i wanted to check what sysctl parameters are currently configured before changing any.

[Patrick M. Hausen]

Does `sysctl -a` work?

[CruxtheNinth]

Does `sysctl -a` work?

yes, that works fine. Strange. Why would showing opaque variables crash the system?

from the manpage:

     The following options are available:

     -A      Equivalent to -o -a (for compatibility).

     -a      List all the currently available values except for those which
             are opaque or excluded from listing via the CTLFLAG_SKIP flag.
             This option is ignored if one or more variable names are
             specified on the command line.

     -o      Show opaque variables (which are normally suppressed).  The
             format and length are printed, as well as a hex dump of the first
             sixteen bytes of the value.

[franco]

I've seen this problem on debug kernels where opening the System: Settings: Tunables page will make the firewall crash. It's surely a FreeBSD problem and if you can report what sysctl panics they will be happy to pick it up I hope.


Cheers,
Franco

[CruxtheNinth]

I've seen this problem on debug kernels where opening the System: Settings: Tunables page will make the firewall crash. It's surely a FreeBSD problem and if you can report what sysctl panics they will be happy to pick it up I hope.


Cheers,
Franco

thank you, i submitted a Bug Report:  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277240