24.1.2_1 Crash and reboot automatically

[Christ666]

Hello Guys! 

First overall, I'm "New" to OPNsense (already worked a little bit with PFSense, but mostly I was using sophos XG home (and alot of other firewall brand for work).

First time production of a OPNsense, on a Partaker S21 (i7 10810U, 16gb RAM, MSATA SSD, 10x I210 Network). Hardware has been stressed, all works perfectly.

While in lab, everything seems to work fine. I went live with it, adjust firewall  rulle, all works fine, on 24.1. Started to install plugin, asked me to update, so it did update to 24.1.2_1, installed my pluggins (zenarmor, ntopng, snmp), configured every thing; Openvon, unbound with DoT, black list for ads, all seems to works fine also.

Today, was my first day back from vacation, and the 2nd day of this FW Live, then issue starts. 2 reboot, no where, with 30-40 mins each between. I saw that the ram usage was 9.6gb. Uninstalled ntopng with elastic search, redis, dropped to 5.4gb. Did put zenarmor on reporting only also. Looks good for all my day then suddently, another reboot.

I've been looking at some log into OPNsense, can't find anything.

I don't know where to search for any error message that could tell me the reason of the reboot (if it's plugin or anything else).

Info: Powerd is configured with Hiadaptive

Any help is greatly appreciate. 

[franco]

Hi there,

Well, to know what crashes the box we need to take a look at the crash report / kernel backtrace but the question is if the system even produces these (requiring a swap partition to dump it for later inspection).


Cheers,
Franco

[Christ666]

Hi and thanks!

Well I saw that the 2gb swap file setting wasn't checked, but in the dashboard, it says there a 8gb SWAP, but not in use (I guess this is the swap for RAM)

During the night, no cable plugged, it crashed (like 4hrs ago)

and I have also a crash reporter Issue but this might be when I shut the device off after I was trying to boot on my USB.

I guess I could try by uninstalling some Plugin, (zenarmor, I know I had uPNP installed but not active), I have  rebellion for dark theme, net-snmp, smart, ddclient.

I'll activate the 2gb swap, then run without mostly any addon and see how it goes.



Edit: I was using native netmap driver... Could this be the issu with zenarmor and I should use emulate?

[franco]

I just have to see the backtrace from the crash reporter to make any assumption on it, but netmap usually doesn't crash.

You don't need the swap file and it doesn't work kernel dumping anyway. The 8 GB swap partition is fine.

You should see at least a crash report from the GUI.


Cheers,
Franco

[Christ666]

I had this only one time, but nto sure if it was when the reboot happened. I'll post it in case, but I'll monitor first for a couple of days, then I'll reinstall the device in my rack and see if it still happen. I'll test first without any zenarmor / ntopng to see if it's better.

System Information:
User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36
FreeBSD 13.2-RELEASE-p10 stable/24.1-n254984-f7b006edfa8 SMP amd64
OPNsense 24.1.2_1 d93dbed68
Plugins os-ddclient-1.21 os-net-snmp-1.5_3 os-ntopng-1.3 os-redis-1.1_2 os-sensei-1.16.3 os-sensei-agent-1.16.5 os-sensei-updater-1.16 os-smart-2.2_4 os-sunnyvalley-1.4_3 os-theme-rebellion-1.8.9 os-upnp-1.5_6
Time Tue, 05 Mar 2024 07:47:25 -0500
OpenSSL 3.0.13
Python 3.9.18
PHP 8.2.15
dmesg.boot:
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
   The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.2-RELEASE-p10 stable/24.1-n254984-f7b006edfa8 SMP amd64
FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c)
VT(efifb): resolution 800x600
CPU: Intel(R) Core(TM) i7-10810U CPU @ 1.10GHz (1600.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0xa0660  Family=0x6  Model=0xa6  Stepping=0
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features3=0xbc000600<MCUOPT,MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x20a0c2b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16468758528 (15705 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs
FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 1 5 7 3 11 10 2 9 6 4 8
wlan: mac acl policy registered
random: entropy device external interface
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x9b70c000-0x9b70c01e
smbios0: Version: 3.2, BCD Revision: 3.2
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
acpi0: <ALASKA A M I >
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 350
Event timer "HPET1" frequency 24000000 Hz quality 340
Event timer "HPET2" frequency 24000000 Hz quality 340
Event timer "HPET3" frequency 24000000 Hz quality 340
Event timer "HPET4" frequency 24000000 Hz quality 340
Event timer "HPET5" frequency 24000000 Hz quality 340
Event timer "HPET6" frequency 24000000 Hz quality 340
Event timer "HPET7" frequency 24000000 Hz quality 340
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xd000-0xd03f mem 0xa8000000-0xa8ffffff,0xa0000000-0xa7ffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xaaf00000-0xaaf0ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <memory, RAM> at device 20.2 (no driver attached)
sdhci_pci0: <Generic SD HCI> mem 0xaaf20000-0xaaf20fff irq 19 at device 20.5 on pci0
sdhci_pci0: 1 slot(s) allocated
pci0: <serial bus> at device 21.0 (no driver attached)
pci0: <serial bus> at device 21.1 (no driver attached)
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <AHCI SATA controller> port 0xd090-0xd097,0xd080-0xd083,0xd060-0xd07f mem 0xaaf14000-0xaaf15fff,0xaaf1c000-0xaaf1c0ff,0xaaf1b000-0xaaf1b7ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pci0: <serial bus> at device 25.0 (no driver attached)
sdhci_pci1: <Generic SD HCI> mem 0xaaf19000-0xaaf19fff at device 26.0 on pci0
sdhci_pci1: 1 slot(s) allocated
mmc0: <MMC/SD bus> on sdhci_pci1
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) I210 (Copper)> port 0xc000-0xc01f mem 0xaad00000-0xaadfffff,0xaae00000-0xaae03fff irq 16 at device 0.0 on pci1
igb0: EEPROM V3.11-0 eTrack 0x80000469
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 4 RX queues 4 TX queues
igb0: Using MSI-X interrupts with 5 vectors
igb0: Ethernet address: 60:be:b4:14:a1:8a
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci2: <ACPI PCI bus> on pcib2
igb1: <Intel(R) I210 (Copper)> port 0xb000-0xb01f mem 0xaaa00000-0xaaafffff,0xaab00000-0xaab03fff irq 17 at device 0.0 on pci2
igb1: EEPROM V3.11-0 eTrack 0x80000469
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 4 RX queues 4 TX queues
igb1: Using MSI-X interrupts with 5 vectors
igb1: Ethernet address: 60:be:b4:14:a1:8b
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci3: <ACPI PCI bus> on pcib3
igb2: <Intel(R) I210 (Copper)> port 0xa000-0xa01f mem 0xaa700000-0xaa7fffff,0xaa800000-0xaa803fff irq 18 at device 0.0 on pci3
igb2: EEPROM V3.11-0 eTrack 0x80000469
igb2: Using 1024 TX descriptors and 1024 RX descriptors
igb2: Using 4 RX queues 4 TX queues
igb2: Using MSI-X interrupts with 5 vectors
igb2: Ethernet address: 60:be:b4:14:a1:8c
igb2: netmap queues/slots: TX 4/1024, RX 4/1024
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci4: <ACPI PCI bus> on pcib4
igb3: <Intel(R) I210 (Copper)> port 0x9000-0x901f mem 0xaa400000-0xaa4fffff,0xaa500000-0xaa503fff irq 19 at device 0.0 on pci4
igb3: EEPROM V3.11-0 eTrack 0x80000469
igb3: Using 1024 TX descriptors and 1024 RX descriptors
igb3: Using 4 RX queues 4 TX queues
igb3: Using MSI-X interrupts with 5 vectors
igb3: Ethernet address: 60:be:b4:14:a1:8d
igb3: netmap queues/slots: TX 4/1024, RX 4/1024
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 29.0 on pci0
pci5: <ACPI PCI bus> on pcib5
igb4: <Intel(R) I210 (Copper)> port 0x8000-0x801f mem 0xaa100000-0xaa1fffff,0xaa200000-0xaa203fff irq 16 at device 0.0 on pci5
igb4: EEPROM V3.11-0 eTrack 0x80000469
igb4: Using 1024 TX descriptors and 1024 RX descriptors
igb4: Using 4 RX queues 4 TX queues
igb4: Using MSI-X interrupts with 5 vectors
igb4: Ethernet address: 60:be:b4:14:a1:8e
igb4: netmap queues/slots: TX 4/1024, RX 4/1024
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 29.1 on pci0
pci6: <ACPI PCI bus> on pcib6
igb5: <Intel(R) I210 (Copper)> port 0x7000-0x701f mem 0xa9e00000-0xa9efffff,0xa9f00000-0xa9f03fff irq 17 at device 0.0 on pci6
igb5: EEPROM V3.11-0 eTrack 0x80000469
igb5: Using 1024 TX descriptors and 1024 RX descriptors
igb5: Using 4 RX queues 4 TX queues
igb5: Using MSI-X interrupts with 5 vectors
igb5: Ethernet address: 60:be:b4:14:a1:8f
igb5: netmap queues/slots: TX 4/1024, RX 4/1024
pcib7: <ACPI PCI-PCI bridge> irq 16 at device 29.4 on pci0
pci7: <ACPI PCI bus> on pcib7
igb6: <Intel(R) I210 (Copper)> port 0x6000-0x601f mem 0xa9b00000-0xa9bfffff,0xa9c00000-0xa9c03fff irq 16 at device 0.0 on pci7
igb6: EEPROM V3.11-0 eTrack 0x80000469
igb6: Using 1024 TX descriptors and 1024 RX descriptors
igb6: Using 4 RX queues 4 TX queues
igb6: Using MSI-X interrupts with 5 vectors
igb6: Ethernet address: 60:be:b4:14:a1:90
igb6: netmap queues/slots: TX 4/1024, RX 4/1024
pcib8: <ACPI PCI-PCI bridge> irq 17 at device 29.5 on pci0
pci8: <ACPI PCI bus> on pcib8
igb7: <Intel(R) I210 (Copper)> port 0x5000-0x501f mem 0xa9800000-0xa98fffff,0xa9900000-0xa9903fff irq 17 at device 0.0 on pci8
igb7: EEPROM V3.11-0 eTrack 0x80000469
igb7: Using 1024 TX descriptors and 1024 RX descriptors
igb7: Using 4 RX queues 4 TX queues
igb7: Using MSI-X interrupts with 5 vectors
igb7: Ethernet address: 60:be:b4:14:a1:91
igb7: netmap queues/slots: TX 4/1024, RX 4/1024
pcib9: <ACPI PCI-PCI bridge> irq 18 at device 29.6 on pci0
pci9: <ACPI PCI bus> on pcib9
igb8: <Intel(R) I210 (Copper)> port 0x4000-0x401f mem 0xa9500000-0xa95fffff,0xa9600000-0xa9603fff irq 18 at device 0.0 on pci9
igb8: EEPROM V3.11-0 eTrack 0x80000469
igb8: Using 1024 TX descriptors and 1024 RX descriptors
igb8: Using 4 RX queues 4 TX queues
igb8: Using MSI-X interrupts with 5 vectors
igb8: Ethernet address: 60:be:b4:14:a1:92
igb8: netmap queues/slots: TX 4/1024, RX 4/1024
pcib10: <ACPI PCI-PCI bridge> irq 19 at device 29.7 on pci0
pci10: <ACPI PCI bus> on pcib10
igb9: <Intel(R) I210 (Copper)> port 0x3000-0x301f mem 0xa9200000-0xa92fffff,0xa9300000-0xa9303fff irq 19 at device 0.0 on pci10
igb9: EEPROM V3.11-0 eTrack 0x80000469
igb9: Using 1024 TX descriptors and 1024 RX descriptors
igb9: Using 4 RX queues 4 TX queues
igb9: Using MSI-X interrupts with 5 vectors
igb9: Ethernet address: 60:be:b4:14:a1:93
igb9: netmap queues/slots: TX 4/1024, RX 4/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
hdac0: <Intel Comet Lake-LP HDA Controller> mem 0xaaf10000-0xaaf13fff,0xa9000000-0xa90fffff irq 16 at device 31.3 on pci0
pci0: <serial bus> at device 31.5 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
acpi_syscontainer0: <System Container> on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hwpstate_intel0: <Intel Speed Shift> on cpu0
hwpstate_intel1: <Intel Speed Shift> on cpu1
hwpstate_intel2: <Intel Speed Shift> on cpu2
hwpstate_intel3: <Intel Speed Shift> on cpu3
hwpstate_intel4: <Intel Speed Shift> on cpu4
hwpstate_intel5: <Intel Speed Shift> on cpu5
hwpstate_intel6: <Intel Speed Shift> on cpu6
hwpstate_intel7: <Intel Speed Shift> on cpu7
hwpstate_intel8: <Intel Speed Shift> on cpu8
hwpstate_intel9: <Intel Speed Shift> on cpu9
hwpstate_intel10: <Intel Speed Shift> on cpu10
hwpstate_intel11: <Intel Speed Shift> on cpu11
Timecounter "TSC" frequency 1607995459 Hz quality 1000
Timecounters tick every 1.000 msec
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
mmc0: No compatible cards found on bus
hdacc0: <Intel Kaby Lake HDA CODEC> at cad 2 on hdac0
hdaa0: <Intel Kaby Lake Audio Function Group> at nid 1 on hdacc0
pcm0: <Intel Kaby Lake (HDMI/DP 8ch)> at nid 3 on hdaa0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <LITEONIT LMT-128M6M mSATA 128GB DM7110F> ATA8-ACS SATA 3.x device
ada0: Serial Number TW0T8MRJ550854421001
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 122104MB (250069680 512 byte sectors)
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
WARNING: / was not properly dismounted
WARNING: /: mount pending error: blocks 456 files 0
Dual Console: Video Primary, Serial Secondary
uhub0: 18 ports with 18 removable, self powered
ugen0.2: <Logitech USB Receiver> at usbus0
ukbd0 on uhub0
ukbd0: <Logitech USB Receiver, class 0/0, rev 2.00/12.10, addr 1> on usbus0
kbd2 at ukbd0

[franco]

That's only the aux information about your system. We'd need "textdump" first and foremost.

But at least it looks like it crashed indeed:

> Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
> WARNING: / was not properly dismounted
> WARNING: /: mount pending error: blocks 456 files 0


Cheers,
Franco

[Christ666]

Now with 24.1.3_1, it's been up for more than 2 days, withotu any problem.

I still didn't reinstall zenarmor yet, or ntopng. I'll wait a few days to reinstall them and see. So far so good

[Christ666]

Reinstalled zenarmor, during elastic search installation, my FW crashed...


I've attached the crash reporter.

Now I did:

pkg remove elasticsearch5

cd /usr/local/etc

rm -rf elasticsearch

Should I use mogo or Elastic search? seems that it's why I had crash, even with ntopng