Help please.. OPNsense box not booting anymore after power outage

[defcon8]

Hi,

I'm running a Bytemine NA-320FL with OPNSense. Today we had a power outage, after which i'm unable to boot my server anymore. I also did install a lot of updates recenetly, after which it runned fine for several weeks, but I can't remember if the machine was rebooted after the updates, so it could also be caused by some update :( Either way my whole network is down. Can anyone please help?

This machine has no VGA, so I attached the serial console and there it outputs the following:

====================

Code Select Expand


+=========================================+
|                                         |
|  1. Boot Multi User [Enter]             |
|  2. Boot [S]ingle User                  |
|  3. [Esc]ape to loader prompt           |
|  4. Reboot                              |
|                                         |
|  Options:                               |
|  5. [K]ernel: kernel (1 of 2)           |
|  6. Configure Boot [O]ptions...         |
|                                         |
|                                         |
|                                         |
+=========================================+


/boot/kernel/kernel text=0x1846cd4 data=0x1e39e8+0x76ed60 syms=[0x8+0x18fcc0+0x8+0x1ae0cb]
/boot/entropy size=0x1000
can't find 'carp'
/boot/kernel/if_bridge.ko size 0xf998 at 0x2892000
loading required module 'bridgestp'
can't find 'bridgestp'
/boot/kernel/if_enc.ko size 0x3e78 at 0x2892000
/boot/kernel/if_gre.ko size 0xb1c0 at 0x2896000
/boot/kernel/if_lagg.ko size 0x16008 at 0x28a2000
/boot/kernel/if_tap.ko size 0x8b60 at 0x28b9000
/boot/kernel/pf.ko size 0x583b8 at 0x28c2000
/boot/kernel/pflog.ko size 0x2af8 at 0x291b000
/boot/kernel/pfsync.ko size 0xebd0 at 0x291e000
Booting...
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
Copyright (c) 2013-2019 The HardenedBSD Project.
Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.1-RELEASE-p16-HBSD #0  b531d3958f5(stable/21.1)-dirty: Tue Apr 20 11:00:08 CEST 2021
    root@sensey:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
VT(vga): resolution 640x480
HardenedBSD: initialize and check features (__HardenedBSD_version 1200059 __FreeBSD_version 1201000).
CPU: Intel(R) Atom(TM) CPU D510   @ 1.66GHz (1662.53-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x106ca  Family=0x6  Model=0x1c  Stepping=10
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x40e31d<SSE3,DTES64,MON,DS_CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 2147483648 (2048 MB)
avail memory = 2031382528 (1937 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <091510 APIC1106>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
random: unblocking device.
ioapic0: Changing APIC ID to 4
ioapic0 <Version 2.0> irqs 0-23 on motherboard
Launching APs: 2 1 3
Timecounter "TSC" frequency 1662533640 Hz quality 1000
wlan: mac acl policy registered
random: entropy device external interface
kbd1 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff8128e7c0, 0) error 19
000.000054 [4336] netmap_init               netmap: loaded module
[ath_hal] loaded
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <091510 XSDT1106> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x7800-0x7807 mem 0xfe500000-0xfe57ffff,0xd0000000-0xdfffffff,0xfe400000-0xfe4fffff irq 16 at device 2.0 on pci0
agp0: <Intel Pineview SVGA controller> on vgapci0
agp0: aperture size is 256M, detected 8188k stolen memory
vgapci0: Boot video device
pcib1: <ACPI PCI-PCI bridge> irq 22 at device 28.0 on pci0
pcib1: [GIANT-LOCKED]
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection> port 0x9c00-0x9c1f mem 0xfe6e0000-0xfe6fffff,0xfe6dc000-0xfe6dffff irq 16 at device 0.0 on pci1
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:60:e0:60:a7:c1
em0: netmap queues/slots: TX 1/1024, RX 1/1024
pcib2: <ACPI PCI-PCI bridge> irq 23 at device 28.1 on pci0
pcib2: [GIANT-LOCKED]
pci2: <ACPI PCI bus> on pcib2
em1: <Intel(R) PRO/1000 Network Connection> port 0xac00-0xac1f mem 0xfe7e0000-0xfe7fffff,0xfe7dc000-0xfe7dffff irq 17 at device 0.0 on pci2
em1: Using 1024 TX descriptors and 1024 RX descriptors
em1: Using an MSI interrupt
em1: Ethernet address: 00:60:e0:60:a7:c2
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib3: <ACPI PCI-PCI bridge> irq 20 at device 28.2 on pci0
pcib3: [GIANT-LOCKED]
pci3: <ACPI PCI bus> on pcib3
em2: <Intel(R) PRO/1000 Network Connection> port 0xbc00-0xbc1f mem 0xfe8e0000-0xfe8fffff,0xfe8dc000-0xfe8dffff irq 18 at device 0.0 on pci3
em2: Using 1024 TX descriptors and 1024 RX descriptors
em2: Using an MSI interrupt
em2: Ethernet address: 00:60:e0:60:a7:c3
em2: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <ACPI PCI-PCI bridge> irq 21 at device 28.3 on pci0
pcib4: [GIANT-LOCKED]
pci4: <ACPI PCI bus> on pcib4
em3: <Intel(R) PRO/1000 Network Connection> port 0xcc00-0xcc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 19 at device 0.0 on pci4
em3: Using 1024 TX descriptors and 1024 RX descriptors
em3: Using an MSI interrupt
em3: Ethernet address: 00:60:e0:60:a7:c4
em3: netmap queues/slots: TX 1/1024, RX 1/1024
pcib5: <ACPI PCI-PCI bridge> irq 22 at device 28.4 on pci0
pcib5: [GIANT-LOCKED]
pci5: <ACPI PCI bus> on pcib5
em4: <Intel(R) PRO/1000 Network Connection> port 0xdc00-0xdc1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 16 at device 0.0 on pci5
em4: Using 1024 TX descriptors and 1024 RX descriptors
em4: Using an MSI interrupt
em4: Ethernet address: 00:60:e0:60:a7:c5
em4: netmap queues/slots: TX 1/1024, RX 1/1024
pcib6: <ACPI PCI-PCI bridge> irq 23 at device 28.5 on pci0
pcib6: [GIANT-LOCKED]
pci6: <ACPI PCI bus> on pcib6
em5: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfebe0000-0xfebfffff,0xfebdc000-0xfebdffff irq 17 at device 0.0 on pci6
em5: Using 1024 TX descriptors and 1024 RX descriptors
em5: Using an MSI interrupt
em5: Ethernet address: 00:60:e0:60:a7:c6
em5: netmap queues/slots: TX 1/1024, RX 1/1024
uhci0: <Intel 82801H (ICH8) USB controller USB-A> port 0x8000-0x801f irq 23 at device 29.0 on pci0
uhci0: LegSup = 0x2f00
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 82801H (ICH8) USB controller USB-B> port 0x7c00-0x7c1f irq 19 at device 29.1 on pci0
uhci1: LegSup = 0x2f00
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 82801H (ICH8) USB controller USB-C> port 0x7880-0x789f irq 18 at device 29.2 on pci0
uhci2: LegSup = 0x2f00
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
ehci0: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> mem 0xfe5ff000-0xfe5ff3ff irq 23 at device 29.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
usbus3: 480Mbps High Speed USB v2.0
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci7: <ACPI PCI bus> on pcib7
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH8M UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ahci0: <Intel ICH8M AHCI SATA controller> port 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x841f mem 0xfe5ff800-0xfe5fffff irq 18 at device 31.2 on pci0
ahci0: AHCI v1.10 with 3 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahciem0: <AHCI enclosure management bridge> at channel 2147483647 on ahci0
device_attach: ahciem0 attach returned 6
acpi_button0: <Power Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
Timecounters tick every 1.000 msec
ugen1.1: <Intel UHCI root HUB> at usbus1
ugen3.1: <Intel EHCI root HUB> at usbus3
uhub0: ugen2.1: <Intel UHCI root HUB> at usbus2
ugen0.1: <Intel UHCI root HUB> at usbus0
<Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
ada0: <ST9750422AS 0001BSM1> ATA8-ACS SATA 2.x device
ada0: Serial Number 5WS4LHZ2
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 715404MB (1465149168 512 byte sectors)
ada0: quirks=0x1<4K>
uhub0: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
Root mount waiting for: usbus3
Root mount waiting for: usbus3
Root mount waiting for: usbus3
uhub1: 6 ports with 6 removable, self powered

Loader variables:

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:zroot/ROOT/default
        cd9660:/dev/cd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>


============================

When I press the ? key it displays:

============================

Code Select Expand

List of GEOM managed disk devices:

  diskid/DISK-5WS4LHZ2a ufs/OPNsense_Nanoa ufsid/5f220d941d79d061a diskid/5WS4LHZ2 ufs/OPNsense_Nano ufsid/5f220d941d79d061 ada0a ada0

==============================

I tried to mount that disk:

==============================

Code Select Expand

mountroot> ufs:/dev/ada0a
Trying to mount root from ufs:/dev/ada0a []...
WARNING:  was not properly dismounted
2021-06-08T21:09:41.576473+00:00  init 96603 - - can't access /etc/rc: No such file or directory
sh: cannot open /etc/rc: No such file or directory
Enter full pathname of shell or RETURN for /bin/sh:
Cannot read termcap database;
using dumb terminal settings.
# ls
.abi_hint                       acme_anchor_rules               dev                             media                           sbin
.cshrc                          acme_anchor_setup               entropy                         mnt                             sys
.probe.for.install.media        bin                             etc                             net                             syslog-ng.core
.profile                        boot                            home                            proc                            tmp
.rnd                            boot.config                     lib                             rescue                          usr
COPYRIGHT                       conf                            libexec                         root                            var
#

==============================

There seems to be a valid filesystem, for example in the config directory there is a config.xml showing my entire OPNsense config. So I think all data is still there.. it just won't boot into the OPNsense webinterface like it used to.

Please help.... :(

[pankaj]

try running (in single user mode):

fsck -y </dev/<root_partition>

[franco]

> sh: cannot open /etc/rc: No such file or directory

Best to reinstall. Who knows how many OS files got corrupted or purged.


Cheers,
Franco