Help please.. OPNsense box not booting anymore after power outage

[defcon8]

Hi,

I'm running a Bytemine NA-320FL with OPNSense. Today we had a power outage, after which i'm unable to boot my server anymore. I also did install a lot of updates recenetly, after which it runned fine for several weeks, but I can't remember if the machine was rebooted after the updates, so it could also be caused by some update Either way my whole network is down. Can anyone please help?

This machine has no VGA, so I attached the serial console and there it outputs the following:

====================

Code: [Select]

+=========================================+
 |                                         |
 |  1. Boot Multi User [Enter]             |
 |  2. Boot [S]ingle User                  |
 |  3. [Esc]ape to loader prompt           |
 |  4. Reboot                              |
 |                                         |
 |  Options:                               |
 |  5. [K]ernel: kernel (1 of 2)           |
 |  6. Configure Boot [O]ptions...         |
 |                                         |
 |                                         |
 |                                         |
 +=========================================+


/boot/kernel/kernel text=0x1846cd4 data=0x1e39e8+0x76ed60 syms=[0x8+0x18fcc0+0x8+0x1ae0cb]
/boot/entropy size=0x1000
can't find 'carp'
/boot/kernel/if_bridge.ko size 0xf998 at 0x2892000
loading required module 'bridgestp'
can't find 'bridgestp'
/boot/kernel/if_enc.ko size 0x3e78 at 0x2892000
/boot/kernel/if_gre.ko size 0xb1c0 at 0x2896000
/boot/kernel/if_lagg.ko size 0x16008 at 0x28a2000
/boot/kernel/if_tap.ko size 0x8b60 at 0x28b9000
/boot/kernel/pf.ko size 0x583b8 at 0x28c2000
/boot/kernel/pflog.ko size 0x2af8 at 0x291b000
/boot/kernel/pfsync.ko size 0xebd0 at 0x291e000
Booting...
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
Copyright (c) 2013-2019 The HardenedBSD Project.
Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.1-RELEASE-p16-HBSD #0  b531d3958f5(stable/21.1)-dirty: Tue Apr 20 11:00:08 CEST 2021
    root@sensey:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
VT(vga): resolution 640x480
HardenedBSD: initialize and check features (__HardenedBSD_version 1200059 __FreeBSD_version 1201000).
CPU: Intel(R) Atom(TM) CPU D510   @ 1.66GHz (1662.53-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x106ca  Family=0x6  Model=0x1c  Stepping=10
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x40e31d<SSE3,DTES64,MON,DS_CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 2147483648 (2048 MB)
avail memory = 2031382528 (1937 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <091510 APIC1106>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
random: unblocking device.
ioapic0: Changing APIC ID to 4
ioapic0 <Version 2.0> irqs 0-23 on motherboard
Launching APs: 2 1 3
Timecounter "TSC" frequency 1662533640 Hz quality 1000
wlan: mac acl policy registered
random: entropy device external interface
kbd1 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff8128e7c0, 0) error 19
000.000054 [4336] netmap_init               netmap: loaded module
[ath_hal] loaded
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <091510 XSDT1106> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x7800-0x7807 mem 0xfe500000-0xfe57ffff,0xd0000000-0xdfffffff,0xfe400000-0xfe4fffff irq 16 at device 2.0 on pci0
agp0: <Intel Pineview SVGA controller> on vgapci0
agp0: aperture size is 256M, detected 8188k stolen memory
vgapci0: Boot video device
pcib1: <ACPI PCI-PCI bridge> irq 22 at device 28.0 on pci0
pcib1: [GIANT-LOCKED]
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection> port 0x9c00-0x9c1f mem 0xfe6e0000-0xfe6fffff,0xfe6dc000-0xfe6dffff irq 16 at device 0.0 on pci1
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:60:e0:60:a7:c1
em0: netmap queues/slots: TX 1/1024, RX 1/1024
pcib2: <ACPI PCI-PCI bridge> irq 23 at device 28.1 on pci0
pcib2: [GIANT-LOCKED]
pci2: <ACPI PCI bus> on pcib2
em1: <Intel(R) PRO/1000 Network Connection> port 0xac00-0xac1f mem 0xfe7e0000-0xfe7fffff,0xfe7dc000-0xfe7dffff irq 17 at device 0.0 on pci2
em1: Using 1024 TX descriptors and 1024 RX descriptors
em1: Using an MSI interrupt
em1: Ethernet address: 00:60:e0:60:a7:c2
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib3: <ACPI PCI-PCI bridge> irq 20 at device 28.2 on pci0
pcib3: [GIANT-LOCKED]
pci3: <ACPI PCI bus> on pcib3
em2: <Intel(R) PRO/1000 Network Connection> port 0xbc00-0xbc1f mem 0xfe8e0000-0xfe8fffff,0xfe8dc000-0xfe8dffff irq 18 at device 0.0 on pci3
em2: Using 1024 TX descriptors and 1024 RX descriptors
em2: Using an MSI interrupt
em2: Ethernet address: 00:60:e0:60:a7:c3
em2: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <ACPI PCI-PCI bridge> irq 21 at device 28.3 on pci0
pcib4: [GIANT-LOCKED]
pci4: <ACPI PCI bus> on pcib4
em3: <Intel(R) PRO/1000 Network Connection> port 0xcc00-0xcc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 19 at device 0.0 on pci4
em3: Using 1024 TX descriptors and 1024 RX descriptors
em3: Using an MSI interrupt
em3: Ethernet address: 00:60:e0:60:a7:c4
em3: netmap queues/slots: TX 1/1024, RX 1/1024
pcib5: <ACPI PCI-PCI bridge> irq 22 at device 28.4 on pci0
pcib5: [GIANT-LOCKED]
pci5: <ACPI PCI bus> on pcib5
em4: <Intel(R) PRO/1000 Network Connection> port 0xdc00-0xdc1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 16 at device 0.0 on pci5
em4: Using 1024 TX descriptors and 1024 RX descriptors
em4: Using an MSI interrupt
em4: Ethernet address: 00:60:e0:60:a7:c5
em4: netmap queues/slots: TX 1/1024, RX 1/1024
pcib6: <ACPI PCI-PCI bridge> irq 23 at device 28.5 on pci0
pcib6: [GIANT-LOCKED]
pci6: <ACPI PCI bus> on pcib6
em5: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfebe0000-0xfebfffff,0xfebdc000-0xfebdffff irq 17 at device 0.0 on pci6
em5: Using 1024 TX descriptors and 1024 RX descriptors
em5: Using an MSI interrupt
em5: Ethernet address: 00:60:e0:60:a7:c6
em5: netmap queues/slots: TX 1/1024, RX 1/1024
uhci0: <Intel 82801H (ICH8) USB controller USB-A> port 0x8000-0x801f irq 23 at device 29.0 on pci0
uhci0: LegSup = 0x2f00
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 82801H (ICH8) USB controller USB-B> port 0x7c00-0x7c1f irq 19 at device 29.1 on pci0
uhci1: LegSup = 0x2f00
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 82801H (ICH8) USB controller USB-C> port 0x7880-0x789f irq 18 at device 29.2 on pci0
uhci2: LegSup = 0x2f00
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
ehci0: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> mem 0xfe5ff000-0xfe5ff3ff irq 23 at device 29.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
usbus3: 480Mbps High Speed USB v2.0
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci7: <ACPI PCI bus> on pcib7
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH8M UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ahci0: <Intel ICH8M AHCI SATA controller> port 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x841f mem 0xfe5ff800-0xfe5fffff irq 18 at device 31.2 on pci0
ahci0: AHCI v1.10 with 3 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahciem0: <AHCI enclosure management bridge> at channel 2147483647 on ahci0
device_attach: ahciem0 attach returned 6
acpi_button0: <Power Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
Timecounters tick every 1.000 msec
ugen1.1: <Intel UHCI root HUB> at usbus1
ugen3.1: <Intel EHCI root HUB> at usbus3
uhub0: ugen2.1: <Intel UHCI root HUB> at usbus2
ugen0.1: <Intel UHCI root HUB> at usbus0
<Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
ada0: <ST9750422AS 0001BSM1> ATA8-ACS SATA 2.x device
ada0: Serial Number 5WS4LHZ2
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 715404MB (1465149168 512 byte sectors)
ada0: quirks=0x1<4K>
uhub0: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
Root mount waiting for: usbus3
Root mount waiting for: usbus3
Root mount waiting for: usbus3
uhub1: 6 ports with 6 removable, self powered

Loader variables:

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:zroot/ROOT/default
        cd9660:/dev/cd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>

============================

When I press the ? key it displays:

============================

Code: [Select]

List of GEOM managed disk devices:

  diskid/DISK-5WS4LHZ2a ufs/OPNsense_Nanoa ufsid/5f220d941d79d061a diskid/5WS4LHZ2 ufs/OPNsense_Nano ufsid/5f220d941d79d061 ada0a ada0
==============================
 
I tried to mount that disk:

==============================

Code: [Select]

mountroot> ufs:/dev/ada0a
Trying to mount root from ufs:/dev/ada0a []...
WARNING:  was not properly dismounted
2021-06-08T21:09:41.576473+00:00  init 96603 - - can't access /etc/rc: No such file or directory
sh: cannot open /etc/rc: No such file or directory
Enter full pathname of shell or RETURN for /bin/sh:
Cannot read termcap database;
using dumb terminal settings.
# ls
.abi_hint                       acme_anchor_rules               dev                             media                           sbin
.cshrc                          acme_anchor_setup               entropy                         mnt                             sys
.probe.for.install.media        bin                             etc                             net                             syslog-ng.core
.profile                        boot                            home                            proc                            tmp
.rnd                            boot.config                     lib                             rescue                          usr
COPYRIGHT                       conf                            libexec                         root                            var
#
==============================

There seems to be a valid filesystem, for example in the config directory there is a config.xml showing my entire OPNsense config. So I think all data is still there.. it just won't boot into the OPNsense webinterface like it used to.

Please help....

[pankaj]

try running (in single user mode):

fsck -y </dev/<root_partition>

[franco]

> sh: cannot open /etc/rc: No such file or directory

Best to reinstall. Who knows how many OS files got corrupted or purged.


Cheers,
Franco