Upgrade DEC-850 to 22.1.4_1 failed

[N0b0dy1985]

I attempted to remotely upgrade a DEC-850 that I use for a home firewall. The device didn't come back up after a reboot. When I arrived back home, there was predictably no internet service and the FW couldn't be reached over the network. Serial console was unresponsive, and I had to resort to ungracefully power-cycling the box.

The device was still failing to boot. I eventually resorted to trying to reinstall from USB. This is what I see in the console:

Code Select Expand

uart0: <8250 or 16450 or compatible> port 0x3f8-0x3ff irq 3 flags 0x10 on acpi0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounter "TSC" frequency 1497188188 Hz quality 1000
Timecounters tick every 1.000 msec
Trying to mount root from ufs:/dev/ufs/OPNsense_Install [ro,noatime]...
ugen0.1: <0x1022 XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Root mount waiting for: usbus0 CAM
uhub0: 8 ports with 8 removable, self powered
Root mount waiting for: usbus0 CAM
ugen0.2: <SanDisk Cruzer> at usbus0
umass0 on uhub0
umass0: <SanDisk Cruzer, class 0/0, rev 2.00/2.00, addr 1> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x4100
umass0:1:0: Attached to scbus1
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <TS256GMTS952T2 02J0T4GB> ACS-2 ATA SATA 3.x device
ada0: Serial Number G568720070
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 244198MB (500118192 512 byte sectors)
da0 at umass-sim0 bus 0 scbus1 target 0 lun 0
da0: <SanDisk Cruzer 8.02> Removable Direct Access SCSI device
da0: Serial Number 432440190D2010EE
da0: 40.000MB/s transfers
uart0: <8250 or 16450 or compatible> port 0x3f8-0x3ff irq 3 flags 0x10 on acpi0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounter "TSC" frequency 1497188188 Hz quality 1000
Timecounters tick every 1.000 msec
Trying to mount root from ufs:/dev/ufs/OPNsense_Install [ro,noatime]...
ugen0.1: <0x1022 XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Root mount waiting for: usbus0 CAM
uhub0: 8 ports with 8 removable, self powered
Root mount waiting for: usbus0 CAM
ugen0.2: <SanDisk Cruzer> at usbus0
umass0 on uhub0
umass0: <SanDisk Cruzer, class 0/0, rev 2.00/2.00, addr 1> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x4100
umass0:1:0: Attached to scbus1
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <TS256GMTS952T2 02J0T4GB> ACS-2 ATA SATA 3.x device
ada0: Serial Number G568720070
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 244198MB (500118192 512 byte sectors)
da0 at umass-sim0 bus 0 scbus1 target 0 lun 0
da0: <SanDisk Cruzer 8.02> Removable Direct Access SCSI device
da0: Serial Number 432440190D2010EE
da0: 40.000MB/s transfers
da0: 15283MB (31301631 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
mountroot: waiting for device /dev/ufs/OPNsense_Install...
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
igb0: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb0 (ifp 0xfffff80001ad1800), ignoring.
igb3: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb3 (ifp 0xfffff80001b72000), ignoring.
intsmb0: <AMD FCH SMBus Controller> at device 20.0 on pci0
smbus0: <System Management Bus> on intsmb0
lo0: link state changed to UP
igb3: link state changed to UP
igb0: link state changed to UP
pflog0: permanently promiscuous mode enabled
da0: 15283MB (31301631 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
mountroot: waiting for device /dev/ufs/OPNsense_Install...
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
igb0: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb0 (ifp 0xfffff80001ad1800), ignoring.
igb3: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb3 (ifp 0xfffff80001b72000), ignoring.
intsmb0: <AMD FCH SMBus Controller> at device 20.0 on pci0
smbus0: <System Management Bus> on intsmb0
lo0: link state changed to UP
igb3: link state changed to UP
igb0: link state changed to UP
pflog0: permanently promiscuous mode enabled

Boot hangs at pflog0: permanently promiscuous mode enabled. I have verified the SHA256 hash of the file I downloaded, and tried a different USB memory stick with the same results. Any ideas on what's going on, or what to try next?

[Raketenmeyer]

There are new images available and it looks like you have to disable "legacy UART support" in the BIOS. Maybe that is the reason for your problems. Please have a look at https://forum.opnsense.org/index.php?topic=27432.msg133502#msg133502

[franco]

> pflog0: permanently promiscuous mode enabled

Actually, this means the box booted fine but the serial console is not available.


Cheers,
Franco

[N0b0dy1985]

Yep. Thanks Raketenmeyer and Franco. That allowed me to move pass the 'stuck boot' non-issue.  :P

When I got the FW fully booted, I still couldn't reach the GUI. I statically configured a laptop with an IP address, and was able to get into the GUI and saw that a lot of services failed to start, DHCP server being among them. I took me another reboot to get everything back up and running.

Not sure why so many services failed to start after the upgrade yesterday, but that was why I had no connectivity when I got home last night.

[franco]

A bit strange indeed. The initial upgrade to 22.1 must have gone well in contrast, otherwise you would not have ended up in 22.1.4_1. The jump to 22.1.4 is considerably smaller so I wonder what happened. The error was likely on the console and that's a bit unfortunate but at least if it happens again it can be traced.

I would suggest to do a health check from the firmware status page to be sure it installed everything correctly or to see if there are file system problems.


Cheers,
Franco