HELP! opnsense stuck in upgrade to 20.7? (-> dying hardware)

[NUeB]

I'm shocked: 5 yrs no problems with opnsense and now:

I started the upgrade to opnsense 20.7 about 45 minutes ago, the router (PC Engines apu1d4) is not responding since.

When I try to connect via serial connetion I just get

Code: [Select]

$ screen /dev/ttyS0 115200
[screen is terminating]

on any serial device (PC runs on manjaro linux).

What shall I do? Hard reset the router by pulling the power plug?

Any help appreciated.

[franco]

What was the last thing you saw on the console?


Cheers,
Franco

[NUeB]

I started the upgrade via web interface, it remains like shown in the screenshot attached.

I tried connecting via serial later, when I got no further response from the routers web interface.

[franco]

Unfortunately that's not the console output. If you think the device is not processing any more reset power and watch the console for bootup / errors.

The upgrade may only be carried out partially. Some people have ran into these premature boot situations and ended up with partial upgrade, but opnsense-bootstrap fixed that.


Cheers,
Franco

[NUeB]

I can access the router via serial again! 

I also updated it to     

Code: [Select]

OPNsense 20.7.1-amd64
FreeBSD 12.1-RELEASE-p8-HBSD
OpenSSL 1.1.1g 21 Apr 2020
But it still keeps stuck in boot process:



I have no idea what da0 is and if it was there before the upgrade.

I then hit [Ctrl]+[C] and [Enter] to get to:




Hitting [Ctrl]+[D] now makes the machine to continue booting until it is up and (as far as I can tell) fully functioning.




What's to be done to have my router boot again without manual intervention?

[banym]

Do you have a USB Flash stick connected to the Firewall?
Try to remove it and reboot if you don't need it to be connected.

[NUeB]

No, there's no USB stick.

[banym]

The log says something on port 5 "Generic Flash"

For me it looks like you have one sata SSD as ada0 and some second ssd or usb device that is connected as da0. This second device maybe is a usb cdrom or something else. Try to disconnect everything and find what device is causing the problem.

[NUeB]

Sorry, I should have mentioned, that there is no USB stick or any other device attached to the router.

It only has one single internal M2 SSD, no other drives, extension cards or else. The hardware was not modified after I set it up 2015.

And something's gone really wrong… I can not login to the console after the FW is up for some time.

[franco]

Sorry, looks like the SSD is dying.


Cheers,
Franco

[NUeB]

@franco - I guess that makes sense.  Thanks.

Best idea (to me) seems be to buy a complete new hardware which I can set up as long as the old one's still running. I'm in home office at least until October, so this might minimize risk to be offline for several days.

Is migration as easy as

• backup configuration of the old FW (i do so regularly)
• set up new hardware with fresh installed opnsense 20.7
• upload configuration to new FW
• use new FW

? Or is there any how-to-migrate instructions I can follow?

Any opinions on the PC Engines APU4D4 as a replacement? Old APU1D4 was never running high load; the APU4D4 is faster (and is even cheaper today).

[franco]

Yes, only thing to watch out for is the naming of the network driver. You need to manually edit the config.xml to make it work directly on the subsequent boot.

This is a simple search and replace in the XML file.


Cheers,
Franco

[NUeB]

Thanks. 

New hardware is ordered, I'll keep you updated...

[franco]

Yes please, feel free to update this thread

[NUeB]

The new hardware is here… and running! 

Installation from USB and via serial console was absolutely simple. Then I downloaded the configuration from the new box and compared it to the one saved from the old box. — I just had to change the interfaces' device names (like from <if>re0</if>
to <if>igb0</if> and so on) and upload that modified configuration to the firewall.

After a restart, it came up and looked just like the old firewall. After allowing the new MAC to access WAN in the FritzBox, everything is working as supposed.

So, migration really was an easy job. Thanks again, @franco for pointing me to the hardware error and to everyone in OPNsense for their support!