OPNsense Forum

Archive => 20.7 Legacy Series => Topic started by: NUeB on August 18, 2020, 04:05:35 pm

Title: HELP! opnsense stuck in upgrade to 20.7? (-> dying hardware)
Post by: NUeB on August 18, 2020, 04:05:35 pm
I'm shocked: 5 yrs no problems with opnsense and now:

I started the upgrade to opnsense 20.7 about 45 minutes ago, the router (PC Engines apu1d4) is not responding since.

When I try to connect via serial connetion I just get

Code: [Select]
$ screen /dev/ttyS0 115200
[screen is terminating]

on any serial device (PC runs on manjaro linux).

What shall I do? Hard reset the router by pulling the power plug?

Any help appreciated.
Title: Re: HELP! opnsense stuck in upgrade to 20.7?
Post by: franco on August 18, 2020, 04:11:47 pm
What was the last thing you saw on the console?


Cheers,
Franco
Title: Re: HELP! opnsense stuck in upgrade to 20.7?
Post by: NUeB on August 18, 2020, 04:17:39 pm
I started the upgrade via web interface, it remains like shown in the screenshot attached.

I tried connecting via serial later, when I got no further response from the routers web interface.
Title: Re: HELP! opnsense stuck in upgrade to 20.7?
Post by: franco on August 18, 2020, 05:34:58 pm
Unfortunately that's not the console output. If you think the device is not processing any more reset power and watch the console for bootup / errors.

The upgrade may only be carried out partially. Some people have ran into these premature boot situations and ended up with partial upgrade, but opnsense-bootstrap fixed that.


Cheers,
Franco
Title: Re: HELP! opnsense stuck in upgrade to 20.7?
Post by: NUeB on August 18, 2020, 05:47:29 pm
I can access the router via serial again!  :)

I also updated it to     
Code: [Select]
OPNsense 20.7.1-amd64
FreeBSD 12.1-RELEASE-p8-HBSD
OpenSSL 1.1.1g 21 Apr 2020

But it still keeps stuck in boot process:

(https://tmp.nueb.net/nopurge/screenshot_serial_01.png)

I have no idea what da0 is and if it was there before the upgrade.

I then hit [Ctrl]+[C] and [Enter] to get to:

(https://tmp.nueb.net/nopurge/screenshot_serial_02.png)


Hitting [Ctrl]+[D] now makes the machine to continue booting until it is up and (as far as I can tell) fully functioning.


(https://tmp.nueb.net/nopurge/screenshot_serial_03.png)

What's to be done to have my router boot again without manual intervention?
Title: Re: HELP! opnsense stuck in upgrade to 20.7?
Post by: banym on August 18, 2020, 06:34:50 pm
Do you have a USB Flash stick connected to the Firewall?
Try to remove it and reboot if you don't need it to be connected.
Title: Re: HELP! opnsense stuck in upgrade to 20.7?
Post by: NUeB on August 18, 2020, 06:41:46 pm
No, there's no USB stick.
Title: Re: HELP! opnsense stuck in upgrade to 20.7?
Post by: banym on August 18, 2020, 07:00:38 pm
The log says something on port 5 "Generic Flash"

For me it looks like you have one sata SSD as ada0 and some second ssd or usb device that is connected as da0. This second device maybe is a usb cdrom or something else. Try to disconnect everything and find what device is causing the problem.
Title: Re: HELP! opnsense stuck in upgrade to 20.7?
Post by: NUeB on August 18, 2020, 07:17:57 pm
Sorry, I should have mentioned, that there is no USB stick or any other device attached to the router.

It only has one single internal M2 SSD, no other drives, extension cards or else. The hardware was not modified after I set it up 2015.

And something's gone really wrong… I can not login to the console after the FW is up for some time.
Title: Re: HELP! opnsense stuck in upgrade to 20.7?
Post by: franco on August 19, 2020, 08:52:06 am
Sorry, looks like the SSD is dying.


Cheers,
Franco
Title: Re: HELP! opnsense stuck in upgrade to 20.7?
Post by: NUeB on August 19, 2020, 10:06:06 am
@franco - I guess that makes sense.  :( Thanks.

Best idea (to me) seems be to buy a complete new hardware which I can set up as long as the old one's still running. I'm in home office at least until October, so this might minimize risk to be offline for several days.

Is migration as easy as
? Or is there any how-to-migrate instructions I can follow?

Any opinions on the PC Engines APU4D4 as a replacement? Old APU1D4 was never running high load; the APU4D4 is faster (and is even cheaper today).
Title: Re: HELP! opnsense stuck in upgrade to 20.7?
Post by: franco on August 19, 2020, 10:43:25 am
Yes, only thing to watch out for is the naming of the network driver. You need to manually edit the config.xml to make it work directly on the subsequent boot.

This is a simple search and replace in the XML file.


Cheers,
Franco
Title: Re: HELP! opnsense stuck in upgrade to 20.7?
Post by: NUeB on August 19, 2020, 10:52:40 am
Thanks.  :)

New hardware is ordered, I'll keep you updated...
Title: Re: HELP! opnsense stuck in upgrade to 20.7? (-> dying hardware)
Post by: franco on August 19, 2020, 10:57:01 am
Yes please, feel free to update this thread :)
Title: Re: HELP! opnsense stuck in upgrade to 20.7? (-> dying hardware)
Post by: NUeB on August 22, 2020, 06:02:35 pm
The new hardware is here… and running!  :)

Installation from USB and via serial console was absolutely simple. Then I downloaded the configuration from the new box and compared it to the one saved from the old box. — I just had to change the interfaces' device names (like from <if>re0</if>
to <if>igb0</if> and so on) and upload that modified configuration to the firewall.

After a restart, it came up and looked just like the old firewall. After allowing the new MAC to access WAN in the FritzBox, everything is working as supposed.

So, migration really was an easy job. Thanks again, @franco for pointing me to the hardware error and to everyone in OPNsense for their support!
Title: Re: HELP! opnsense stuck in upgrade to 20.7? (-> dying hardware)
Post by: meazz1 on August 22, 2020, 08:30:33 pm
I have seen similar problem with a 100 gig ssd.
I installed a sata hdd and never seen the issue.
Title: Re: HELP! opnsense stuck in upgrade to 20.7? (-> dying hardware)
Post by: franco on August 23, 2020, 05:20:37 pm
Yay, happy to hear :)


Cheers,
Franco
Title: Re: HELP! opnsense stuck in upgrade to 20.7? (-> dying hardware)
Post by: NUeB on August 28, 2020, 12:48:42 pm
Final update: @franco — your guess was absolutely right, it was the SSD in the old firewall.

I now installed a new SSD to the old FW and set up OPNsense with the latest configuration & updates. It's running perfectly fine. So now I have a hardware backup, just in case…  :)