Error during Upgrade from 19.1.2 to .4: can't use sudo nor reach the WebIF

[mircsicz]

Hi all,

just upgraded a bunch of machines using SSH. I've seen similar probs during 18.7 upgrades but never took the time to post...



Being dropped to such a limited shell I'm kinda lost, I can't use sudo or su, so I can't reboot the machine. And as the webinterface doesn't load through my usual SSH tunnel I can't even reboot from there!!! So pls help! :-(

If I remember correctly I didn't post when having that issue in 18.7 because back then I could use the webinterface.

[franco]

This looks damaged for one reason or another. Try forcing a cold reboot. But probably better to have an image ready to do a config import + install.

If you know the root password you can also try "su" to see if this can be recovered from the running state:

# opnsense-update -f


Cheers,
Franco

[mircsicz]

Hi Franco, nope sure I have tried su with the root passwd... But guess what!

And as I said I've had that a few times with 18.7 upgrades too, all of them worked fine after the webif reboot...

Greetz

[mitsos]

What hardware are you running this on?

The screenshot almost made my mind blow up. You are connecting through your DDNS, which means everything is working up to that point: DDNS got updated, forwarding rules work (I'm pretending I didn't see you open up your firewall's remote admin to the world), but the system looks barely functional.

Since you said you've seen this a couple of times, I would say that it's either a bad RAM stick or bad HDD/SSD (SSD should have reverted to a read only state if it failed, could explain why half of the things are working and half aren't).

I would start with a hardware diagnostic first, your hardware looks haunted 

[franco]

So this is clearly an instance where the full upgrade procedure (1-3) went through:

1. update all packages including GUI and command line updater
2. run command line updater to install new kernel files
3. run command line updater to install new base files
4. reboot*

From the HardenedBSD message of the day you can see the base update (3) went fine so that's assuming it got this far.

sudo works in these situations since https://github.com/opnsense/update/commit/ad150cfa77 so that's not the problem here on 19.1... what seems to be an issue is the step (1) which could have a malfunction and thus can't execute PHP so everything starts failing including reboot (4).

The likely problem, however, would be a dying disk which could affect the base and kernel files as well ranging from all sorts of operational issues to boot not working because the kernel is missing or damaged. You should do regular health audits from the GUI before and after upgrades to make sure all files are in their correct state and none are missing.


Cheers,
Franco

[mircsicz]

Most of my Router's are build on APU's.

But those two sadly aren't, but I also had this on APU's with SSD's that are still used until today...

One of them is a year old the other, is way older... As it happened with a new and an old one and happened before I hope it's not the hard drive...

Anyways the new one is 250km's away, the old one only 15mins driving! I'll probably reset the old one during the next week!

Is there any data that would help you debug the issue?

EDIT: just found a third system (my father in law's ...), this time an APU with a 6month old SSD:


As you can see in the screener on that one the webinterface still works! And I'm only seating only a few meter's away and just loaded the latest installer image. I'ld bet that after the cold boot it will be fine...