"Danger. Unexpected error, check log for details" during 25.7.6 upgrade

[drosophila]

The 403 is expected, that always happens when the webgui is restarted: you're forcefully logged out, all access is rightly denied. Just go back to the index page, log in and done, like you did, nothing to be concerned about. You could run an audit. If this finds any missing or corrupt files, it should correct or at least report them, unless they're config files which probably aren't being checked (but a check would be possible for presence and syntax at least; semantics would also be possible, like if everything required is there and nothing extraneous, and even if the contents make sense, like if you have DHCP and a static IP configured for the same interface (unless that's a supported feature of course)).
So, everyone for whom the box came back from a 404, did you perchance look at the CPU usage? I had waited 30 minutes and the CPU was 100% idle before I killed it. The problem is that I'm using a thumbdrive so I have no indication of disk activity, and also it is very slow. The previous update took 2 hours to complete, and the CPU was mostly idle but not entirely (between 85% and 96% idle), but never 100% so I knew it hadn't died. I may try the update again and just leave it sit overnight in case it actually is just having a coffee break or something. ;)

I am on the -nano image, maybe there's a connection between working and failing and partially failing updates there?

[Wyzard]

I had what seems to be the same problem while upgrading from 25.7.5: error popup on the update screen while installing the upgrade, 404 when I reloaded the update page, 403 from the root URL (which should've given me either the dashboard or the login page).  But it came back after a few minutes.

Since earlier posts here have expressed concern about whether the core package might be broken by this upgrade, I downloaded a config backup and the latest installer as a precaution, then rebooted my router as a test.  It started up successfully and seems to be working as usual.

[razamatan]

i'm having this issue.  how do i recover?

[drosophila]

You simply wait, as I've found out:
it seems like the upgrade indeed was just taking a coffee break. I've restored and then re-run the update. The GUI stopped responding at the exact same spot, but since this time I had opened a root shell, I was able to see that it indeed was and still is doing stuff. It's literally crawling around at about 0.1MB/s, mostly stuck waiting on block I/O. In /var/log/pkg/pkg<xxxxxxx>.log I can see that it still is making progress, so even though the GUI still is 404, it's doing its thing. Obviously, disk speed is the abcolutely determining factor here, on fast drives you might not even notice the outage when half the root fs isn't there, while on slower drives it'll be more likely to hit the window of opportunity. Having seen all that I'm pretty confident that even my snail will come back to life eventually. I'll need another storage device since I can't have day-long outages every couple weeks once I actually deploy it (well, I can if the core function stays active). But this trickle-writes probably are going to kill the thumbdrive sooner rather than later, even with /var and /tmp mounted as tmpfs as they are in -nano.
Reveal: I'm used to XigmaNAS (embedded), which does its updates as one large file in a couple of minutes at most, and otherwise runs completely in memory, so I sort of expected the same from -nano, especially given that bad actors might somehow manage to write to the filesystem. With a memory-only fs, it'll just be one reboot away from a clean system. OPNsense seems to be more hdd-centric with its package system, I'll need to adjust to this.

[razamatan]

i have a fast nvme drive, and i think my update was broken for other reasons.  i followed https://forum.opnsense.org/index.php?topic=49437.15 to fully recover.

[drosophila]

By chance I checked my still open shell and saw "pkg-static 92615 - [meta sequenceId="1"] opnsense-25.7.6 installed". Reloading the also still open browser window indeed brought back the dashboard like nothing ever happened. It's still not done yet but probably will end up finishing in due time. Apparently I wasn't patient enough last time.

BTW: it's a bit bothersome that one doesn't seem to be able to install missing plugins when an update is pending, at least it said I need to update first. Since I had recovered using a vanilla download, my reloaded config made it miss the plugins (it's working though). I'd have preferred to install them before retrying the update, though they're not essential.