[Solved] Attempt to Upgrade to Opnsense 26.7 hangs on reboot at "masks"

[jll544]

Allegedly this was fixed in late 15.1 RC cycle: https://github.com/opnsense/src/commit/56e59980b67

But the fix affects loader.efi, so it's not actually effective until the bootloader gets updated in ESP: https://www.freebsd.org/releases/15.1R/upgrading/#upgrade-loader-uefi

That's not something that happens as part of the OPNsense upgrade (should it?).

[franco]

You're right. Doesn't change the fact that everyone including 26.7 would be affected either way by it unless they did a clean reinstall.


Cheers,
Franco

[nero355]

Allegedly this was fixed in late 15.1 RC cycle: https://github.com/opnsense/src/commit/56e59980b67

But the fix affects loader.efi, so it's not actually effective until the bootloader gets updated in ESP: https://www.freebsd.org/releases/15.1R/upgrading/#upgrade-loader-uefi

That's not something that happens as part of the OPNsense upgrade (should it?).

Patrick and meyergru have made some excellent instructions about upgrading the bootloader in a couple of topics so I am sure they can help you out in one of those topics :)

System > Settings > Administration > Console

Yes.

I don't have the exact links anymore so could one of you do that part ?

Thnx! in advance ;)

[meyergru]

https://forum.opnsense.org/index.php?msg=243083

However, the process is different for BIOS and UEFI boot, sometimes it depends on the type of boot disk and / or partitioning.

This is the reason why there is no automatic bootloader update (or upgrade of zpool options, FWIW). And if you want to update like depicted in the link, you need to have booted the NEW system first, which sometimes does not work, so it is a catch-22. That is why Franco wrote:

Doesn't change the fact that everyone including 26.7 would be affected either way by it unless they did a clean reinstall.

[jagged]

I thought I'd share a related experience, though I didn't experience a direct crash. I have a CWWK S7 Pro (Intel N150) that was running 26.1.10_11. I had no issues with the `os-cpu-microcode-intel` package until attempting the upgrade to 26.7.

After rebooting, the upgrade process would start and then fail (log snippet below). I ended up power cycling the device, as I had no KVM connected to it. This left the system in an odd state where it was running the 15.1 kernel, but the OPNsense version was still showing as 26.1.10.
 
I ended up removing the `os-cpu-microcode-intel` package and then running `opnsense-update -u` from the command line. The normal upgrade process was just reporting "nothing to do."

Code Select Expand

[12/360] Extracting os-upnp-1.9: ....... done
Reloading plugin configuration
Flushing all caches...done.
Configuring system logging...done.
Reloading template OPNsense/Syslog: configd socket missing (@/var/run/configd.socket)
pkg-static: POST-INSTALL script failed
[13/360] Upgrading pciids from 20260522 to 20260624...
[13/360] Extracting pciids-20260624: ..... done
[14/360] Reinstalling libpci-3.15.0...
[14/360] Extracting libpci-3.15.0: .......... done
[15/360] Reinstalling x86info-1.31.s03_1...
[15/360] Extracting x86info-1.31.s03_1: ....... done
[16/360] Reinstalling os-cpu-microcode-intel-1.1...
[16/360] Extracting os-cpu-microcode-intel-1.1: .. done
Reloading firmware configuration
failed waiting for configd (doesn't seem to be running)
failed waiting for configd (doesn't seem to be running)
failed waiting for configd (doesn't seem to be running)


[wide]

I have same issue right after os-cpu-microcode-intel-1.1 update

Code Select Expand

Reinstalling os-cpu-microcode-intel-1.1...
[22/392] Extracting os-cpu-microcode-intel-1.1: .. done
Reloading firmware configuration
failed waiting for configd (doesn't seem to be running)
failed waiting for configd (doesn't seem to be running)
failed waiting for configd (doesn't seem to be running)
failed waiting for configd (doesn't seem to be running)
Opnsense is running on top of the latest Proxmox version 9.2.4, using SeaBIOS and UFS2 filesystem.

[nero355]

I have same issue right after os-cpu-microcode-intel-1.1 update

Opnsense is running on top of the latest Proxmox version 9.2.4, using SeaBIOS and UFS2 filesystem.

Why not let Proxmox handle it ?!

[meyergru]

Opnsense is running on top of the latest Proxmox version 9.2.4, using SeaBIOS and UFS2 filesystem.

...which in turn prevents a CPU microcode update in a VM for security reasons, so that is useless, if not even harmful. If you want the upgrade, install "intel-microcode" under PVE and uninstall the OpnSense plugin. That is one of the first recommendations here.

[wide]

I have same issue right after os-cpu-microcode-intel-1.1 update

Opnsense is running on top of the latest Proxmox version 9.2.4, using SeaBIOS and UFS2 filesystem.

Why not let Proxmox handle it ?!

Had to restore Opnsense VM from backup taken just before upgrade since this failed upgrade messed up system completely. After removing os-cpu-microcode-intel-1.1 plugin before the upgrade the upgrade went without issues. Yes Proxmox now handles the microcode update. This os-cpu-microcode-intel plugin had been installed maybe to 23.1 Opnsense and haven't caused any issues before this 26.7 upgrade.

[Patrick M. Hausen]

You cannot apply CPU microcode updates from inside a VM at all. Installing the plugin is nonsense for virtualised systems. The hypervisor host must do this.

[wide]

You cannot apply CPU microcode updates from inside a VM at all. Installing the plugin is nonsense vor virtualised systems. The hypervisor host must do this.

Thanks for clarifying this. My point was that this os-cpu-microcode-intel plugin hasn't caused any issues from 23.1 to 26.1 until now with 26.7 where it completely broke the update process and forced to recover the system from backup.

[Patrick M. Hausen]

Yes, because there is a bug in the FreeBSD boot loader that might make the system hang with early loading of the ucode updates, regardless if the updates are ever successfully applied to the CPU ;-)

[nero355]

Yes Proxmox now handles the microcode update.

Good, because the reason I mentioned it was the same as these posts :

...which in turn prevents a CPU microcode update in a VM for security reasons, so that is useless, if not even harmful.
If you want the upgrade, install "intel-microcode" under PVE and uninstall the OpnSense plugin.

You cannot apply CPU microcode updates from inside a VM at all.

Installing the plugin is nonsense for virtualised systems. The hypervisor host must do this.

;)