I am running the latest Opnsense and decided to run zpool upgrade. Here was the result:
Quoteroot@router:~ # zpool upgrade zroot
This system supports ZFS pool feature flags.
Enabled the following features on 'zroot':
userobj_accounting
encryption
project_quota
allocation_classes
resilver_defer
bookmark_v2
redaction_bookmarks
redacted_datasets
bookmark_written
log_spacemap
livelist
device_rebuild
zstd_compress
draid
Pool 'zroot' has the bootfs property set, you might need to update
the boot code. See gptzfsboot and loader.efi for details.
Is there anything I need to do, or can I reboot? I am worried that I'm going to brick my opnsense install by not updating "the boot code".
You should update the boot code. The details are in the mentioned man pages. ;)
OK, let's start with the output of zpool status and gpart show, please. Do you know if your system boots legacy or EFI?
What feature or flag you need desperately ? Playing with zpool upgrade without backup is something not recommended.
I would definelty not upgrade zroot atleast.
Quote from: os914964619 on July 11, 2022, 05:15:51 PM
I am running the latest Opnsense and decided to run zpool upgrade. Here was the result:
Quoteroot@router:~ # zpool upgrade zroot
This system supports ZFS pool feature flags.
Enabled the following features on 'zroot':
userobj_accounting
encryption
project_quota
allocation_classes
resilver_defer
bookmark_v2
redaction_bookmarks
redacted_datasets
bookmark_written
log_spacemap
livelist
device_rebuild
zstd_compress
draid
Pool 'zroot' has the bootfs property set, you might need to update
the boot code. See gptzfsboot and loader.efi for details.
Is there anything I need to do, or can I reboot? I am worried that I'm going to brick my opnsense install by not updating "the boot code".
Any fresh install will create a zpool with all these features enabled, so why not upgrade your existing one if only to get rid of that message in `zpool status`?
I prefer consistent state.
I know I revive a very old thread, however, a word of warning and advice is due:
Do not upgrade your boot zpool unless you have prepared the system first - I found out the hard way today!
I now know for a fact that OpnSense does not update the boot settings for either UEFI or for BIOS booting after initial installation. You can verify yourself by looking at /boot/loader.efi (which is the updated current UEFI boot loader) and comparing it to /boot/efi/efi/boot/bootx64.efi and /boot/efi/efi/freebsd/loader.efi, the two latter of which are created only once at initial installation, thus /boot/loader.efi will most likely differ (do not worry if the newer files are actually smaller).
Same goes for the files in the gptzfsboot partition for BIOS booting.
For me, it was with the new FreeBSD 14.1 kernel on 24.7.x that had a few new zpool features:
# zpool upgrade zroot
This system supports ZFS pool feature flags.
Enabled the following features on 'zroot':
edonr
zilsaxattr
head_errlog
blake3
block_cloning
vdev_zaps_v2
Pool 'zroot' has the bootfs property set, you might need to update
the boot code. See gptzfsboot(8) and loader.efi(8) for details.
The incompatible feature that stopped booting for me was head_errlog - just because the zpool was not recognized by the old boot file.
To avoid this happening, say your boot drive is on da0, you will have to do the following for UEFI (before you try to reboot!):
cp /boot/loader.efi /boot/efi/efi/boot/bootx64.efi
cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi
and for BIOS (you can do both just in case):
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 da0
Replace "da0" with your specific boot disk (find out the name with goem disk list) and note "-i 2" instead of "-i 1" from the documentation, because the BIOS boot partition is 2 instead of 1 for OpnSense, because it is set up for dual booting (partition 1 is for UEFI boot).
If the worst has happened and your system does not boot any more, you can either reinstall OpnSense from scratch or create a FAT32/VFAT USB stick with a correct loader.efi and copy that to the EFI partition into the correct locations (this is tedious, however). IDK how to fix it with BIOS boot.
Quote from: meyergru on August 05, 2024, 07:19:18 PM
If the worst has happened and your system does not boot any more, you can either reinstall OpnSense from scratch or create a FAT32/VFAT USB stick with a correct loader.efi and copy that to the EFI partition into the correct locations (this is tedious, however). IDK how to fix it with BIOS boot.
Boot a FreeBSD 14.1 CD/DVD image, escape to shell, use the same gpart command you gave above to install the boot loader.
Similar for EFI:
mount -t msdosfs /dev/da0p1 /mnt
cp /boot/loader.efi /mnt/efi/boot/bootx64.efi
cp /boot/loader.efi /mnt/efi/freebsd/loader.efi
umount /mnt
Quote from: Patrick M. Hausen on August 05, 2024, 07:31:35 PM
Quote from: meyergru on August 05, 2024, 07:19:18 PM
If the worst has happened and your system does not boot any more, you can either reinstall OpnSense from scratch or create a FAT32/VFAT USB stick with a correct loader.efi and copy that to the EFI partition into the correct locations (this is tedious, however). IDK how to fix it with BIOS boot.
Boot a FreeBSD 14.1 CD/DVD image, escape to shell, use the same gpart command you gave above to install the boot loader.
Similar for EFI:
mount -t msdosfs /dev/da0p1 /mnt
cp /boot/loader.efi /mnt/efi/boot/bootx64.efi
cp /boot/loader.efi /mnt/efi/freebsd/loader.efi
umount /mnt
Yup. After having someone at the remote site connect a display, keyboard and having no FreeBSD iso (and no CD-ROM either). I had to prepare a USB stick for download. Then, they had no internet to download that. Then, they had a Mac and no USB stick... been there, done that - not funny in any case.
I opened a feature request (https://github.com/opnsense/core/issues/7726), although I know it was my own fault...
Quote from: meyergru on August 05, 2024, 07:37:19 PM
I opened a feature request (https://github.com/opnsense/core/issues/7726), although I know it was my own fault...
Are you sure we want that? OPNsense does not necessarily run with the same partition scheme everywhere. Thanks to the bootstrap install method there could be any fancy partition and zpool layout.
Think e.g. Hetzner: Linux rescue system --> depenguinator --> opnsense-bootstrap.sh
Honestly I for one would rather install without the legacy boot partition and EFI only if the system can EFI boot. If you find both, you never know which one to update so you always have to do both just in case. The admin should set the one correct way at installation time, IMHO.
That very topic is currently discussed on the freebsd-arm mailing list. There seems to be no "one size fits all" solution, unfortunately.
Kind regards,
Patrick
P.S. Of course we could
define and document that OPNsense will always have an EFI partition in p1 and a legacy boot partition in p2 and then just automatically update. And if you insist on rolling your own installation you better stick to that partition scheme. I would be perfectly fine with that. Truenas (for example) is no different: always refer to drives by partition UUID when manipulating your pool(s) - if you don't, bad things happen.
P.P.S. That still does not solve the case of mirrored or otherwise redundant installations.
IIRC FreeBSD wanted to work towards being able to handle the boot code updates at some point. For EFI they were already adding /boot/efi to the running systems by default in some prior upgrade but there appears to be no boot partition upgrade code yet and it's harder for even older system that will not have all the mount points set. It's a slow grind and the risks are quite real either way.
I don't necessarily want this solved on our end before knowing what the actual plan is in FreeBSD (something within pkg-base perhaps?). We will have to wait and see...
Cheers,
Franco
Yes, probably you need to have some intelligence to do this to fit all needs...
I for one like that the images provided by OPN have both EFI and legacy BIOS so a single download/image can be used on either type of system. That said, I agree a documentation could be enhanced to better inform, because it is very true that once installed, it's hard to know in an emergency and needs to update both.
I suppose the alternative to provide each image X 2, one for BIOS and one for EFI could deal with it in another way.
You only need to update either BIOS or UEFI, because your specific system only uses one. But just for good measure, I would always update both types.
Quote from: cookiemonster on August 05, 2024, 10:34:15 PM
I for one like that the images provided by OPN have both EFI and legacy BIOS so a single download/image can be used on either type of system. That said, I agree a documentation could be enhanced to better inform, because it is very true that once installed, it's hard to know in an emergency and needs to update both.
I suppose the alternative to provide each image X 2, one for BIOS and one for EFI could deal with it in another way.
Not sure what you meant to say but we actually do have both images and installation being dual-boot? Serial, vga, dvd plus the hybrid ZFS or UFS install modes...
Cheers,
Franco
Quote from: meyergru on August 06, 2024, 12:02:09 AM
But just for good measure, I would always update both types.
For good measure, I'd never run zpool upgrade on OPNsense because it brings virtually nothing useful, only potentially makes your system not bootable any more.
After you install fresh, you will have exactly those zpool features active anyway, useful or not.
> For good measure, I'd never run zpool upgrade on OPNsense because it brings virtually nothing useful, only potentially makes your system not bootable any more.
I second that. Ironically enough the FreeBSD 14.0 release notes[1] state:
If the root is on a ZFS file system, updating the boot loader is mandatory if the pool is to be upgraded, and the boot loader update must be done first. Note that ZFS pool upgrades are not recommended for root file systems in most cases, but updating the boot loader can avoid making the system unbootable if the pool is upgraded in the future.
Cheers,
Franco
[1] https://www.freebsd.org/releases/14.0R/relnotes/
Quote from: meyergru on August 06, 2024, 12:02:09 AM
You only need to update either BIOS or UEFI, because your specific system only uses one. But just for good measure, I would always update both types.
Of course. As I said, in an emergency having both in the fs just adds to the elements to consider.
Quote from: franco on August 06, 2024, 07:24:07 AM
Quote from: cookiemonster on August 05, 2024, 10:34:15 PM
I for one like that the images provided by OPN have both EFI and legacy BIOS so a single download/image can be used on either type of system. That said, I agree a documentation could be enhanced to better inform, because it is very true that once installed, it's hard to know in an emergency and needs to update both.
I suppose the alternative to provide each image X 2, one for BIOS and one for EFI could deal with it in another way.
Not sure what you meant to say but we actually do have both images and installation being dual-boot? Serial, vga, dvd plus the hybrid ZFS or UFS install modes...
Cheers,
Franco
Yes exactly. Each is dual.
Quote from: franco on August 06, 2024, 09:30:31 AM
but updating the boot loader can avoid making the system unbootable if the pool is upgraded in the future.
Lol... seems kinda the other way round. Pretty much everyone (FreeNAS/TrueNAS, NAS4Free/XigmaNAS explicitly advises against upgrading the root pool feature flags -- since it only causes breakage.
Everyone has their reasons to do one thing or another. By reviving this thread, I only wanted to make it clear that it is really dangerous to "just" update the zpool features, which was not what I got from the thread so far - however dumb the move may have been on my part in the first place.
While it was clear to me that I give up backwards-compatibility by upgrading the pool (i.e. I cannot use that pool with an older ZFS/kernel version), I was not aware that the boot files are never getting updated by the OpnSense upgrade process. This is totally different with most Linuxes that use ZFS, like with my Proxmox host and the negative outcome would be prevented if that was the case for OpnSense as well, and thus I made a feature request.
I also was not aware that this is already being discussed upstream and Franco is waiting for a fix there.
However, OpnSense's approach is a little different from standard FreeBSD, by using both an EFI and a GTPBOOT partition as is indicated by the different partition index 2 for updating gptzfsboot.
Quote from: meyergru on August 06, 2024, 11:33:20 AM
I was not aware that the boot files are never getting updated by the OpnSense upgrade process. This is totally different with most Linuxes that use ZFS, [...]
Neither does FreeBSD update the boot loader - legacy boot or EFI - when the OS is updated. And the devs are aware that this is a problem waiting to turn into a disaster happening to the unsuspicious admin. There is just no one-size-fits-all solution, yet. Not even a clear consensus about the right approach, if I am not mistaken.
Before the proliferation of EFI the `zpool upgrade` command would print a message with the more or less correct gpart invocation to update the boot loader. This message was removed because today it is in itself dangerous - if followed by the letter without thinking people will overwrite their EFI partition with gptzfsboot.
I'll report back from EuroBSDCon in September ;)
With most Linux distributions it's "everything is GRUB", so much less opportunities for error. Caveat: if you follow the OpenZFS howtos to install e.g. Debian with ZFS boot and root, never upgrade your boot pool! The version of GRUB in Debian cannot boot with all feature flags enabled. GRUB is notoriously lagging behind ZFS development.
One might be tempted to use ZFSBootMenu instead, but then you are on your own again.
https://github.com/zbm-dev/zfsbootmenu
Er, what happens if you run zfs upgrade zroot and THEN update the bootloader? Are you screwed?
It's a safety precaution doing it in this order. If you update the boot code the old zroot still boots. The other way around you need to do both otherwise it stops booting. For whatever reason zfs-upgrade could potentially fail to run cleanly.
Cheers,
Franco
Quote from: meyergru on August 05, 2024, 07:37:19 PMQuote from: Patrick M. Hausen on August 05, 2024, 07:31:35 PMQuote from: meyergru on August 05, 2024, 07:19:18 PMIf the worst has happened and your system does not boot any more, you can either reinstall OpnSense from scratch or create a FAT32/VFAT USB stick with a correct loader.efi and copy that to the EFI partition into the correct locations (this is tedious, however). IDK how to fix it with BIOS boot.
Boot a FreeBSD 14.1 CD/DVD image, escape to shell, use the same gpart command you gave above to install the boot loader.
Similar for EFI:
mount -t msdosfs /dev/da0p1 /mnt
cp /boot/loader.efi /mnt/efi/boot/bootx64.efi
cp /boot/loader.efi /mnt/efi/freebsd/loader.efi
umount /mnt
Yup. After having someone at the remote site connect a display, keyboard and having no FreeBSD iso (and no CD-ROM either). I had to prepare a USB stick for download. Then, they had no internet to download that. Then, they had a Mac and no USB stick... been there, done that - not funny in any case.
I opened a feature request (https://github.com/opnsense/core/issues/7726), although I know it was my own fault...
I would sincerely like to thank you both for saving my life with your info... Upgrade broke my system and it seemed the backups stopped a while back, still get cold shivers thinking about it.
Also this uncovered that my zfs mirror wasn't properly working and the mirrors p1 was faulty, after copying the files a DD from the working p1 to the faulty fixed that as well.
Thank you!