Classic `zpool upgrade zroot` mistake but haven't rebooted yet

[morik_opnsense]

Hello Patrick et al,

I have a classic RTFM case for which I was hoping to solicit your help. Running OPNsense 25.10_2-amd64 w/ FreeBSD 14.3-RELEASE-p4 on a zfs single disk decision DEC4040 appliance. As part of a recent zpool status check, I encountered a message to upgrade zpool to enable new features. Rather than doing my homework as to whether I need said features or not, I simply issued `zpool upgrade zroot` - a no-no. Now, here we are.

Code Select Expand

zpool status
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
config:

NAME        STATE     READ WRITE CKSUM
zroot       ONLINE       0     0     0
  nda0p3    ONLINE       0     0     0

errors: No known data errors


zpool upgrade zroot
This system supports ZFS pool feature flags.

Enabled the following features on 'zroot':
  edonr
  zilsaxattr
  head_errlog
  blake3
  block_cloning
  vdev_zaps_v2

Pool 'zroot' has the bootfs property set, you might need to update
the boot code. See gptzfsboot(8) and loader.efi(8) for details.

I read this post (and also other related ones). As I have yet to reboot the appliance, I had a few questions. My system info can be found below.

Code Select Expand

zpool status
  pool: zroot
 state: ONLINE
config:

NAME        STATE     READ WRITE CKSUM
zroot       ONLINE       0     0     0
  nda0p3    ONLINE       0     0     0

errors: No known data errors

gpart show
=>         3  2000409253  nda0  GPT  (954G)
           3      532480     1  efi  (260M)
      532483         311     2  freebsd-boot  (156K)
      532794  1981808640     3  freebsd-zfs  (945G)
  1982341434    18067822     4  freebsd-swap  (8.6G)


dmesg | grep -i "bios"
[1] smbios0: <System Management BIOS> at iomem 0x7945c000-0x7945c017
[1] smbios0: Entry point: v3 (64-bit), Version: 3.0
[1] usbus0: waiting for BIOS to give up control

dmesg | grep -i "efi"
[1] efirtc0: <EFI Realtime Clock>
[1] efirtc0: registered as a time-of-day clock, resolution 1.000000s


The following efi files exist in /boot:

Code Select Expand

ll /boot/*.efi
-r-xr-xr-x  1 root wheel 157184 Oct  7 04:24 /boot/boot1.efi*
-r-xr-xr-x  1 root wheel 110080 Oct  7 04:24 /boot/gptboot.efi*
-r-xr-xr-x  2 root wheel 658944 Oct  7 04:24 /boot/loader.efi*
-r--r--r--  1 root wheel  13653 Oct  7 04:24 /boot/loader.help.efi
-r-xr-xr-x  1 root wheel 569856 Oct  7 04:24 /boot/loader_4th.efi*
-r-xr-xr-x  1 root wheel 615424 Oct  7 04:24 /boot/loader_ia32.efi*
-r-xr-xr-x  2 root wheel 658944 Oct  7 04:24 /boot/loader_lua.efi*
-r-xr-xr-x  1 root wheel 516096 Oct  7 04:24 /boot/loader_simp.efi*

ll /boot/efi/efi/boot
total 870
-r-xr-xr-x  1 root wheel 890368 Oct 25  2022 bootx64.efi


In reading through Patrick's post (msg#5), the code mentions two files and directories namely

Code Select Expand

# update legacy boot code
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda0
# update EFI boot code
mount -t msdosfs /dev/nda0p1 /mnt
mkdir -p /mnt/efi/boot /mnt/efi/freebsd
cp /boot/loader.efi /mnt/efi/boot/bootx64.efi
cp /boot/loader.efi /mnt/efi/freebsd/loader.efi
umount /mnt


Where should I get the Freebsd14.2 loader.efifrom? and which directory should I copy it to?

I found FreeBSD 14.3 image. II use macOS as my primary workstation. So, I doubt that I can mount a UFS system like freebsd onto MacBook (as a volume) or even dd it into a usb disk to then read on macos to peek into /boot folder. My /boot has efi/efi/boot but not /efi/boot /efi/freebsd. Am I doomed?

[Patrick M. Hausen]

The loader you already have in /boot/loader.efi is the correct one for your version of FreeBSD and ZFS. Copy this to your EFI partition following my instructions in the other thread.

Also update your legacy boot partition with gpart if you have one. All necessary files are of course already on your system, they just need to be installed in the special boot partitions.

[morik_opnsense]

Thank you, Patrick. My apologies for sounding dense, but a clarification to below mentioned would be appreciated.

The loader you already have in /boot/loader.efi is the correct one for your version of FreeBSD and ZFS. Copy this to your EFI partition following my instructions in the other thread.

Recommendation was:

Code Select Expand

1. mount -t msdosfs /dev/nda0p1 /mnt
2. mkdir -p /mnt/efi/boot /mnt/efi/freebsd
3. cp /boot/loader.efi /mnt/efi/boot/bootx64.efi
4. cp /boot/loader.efi /mnt/efi/freebsd/loader.efi
5. umount /mnt

I understand that when loading w/ FreeBSD installer CD/DVD, /1/ mounts the disk, and /2/ creates EFI's boot partition for both UEFI and legacy. Existing mount on my node reveals /boot/efi to be mounting /dev/gpt/efifs.

Code Select Expand

root@MorikCage:~ %# mount
zroot/ROOT/24.4.2 on / (zfs, local, nfsv4acls)
devfs on /dev (devfs)
/dev/gpt/efifs on /boot/efi (msdosfs, local)
zroot on /zroot (zfs, local, nfsv4acls)
zroot/tmp on /tmp (zfs, local, nosuid, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noexec, nosuid, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, nosuid, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noexec, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noexec, nosuid, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, nosuid, nfsv4acls)
devfs on /var/dhcpd/dev (devfs)
devfs on /var/unbound/dev (devfs)
/usr/local/lib/python3.11 on /var/unbound/usr/local/lib/python3.11 (nullfs, local, read-only)
/lib on /var/unbound/lib (nullfs, local, read-only)

Therefore, am I correct in saying that /boot/efi/efi/boot in the current system equates to /mnt/efi/boot per above. If case be then /2-4/ for me should look like:

Code Select Expand

2'. mkdir -p /boot/efi/efi/freebsd
3'. cp /boot/loader.efi /boot/efi/efi/boot/bootx64.efi
4'. cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi

Then, as to updating legacy boot, the recommendation was:

Code Select Expand

# update legacy boot code
0. gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda0

For my case, freebsd-boot is on nda0p2 (based on this):

Code Select Expand

gpart show
=>         3  2000409253  nda0  GPT  (954G)
           3      532480     1  efi  (260M)
      532483         311     2  freebsd-boot  (156K)
      532794  1981808640     3  freebsd-zfs  (945G)
  1982341434    18067822     4  freebsd-swap  (8.6G)

So, command /0/ would be the same for me.

Code Select Expand

# update legacy boot code
0. gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda0


Interestingly though, based on the sysctl on [/i](kern.boottrace), I'm guessing that the system is using legacy not EFI for loading stage 1+2 boot? Which would mean that legacy bios update is in need of update for sure?

Code Select Expand

sysctl kern | grep boot
kern.boottime: { sec = 1761201281, usec = 365845 } Wed Oct 22 23:34:41 2025
kern.bootfile: /boot/kernel/kernel
kern.boot_tag: ---<<BOOT>>---
kern.reboot_wait_time: 0
kern.panic_reboot_wait_time: 15
kern.module_path: /boot/kernel;/boot/modules;/boot/dtb;/boot/dtb/overlays
kern.boottrace.table_size: 0
kern.boottrace.shutdown_trace_threshold: 0
kern.boottrace.shutdown_trace: 0
kern.boottrace.enabled: 0
kern.boottrace.shuttrace:
kern.boottrace.runtrace:
kern.boottrace.boottrace:
1 PART nda0p2 159232 512 i 2 o 272631296 ty freebsd-boot xs GPT xt 83bd6b9d-7f41-11dc-be0b-001560b84f0f
2 LABEL gpt/bootfs 159232 512 i 0 o 0
z0xfffff80002818200 [shape=hexagon,label="gpt/bootfs\nr0w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff8000386c600 [shape=box,label="DEV\ngpt/bootfs\nr#4"];
    <type>freebsd-boot</type>
    <label>bootfs</label>
  <name>gpt/bootfs</name>
      <name>gpt/bootfs</name>
kern.geom.eli.boot_passcache: 1
kern.vt.kbd_reboot: 1
kern.cam.boot_delay: 0