Was running 25.1 for a week+ now. But from the logs I was seeing wierd kernel errors, "stack overflow; terminated", and intermittent disconnects on the WAN interface.
From console I ran a "zpool scrub zroot" and waited for that to finish, thinking maybe there was some corruption (SSD). That came back clean. Then it was prompting me about new features, so I did "zpool upgrade -a", which turned on a bunch of features.
I then proceeded to upgrade to 25.1.1 (through GUI), it did it's usual thing but didn't come back up on reboot.
It won't boot now. Looks like I need to somehow boot from USB stick and either replace the EFI boot loader, or disable the feature in the subject line. Maybe disable other ZFS features.
How do I do that? This doesn't work: https://forums.freebsd.org/threads/14-0-release-zfs-features-gotcha.91085/
Here's console output:
Consoles: EFI console
ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2
FreeBSD/amd64 EFI loader, Revision 1.1
(Mon Aug 2 13:18:28 CEST 2021 root@sensey)
Command line arguments: loader.efi
EFI version: 2.70
EFI Firmware: American Megatrends (rev 5.12)
Console: efi (0x1000)
Load Path: \EFI\BOOT\BOOTX64.EFI
Load Device: PciRoot(0x0)/Pci(0x17,0x0)/Sata(0x0,0xFFFF,0x0)/HD(1,GPT,0BCAF8FD-21A3-11EC-9038-009BC8F03B2A,0x28,0x64000)
BootOrder: 0001 0000
Trying ESP: PciRoot(0x0)/Pci(0x17,0x0)/Sata(0x0,0xFFFF,0x0)/HD(1,GPT,0BCAF8FD-21A3-11EC-9038-009BC8F03B2A,0x28,0x64000)
Setting currdev to disk1p1:
Trying: PciRoot(0x0)/Pci(0x17,0x0)/Sata(0x0,0xFFFF,0x0)/HD(2,GPT,0BDAD07B-21A3-11EC-9038-009BC8F03B2A,0x64028,0x400)
Setting currdev to disk1p2:
Trying: PciRoot(0x0)/Pci(0x17,0x0)/Sata(0x0,0xFFFF,0x0)/HD(3,GPT,0BE6A7AF-21A3-11EC-9038-009BC8F03B2A,0x64800,0x1000000)
Setting currdev to disk1p3:
Trying: PciRoot(0x0)/Pci(0x17,0x0)/Sata(0x0,0xFFFF,0x0)/HD(4,GPT,0BF000D1-21A3-11EC-9038-009BC8F03B2A,0x1064800,0xDE17800)
Setting currdev to disk1p4:
Failed to find bootable partition
Press any key to interrupt reboot in 3 seconds
can't load 'kernel'
Type '?' for a list of commands, 'help' for more detailed help.
OK
You could simply reinstall 25.1, import the configuration
Whenever you upgrade your zpool you must update the boot loader. That is a manual process, unfortunately.
See this thread for details:
https://forum.opnsense.org/index.php?topic=29304.0
Sigh. Too late, but thanks for the pointer. Maybe will save somebody else the trouble.
I tried a FreeBSD fix and only updated in the EFI partition (bootx64.efi) and gave up after that didn't work. Didn't copy over loader.efi into the root partition (2).
Seems like a gap in the upgrade process to not update the EFI boot loader automatically.