ZFS snapshots and rollback question

[Mks]

Dear all,

may you can support, I'm a bit confused.

I've installed the most recent version of opnsense with ZFS. Main reason is the possibility of snapshots.
There is one pool created "zroot", I try to create and rollback a snapshot but without success, may you have some ideas. Here some code (clean install for test purposes)

Code: [Select]

root@OPNsense:/ # zfs list -t all
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot                904M  48.5G    88K  /zroot
zroot/ROOT           900M  48.5G    88K  none
zroot/ROOT/default   900M  48.5G   900M  /
zroot/tmp            152K  48.5G   152K  /tmp
zroot/usr            352K  48.5G    88K  /usr
zroot/usr/home        88K  48.5G    88K  /usr/home
zroot/usr/ports       88K  48.5G    88K  /usr/ports
zroot/usr/src         88K  48.5G    88K  /usr/src
zroot/var           1.36M  48.5G    88K  /var
zroot/var/audit       88K  48.5G    88K  /var/audit
zroot/var/crash       88K  48.5G    88K  /var/crash
zroot/var/log        956K  48.5G   956K  /var/log
zroot/var/mail        88K  48.5G    88K  /var/mail
zroot/var/tmp         88K  48.5G    88K  /var/tmp
root@OPNsense:/ # echo "test1" > test.txt
root@OPNsense:/ # cat test.txt
test1
root@OPNsense:/ # zfs snap -r zroot@snap1
root@OPNsense:/ # echo "test2" > test.txt
root@OPNsense:/ # cat test.txt
test2
root@OPNsense:/ # zfs list -t all
NAME                       USED  AVAIL  REFER  MOUNTPOINT
zroot                      904M  48.5G    88K  /zroot
zroot@snap1                   0      -    88K  -
zroot/ROOT                 900M  48.5G    88K  none
zroot/ROOT@snap1              0      -    88K  -
zroot/ROOT/default         900M  48.5G   900M  /
zroot/ROOT/default@snap1    56K      -   900M  -
zroot/tmp                  152K  48.5G   152K  /tmp
zroot/tmp@snap1               0      -   152K  -
zroot/usr                  352K  48.5G    88K  /usr
zroot/usr@snap1               0      -    88K  -
zroot/usr/home              88K  48.5G    88K  /usr/home
zroot/usr/home@snap1          0      -    88K  -
zroot/usr/ports             88K  48.5G    88K  /usr/ports
zroot/usr/ports@snap1         0      -    88K  -
zroot/usr/src               88K  48.5G    88K  /usr/src
zroot/usr/src@snap1           0      -    88K  -
zroot/var                 1.43M  48.5G    88K  /var
zroot/var@snap1               0      -    88K  -
zroot/var/audit             88K  48.5G    88K  /var/audit
zroot/var/audit@snap1         0      -    88K  -
zroot/var/crash             88K  48.5G    88K  /var/crash
zroot/var/crash@snap1         0      -    88K  -
zroot/var/log             1.00M  48.5G   956K  /var/log
zroot/var/log@snap1         72K      -   956K  -
zroot/var/mail              88K  48.5G    88K  /var/mail
zroot/var/mail@snap1          0      -    88K  -
zroot/var/tmp               88K  48.5G    88K  /var/tmp
zroot/var/tmp@snap1           0      -    88K  -
root@OPNsense:/ # zfs rollback -r zroot@snap1
root@OPNsense:/ # cat test.txt
test2

After the rollback my assumption was that "test1" is the output?
My goal is to create before any update an snapshot to be able to rollback to the previous state if anything unexpected happens.

br

[Patrick M. Hausen]

There is no recursive flag for zfs rollback. You need to rollback all the individual filesystem manually.
Specifically in your case - probably - zroot/ROOT/default@snap1.

The "-r" flag for zfs rollback does not mean "recursive". Read `man zfs-rollback` for details.

I frequently use something like this:

Code: [Select]

zfs list -t snap -h | awk '/@snap1/ { printf "zfs rollback %s\n", $1 }' | sh
But ...

you might want to look into boot environments. They take care of the snapshots and allow you to give fancy names to your various versions, even boot into past ones if you have console access.

Code: [Select]

# list current BEs
bectl list
# assume we are running 21.7 and the major update to 22.1 is waiting
# rename "default" to "21.7"
bectl rename default 21.7
# create new BE for the new version
bectl create 22.1
# activate new BE for next reboot, then reboot into it
bectl activate 22.1
reboot
# now perform UI update
# after reboot 2 BEs will be present: 21.7 and 22.1 - you can pick them at the boot loader prompt if necessary
bectl list

You can do the same with minor versions, of course. All the work is already been done. Enjoy

[Mks]

Thanks, great!

[tmanok]

But ...

you might want to look into boot environments. They take care of the snapshots and allow you to give fancy names to your various versions, even boot into past ones if you have console access.

Code: [Select]

# list current BEs
bectl list
# assume we are running 21.7 and the major update to 22.1 is waiting
# rename "default" to "21.7"
bectl rename default 21.7
# create new BE for the new version
bectl create 22.1
# activate new BE for next reboot, then reboot into it
bectl activate 22.1
reboot
# now perform UI update
# after reboot 2 BEs will be present: 21.7 and 22.1 - you can pick them at the boot loader prompt if necessary
bectl list

You can do the same with minor versions, of course. All the work is already been done. Enjoy

Holy crapperjacks that's awesome! Was bectl made by/for OPNSense or is this a FreeBSD tool that I have yet to come across? Well done whoever made it, like a cross between installing a new Linux kernel and making a VM snapshot, hah!

[mimugmail]

Is it worth to build a plugin around it?
@pmhausen: is there a way to define a fallback if Default doesnt boot?

[Patrick M. Hausen]

Holy crapperjacks that's awesome! Was bectl made by/for OPNSense or is this a FreeBSD tool that I have yet to come across? Well done whoever made it, like a cross between installing a new Linux kernel and making a VM snapshot, hah!

Standard FreeBSD since ... well almost since ZFS. There was the beadm uitility written in shell, which mimicked the Solaris tool of the same name. Was replaced by bectl written in C and using libzfs in 12.x or 11.x, even, can't remember.

Kind regards,
Patrick

[Patrick M. Hausen]

Is it worth to build a plugin around it?
@pmhausen: is there a way to define a fallback if Default doesnt boot?

You can always set a BE to "boot once" and if that fails have a sneaker admin power cycle the machine and it will boot into the last active one, which is hopefully "known good".

So probably:

Code: [Select]

bectl create new-version
bectl activate -t new-version # only activates for one boot
shutdown -r now
# do update
bectl activate -t new-version
shutdown -r now
# check if it works, if ok, then
bectl activate new-version # activate permanently
You might want to look at TrueNAS CORE and the System > Boot section of the UI for inspiration 

[Greelan]

Hearing about boot environments is a revelation to me too! It’s prompted me to do some more reading, and it seems amazing. Another reason why I’m glad I recently did the conversion to ZFS for my OPNsense installation

[senser]

I wonder if the -r flag is needed to be able to go back to a previous system version.
eg bectl create -r 22.1

Anyone knows?

[tiermutter]

Patrick described the workfkow in this thread:
https://forum.opnsense.org/index.php?topic=25540.msg122750

[Patrick M. Hausen]

I wonder if the -r flag is needed to be able to go back to a previous system version.
eg bectl create -r 22.1

In general - no. The dataset layout of FreeBSD on ZFS is carefully tailored so that everything impacted by an update is in one single dataset - the boot environment.

Things like users' home directories and all of /var/* are excluded, because they might contain valuable continuously changing application data that you want to keep even in case you do a rollback of an update gone wrong.

In the OPNsense case there might occacionally be a problem with a particular service, when data structures in /var/* are changed by an update and the older version cannot cope with that. In that case you need to investigate individual services.

But nothing keeps you from doing an extra `zfs snapshot ...` for all the datasets you might want to roll back in addition to the boot environment.

HTH,
Patrick

[senser]

In general - no. The dataset layout of FreeBSD on ZFS is carefully tailored so that everything impacted by an update is in one single dataset - the boot environment.

Things like users' home directories and all of /var/* are excluded, because they might contain valuable continuously changing application data that you want to keep even in case you do a rollback of an update gone wrong.

Thats pretty cool! Thanks for the info!