So with ZFS file system and bectl we can easily create snapshots of our system and always go pack to the point of time when we took the snapshot. I am familiar with snapshots with BTRFS but on OPNsense I'm very confused about them.
Here's the thing. In my experience all changes after the latest snapshot are saved and the latest snapshot + changes is the current status after a reboot. When I create a new snapshot with bectl, the active snapshot stays the currently running snapshot - not the latest snapshot.
What happens if I take a snapshot, do some changes on the system, then activate the new snapshot and reboot? Are my changes gone because I bootet into the snapshot I made before rebooting the system? Or are the changes applied to the latest snapshot? Or are my changes maybe gone?
I just can't wrap my head around it. It gives me headache the more I think about it and try to understand. Could you please explain to me how this works? In case I ever have to revert back to an old snapshot I wanna make sure to make it right.
Quote from: Vexz on August 10, 2022, 08:26:38 AM
What happens if I take a snapshot, do some changes on the system, then activate the new snapshot and reboot? Are my changes gone because I bootet into the snapshot I made before rebooting the system?
Exactly. Although the "current snapshot" (if you want to phrase it that way) still exists and if you reboot a second time into that, your changes are back.
The boot environments are really snapshot backed r/w clones. The active one, too. So you can boot back and forth as you desire. Each BE keeps the changes you make while the BE is active.
HTH,
Patrick
Aah! I think I get it now. So I can view the snapshots as some kind of individual "sessions" that I can boot into and they always keep all their changes.
Thank you very much for the clarification. :)
Hi, some time ago I've created a TUI wrapper around beadm/bectl to ease the Boot Environments backups(exports) to either local or remote servers through SSH.
Each time I upgrade either FreeBSD and OPNsense to a major version I backup the current BE to my server and and after backup completes I create a new BE and reboot into it to perform the upgrade.
Here is the little utility: Bemanager (https://forum.opnsense.org/index.php?topic=4255.msg40745#msg40745)
Here is the source: Bemanager at GitHub (https://github.com/JRGTH/bemanager)
There is no manual on how to configure it but the sample config have some hints.
Notes:
Be aware that after upgrading to a major version, if the user upgrades the bootcode GPT/BIOS/EFI, in order to rollback to a previous BE, the user must mount the previous BE containing the older bootcode and perform a downgrade from its files under "/boot", however if the user upgrades the ZFS pool feature flags, this may lock-down the ability to easily rollback to a previous BE*.
*The solution is to backup the wanted previous BE, then reinstall a FreeBSD/OPNsense version matching the previous version then import back the Boot Environment regardless of the disk layout, e.g. I've exported a BE from a single disk to later import that BE to a mirrored setup with good results.
Unfortunately haven't enough time to read the plugin development manual to make an Boot Environments manager plugin for the OPNsense GUI to perform basic tasks from.
ZFS Boot Environments Reloaded (https://vermaden.files.wordpress.com/2018/11/nluug-zfs-boot-environments-reloaded-2018-11-15.pdf) by Sławomir Wojciech Wojtczak (vermaden)
Regards
Boot loaders are at least supposed to always be backwards compatible. If you ever upgrade your zpool, you need to upgrade your boot loader, too. But a downgrade should never be necessary.
Quote from: pmhausen on August 10, 2022, 01:04:20 PM
Boot loaders are at least supposed to always be backwards compatible. If you ever upgrade your zpool, you need to upgrade your boot loader, too. But a downgrade should never be necessary.
Yeah it just happens that I've ran into boot issues/broken EFI console in the past and and I'm still skeptical about bootcode upgrade, though I think I should make some testing on VM's in this regards.
Also agreed that a downgrade should never happens, but sometimes jumping into bleeding edge makes you wanting to rollback when you later discover that something can't be easily fixable on time.
Regards
I figured I'd use my old topic (this one right here) to ask another question about boot environments.
So yesterday I deleted an old snapshot with the -o parameter to delete the origin.
Used command: bectl destroy -o <snapshot>
Actually I couldn't find out exactly what it means to "delete the origin" but everyone says you should always do it because there's no reason why you wouldn't want that.
Now after I deleted the old snapshot and rebooted my OPNsense many configurations were gone and I was on an old configuration state.
Do you think that happened because of the deleted origin? Will it keep my latest configurations if I don't delete the origin of snapshots?
Quote from: Jose on August 10, 2022, 12:53:53 PM
Hi, some time ago I've created a TUI wrapper around beadm/bectl to ease the Boot Environments backups(exports) to either local or remote servers through SSH.
Each time I upgrade either FreeBSD and OPNsense to a major version I backup the current BE to my server and and after backup completes I create a new BE and reboot into it to perform the upgrade.
This should be integrated in the OPNsense update process. But a plugin would suffice too...maybe @mimugmail could help. :)
Questions:
1. If I understood correctly the BEs will also appear in the FreeBSD loader right? This is fantastic.
2. I'm reading the sample config file, but some options are not really clear, ins there any additional doc I can read about those with some examples for their usage?
3. Is there a substantial difference between bectl vs beadm?
Thanks a lot for your work Jose, hope this can become an integrated plugin one day.
I explained the whole mechanism here:
https://forum.opnsense.org/index.php?topic=25540.msg122731#msg122731
beadm is the old deprecated tool that was created as a copy of Solaris beadm and available as a port/package. bectl is the new official one in the FreeBSD base system.
Thank you so much for that post.
I would still like to test Jose's script/menu so will also wait for his reply, but it's nice to know how it works, thanks again.
Would a test inside a VM not work for you?
I for one fail to see what would be the benefit of creating a plugin for it. It is a single file download and the single-line to get it installed is already provided.
I didn't think of a plugin for the installation, but for the management part. The menu, etc.
I think that if OPNsense installation is made with ZFS, this kind of functionality should be part of the main functionalities, or at least as an official plugin. But it's just my personal opinion. :)
Quote from: pmhausen on January 22, 2023, 04:17:18 PM
I explained the whole mechanism here:
https://forum.opnsense.org/index.php?topic=25540.msg122731#msg122731
I finally had the chance to test the process you described with 23.1 final. I didn't dare to try rebooting to 23.1r2 though. :)
I think it's much easier from shell vs bemanager, the only thing I'd like to do is backing up these snaps to an external NFS share, and what is the process if I wanted to restore it from there, in case the hw fails completely. After all these snaps live on the local ssd storage.
Thank you again for explaining the process, it's an invaluable functionality, IMHO.
is this somewhere in menu, with snap actions defined after ①update, ②boot, ③package install? Or at least an action.d item? It only makes sense if this is defined somewhere I guess.
It's all command line tools, most importantly bectl.
It looks like the system doesn't use much of ZFS, it only formats the drive and that's it? It doesn't even set trimming:o
Is there any 'solution' to boot environments published?
Running CLI commands isn't useful, you will not have access to CLI once you need the snapshot, obviously. Same with the snapshot creation, you will not have it, because it's not hooked up to pkg updates or you simply forget.
So I think about install hook + boot menu restore as a very first step.
① Install hook can be anything slapped into rc.syshook.d/upgrade folder I guess. Easy but also needs maintenance and bectl doesn't offer any rules like number of snapshot clones (something like zfs set snapshot_count=5).
② I didn't notice this in the boot menu:
I create a BE before every major update. I have never had an update lock me out of SSH, so I could always roll back if anything wasn't working.
Quote from: Patrick M. Hausen on December 16, 2023, 05:36:09 PM
I create a BE before every major update. I have never had an update lock me out of SSH, so I could always roll back if anything wasn't working.
I could not, but good for you. Boot selection would be the perfect solution if not the whole point of BE. If it works, it makes sense to scripts some hooks👍The question remains, can it be turned on? I've searched for another hour without success.
Quote from: 36thchamber on December 16, 2023, 09:59:57 PM
I could not, but good for you.
What's keeping you from making
bectl create before-upgrade-20231216 a regular part of your update routine? That's all it takes. You can always boot into that snapshot via the console menu afterwards. What you found in the boot menu does work. You need physical or at least remote serial or remote KVM access to use that, of course.
There's one active and by default also active at the next reboot BE at all times. The OPNsense system does not really care about that - I get your point. But you can really just
bectl create ... before every update and then the state before the update will be available via the FreeBSD boot menu. It cannot get much simpler apart from the fact that you need to use the CLI.
HTH,
Patrick
Maybe I don't understand the problem, but I am doing the same as Patrick before every update and I am able to go back all the times using CLI.
However, local snapshots/BE can never be used when system is no longer reachable by any ways, for sure.
For such cases I export those BE using bemanager and I am able to restore within a few minutes. Did this a few weeks ago when I changed hardware... I simply had to install fresh sense, activate SSH, install bemanager and import BE. As I remember this all took just a little longer than a cigarette..
Quote from: Patrick M. Hausen on December 16, 2023, 10:22:28 PM
What's keeping you from making bectl create before-upgrade-20231216 a regular part of your update routine? That's all it takes.
so the answer to that question is yes, that's what I wanted to double check. It's only visible when multiple snapshots exist as i couldn't see it. Now it's time to create a hook. I would not run it manually because that's should be hooked to installation as on every other OS. Meanwhile, i created automatic ZFS handling. ZFS was just balooning in size, without snapshots, which is a waste of space. Now it does snapshotting daily/hourly, cleanup and emergency cleanup. That's because ZFS soon took 100% of the harddrive despite zfs snapshout_limit & zpool feature@filesystem_limits=active, rendering OS frozen very quick in the process. Luckily it's without its famous bugs like 'pool can't operate when full', and it didn't kernel panic yet. It's just producing some silent errors which it can't autoheal like it never could. Let's see how long it will last. Defly one of my least favorite fs, but better than UFS with PHP:)