Safely upgrade to 22.1 beta and possibly roll back - boot environments FTW!

[Patrick M. Hausen]

Also, I thought it worth mentioning that I found beadm more mature than bectl, but it's been a while since I checked out the latter.

beadm is the predecessor to bectl and deprecated. It's a shell script that uses zfs commands and started life as a clone of Solaris' command of the same name. bectl is implemented in C, uses libzfs, and is part of base since at least FreeBSD 12.

[benyamin]

beadm is the predecessor to bectl and deprecated. It's a shell script that uses zfs commands and started life as a clone of Solaris' command of the same name. bectl is implemented in C, uses libzfs, and is part of base since at least FreeBSD 12.

All the ZFS magic came from Solaris of course.  ;)

bectl still had problems early last year when I last used it. I know I had the issue with destroying BEs, which looking at now, seems to have been resolved 10 January 2020 in 12.1 r356593.

bectl is backwards-compatible with the beadm command set. IIRC, bectl does have some cool features that can be used to test new BEs, e.g. booting once into a new BE, including with new boot options, so you can test the BE before committing to it; if the boot fails (or otherwise restarts), it reverts to the BE marked for reboot.

Thanks for trying out BEs with OPNsense, Patrick. Pleasing to see Franco paying attention too. This has the potential to automate recovery on bad upgrades (and much more no doubt). Provided the logic around promotion and destruction is well managed it could be highly advantageous.

[Mks]

Dear all ZFS experts,

I want to follow up on this post, may you can clarify.

..... Both /usr and /var are not placed under the pool root, which IMHO, is not justified.

To fix that, issue something akin to the following:

Code Select Expand

zfs rename -u zroot/usr zroot/ROOT/default/usr
zfs rename -u zroot/var zroot/ROOT/default/var

Then zfs list should show you the magic.  ;)

So, is there anything to to improve to ensure the BE rollback is fully working and performs a "complete" rollback of the sense in case of an update issue?

e.g my setup looks like this:

Code Select Expand


zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
zroot                4.16G  45.2G    88K  /zroot
zroot/ROOT           3.24G  45.2G    88K  none
zroot/ROOT/21.7.3_3     8K  45.2G  1.79G  /
zroot/ROOT/21.7.5    3.24G  45.2G  1.48G  /
zroot/tmp             152K  45.2G   152K  /tmp
zroot/usr             352K  45.2G    88K  /usr
zroot/usr/home         88K  45.2G    88K  /usr/home
zroot/usr/ports        88K  45.2G    88K  /usr/ports
zroot/usr/src          88K  45.2G    88K  /usr/src
zroot/var             932M  45.2G    88K  /var
zroot/var/audit        88K  45.2G    88K  /var/audit
zroot/var/crash        88K  45.2G    88K  /var/crash
zroot/var/log         931M  45.2G   931M  /var/log
zroot/var/mail        112K  45.2G   112K  /var/mail
zroot/var/tmp          96K  45.2G    96K  /var/tmp


Thanks for clarification?

br

[Patrick M. Hausen]

How did you install to end up with zroot/usr mounted? That should not be the case and will prevent a successful rollback. Neither the yfreeBSD nor the OPNsense installer will create a system like this.

[Mks]

How did you install to end up with zroot/usr mounted? That should not be the case and will prevent a successful rollback. Neither the yfreeBSD nor the OPNsense installer will create a system like this.

Thanks for pointing that out, but to be honest I followed the default installer. (really ;) ), or let's phrase it like this, I'm not aware that I made material changes to the default installer setup especially when it comes to ZFS setup.

So any chance to solve this?

Br

[Patrick M. Hausen]

Sorry - my bad. Of course "zfs list" will list the dataset and the mountpoint. You can check with "df" if it is really mounted. If not, all is well. Sorry for the confusion.

A BE rollback will restore everything that is on the "/" filesystem. So again refer to the output of "df". Everything that is in its own dataset will not be rolled back.

[Mks]

Sorry - my bad. Of course "zfs list" will list the dataset and the mountpoint. You can check with "df" if it is really mounted. If not, all is well. Sorry for the confusion.
....

;D Men, I've already installed a VM to doublecheck  ;).

Default install, should be fine:

Code Select Expand


df -ht zfs
Filesystem           Size    Used   Avail Capacity  Mounted on
zroot/ROOT/21.7.5     47G    1.5G     45G     3%    /
tmpfs                 18G    4.7M     18G     0%    /tmp
zroot                 45G     88K     45G     0%    /zroot
zroot/var/crash       45G     88K     45G     0%    /var/crash
zroot/usr/ports       45G     88K     45G     0%    /usr/ports
zroot/usr/src         45G     88K     45G     0%    /usr/src
zroot/var/log         46G    934M     45G     2%    /var/log
zroot/usr/home        45G     88K     45G     0%    /usr/home
zroot/var/audit       45G     88K     45G     0%    /var/audit
zroot/var/tmp         45G     96K     45G     0%    /var/tmp
zroot/var/mail        45G    112K     45G     0%    /var/mail


Thanks

[benyamin]

How did you install to end up with zroot/usr mounted? That should not be the case and will prevent a successful rollback. Neither the yfreeBSD nor the OPNsense installer will create a system like this.

I'm pretty sure the /usr mount is a result of canmount=off, with ZFS providing a "dummy" mount for the other /usr/X datasets.

[benyamin]

So, is there anything to to improve to ensure the BE rollback is fully working and performs a "complete" rollback of the sense in case of an update issue?

IMHO, the only way to ensure a complete rollback using BE is to make /usr and /var part of the root dataset. It's what I do, it works.

If OPNsense core, or any plugin(s), write outside the BE root dataset, BE may not successfully rollback. Also, as I previously mentioned, if /var/db/pkg or any other /var/db/* is outside the BE root dataset, then you should expect problems. The same goes for any HBSD-specific requirements that might exist around /usr/src and /usr/ports.

[Patrick M. Hausen]

@benyamin - /var/db/* is part of the BE without any manipulation. /var is created with canmount=off, too and the only datasets outside of the BE are:

Code Select Expand


zroot/usr/home    247183612      88 247183524     0%    /usr/home
zroot/tmp         247183788     264 247183524     0%    /tmp
zroot/var/audit   247183612      88 247183524     0%    /var/audit
zroot/var/mail    247183636     112 247183524     0%    /var/mail
zroot/var/crash   247183612      88 247183524     0%    /var/crash
zroot/var/log     247451252  267728 247183524     0%    /var/log
zroot/var/tmp     247183612      88 247183524     0%    /var/tmp


I already deleted /usr/src and /usr/ports because they are not used at all - the FreeBSD installer creates them for good measure.

HTH,
Patrick

[benyamin]

@benyamin - /var/db/* is part of the BE without any manipulation. /var is created with canmount=off, too..

On /var/db/*, I've definitely had breakage with /var/db/pkg/* in the past...

Looks like canmount=off was added to /var in September 2014, so probably made it into 10.1-RELEASE. The relevant bug report can be found here.

Crikey, I'm showing my age...! 8^d

Just to go over them once more for the sake of completeness:

Code Select Expand

zroot/tmp       /tmp       - volatile temp files
zroot/usr       /usr       - dummy mount
zroot/usr/home  /usr/home  - do OPNsense upgrades touch this?
zroot/usr/ports /usr/ports - in-place hbsd-update?
zroot/usr/src   /usr/src   - in-place hbsd-update?
zroot/var       /var       - dummy mount
zroot/var/audit /var/audit - auditd log files - in use?
zroot/var/crash /var/crash - separate partition?
zroot/var/log   /var/log   - circular logging changes? separate partition?
zroot/var/mail  /var/mail  - separate partition?
zroot/var/tmp   /var/tmp   - non-volatile temp files


On temp file directories: No benefit to /tmp being in BE, /var/tmp however, is persistent. Thus there is some potential for breakage.

Dummy mounts are as a result of canmount=off to provide a contiguous path for subsequent datasets.

Does /usr/home ever get touched? In BE /home is a symlink to /usr/home, yes? Any plugins ever touch it?

Does OPNsense make use of hbsd-update? hbsd-update makes use of BE already. Depending on the hardware and installed ports, I also update /usr/src and /usr/ports (via git) too. In some builds these are on separate partitions so as not to include in images, e.g. in NanoBSD images.

Does OPNsense make use of auditd? Will auditd log files always play well with previous versions of the daemon?

With /var/crash, /var/mail and /var/log, isn't the justification for different datasets here because these mounts would typically be on separate partitions or slices? Perhaps this is true for most of these mounts anyway.

Along similar lines, why separate them from the BE root pool anyway? I know there is a case for using defaults, but at the same time it is worth considering if the justification for those defaults is applicable in this use scenario.

EDIT: Corrected extremely misleading typo... ;)

[Patrick M. Hausen]

Does /usr/home ever get touched? In BE /home is a symlink to /usr/home, yes? Any plugins ever touch it?

That's where user accounts have their home directories. We are a team of four and of course we have individual accounts for UI and SSH. I want them persistent regardless of updates/rollbacks.

Does OPNsense make use of hbsd-update?

To my knowledge - no. But @franco can probably confirm or deny.

Depending on the hardware and installed ports, I also update /usr/src and /usr/ports (via git) too. In some builds these are on separate partitions so as not to include in images, e.g. in NanoBSD images.

You use src and ports on OPNsense firewalls? OK ...
What is your argument here? Do you think they should be part of the BE and potentially rolled back? I don't.

Does OPNsense make use of auditd? Will auditd log files always play well with previous versions of the daemon?

Again, not that I knew.

With /var/crash, /var/mail and /var/log, isn't the justification for different datasets here because these mounts would typically be on separate partitions or slices? Perhaps this is true for most of these mounts anyway.

There are no partitions and slices in ZFS. These are separate because you (in the regular case) don't want your logfiles and emails to vanish when you revert an update via BE - simple as that.

Along similar lines, why separate them from the BE root pool anyway? I know there is a case for using defaults, but at the same time it is worth considering if the justification for those defaults is applicable in this use scenario.

They are not separate from the root pool. There is only one pool named `zroot`. Please keep the terminology consistent for the sake of people not familiar with ZFS. They are not part of the `zroot/ROOT/<some-be>` dataset.

I would always argue to keep things exactly that way unless there is a compelling reason to change anything. Stick to upstream defaults wherever possible, so you don't need to maintain deliberate changes and you won't get surprised by upstream changes should they occur.

The FreeBSD setup is fine. I delete /usr/src and /usr/ports after installation - they are not used on OPNsense. For development I use a Vagrant project.

[benyamin]

Does /usr/home ever get touched? In BE /home is a symlink to /usr/home, yes? Any plugins ever touch it?

That's where user accounts have their home directories. We are a team of four and of course we have individual accounts for UI and SSH. I want them persistent regardless of updates/rollbacks.

You wouldn't be populating home directories on a production firewall, surely... Any nice utilities for black hat use? ;)

You use src and ports on OPNsense firewalls? OK ...
What is your argument here? Do you think they should be part of the BE and potentially rolled back? I don't.

No, I don't use them on production installs of OPNsense. I do on occasion in dev/test.

It was in the context of my comment: "If I'm developing an appliance..." This was in relation to building and updating appliances. In some cases, you need the src and ports trees (or part thereof) to have a successful automated upgrade, especially when using HBSD. As such, yes, I want them rolled back so any git pulls are reverted too. The alternative is to provide signed binaries, but then you need to build them and provide them don't you (rather than using the user's compute resource and the git CDN).

I was just providing an example of why it can be advantageous to have those trees in the BE.

With /var/crash, /var/mail and /var/log, isn't the justification for different datasets here because these mounts would typically be on separate partitions or slices? Perhaps this is true for most of these mounts anyway.

There are no partitions and slices in ZFS. These are separate because you (in the regular case) don't want your logfiles and emails to vanish when you revert an update via BE - simple as that.

My point here was that historically these mounts were separate partitions to avoid filling up the root filesystem. Sometimes for performance too.

For a workstation, I guess you might want the spooler to survive and logs might be of some limited value. I presume this is what you mean by "in the regular case", and in such scenarios it does make some sense to keep them independent of BE changes (to varying degrees).

However, for an appliance, or even a server:

• I wouldn't give two hoots about the spooler. I would happily presume it has forwarded any mail to a MTA elsewhere and is busily consuming 0B (as it should).
• As for logs, I would be tarring them up and sending them elsewhere for analysis via scp or sftp (using a catch-all script), before rebooting into the previous BE and returning the system to service.

tbh, in most cases I simply don't care if any of that is lost. I want the system back up and operating at the end of the change window.

Along similar lines, why separate them from the BE root pool anyway?

They are not separate from the root pool. There is only one pool named `zroot`. Please keep the terminology consistent for the sake of people not familiar with ZFS. They are not part of the `zroot/ROOT/<some-be>` dataset.

My bad. Just a typo brought forward from my previous point which I skipped over. I'll correct it so others aren't confused.

I know there is a case for using defaults, but at the same time it is worth considering if the justification for those defaults is applicable in this use scenario.

I would always argue to keep things exactly that way unless there is a compelling reason to change anything. Stick to upstream defaults wherever possible, so you don't need to maintain deliberate changes and you won't get surprised by upstream changes should they occur.

Generally speaking, I would agree to a certain degree.

However, it is noteworthy that in this case, the easiest way to ensure there are no surprises is to lump everything in the BE.

The FreeBSD setup is fine. I delete /usr/src and /usr/ports after installation - they are not used on OPNsense.

Unless you do need to build something on a dev/test install of OPNsense, in which case, there might be compelling reasons to include/exclude those trees in/from the BE depending on the use scenario.

[Patrick M. Hausen]

You wouldn't be populating home directories on a production firewall, surely... Any nice utilities for black hat use? ;)

How do you expect me to logon?

Code Select Expand

ry93@kagate1:~ % id
uid=2001(ry93) gid=0(wheel) groups=0(wheel),1999(admins)
ry93@kagate1:~ % ls -l .ssh/
total 5
-rw-r-----  1 ry93  nobody  836 Nov 12 18:14 authorized_keys
ry93@kagate1:~ %


Root login disabled, sudo enabled, individual account for each admin. Best practice as far as I know.

[benyamin]

Root login disabled, sudo enabled, individual account for each admin. Best practice as far as I know.

Yeah, just don't drop anything in your home directory. Keys are ok. Same for shell config, history (if policy permits it), etc. Anything else should be deleted on logout. It's often one of the first things checked during audits / pen testing.

It's shocking how many perimeter system home dirs have a file called system_passwords or a script with embedded scp/rsync/sftp/etc. creds. Even without creds, it can give an attacker insight into operations and help locate creds in flight in packet captures, etc. Be careful with cron for same reason...

In the pressure of troubleshooting an outage, sometimes info leaks in home dirs. It's just one of those things. It pays to cleanup on logout - whether manually or automated.

I often ask: Would you post the output of the following?

Code Select Expand

ls -ltrRha ~