OPNsense Forum
Archive => 22.1 Legacy Series => Topic started by: Patrick M. Hausen on November 11, 2021, 07:34:10 pm
-
Prerequisites
- Installation with ZFS
- Running 21.7.4
Show current boot environments - this is the state when you never have used boot environments before
root@opnsense:~ # bectl list
BE Active Mountpoint Space Created
default NR / 775M 2021-07-31 21:46
N means "active now"
R means "active on reboot"
Create a backup BE for 21.7.4 and rename the active one to 21.7.5
root@opnsense:~ # bectl create 21.7.4
root@opnsense:~ # bectl rename default 21.7.5
root@opnsense:~ # bectl list
BE Active Mountpoint Space Created
21.7.4 - - 71.3M 2021-11-11 18:24
21.7.5 NR / 775M 2021-07-31 21:46
Now do your update to 21.7.5 in the UI as always. All changes will end up in the BE 21.7.5. If there is a problem with that and SSH is still working you can always do:
root@opnsense:~ # bectl activate 21.7.4
root@opnsense:~ # bectl list
BE Active Mountpoint Space Created
21.7.4 R - 71.3M 2021-11-11 18:24
21.7.5 N / 775M 2021-07-31 21:46
root@opnsense:~ # shutdown -r now
and boot back into 21.7.4 as if nothing ever happenend.
With the update to 21.7.5 as a warm up exercise let's try the beta ...
Rename the active BE for the new version and create a backup BE for the current one
root@opnsense:~ # bectl rename 21.7.5 22.1.b1
root@opnsense:~ # bectl create 21.7.5
root@opnsense:~ # bectl list
BE Active Mountpoint Space Created
21.7.4 - - 775M 2021-11-11 18:24
21.7.5 - - 71.3M 2021-11-11 19:07
22.1.b1 NR / 2.15G 2021-07-31 21:46
Now upgrade to the beta knowing that you can always rollback to 21.7.5.
If SSH does not work for some reason, you can pick boot environments at the loader prompt when you connect to the console (serial/IPMI/VGA/...)
Happy beta testing!
Patrick
-
Super, thanks for this! :)
-
As if I had had a foreboding. 22.1.b1 breaks my PPPoE uplink. Although this is a private system, it's still "sort of" production - at least for me and all people in the household. So no time to debug this on this particular installation.
I rolled back to 21.7.5. But then half of my services would not start at boot. AdGuard, BIND, WireGuard ... all red in the dashboard. Although I could start all of them manually.
Does the update to 22.1 change major things in /var that might not be part of the boot environment?
The BE contains (and rolls back/forward when switching) all of / and the directories below except:
zroot/tmp 247375828 240 247375588 0% /tmp
zroot/var/log 247657724 282136 247375588 0% /var/log
zroot/var/crash 247375676 88 247375588 0% /var/crash
zroot/var/audit 247375676 88 247375588 0% /var/audit
zroot/usr/home 247375676 88 247375588 0% /usr/home
zroot/var/mail 247375700 112 247375588 0% /var/mail
zroot/usr/ports 247375676 88 247375588 0% /usr/ports
zroot/var/tmp 247375676 88 247375588 0% /var/tmp
zroot/usr/src 247375676 88 247375588 0% /usr/src
Anyway, rolling back to 21.7.4, re-applying the update to 21.7.5. All up and well now. I'll check out the beta in my Vagrant environment first, I guess.
-
22.1 discontinues clog-style logging under /var/log. That might be it.
Cheers,
Franco
-
I disabled that right after installation of 21.7 ... but possibly.
-
Nevertheless I made a note to look at this more. I'm all here for beta reports. :)
Cheers,
Franco
-
I didn't use BE but update went fine and I updated the community repo for FreeBSD13 so your AdGuard Home should upgrade and run too :)
Finally also Kibana and Elastic compile again.
-
Is there a downloadable ISO for 22.1 b1?
-
I didn't use BE but update went fine and I updated the community repo for FreeBSD13 so your AdGuard Home should upgrade and run too :)
Finally also Kibana and Elastic compile again.
Upgraded to the beta with your repo configured ,but getting this :
Updating mimugmail repository catalogue...
pkg: https://opn-repo.routerperformance.net/repo/FreeBSD:13:amd64/meta.txz: Not Found
repository mimugmail has no meta file, using default settings
pkg: https://opn-repo.routerperformance.net/repo/FreeBSD:13:amd64/packagesite.txz: Not Found
Unable to update repository mimugmail
Error updating repositories!
What would be the best way to fix it ?
-
Is there a downloadable ISO for 22.1 b1?
Images (as well as stable packages) will be available with 22.1 release candidates.
Cheers,
Franco
-
Thanks for showing how easy it could be to use Boot Environments.
Did you mean
root@opnsense:~ # bectl rename 21.7.5 22.1.b1by any chance? I think so as it is what it shows when listing all BEs afterwards.
root@opnsense:~ # bectl rename 21.7.5 21.1.b1
root@opnsense:~ # bectl create 21.7.5
root@opnsense:~ # bectl list
BE Active Mountpoint Space Created
21.7.4 - - 775M 2021-11-11 18:24
21.7.5 - - 71.3M 2021-11-11 19:07
22.1.b1 NR / 2.15G 2021-07-31 21:46
-
Yes, of course. I'll fix the original article, thanks.
I had already done all of this, then came up with the idea to write a forum post about it. ;)
-
I've been using ZFS BE in my HBSD builds for a while now.
I don't think much of the default ZFS install on FreeBSD or HBSD. Both /usr and /var are not placed under the pool root, which IMHO, is not justified.
To fix that, issue something akin to the following:
zfs rename -u zroot/usr zroot/ROOT/default/usr
zfs rename -u zroot/var zroot/ROOT/default/var
Then zfs list should show you the magic. ;)
-
I've been using ZFS BE in my HBSD builds for a while now.
I don't think much of the default ZFS install on FreeBSD or HBSD. Both /usr and /var are not placed under the pool root, which IMHO, is not justified.
This is intentional for /var - you are free to disagree with that decision. I don't.
But /usr is placed under the BE root. The /usr dataset is created with canmount=off, which means all files placed under /usr/bin|lib|... are in the BE. This configuration is necessary to facilitate keeping /usr/local, /usr/src, ... separate from the BE.
Kind regards,
Patrick
-
But /usr is placed under the BE root. The /usr dataset is created with canmount=off, which means all files placed under /usr/bin|lib|... are in the BE. This configuration is necessary to facilitate keeping /usr/local, /usr/src, ... separate from the BE.
That is true and worth pointing out.
I should have said "in the context of an appliance"... 8^d
In that context, I don't think there is any advantage to keeping /var and /usr/X datasets separate from the BE. In my experience, I prefer /usr/local, /var/db/pkg, etc. to be protected under the BE. The same for /usr/src and /usr/ports if running HBSD debug kernels with INVARIANTS.
If I'm developing an appliance, and the need arises for a persistent mount outside the BE, I use a separate partition for that purpose, e.g. a src repo that survives snapshot reversions.
The rename commands I posted are just the quickest means to an end. You could certainly be more creative.
If it was a workstation / desktop install, maybe a server (depending on purpose), I would feel differently about it.
Also, I thought it worth mentioning that I found beadm more mature than bectl, but it's been a while since I checked out the latter.
-
Also, I thought it worth mentioning that I found beadm more mature than bectl, but it's been a while since I checked out the latter.
beadm is the predecessor to bectl and deprecated. It's a shell script that uses zfs commands and started life as a clone of Solaris' command of the same name. bectl is implemented in C, uses libzfs, and is part of base since at least FreeBSD 12.
-
beadm is the predecessor to bectl and deprecated. It's a shell script that uses zfs commands and started life as a clone of Solaris' command of the same name. bectl is implemented in C, uses libzfs, and is part of base since at least FreeBSD 12.
All the ZFS magic came from Solaris of course. ;)
bectl still had problems early last year when I last used it. I know I had the issue with destroying BEs, which looking at now, seems to have been resolved 10 January 2020 in 12.1 r356593.
bectl is backwards-compatible with the beadm command set. IIRC, bectl does have some cool features that can be used to test new BEs, e.g. booting once into a new BE, including with new boot options, so you can test the BE before committing to it; if the boot fails (or otherwise restarts), it reverts to the BE marked for reboot.
Thanks for trying out BEs with OPNsense, Patrick. Pleasing to see Franco paying attention too. This has the potential to automate recovery on bad upgrades (and much more no doubt). Provided the logic around promotion and destruction is well managed it could be highly advantageous.
-
Dear all ZFS experts,
I want to follow up on this post, may you can clarify.
..... Both /usr and /var are not placed under the pool root, which IMHO, is not justified.
To fix that, issue something akin to the following:
zfs rename -u zroot/usr zroot/ROOT/default/usr
zfs rename -u zroot/var zroot/ROOT/default/var
Then zfs list should show you the magic. ;)
So, is there anything to to improve to ensure the BE rollback is fully working and performs a "complete" rollback of the sense in case of an update issue?
e.g my setup looks like this:
zfs list
NAME USED AVAIL REFER MOUNTPOINT
zroot 4.16G 45.2G 88K /zroot
zroot/ROOT 3.24G 45.2G 88K none
zroot/ROOT/21.7.3_3 8K 45.2G 1.79G /
zroot/ROOT/21.7.5 3.24G 45.2G 1.48G /
zroot/tmp 152K 45.2G 152K /tmp
zroot/usr 352K 45.2G 88K /usr
zroot/usr/home 88K 45.2G 88K /usr/home
zroot/usr/ports 88K 45.2G 88K /usr/ports
zroot/usr/src 88K 45.2G 88K /usr/src
zroot/var 932M 45.2G 88K /var
zroot/var/audit 88K 45.2G 88K /var/audit
zroot/var/crash 88K 45.2G 88K /var/crash
zroot/var/log 931M 45.2G 931M /var/log
zroot/var/mail 112K 45.2G 112K /var/mail
zroot/var/tmp 96K 45.2G 96K /var/tmp
Thanks for clarification?
br
-
How did you install to end up with zroot/usr mounted? That should not be the case and will prevent a successful rollback. Neither the yfreeBSD nor the OPNsense installer will create a system like this.
-
How did you install to end up with zroot/usr mounted? That should not be the case and will prevent a successful rollback. Neither the yfreeBSD nor the OPNsense installer will create a system like this.
Thanks for pointing that out, but to be honest I followed the default installer. (really ;) ), or let's phrase it like this, I'm not aware that I made material changes to the default installer setup especially when it comes to ZFS setup.
So any chance to solve this?
Br
-
Sorry - my bad. Of course "zfs list" will list the dataset and the mountpoint. You can check with "df" if it is really mounted. If not, all is well. Sorry for the confusion.
A BE rollback will restore everything that is on the "/" filesystem. So again refer to the output of "df". Everything that is in its own dataset will not be rolled back.
-
Sorry - my bad. Of course "zfs list" will list the dataset and the mountpoint. You can check with "df" if it is really mounted. If not, all is well. Sorry for the confusion.
....
;D Men, I've already installed a VM to doublecheck ;).
Default install, should be fine:
df -ht zfs
Filesystem Size Used Avail Capacity Mounted on
zroot/ROOT/21.7.5 47G 1.5G 45G 3% /
tmpfs 18G 4.7M 18G 0% /tmp
zroot 45G 88K 45G 0% /zroot
zroot/var/crash 45G 88K 45G 0% /var/crash
zroot/usr/ports 45G 88K 45G 0% /usr/ports
zroot/usr/src 45G 88K 45G 0% /usr/src
zroot/var/log 46G 934M 45G 2% /var/log
zroot/usr/home 45G 88K 45G 0% /usr/home
zroot/var/audit 45G 88K 45G 0% /var/audit
zroot/var/tmp 45G 96K 45G 0% /var/tmp
zroot/var/mail 45G 112K 45G 0% /var/mail
Thanks
-
How did you install to end up with zroot/usr mounted? That should not be the case and will prevent a successful rollback. Neither the yfreeBSD nor the OPNsense installer will create a system like this.
I'm pretty sure the /usr mount is a result of canmount=off, with ZFS providing a "dummy" mount for the other /usr/X datasets.
-
So, is there anything to to improve to ensure the BE rollback is fully working and performs a "complete" rollback of the sense in case of an update issue?
IMHO, the only way to ensure a complete rollback using BE is to make /usr and /var part of the root dataset. It's what I do, it works.
If OPNsense core, or any plugin(s), write outside the BE root dataset, BE may not successfully rollback. Also, as I previously mentioned, if /var/db/pkg or any other /var/db/* is outside the BE root dataset, then you should expect problems. The same goes for any HBSD-specific requirements that might exist around /usr/src and /usr/ports.
-
@benyamin - /var/db/* is part of the BE without any manipulation. /var is created with canmount=off, too and the only datasets outside of the BE are:
zroot/usr/home 247183612 88 247183524 0% /usr/home
zroot/tmp 247183788 264 247183524 0% /tmp
zroot/var/audit 247183612 88 247183524 0% /var/audit
zroot/var/mail 247183636 112 247183524 0% /var/mail
zroot/var/crash 247183612 88 247183524 0% /var/crash
zroot/var/log 247451252 267728 247183524 0% /var/log
zroot/var/tmp 247183612 88 247183524 0% /var/tmp
I already deleted /usr/src and /usr/ports because they are not used at all - the FreeBSD installer creates them for good measure.
HTH,
Patrick
-
@benyamin - /var/db/* is part of the BE without any manipulation. /var is created with canmount=off, too..
On /var/db/*, I've definitely had breakage with /var/db/pkg/* in the past...
Looks like canmount=off was added to /var in September 2014, so probably made it into 10.1-RELEASE. The relevant bug report can be found here (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193971).
Crikey, I'm showing my age...! 8^d
Just to go over them once more for the sake of completeness:
zroot/tmp /tmp - volatile temp files
zroot/usr /usr - dummy mount
zroot/usr/home /usr/home - do OPNsense upgrades touch this?
zroot/usr/ports /usr/ports - in-place hbsd-update?
zroot/usr/src /usr/src - in-place hbsd-update?
zroot/var /var - dummy mount
zroot/var/audit /var/audit - auditd log files - in use?
zroot/var/crash /var/crash - separate partition?
zroot/var/log /var/log - circular logging changes? separate partition?
zroot/var/mail /var/mail - separate partition?
zroot/var/tmp /var/tmp - non-volatile temp files
On temp file directories: No benefit to /tmp being in BE, /var/tmp however, is persistent. Thus there is some potential for breakage.
Dummy mounts are as a result of canmount=off to provide a contiguous path for subsequent datasets.
Does /usr/home ever get touched? In BE /home is a symlink to /usr/home, yes? Any plugins ever touch it?
Does OPNsense make use of hbsd-update? hbsd-update makes use of BE already. Depending on the hardware and installed ports, I also update /usr/src and /usr/ports (via git) too. In some builds these are on separate partitions so as not to include in images, e.g. in NanoBSD images.
Does OPNsense make use of auditd? Will auditd log files always play well with previous versions of the daemon?
With /var/crash, /var/mail and /var/log, isn't the justification for different datasets here because these mounts would typically be on separate partitions or slices? Perhaps this is true for most of these mounts anyway.
Along similar lines, why separate them from the BE root pool anyway? I know there is a case for using defaults, but at the same time it is worth considering if the justification for those defaults is applicable in this use scenario.
EDIT: Corrected extremely misleading typo... ;)
-
Does /usr/home ever get touched? In BE /home is a symlink to /usr/home, yes? Any plugins ever touch it?
That's where user accounts have their home directories. We are a team of four and of course we have individual accounts for UI and SSH. I want them persistent regardless of updates/rollbacks.
Does OPNsense make use of hbsd-update?
To my knowledge - no. But @franco can probably confirm or deny.
Depending on the hardware and installed ports, I also update /usr/src and /usr/ports (via git) too. In some builds these are on separate partitions so as not to include in images, e.g. in NanoBSD images.
You use src and ports on OPNsense firewalls? OK ...
What is your argument here? Do you think they should be part of the BE and potentially rolled back? I don't.
Does OPNsense make use of auditd? Will auditd log files always play well with previous versions of the daemon?
Again, not that I knew.
With /var/crash, /var/mail and /var/log, isn't the justification for different datasets here because these mounts would typically be on separate partitions or slices? Perhaps this is true for most of these mounts anyway.
There are no partitions and slices in ZFS. These are separate because you (in the regular case) don't want your logfiles and emails to vanish when you revert an update via BE - simple as that.
Along similar lines, why separate them from the BE root pool anyway? I know there is a case for using defaults, but at the same time it is worth considering if the justification for those defaults is applicable in this use scenario.
They are not separate from the root pool. There is only one pool named `zroot`. Please keep the terminology consistent for the sake of people not familiar with ZFS. They are not part of the `zroot/ROOT/<some-be>` dataset.
I would always argue to keep things exactly that way unless there is a compelling reason to change anything. Stick to upstream defaults wherever possible, so you don't need to maintain deliberate changes and you won't get surprised by upstream changes should they occur.
The FreeBSD setup is fine. I delete /usr/src and /usr/ports after installation - they are not used on OPNsense. For development I use a Vagrant project.
-
Does /usr/home ever get touched? In BE /home is a symlink to /usr/home, yes? Any plugins ever touch it?
That's where user accounts have their home directories. We are a team of four and of course we have individual accounts for UI and SSH. I want them persistent regardless of updates/rollbacks.
You wouldn't be populating home directories on a production firewall, surely... Any nice utilities for black hat use? ;)
You use src and ports on OPNsense firewalls? OK ...
What is your argument here? Do you think they should be part of the BE and potentially rolled back? I don't.
No, I don't use them on production installs of OPNsense. I do on occasion in dev/test.
It was in the context of my comment: "If I'm developing an appliance..." This was in relation to building and updating appliances. In some cases, you need the src and ports trees (or part thereof) to have a successful automated upgrade, especially when using HBSD. As such, yes, I want them rolled back so any git pulls are reverted too. The alternative is to provide signed binaries, but then you need to build them and provide them don't you (rather than using the user's compute resource and the git CDN).
I was just providing an example of why it can be advantageous to have those trees in the BE.
With /var/crash, /var/mail and /var/log, isn't the justification for different datasets here because these mounts would typically be on separate partitions or slices? Perhaps this is true for most of these mounts anyway.
There are no partitions and slices in ZFS. These are separate because you (in the regular case) don't want your logfiles and emails to vanish when you revert an update via BE - simple as that.
My point here was that historically these mounts were separate partitions to avoid filling up the root filesystem. Sometimes for performance too.
For a workstation, I guess you might want the spooler to survive and logs might be of some limited value. I presume this is what you mean by "in the regular case", and in such scenarios it does make some sense to keep them independent of BE changes (to varying degrees).
However, for an appliance, or even a server:
- I wouldn't give two hoots about the spooler. I would happily presume it has forwarded any mail to a MTA elsewhere and is busily consuming 0B (as it should).
- As for logs, I would be tarring them up and sending them elsewhere for analysis via scp or sftp (using a catch-all script), before rebooting into the previous BE and returning the system to service.
tbh, in most cases I simply don't care if any of that is lost. I want the system back up and operating at the end of the change window.
Along similar lines, why separate them from the BE root pool anyway?
They are not separate from the root pool. There is only one pool named `zroot`. Please keep the terminology consistent for the sake of people not familiar with ZFS. They are not part of the `zroot/ROOT/<some-be>` dataset.
My bad. Just a typo brought forward from my previous point which I skipped over. I'll correct it so others aren't confused.
I know there is a case for using defaults, but at the same time it is worth considering if the justification for those defaults is applicable in this use scenario.
I would always argue to keep things exactly that way unless there is a compelling reason to change anything. Stick to upstream defaults wherever possible, so you don't need to maintain deliberate changes and you won't get surprised by upstream changes should they occur.
Generally speaking, I would agree to a certain degree.
However, it is noteworthy that in this case, the easiest way to ensure there are no surprises is to lump everything in the BE.
The FreeBSD setup is fine. I delete /usr/src and /usr/ports after installation - they are not used on OPNsense.
Unless you do need to build something on a dev/test install of OPNsense, in which case, there might be compelling reasons to include/exclude those trees in/from the BE depending on the use scenario.
-
You wouldn't be populating home directories on a production firewall, surely... Any nice utilities for black hat use? ;)
How do you expect me to logon?
ry93@kagate1:~ % id
uid=2001(ry93) gid=0(wheel) groups=0(wheel),1999(admins)
ry93@kagate1:~ % ls -l .ssh/
total 5
-rw-r----- 1 ry93 nobody 836 Nov 12 18:14 authorized_keys
ry93@kagate1:~ %
Root login disabled, sudo enabled, individual account for each admin. Best practice as far as I know.
-
Root login disabled, sudo enabled, individual account for each admin. Best practice as far as I know.
Yeah, just don't drop anything in your home directory. Keys are ok. Same for shell config, history (if policy permits it), etc. Anything else should be deleted on logout. It's often one of the first things checked during audits / pen testing.
It's shocking how many perimeter system home dirs have a file called system_passwords or a script with embedded scp/rsync/sftp/etc. creds. Even without creds, it can give an attacker insight into operations and help locate creds in flight in packet captures, etc. Be careful with cron for same reason...
In the pressure of troubleshooting an outage, sometimes info leaks in home dirs. It's just one of those things. It pays to cleanup on logout - whether manually or automated.
I often ask: Would you post the output of the following?
ls -ltrRha ~