Hi guys,
I've been having problems with Unbound crashing since I moved to 19.1, so I want to do a fresh install of 19.1.4. Problem is I usually install on ZFS & I'm not seeing a way to do so in 19.1.4.
Is this feature missing in 19.1.4, or am I just missing something obvious?
-Will
There are (as has been for a number of years) not enough available resources to implement ZFS installer yet.
Cheers,
Franco
RE unbound on 19.1, are you using LibreSSL?
Then this might help:
https://forum.opnsense.org/index.php?topic=11657.msg55526#msg55526
With regards to ZFS: I've just done 2 conversions from vanilla "FreeBSD on ZFS" to OPNSense (opnsense-bootstrap) and they both work as expected, ran the script, rebooted, was in webgui.
QuoteWith regards to ZFS: I've just done 2 conversions from vanilla "FreeBSD on ZFS" to OPNSense (opnsense-bootstrap) and they both work as expected, ran the script, rebooted, was in webgui.
Can you write short guide/tutorial how to use ZFS in OPNsense? Did you install FreeBSD first and then all this OPNsense stuff or did you have a script to convert OPNsense to ZFS?
ZFS should be installed on system that has ecc ram and 8GB min. Do you have such system ? If you don't have such system then you might have any stable OS filesystem or even leads to filesystem corruption.
QuoteZFS should be installed on system that has ecc ram and 8GB min. Do you have such system?
I have 512GB ram. Think this will not be an issue. I just wonder how an ZFS installation would work.
Wow 512GB RAM !!! ZFS really loves it . Is it a firewall do you use squid or some caching memeroy devices. We can actually contribute ZFS installer if there is a strong use case for it be be part of firewall.
Yes, we run squid and sensei. There are several 10GbE chelsio cards in this machine. Also has dual XEON. ;)
ZFS is designed such a way that it will actually take away all your free memeroy space. what ZFS features you will be using it ? How much is your system free memory at peak usage ?
Quote from: hbc on April 01, 2019, 09:26:35 AM
QuoteWith regards to ZFS: I've just done 2 conversions from vanilla "FreeBSD on ZFS" to OPNSense (opnsense-bootstrap) and they both work as expected, ran the script, rebooted, was in webgui.
Can you write short guide/tutorial how to use ZFS in OPNsense? Did you install FreeBSD first and then all this OPNsense stuff or did you have a script to convert OPNsense to ZFS?
FreeBSD was already installed on ZFS, then it was a simple copy/paste from https://github.com/opnsense/update#opnsense-bootstrap, so the quick guide is:
1) Get FreeBSD installed on ZFS
2) Run the script ;D
Has anyone taken a look at the way FreeNAS installs to ZFS?
There are a lot of approaches, some easier, some harder. The currently favoured approach is to switch our old "bsdinstaller" to the current FreeBSD "bsdinstall".
The issue is simply nobody offers the time to implement the ZFS installer as of yet.
Cheers,
Franco
Yes, I came here also to get opnsense installed on zfs.
My use case :
Opnsense on bare metal as internet router.
Since it's up all the time (super low power chip!), I want to run a few lightweight vm in bhyve (ie: database for sensor readings).
I guess I will take the "opnsense-bootstrap" road then. By the way, is it supported and feature identical ? Thanks
+1 to get it on the installer !
Or, you could just provide a usb installed image first. We could just make a small script to create zfs array, copy usb dataset to it and bootstrap it.
Thanks
Just curious, is the opnsense-bootstrap method still supported?
And I assume future updates don't care about whether the box was originally an opnsense install or was migrated from a stock FreeBSD install?
Yes and yes.
12.1 adoption is a bit stalled for HBSD-related reasons so installer work is also stalled. It was planned to bring ZFS install support to 12.1 (20.1) but at the moment time is running out on that goal.
Cheers,
Franco
Thanks - just got around to this today, was really easy:
- backup config
- grab another drive (I suspect with all the file corruption I've been seeing around my kernel panics, the drive has to be bad even if SMART isn't showing anything), stick it in any PC laying around
- Install FreeBSD 11.2
- "pkg add" the nss cert stuff and git
- run the bootstrap script
- upload old config
- pop drive into firewall
And with zfs, I am hoping that even if the box panics for any reason I've got a better chance of not having corrupted files in the base.
Now I'm going to fiddle with zfs a bit and do the "poor man's RAID" and set "copies=2" on a few of the filesystems...
So this is kind of crazy. Got a few panics, but no swap was enabled (forgot to change ada2 to ada0 in fstab).
'zpool scrub' shows me no errors, but the health check shows some bad checksums, and I found the dashboard was not loading because "interfaces.inc" just had some random garbage sprinkled in the file. And the dashboard choking on those errors happened while the system was up and running.
Health check has this, not sure if this is corruption or if these are leftovers from the initial FreeBSD install:
***GOT REQUEST TO AUDIT HEALTH***
>>> Check installed kernel version
Version 19.7.7 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 19.7.7 is correct.
>>> Check for missing or altered base files
Error 2 ocurred.
lib/libcrypto.so.8:
sha256digest (0xbfe218c482ce5e1bcce776e265ee46b61a2425c91523de5a6472e1d3a39775e9, 0x1a55439f97072bbc6de73d19bbaa6e71e289a91fe86d9971af0483b3795c20b4)
usr/lib/libarchive.so.6:
sha256digest (0x917956bf32b66a471e0b8d6d0a4ecd2c3deaafd037a6778fc4c55cf0590c334a, 0xb2767e3b22acc19e0f40a05e74a5892a86095e4bd59e156c01fb60b2d71675f0)
usr/bin/awk:
sha256digest (0xc08dd78ff24e3d9d8484f55aeedd4105c3de8cd65be29db6b02aef61363795a9, 0xe3d9ce8f7c16df56f399543462f8bc73facfd012ce6af77543f0088e416aa325)
usr/bin/openssl:
sha256digest (0x9ce124fcee5cff90a2869273e48e9910b24873d454eb2f8af4c12258254ed391, 0x81f4e33a0e22fa02a9d4ccf2308074c8d15d3b5c0f1a8926ccce7c5413086bf6)
usr/bin/nawk:
sha256digest (0xc08dd78ff24e3d9d8484f55aeedd4105c3de8cd65be29db6b02aef61363795a9, 0xe3d9ce8f7c16df56f399543462f8bc73facfd012ce6af77543f0088e416aa325)
>>> Check for and install missing package dependencies
/usr/lib/libarchive.so.6: Undefined symbol "@FBSD_1.0"
>>> Check for missing or altered package files
/usr/lib/libarchive.so.6: Undefined symbol "@FBSD_1.0"
***DONE***
Might have to spend a few months on that other firewall distribution (if they still let me run w/o AESNI, ha!), if for no other reason than to regain my sanity.
Hello, I also use here the bootstrap installer for OPNsense ZFS install and is working just fine. ;)
However looks like there's still many users requesting for a built-in ZFS installer, especially newcomers and/or users afraid of the command line, or simply an online installation is not possible.
Short story:
I was also thinking on doing something similar for NAS4Free/XigmaNAS few years ago, adding the bsdinstall, but since its main Embedded roots don't have much of acceptance, so I created a very simple ZFS installer single script solution(already added in base) to keep a small footprint and file count as minimum as possible, yet its process is based from latest bsdinstall with Boot Environments compliant which is a must.
Simple single script method:
Here is a small video (https://drive.google.com/file/d/1DBiK8KKeJDVyqYfYXJNZ0-_gSrH6wQ2s/view?usp=sharing) showing how it works in OPNsense with very small edit, the little script does offers to install in ZFS Stripe, Mirror and RAID10, as well as for Swap geli encryption option, currently supports MBR, GPT, UEFI and GPT+UEFI install options during its dialog driven installation.
I would be happy enough to contribute the ZFS installer to the OPNsense devs so they can update/modify/adapt as needed. :)
Datasets creation process by the installer(same as bsdinstall):
zfs create -o mountpoint=none "zroot/ROOT"
zfs create -o mountpoint=/ "zroot/ROOT/default"
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off "zroot/tmp"
zfs create -o mountpoint=/usr -o canmount=off "zroot/usr"
zfs create -o setuid=off "zroot/usr/ports"
zfs create -o mountpoint=/var -o canmount=off "zroot/var"
zfs create -o exec=off -o setuid=off "zroot/var/audit"
zfs create -o exec=off -o setuid=off "zroot/var/crash"
zfs create -o exec=off -o setuid=off "zroot/var/log"
zfs create -o atime=on "zroot/var/mail"
zfs create -o setuid=off "zroot/var/tmp"
Here is the Disk usage widget after new install boot:
(https://drive.google.com/uc?export=download&id=18t0tT7i61vk3zlJeCaiHfmE-lA9Y5VUW)
P.S. Please fast forward the video during Unbound DNS after reboot to see the zpool/disk gpart layouts.
The BSDINSTALL method:
Here is a small video (https://drive.google.com/file/d/1HVWIW0QehyDv8kr3NmRuz4v-pCSajyPy/view?usp=sharing) showing a very simple install progress using the "bsdinstall" approach, it just need for the distfiles to be placed in "/usr/freebsd-dist" with a sane MANIFEST generated file, then explicitly add the mandatory files under bsdinstall/auto to only show optionals in the dialog if any such debug/extras dist files, additionally place customs in the bsdinstall/config to properly generate/append to config files.
One drawback is that this method required for the OPNsense distfiles to be included in the distribution as expected, however a small edit in the bsdinstall can take the files already present in the ISO and copy them to the target media like in the previous single script method, though I still prefer the bsdinstall personally and follow standards whenever possible.
Here is the test examples for reference. (https://drive.google.com/drive/folders/186MzlY_MI5qB2Nn2gC6YgyYAlXhhjWNZ?usp=sharing)
Regards
Jose, that looks great. I agree with you that the BSD Install method looks best. It's more consistent for users (that's the method adopted by other projects based on FreeBSD such as FreeNAS and pfSense) and I expect it should be easier to maintain (as we can get changes from FreeBSD as they adjust their installer script).
Quote from: Jose on December 03, 2019, 06:55:27 PM
I would be happy enough to contribute the ZFS installer to the OPNsense devs so they can update/modify/adapt as needed. :)
There are currently not enough developers to implement all the desired features in OPNsense, but the source code is on github (https://github.com/opnsense) and all are welcome to contribute pull requests. It's spread across multiple repositories, but here's the repo for the installer (https://github.com/opnsense/bsdinstaller). Would you be willing to create a pull request for this?
The bsdinstall-based installer is here... https://github.com/opnsense/installer
Cheers,
Franco