OPNsense Forum

Archive => 19.1 Legacy Series => Topic started by: survive on March 31, 2019, 05:00:34 am

Title: 19.1.4 ISO - ZFS install option missing?
Post by: survive on March 31, 2019, 05:00:34 am
Hi guys,

I've been having problems with Unbound crashing since I moved to 19.1, so I want to do a fresh install of 19.1.4. Problem is I usually install on ZFS & I'm not seeing a way to do so in 19.1.4.

Is this feature missing in 19.1.4, or am I just missing something obvious?

-Will
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: franco on March 31, 2019, 12:35:43 pm
There are (as has been for a number of years) not enough available resources to implement ZFS installer yet.


Cheers,
Franco
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: chemlud on March 31, 2019, 05:18:34 pm
RE unbound on 19.1, are you using LibreSSL?

Then this might help:

https://forum.opnsense.org/index.php?topic=11657.msg55526#msg55526
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: mitsos on March 31, 2019, 11:17:58 pm
With regards to ZFS: I've just done 2 conversions from vanilla "FreeBSD on ZFS" to OPNSense (opnsense-bootstrap) and they both work as expected, ran the script, rebooted, was in webgui.
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: hbc on April 01, 2019, 09:26:35 am
Quote
With regards to ZFS: I've just done 2 conversions from vanilla "FreeBSD on ZFS" to OPNSense (opnsense-bootstrap) and they both work as expected, ran the script, rebooted, was in webgui.

Can you write short guide/tutorial how to use ZFS in OPNsense? Did you install FreeBSD first and then all this OPNsense stuff or did you have a script to convert OPNsense to ZFS?
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: rackg on April 02, 2019, 04:18:37 pm
ZFS should be installed on system that has ecc ram and 8GB min. Do you have such system ? If you don't have such system then you might have any stable OS filesystem or even leads to filesystem corruption.
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: hbc on April 02, 2019, 04:22:50 pm
Quote
ZFS should be installed on system that has ecc ram and 8GB min. Do you have such system?

I have 512GB ram. Think this will not be an issue. I just wonder how an ZFS installation would work.
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: rackg on April 02, 2019, 04:28:03 pm
Wow 512GB RAM !!! ZFS really loves it . Is it a firewall do you use squid or some caching memeroy devices. We can actually contribute ZFS installer if there is a strong use case for it be be part of firewall.
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: hbc on April 02, 2019, 04:32:31 pm
Yes, we run squid and sensei. There are several 10GbE chelsio cards in this machine. Also has dual XEON.  ;)
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: rackg on April 02, 2019, 04:40:13 pm
ZFS is designed such a way that it will actually take away all your free memeroy space. what ZFS features you will be using it ? How much is your system free memory at peak usage ?
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: mitsos on April 02, 2019, 10:01:49 pm
Quote
With regards to ZFS: I've just done 2 conversions from vanilla "FreeBSD on ZFS" to OPNSense (opnsense-bootstrap) and they both work as expected, ran the script, rebooted, was in webgui.

Can you write short guide/tutorial how to use ZFS in OPNsense? Did you install FreeBSD first and then all this OPNsense stuff or did you have a script to convert OPNsense to ZFS?

FreeBSD was already installed on ZFS, then it was a simple copy/paste from https://github.com/opnsense/update#opnsense-bootstrap, so the quick guide is:
1) Get FreeBSD installed on ZFS
2) Run the script  ;D
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: temporal-agent on April 20, 2019, 11:18:50 pm
Has anyone taken a look at the way FreeNAS installs to ZFS?
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: franco on April 21, 2019, 07:47:50 pm
There are a lot of approaches, some easier, some harder. The currently favoured approach is to switch our old "bsdinstaller" to the current FreeBSD "bsdinstall".

The issue is simply nobody offers the time to implement the ZFS installer as of yet.


Cheers,
Franco
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: rekriux on July 19, 2019, 03:32:50 am
Yes, I came here also to get opnsense installed on zfs.

My use case :

Opnsense on bare metal as internet router.
Since it's up all the time (super low power chip!), I want to run a few lightweight vm in bhyve (ie: database for sensor readings).

I guess I will take the "opnsense-bootstrap" road then. By the way, is it supported and feature identical ? Thanks

+1 to get it on the installer !

Or, you could just provide a usb installed image first. We could just make a small script to create zfs array, copy usb dataset to it and bootstrap it.

Thanks
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: sporkman on November 15, 2019, 03:36:01 am
Just curious, is the opnsense-bootstrap method still supported?

And I assume future updates don't care about whether the box was originally an opnsense install or was migrated from a stock FreeBSD install?
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: franco on November 15, 2019, 12:20:56 pm
Yes and yes.

12.1 adoption is a bit stalled for HBSD-related reasons so installer work is also stalled. It was planned to bring ZFS install support to 12.1 (20.1) but at the moment time is running out on that goal.


Cheers,
Franco
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: sporkman on November 25, 2019, 02:14:47 am
Thanks - just got around to this today, was really easy:

- backup config
- grab another drive (I suspect with all the file corruption I've been seeing around my kernel panics, the drive has to be bad even if SMART isn't showing anything), stick it in any PC laying around
- Install FreeBSD 11.2
- "pkg add" the nss cert stuff and git
- run the bootstrap script
- upload old config
- pop drive into firewall

And with zfs, I am hoping that even if the box panics for any reason I've got a better chance of not having corrupted files in the base.

Now I'm going to fiddle with zfs a bit and do the "poor man's RAID" and set "copies=2" on a few of the filesystems...
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: sporkman on November 25, 2019, 11:08:33 am
So this is kind of crazy. Got a few panics, but no swap was enabled (forgot to change ada2 to ada0 in fstab).

'zpool scrub' shows me no errors, but the health check shows some bad checksums, and I found the dashboard was not loading because "interfaces.inc" just had some random garbage sprinkled in the file. And the dashboard choking on those errors happened while the system was up and running.

Health check has this, not sure if this is corruption or if these are leftovers from the initial FreeBSD install:

Code: [Select]
***GOT REQUEST TO AUDIT HEALTH***
>>> Check installed kernel version
Version 19.7.7 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 19.7.7 is correct.
>>> Check for missing or altered base files
Error 2 ocurred.
lib/libcrypto.so.8:
sha256digest (0xbfe218c482ce5e1bcce776e265ee46b61a2425c91523de5a6472e1d3a39775e9, 0x1a55439f97072bbc6de73d19bbaa6e71e289a91fe86d9971af0483b3795c20b4)
usr/lib/libarchive.so.6:
sha256digest (0x917956bf32b66a471e0b8d6d0a4ecd2c3deaafd037a6778fc4c55cf0590c334a, 0xb2767e3b22acc19e0f40a05e74a5892a86095e4bd59e156c01fb60b2d71675f0)
usr/bin/awk:
sha256digest (0xc08dd78ff24e3d9d8484f55aeedd4105c3de8cd65be29db6b02aef61363795a9, 0xe3d9ce8f7c16df56f399543462f8bc73facfd012ce6af77543f0088e416aa325)
usr/bin/openssl:
sha256digest (0x9ce124fcee5cff90a2869273e48e9910b24873d454eb2f8af4c12258254ed391, 0x81f4e33a0e22fa02a9d4ccf2308074c8d15d3b5c0f1a8926ccce7c5413086bf6)
usr/bin/nawk:
sha256digest (0xc08dd78ff24e3d9d8484f55aeedd4105c3de8cd65be29db6b02aef61363795a9, 0xe3d9ce8f7c16df56f399543462f8bc73facfd012ce6af77543f0088e416aa325)
>>> Check for and install missing package dependencies
/usr/lib/libarchive.so.6: Undefined symbol "@FBSD_1.0"
>>> Check for missing or altered package files
/usr/lib/libarchive.so.6: Undefined symbol "@FBSD_1.0"
***DONE***

Might have to spend a few months on that other firewall distribution (if they still let me run w/o AESNI, ha!), if for no other reason than to regain my sanity.
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: Jose on December 03, 2019, 06:55:27 pm
Hello, I also use here the bootstrap installer for OPNsense ZFS install and is working just fine.  ;)
However looks like there's still many users requesting for a built-in ZFS installer, especially newcomers and/or users afraid of the command line, or simply an online installation is not possible.

Short story:
I was also thinking on doing something similar for NAS4Free/XigmaNAS few years ago, adding the bsdinstall, but since its main Embedded roots don't have much of acceptance, so I created a very simple ZFS installer single script solution(already added in base) to keep a small footprint and file count as minimum as possible, yet its process is based from latest bsdinstall with Boot Environments compliant which is a must.

Simple single script method:
Here is a small video (https://drive.google.com/file/d/1DBiK8KKeJDVyqYfYXJNZ0-_gSrH6wQ2s/view?usp=sharing) showing how it works in OPNsense with very small edit, the little script does offers to install in ZFS Stripe, Mirror and RAID10, as well as for Swap geli encryption option, currently supports MBR, GPT, UEFI and GPT+UEFI install options during its dialog driven installation.

I would be happy enough to contribute the ZFS installer to the OPNsense devs so they can update/modify/adapt as needed.  :)

Datasets creation process by the installer(same as bsdinstall):
Code: [Select]
zfs create -o mountpoint=none "zroot/ROOT"
zfs create -o mountpoint=/ "zroot/ROOT/default"
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off "zroot/tmp"
zfs create -o mountpoint=/usr -o canmount=off "zroot/usr"
zfs create -o setuid=off "zroot/usr/ports"
zfs create -o mountpoint=/var -o canmount=off "zroot/var"
zfs create -o exec=off -o setuid=off "zroot/var/audit"
zfs create -o exec=off -o setuid=off "zroot/var/crash"
zfs create -o exec=off -o setuid=off "zroot/var/log"
zfs create -o atime=on "zroot/var/mail"
zfs create -o setuid=off "zroot/var/tmp"

Here is the Disk usage widget after new install boot:
(https://drive.google.com/uc?export=download&id=18t0tT7i61vk3zlJeCaiHfmE-lA9Y5VUW)

P.S. Please fast forward the video during Unbound DNS after reboot to see the zpool/disk gpart layouts.

The BSDINSTALL method:
Here is a small video (https://drive.google.com/file/d/1HVWIW0QehyDv8kr3NmRuz4v-pCSajyPy/view?usp=sharing) showing a very simple install progress using the "bsdinstall" approach, it just need for the distfiles to be placed in "/usr/freebsd-dist" with a sane MANIFEST generated file, then explicitly add the mandatory files under bsdinstall/auto to only show optionals in the dialog if any such debug/extras dist files, additionally place customs in the bsdinstall/config to properly generate/append to config files.

One drawback is that this method required for the OPNsense distfiles to be included in the distribution as expected, however a small edit in the bsdinstall can take the files already present in the ISO and copy them to the target media like in the previous single script method, though I still prefer the bsdinstall personally and follow standards whenever possible.

Here is the test examples for reference. (https://drive.google.com/drive/folders/186MzlY_MI5qB2Nn2gC6YgyYAlXhhjWNZ?usp=sharing)

Regards
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: bobpaul on February 01, 2020, 05:48:37 pm
Jose, that looks great. I agree with you that the BSD Install method looks best. It's more consistent for users (that's the method adopted by other projects based on FreeBSD such as FreeNAS and pfSense) and I expect it should be easier to maintain (as we can get changes from FreeBSD as they adjust their installer script).

I would be happy enough to contribute the ZFS installer to the OPNsense devs so they can update/modify/adapt as needed.  :)

There are currently not enough developers to implement all the desired features in OPNsense, but the source code is on github (https://github.com/opnsense) and all are welcome to contribute pull requests. It's spread across multiple repositories, but here's the repo for the installer (https://github.com/opnsense/bsdinstaller). Would you be willing to create a pull request for this?
Title: Re: 19.1.4 ISO - ZFS install option missing?
Post by: franco on February 03, 2020, 10:46:05 am
The bsdinstall-based installer is here... https://github.com/opnsense/installer


Cheers,
Franco