[NOOB] How to properly partition (ZFS) on 2 disks

[erica.vh]

Hello people,
using the [NOOB] tag from Marie-Sophie, I think it's a great thing for us newbies to recognize each-other :-p

As a newb, and reading many post in here, I guess I'm gonna need a lot of log and reports to figure out what/where/how
I would like to setup my system on my NVMe (done) and have my logs & Reports on my second disk (cheaper, expandable) but after a dozen tries, I still can't get it to run at boot.

So NVMe is all default:
Zpool is zroot: nda0p1 is efi, nda0p2 is boot, nda0p3 is 8G swap and nda0p4 is system.
While the SATA disk is
ada0p1 is for swap, ada0p2 for /tmp and ada0p3 for /var

First I tried doing the complete install via the installer/opnsense instal menu, but failed every single time.
Then I gave up, installed on NVMe only, and SSH to zpool attach zroot ada0 (all of them) but none of the mount mounted (through fstab)
Then I zpool create -f tank ada0p1 ada0p2 ada0p3
=> Zpool list shoed the two pool (yeah!)
=> And I added the corresponding entries in fstab, but there again, conflicts (/tmp vs /tmp etc)
 
I'm out of options ...
Any idea, lead, hints ? pretty pleeease

[erica.vh]

recap: nda0p1 (efi), nda0p2 (boot), nda0p3 swp an vdev "zroot" with all the zvols

ok now for my tentative in details:
For the Swap (just in case I run out of 32G RAM)
gpart add -t freebsd-swap -s 16G ada0
swappoff -a
nano /etc/fstab #current swap; add /dev/ada0p1 none swap
swappon -a

Idea: The initial 8G RAM slice on NVMe will be used as L2ARC/Cash ? (is that even a good idea on a FW/router?)
i.e: zfs create zroot/cash; zfs set quota=7G zroot/cash; zfs set mountpoint=/var/cash zroot/cash

Then for the /tmp and /var:
gpart add -t freebsd-zfs -b 33554500 -s 93GB ada0
ada0p2 added

The vdev "ztank"
zpool create -f ztank /dev/adaop2
zpool list shows zroot and ztank with the respective right size

so far so good, now for the plat de resistance ... moving zvol from zroot to ztank
zfs list (to get the axect name of the current zvol)
zfs create -V 32G ztank/tmp
BUT I don't want to preattribute size, so I skip the -V
zfs create ztank/tmp
zfs create ztank/var/audit
=> Parent doesn't exist, so first: zfs create ztank/var
zfs create stank/var/audit
zfs create stank/var/log
zfs create stank/var/tmp

and now, for the actual transfer ...
zfs create snapshot -r zroot/tmp@relocate
zfs create snapshot -r zroot/var/audit@relocate
zfs create snapshot -r zroot/var/log@relocate
zfs create snapshot -r zroot/var/tmp@relocate
------
zfs send -V zroot/tmp@relocate | zfs receive -Fdu ztank/tmp
zfs send -V zroot/var/audit@relocate | zfs receive -Fdu ztank/var/audit
zfs send -V zroot/var/log@relocate | zfs receive -Fdu ztank/var/log
zfs send -V zroot/var/tmp@relocate | zfs receive -Fdu ztank/var/tmp
-------
now ... I can't put these zroot/xxx offline, I can't rename them, I'm wondering how I will get rid of them ?
if I restart the system they will still be there, in a kind of a mirror or extension

I went this far, but now I'm out of idea, as I can't access the system if I put it offline

[EricPerl]

That doesn't look like a noob topic to me.
I've had to deal with a zpool import issue under proxmox (my fault) and that was enough.

FWIW, for a much simpler case (just adding a disk, no ZFS), Franco pointed the OP of this thread to FreeBSD docs...
https://forum.opnsense.org/index.php?topic=8979.0

It looks like you managed to move the swap (or add to it).
Are you sure that was wise?
That SATA drive is likely way slower (500MB/s) than the NVME (depends on PCI slot version and lanes too). Even swapping to NVME would likely affect overall performance.
Would it make sense for ZFS to "cache" data using NVME when that data is already stored in that same NVME?
The cache has to be faster that the underlying storage... Swap is really a last resort.

For the rest, it's beyond me. Beyond the use of ZFS for your purpose (is it worth it?), I'm not sure that what you are trying to move over (tmp, var, ...) aligns with existing mount points in the default install.

[cookiemonster]

> Idea: The initial 8G RAM slice on NVMe will be used as L2ARC/Cash ? (is that even a good idea on a FW/router?)
Not a good idea.

https://klarasystems.com/articles/openzfs-all-about-l2arc/
When should I use L2ARC?

For most users, the answer to this question is simple—you shouldn't. The L2ARC needs system RAM to index it—which means that L2ARC comes at the expense of ARC. Since ARC is an order of magnitude or so faster than L2ARC and uses a much better caching algorithm, you need a rather large and hot working set for L2ARC to become worth having.

In general, if you have budget which could be spent either on more RAM or on CACHE vdev devices—buy the RAM! You shouldn't typically consider L2ARC until you've already maxed out the RAM for your system.

[erica.vh]

That doesn't look like a noob topic to me.
I've had to deal with a zpool import issue under proxmox (my fault) and that was enough.

FWIW, for a much simpler case (just adding a disk, no ZFS), Franco pointed the OP of this thread to FreeBSD docs...
https://forum.opnsense.org/index.php?topic=8979.0

It looks like you managed to move the swap (or add to it).
Are you sure that was wise?
That SATA drive is likely way slower (500MB/s) than the NVME (depends on PCI slot version and lanes too). Even swapping to NVME would likely affect overall performance.
Would it make sense for ZFS to "cache" data using NVME when that data is already stored in that same NVME?
The cache has to be faster that the underlying storage... Swap is really a last resort.

For the rest, it's beyond me. Beyond the use of ZFS for your purpose (is it worth it?), I'm not sure that what you are trying to move over (tmp, var, ...) aligns with existing mount points in the default install.

Yes, you are right, the SATA is way slower that NVMe, but I'm not after speed, rather preserving the NVMe from heavy R/W as the SATA is way cheaper, and I have 32G RAM, so the use of swap will (should) be very limited to rare occasions
The /tmp though will be much more frequent
And, of course, the multiple log/audit/reports, etc ...

As for the link, it is indeed for a different task, swapping the drive, while my need after moving some zvol to a second drive, I want to find a way to shut the ones on the initial drive . which I can't when the system is up, but if I shut down the system, then I no longer have access to the drive

[erica.vh]

> Idea: The initial 8G RAM slice on NVMe will be used as L2ARC/Cash ? (is that even a good idea on a FW/router?)
Not a good idea.

https://klarasystems.com/articles/openzfs-all-about-l2arc/
When should I use L2ARC?

For most users, the answer to this question is simple—you shouldn't. The L2ARC needs system RAM to index it—which means that L2ARC comes at the expense of ARC. Since ARC is an order of magnitude or so faster than L2ARC and uses a much better caching algorithm, you need a rather large and hot working set for L2ARC to become worth having.

In general, if you have budget which could be spent either on more RAM or on CACHE vdev devices—buy the RAM! You shouldn't typically consider L2ARC until you've already maxed out the RAM for your system.

Thank you ! so no cashe on NVMe then ! As for RAM, I've maxed out the capacity of the MB, so it will stay at 32G

[cookiemonster]

Yes, no L2ARC. Even on storage systems using ZFS, the benefit is in particular cases. For a firewall, not for now anyway. Meaning, after a lot of users and lots of services start consuming RAM, then might be worth revisiting.

As for the saving of the fast and expensive SSD from constant logging, you should leave the default logging which is to not log default rules and keep the rest for a max of X number of days.
Once you create your own rules, enable logging only for diagnostics.
With this, there will be a very limited amount of wear. Once you find your drive's official endurance, chances are that you'll find you have years of life in it.
Another option if you really must is to log to ram. The obvious downside is that they don't persist power loss. Personally for the reasons above, fail to see why people would chose that, except when using an embedded install.

[EricPerl]

I guess I should have been more explicit.
Adding a disk is indeed a simple operation, and entirely handled by the underlying distribution (in CLI no less).
In your case, you're trying to change the way FreeBSD mounts FSs and boots.
You might have better chances of getting an answer to your questions on FreeBSD forums.
It's not OPN specific apparently.

Beyond the NVME wear saving measures mentioned by cookiemonster, also consider what it would take to fix that box if the SATA fails. It probably won't boot... Even if you want to swap the drive before it dies, what will it take? Do you have room for another to be able to clone in place? ...

[EricPerl]

FWIW, here's what I think you're signing up for, whether it's to solve your current issue or fix it later if there are issues. It's all theoretical (based on superficial knowledge and very little experience)...

You'd have to:
* get a bootable USB/CD with a freeBSD version compatible with OPNsense's distribution.
* boot from it
* access OPN's storage (zpool import??)
* For each location you want to "relocate", copy the existing content over to the corresponding partition, mount the partition in place of the current location.
* modify fstab accordingly
* you might have to re-import the zpool in OPN afterwards (prompt during the boot process). I've had to do that once...
On top of it, there are hardlink under var/unbound in my current install. I have no clue how those would follow the steps above or if they would have to be recreated.

There are things that I'm willing to tinker with to help. The above is not one of them (stretching current knowledge)...
A wrong step will likely lead to a non-bootable system. If it's fresh, you can restart from scratch but it's time consuming.

Again, at your own risks!