Opnsense in Esxi VM - ZFS?

[GreenMatter]

My opnsense instance runs in esxi vm. I guess it doesn’t make sense to reinstall it using latest, zfs installator when vm doesn’t have direct access to HD’s controller?

[Patrick M. Hausen]

There probably is no one true answer to this but I think the same way. You might want to host your VMs on a system running on ZFS connected to ESXi via iSCSI 

[cookiemonster]

Well this debate is still going on in my head, whether to stay on zfs or reinstall on ufs.

Let me explain my thinking.
I'm a fan of zfs in general but not a fan of it on anything but raw disk drives. Yes it can made to work on networked storage volumes but that's another layer between zfs and the actual disks. This is for the volume management part i.e the management of raw devices, virtual devices, volumes, etc.

Now then, when we think about the next layer, the zfs filesystem, that's what we're gaining by staying on it. A copy on write filesystem less susceptible to power loss for instance.

I'm using OPN on a small APU2 system with only 4 GB of (non ECC) RAM. My concern is the memory used by ARC. So far the usage has gone to 65% of the total. Higher than before, used to consume about 30%. I'm all familiar with the "if memory is available what good is to leave it unused" message. So far I'm keeping an eye on any memory starvation but luckily it hasn't happened.

[Patrick M. Hausen]

I'm a fan of zfs in general but not a fan of it on anything but raw disk drives.

Same here - all my OPNsense appliances are running on ZFS now. No problem on e.g. the apu4d4 so far.
I would install with ZFS on everything with 4GB of memory and up. I am running ZFS on a netbook with 2GB - this is not the problem some people claim it is. It all depends on the amount of storage. So said netbook with a single 275GB SSD - absolutely no problem. Historically people have been trying to build storage appliances with terabytes of spinning disk drives and ridiculously low amounts of memory. That's where these general recommendations come from.

In virtualised environments I tend to use UFS or EXT4 if it's Linux inside the VM - and all my hypervisors are running ZFS, too, so I can snapshot replicate, clone ... the disk images from outside just fine.
ZFS inside a VM running on ZFS on the hypervisor - we have one of those for poudriere, the FreeBSD package build environment. Of course that works, too. You will get a lot of write amplification and probably wasted space, too - just need to be willing to bear that, if there are other reasons for running poudriere virtualised.

Kind regards,
Patrick

[GreenMatter]

Thanks guys for answers and confirmations. I decided to do not passthrough drive's controller in order not to waste entire drive capacity for esxi. Or maybe I'm wrong?

In virtualised environments I tend to use UFS or EXT4 if it's Linux inside the VM - and all my hypervisors are running ZFS, too, so I can snapshot replicate, clone ... the disk images from outside just fine.

What hypervisors run natively on ZFS? I don't think esxi is and never will, but being honest, I've never verified that...

[Patrick M. Hausen]

What hypervisors run natively on ZFS?

TrueNAS.

[maclinuxfree]

and Proxmox

[cookiemonster]

Yes for my ESXi installation I pass through the controller to the VM. Not Freenas/Truenas but xigmanas.
We might be getting the OP confused though. Those are not hypervisors but NAS OSes.
Proxmox and ESXi are.

[Patrick M. Hausen]

Those are not hypervisors but NAS OSes.

Sorry but my TrueNAS runs VMs quite smoothly.

[GreenMatter]

I was choosing between esxi and proxmox. FreeNAS didn't have a good reviews as "hypervisor" and I've never considered it as such. Now it runs in esxi VM with HDD controller passthrough and I use mainly as NAS. And I must admit I prefer to run separate debian/ubuntu instance as docker host than use jails or bhyve. My experience is a lot of issues with network efficiency (internal NAT/VNET in FreeNAS). I have only nextcloud running in jail as I use NAS purely for desktop backups, storage (datastore for esxi as well).
@pmhausen has different experience, can you share jails or bhyve VM network setup?
But that's off topic :-).

[Patrick M. Hausen]

@pmhausen has different experience, can you share jails or bhyve VM network setup?

Just standard bridged networking with VNET for jails and same bridge for VMs. Bridge to VLAN, VLAN on top of lagg, lagg with two ports with LACP into Cisco 2960-L. Just like my OPNsense, all lagg/VLAN here.

Never had a problem with TrueNAS in that area. Yes, in FreeBSD 12.2 bridging is still single core. Expect a huge performance boost when switching to FreeBSD 13.

We run a data centre with dozens of hosts running about 1000 bridged jails with iocage just like TrueNAS does. Definitely robust production ready.

[cookiemonster]

Those are not hypervisors but NAS OSes.

Sorry but my TrueNAS runs VMs quite smoothly.

Yes you are right. It is a Hypervisor.

I was choosing between esxi and proxmox. FreeNAS didn't have a good reviews as "hypervisor" and I've never considered it as such. Now it runs in esxi VM with HDD controller passthrough and I use mainly as NAS. And I must admit I prefer to run separate debian/ubuntu instance as docker host than use jails or bhyve. My experience is a lot of issues with network efficiency (internal NAT/VNET in FreeNAS). I have only nextcloud running in jail as I use NAS purely for desktop backups, storage (datastore for esxi as well).
@pmhausen has different experience, can you share jails or bhyve VM network setup?
But that's off topic :-).

I have a very similar home setup. I also have a few jails and if I need a VM it goes as another VM on ESXi alongside XN. Mostly for Linux ones given that a lot of packages are available for it and not for FreeBSD.

[GreenMatter]

@pmhausen has different experience, can you share jails or bhyve VM network setup?

Just standard bridged networking with VNET for jails and same bridge for VMs. Bridge to VLAN, VLAN on top of lagg, lagg with two ports with LACP into Cisco 2960-L. Just like my OPNsense, all lagg/VLAN here.

Never had a problem with TrueNAS in that area. Yes, in FreeBSD 12.2 bridging is still single core. Expect a huge performance boost when switching to FreeBSD 13.

We run a data centre with dozens of hosts running about 1000 bridged jails with iocage just like TrueNAS does. Definitely robust production ready.

At the moment NAS works on TrueNAS 12.0-U4 so far away from ver. 13. It's set with 2 passthrough Intel NICs which are set similar to your setup in lagg and lagg is parent interface for VLANs and further bridges.
So I need to wait until version 13 becomes available and than try again...

[Patrick M. Hausen]

What exactly are the problems you are having? Maybe you should take this to the TrueNAS forum?

https://www.truenas.com/community/

[GreenMatter]

What exactly are the problems you are having? Maybe you should take this to the TrueNAS forum?

https://www.truenas.com/community/

We were discussing this issue there 
https://www.truenas.com/community/threads/freenas-mini-xl-vnet-jail-bridged-to-vlan-extremely-slow.80878/#post-629450
Sorry for off topic.