OPNsense as VM on HDD pool?

[patrick3000]

I currently have OPNsense installed on bare metal with an SSD NVME boot drive, three physical interfaces, and five VLANs. I'm going to be migrating it to a VM on Truenas SCALE, which is built on Linux Debian and uses KVM for virtualization.

I'm considering installing OPNsense on a 3-disk hard drive mirror ZFS pool. It will also have 16 GB dedicated RAM. However, I'm wondering if installing it in a VM on a hard drive pool could result in latency problems given that hard drives are slow at random reads and writes?

In particular, and here is my main question: after OPNsense boots, does it mostly just stay in memory? If so, then I should be fine. If on the other hand it's constantly doing a bunch of read and write operations to the boot drive, then I'd imagine I could have problems installing it on an HDD pool.

I also have the option of installing it on an SSD pool in Truenas SCALE KVM, but I'd rather not because that pool only has two mirrored SSDs, and one of them is slightly old, so there would be less redundancy than if I put it on the 3-way HDD mirror.

[bartjsmit]

System: Settings: Miscellaneous
Disk/Memory settings

I would log on a different device. Not a bad idea for a security device regardless

Bart...

[Patrick M. Hausen]

@patrick3000 I don't expect any problems. ZFS will keep all "hot" blocks in ARC, anyway. And why would a firewall not run from spinning disks? They were the standard in all servers, even high end ones, not so long ago.

[cookiemonster]

Most of the IO is for logging, the working is in memory as far as I can see. IMHO Nvme is unnecessary, SSDs nice to have and spinning hard disks can be fine too. There have never been requirements for fast IO and many ran firewalls with HDs for ages.
In short, SSDs shall be plenty and mirrored pair is pretty ideal for redundancy.

[cookiemonster]

typing at the same time Patrick(s).

[CJ]

No comment on the performance, but keep in mind that this means your internet will be tied to your TrueNAS.  If you need to take it down for any reason, your entire network goes with it.

Is there a particular problem you're attempting to solve by virtualizing it?

[patrick3000]

Regarding CJ's question "No comment on the performance, but keep in mind that this means your internet will be tied to your TrueNAS.  If you need to take it down for any reason, your entire network goes with it. Is there a particular problem you're attempting to solve by virtualizing it?"

I totally understand your point, and this is a good question which I've thought carefully about, and there are pluses and minuses either way. But in the end I decided to virtualize, mainly because it will allow me eliminate extra hardware and cabling, reduce energy usage, and clear some space on the shelving unit where I have my gear set up. I can't just use one of those mini firewall devices for OPNsense, at least not easily, because one of the adapters is 10gbps SFP+ and another is 2.5gpbs base-T, and finding a mini PC that supports 10gbps (especially) is difficult. So right now, I'm using an old mini Tower for OPNSense. It will be nice to clear it out of there.

Also, two physical computers (one with OPNsense, one with Truenas) means extra hardware to maintain, with twice the components that can fail.

All of that being said, yes your point is certainly valid. Virtualizing OPNsense on Truenas means that if Truenas goes down, my entire network does. That's a downside, but it's a risk I've decided to take, mainly because Truenas SCALE is extremely stable, runs for months with no problems, and is installed on server hardware with ECC memory, and the OPNsense VM will be on a 3x mirror pool, so out of three HDDs, up to two could fail and OPNsense would still function.

[patrick3000]

Patrick M. Hausen and cookiemonster, thanks for your comments.

My only concern is that every once in a while, I do a large, multi-Terabyte transfer of data to or from another device, and the HDDs on Truenas crank for hours, which I assume increases I/O latency. But if OPNsense runs primarily in memory, with mostly just logging to the boot drive, as would make sense given its function, then I doubt that would be a problem for the operation of OPNsense.

[CJ]

Regarding CJ's question "No comment on the performance, but keep in mind that this means your internet will be tied to your TrueNAS.  If you need to take it down for any reason, your entire network goes with it. Is there a particular problem you're attempting to solve by virtualizing it?"

I totally understand your point, and this is a good question which I've thought carefully about, and there are pluses and minuses either way. But in the end I decided to virtualize, mainly because it will allow me eliminate extra hardware and cabling, reduce energy usage, and clear some space on the shelving unit where I have my gear set up. I can't just use one of those mini firewall devices for OPNsense, at least not easily, because one of the adapters is 10gbps SFP+ and another is 2.5gpbs base-T, and finding a mini PC that supports 10gbps (especially) is difficult. So right now, I'm using an old mini Tower for OPNSense. It will be nice to clear it out of there.

Also, two physical computers (one with OPNsense, one with Truenas) means extra hardware to maintain, with twice the components that can fail.

All of that being said, yes your point is certainly valid. Virtualizing OPNsense on Truenas means that if Truenas goes down, my entire network does. That's a downside, but it's a risk I've decided to take, mainly because Truenas SCALE is extremely stable, runs for months with no problems, and is installed on server hardware with ECC memory, and the OPNsense VM will be on a 3x mirror pool, so out of three HDDs, up to two could fail and OPNsense would still function.

You could have just used the quote functionality of the forum. :)

Regarding mini pc that support 2.5g and 10g SFP+, have you looked at the R86S-U4?  It sounds like it fits your use case perfectly.  HomeNetworkGuy reviews it with OPNSense here.  https://homenetworkguy.com/review/gowin-r86s-u4/  I believe he's on the forum but I don't know his username.

In regards to virtualizing OPNSense, I wasn't even referring to hardware failures.  What happens when you need to update TrueNAS?  Or replace a drive for more storage, etc.  Regardless, it's your network, but I wanted to point out the risks.

[patrick3000]

Regarding CJ's question "No comment on the performance, but keep in mind that this means your internet will be tied to your TrueNAS.  If you need to take it down for any reason, your entire network goes with it. Is there a particular problem you're attempting to solve by virtualizing it?"

I totally understand your point, and this is a good question which I've thought carefully about, and there are pluses and minuses either way. But in the end I decided to virtualize, mainly because it will allow me eliminate extra hardware and cabling, reduce energy usage, and clear some space on the shelving unit where I have my gear set up. I can't just use one of those mini firewall devices for OPNsense, at least not easily, because one of the adapters is 10gbps SFP+ and another is 2.5gpbs base-T, and finding a mini PC that supports 10gbps (especially) is difficult. So right now, I'm using an old mini Tower for OPNSense. It will be nice to clear it out of there.

Also, two physical computers (one with OPNsense, one with Truenas) means extra hardware to maintain, with twice the components that can fail.

All of that being said, yes your point is certainly valid. Virtualizing OPNsense on Truenas means that if Truenas goes down, my entire network does. That's a downside, but it's a risk I've decided to take, mainly because Truenas SCALE is extremely stable, runs for months with no problems, and is installed on server hardware with ECC memory, and the OPNsense VM will be on a 3x mirror pool, so out of three HDDs, up to two could fail and OPNsense would still function.

You could have just used the quote functionality of the forum. :)

Regarding mini pc that support 2.5g and 10g SFP+, have you looked at the R86S-U4?  It sounds like it fits your use case perfectly.  HomeNetworkGuy reviews it with OPNSense here.  https://homenetworkguy.com/review/gowin-r86s-u4/  I believe he's on the forum but I don't know his username.

In regards to virtualizing OPNSense, I wasn't even referring to hardware failures.  What happens when you need to update TrueNAS?  Or replace a drive for more storage, etc.  Regardless, it's your network, but I wanted to point out the risks.

Thanks. I didn't know about that mini PC with 2.5gb and 10gb SFP+ support. It looks like a nice option, as most mini PCs lack 10gb ports. Although frankly I'm still leaning toward virtualizing as my Truenas system almost never needs to be taken off-line, but that is the one of the better options I've seen in mini PCs for use as firewalls.

[CJ]

Thanks. I didn't know about that mini PC with 2.5gb and 10gb SFP+ support. It looks like a nice option, as most mini PCs lack 10gb ports. Although frankly I'm still leaning toward virtualizing as my Truenas system almost never needs to be taken off-line, but that is the one of the better options I've seen in mini PCs for use as firewalls.

NP.  I personally don't like virtualizing things like OPNSense both due to convenience and security issues, but like I said, it's your network. :)

[Patrick M. Hausen]

I have run OPNsense virtualised in bhyve on TrueNAS CORE. With PCIe pass through for two dedicated network interfaces I could not get the VM to run stable.

With two emulated VirtIO interfaces connected to two bridges in TrueNAS that were in turn connected to two dedicated hardware interfaces used for the firewall only, I got around 800-900 Mbit/s of throughput with 1 Gbit/s network cards. System running perfectly well.

So all caveats considered - any maintenance of your TrueNAS will mean no Internet - it is certainly possible. I completely understand the appeal of a "single box SOHO/SMB appliance".

TrueNAS CORE 13.1 will come with FreeBSD 13.2 and many minor improvements to bhyve. Possibly I'll try and re-evaluate the PCIe passthrough approach once more.

Good luck,
Patrick

[newsense]

For testing OPNSense can already be bootstrapped on a FreeBSD 13.2 and the experience should be very close if not identical with the upcoming TrueNAS 13.1

The Cobia announcement was a bit concerning, sounded like FreeBSD TrueNAS is on an extended life support. All resources go on Linux and some bits will get backports later on.

[Patrick M. Hausen]

For testing OPNSense can already be bootstrapped on a FreeBSD 13.2 and the experience should be very close if not identical with the upcoming TrueNAS 13.1

The version of FreeBSD in TrueNAS and the version of FreeBSD in an OPNsense VM are not in any way related. Current CORE is based on FreeBSD 13.1, CORE 13.1 will be based on FreeBSD 13.2 - there went a lot of improvements into the hypervisor - that's my point. If OPNsense is FreeBSD 13, 14, Linux or Windows does not matter in a hypervisor environment.

[CJ]

The Cobia announcement was a bit concerning, sounded like FreeBSD TrueNAS is on an extended life support. All resources go on Linux and some bits will get backports later on.

Interesting.  I hadn't seen that.  Been meaning to give Scale another spin but haven't had time.  I don't think Core is on life support, as you say.  Just that it's feature set is going to continue to be limited to storage and sharing.  Which is all I use it for currently.