Virtualization inside OPNsense

[BernhardMM]

Hi, fellow OPNsense users!

We have been using OPNsense for some time now, currently alongside our hardware FW, but intend to move to a HA OPNsense setup soon. We don't want to run OPNsense as a virtual machine on ESXi or any other hypervisor, but also think that it would be a shame to waste good computing power

Is there any good solution to run jails / bhyve inside OPNsense without tampering with the BSD system? I imagine that this would potentially hurt upgrades.

I would want to run a few low prio encoding jobs, which should not bog down the main OPNsense system in case of, say, an attack - and also are encapsulated sufficiently to not pose any security risk.

I have only found one thread (Docker in the OPNsense environment?) which seems to be related - is this really such an unlikely use case?

Cheers,

Bernhard

[heresjody]

Personally I think the reason it’s not used much is the risk of having a less secure setup. As small as it may be, you’re still performing non-essential services on a firewall. Especially when you implement a HA solution, uptime and stability is important. I would use VM’s.

[banym]

Virtualization and firewall for security reasons are not the best combination. It adds a lot of complexity to the system stack.

Since the Intel Spectre and Meltdown security flaws it should be off the table to run critical systems on the same hardware. If you could not trust the hardware to separate the VMs and processes like they should do, it is not possible for a software layer on top to do so.

Beside this security aspect, the problem with sharing resources for your firewall with other VMs on the same box will lead to problems and trouble that never could be debugged. You will hardly find someone to support you or this solution since it is not clear what workloads are performed on the same hardware.

Sharing network resources and simulated device drivers for network cards will add another layer of complexity.

In my opinion it is a bad idea.
If you want to safe power, scale down your network boxes to only what you need.

[BernhardMM]

Personally I think the reason it’s not used much is the risk of having a less secure setup. As small as it may be, you’re still performing non-essential services on a firewall. Especially when you implement a HA solution, uptime and stability is important. I would use VM’s.

Maybe my idea is a bit off, but I thought that Network->OPNsense(physical)->protected VM/jail would be a more secure setup than Network->VM Host(physical)->VM1(OPNsense)&VMx(something else) in case the VM Host can be attacked.

I would not have used that in a production service, but in our office environment the added processing power would have been OK to trade for stability (not security, though with the extremly limited use case, I would not have thought of it as being a big risk)

[BernhardMM]

Virtualization and firewall for security reasons are not the best combination. It adds a lot of complexity to the system stack.

Since the Intel Spectre and Meltdown security flaws it should be off the table to run critical systems on the same hardware. If you could not trust the hardware to separate the VMs and processes like they should do, it is not possible for a software layer on top to do so.

Beside this security aspect, the problem with sharing resources for your firewall with other VMs on the same box will lead to problems and trouble that never could be debugged. You will hardly find someone to support you or this solution since it is not clear what workloads are performed on the same hardware.

Sharing network resources and simulated device drivers for network cards will add another layer of complexity.

In my opinion it is a bad idea.
If you want to safe power, scale down your network boxes to only what you need.

I agree on a general standpoint, but OPNsense in a VM alongside other VMs on the same host seems to be quite widespread - but I don't want to go that route.

Our production system will run only OPNsense on the boxes, but in our office environment I would like to have a similar setup, so we can test new releases / workflows, without it impacting production services and without having to setup a separate lab environment - which would be our other option.

Thank you for your inputs!