Pros and Cons of bare metal versus virtualization home environment

[opnsenseguy]

Assuming hardware isn't a constraint what are some pros and cons of bare metal versus virtualization OPNsense in a home or small business environment?

Here are some that I can think of but would like to have a discussion about my small list and other items that you can think of.

Bare Metal
+ It is how the software is designed.
+ No host OS configuration, updates, etc.
+ All hardware is direct and so less potential diagnosing of issues, tunables, etc.
- Potential waste of hardware resources.
- Subject to OPNsense HA and configuration backup tools.

Virtualization
+ Can easily snapshot or backup the entire OPNsense instance in a click.
+ Can run other complimentary services/applications on same hardware such as pi-hole.
- Networking bridging versus PCI pass-thru can be confusing and complicates setup and design.
- Overall architecture is substantially more complicated.

It seems like not that long ago it was totally frowned upon to use something like Proxmox for OPNsense and now it seems like it is much more accepting and in some cases even recommended.  Again, I would love to have a discussion about this. 

[passeri]

My firewalls, routers, are on bare metal and most likely will remain so for three reasons which relate to my circumstances, not setting out to persuade others.

• Complexity is a basic element of failure risk. Therefore I prefer to minimise it.
• In case of hardware failure, either strategy needs new hardware. In my case that should be on hand. If it is on hand, then it is already configured, and provides also a backup for testing or upgrades.
• I have a constrained space where the firewall ideally goes. Thus I am interested in minimising generated heat, so again a simpler firewall box suits. Other functions are easily run elsewhere.

I do use virtualisation on other boxes as a way of containerising some minor things, none related to the firewall.

[bartjsmit]

Bare Metal
+ All resources are dedicated to OPNsense so it works on simpler hardware

Virtual
+ Monitoring built in
+ Easier clustering (CARP vs HA)
+ Wider hardware support (e.g. USB NIC)
+ Wider software support - backup, alerting, orchestration

[passeri]

An additional comment:

+ Can easily snapshot or backup the entire OPNsense instance in a click.

For Opnsense itself this advantage of virtualisation is negated since the introduction of ZFS snapshots in 24.7. I think it comes down mostly to resource sharing.

For more novel experiments, one can do both.

[OPNenthu]

I'm biased toward bare metal for security and performance.

- Smaller overall attack surface
- Single source of bugs and bug fixes rather than multiple (no dependency on Proxmox & Debian in addition to OPNsense & FreeBSD)
- No "noisy neighbor" VMs sapping performance
- Some router manufacturers now shipping coreboot as an option to mitigate e.g. supply chain key leak issues and Intel ME backdoors
    -- although you are now shifting trust from A to B, you are making a bet that B carries less overall risk

[Trannie]

I'm biased toward bare metal for security and performance.

- Smaller overall attack surface
- Single source of bugs and bug fixes rather than multiple (no dependency on Proxmox & Debian in addition to OPNsense & FreeBSD)
- No "noisy neighbor" VMs sapping performance
- Some router manufacturers now ship coreboot as an option to mitigate e.g. supply chain key leak issues and Intel ME backdoors
    -- although you are now shifting trust from A to B, you are making a bet that B carries less overall risk Block Blast

Thanks for sharing your insight into security and performance when using bare metal! The points you raise about attack surface, error management, and risk from dependencies are really worth considering. Your input is greatly appreciated!

[devilkin]

In my case, I run my 'prod' (aka the one connecting to the internet) OPNsense on bare metal. I do have a virtualised OPNsense for testing stuff in a lab environment.

* Router does one thing does one thing, and does it well
* In case my Proxmox host goes down, it doesn't take everything with it - a hypervisor needs patches + reboots to update kernels