VLANs on hypervisor that hosts OPNsense or in OPNsense itself?

[patrick3000]

I recently moved OPNsense from a bare metal install on a physical machine to a virtual machine running as a guest in Truenas SCALE, a hypervisor that reportedly is similar to Proxmox (although I haven't used Proxmox).

When I moved OPNsense to a virtual machine, I chose to set up the VLANs on the host (Truenas SCALE) and pass each VLAN as a virtual adapter to OPNsense. So, OPNsense doesn't "see" any VLANs at all. All it sees are adapters named vtnet1, vtnet2, etc. But each of those adapters corresponds to a VLAN in Truenas SCALE.

Everything works perfectly, but my concern is that if there is ever a hardware failure and I need to move OPNsense somewhere else, whether to another VM or to a physical machine, then it will be laborious to reconfigure all the VLANs, probably in OPNsense at that point.

So I'm wondering whether I should have instead passed the OPNsense VM virtual adapters that correspond to physical adapters on the host (rather than passing it adapters that correspond to VLANs), and then created the VLANs in OPNsense, which would make for easier portability of the VM. Does anyone know the best practice?

[Patrick M. Hausen]

In ESXi I would do all the virtual and real switching in ESXi and pass virtual interfaces to OPNsense. I do not have any experience with Proxmox so I cannot offer advice in that regard.

But for both hypervisors: if you have enough interfaces in your server to use PCIe passthrough, then do that. You have a clean separation, can use bridging, lagg, VLANs in OPNsense as you like, get full performance of the hardware ...

Only downside: traffic from hypervisor through OPNsense will have to pass through your switch. But that would be the case if hypervisor and OPNsense were two separate systems, too.

[patrick3000]

Thanks. That's an interesting thought to use PCIe pass through which, if I understand it, would give the VM real adapters rather than virtual. I'll look into that as a possibility.

Also, one clarification, I don't use Proxmox as my hypervisor. I use Truenas SCALE. But for this purpose, I believe they're almost the same (both built on Debian, etc.).

[Patrick M. Hausen]

Sorry for not paying attention to your post. Most people who come here for virtualisation advice use Proxmox.

I run an OPNsense in TrueNAS CORE with PCIe passthrough. Works like a charm. SCALE should do just as well.

Code Select Expand

┌──────────────────────────────────────────────────────────┐
│                                                          │
│                           TrueNAS CORE                   │
│          ┌────────────────────────────────────────────┐  │
│          │                                            │  │
│          │                           OPNsense VM      │  │
│          │       ┌ ─ ─ ─ ─ ─ ┐   ┌──────────────────┐ │  │
│          │    ┌───────────┐      │                  │ │  │
│          │ ┌──┴────────┐  │  │   │  LAN        WAN  │ │  │
│          │ │           │  │      │┌─────┐    ┌─────┐│ │  │
│          │ │ VMs/jails │  │  │   ││ ix0 │    │ ix1 ││ │  │
│          │ │           │  ├ ─    │└─────┘    └─────┘│ │  │
│          │ │           ├──┘      │   ▲          ▲   │ │  │
│          │ └────────┬──┘         └───┼──────────┼───┘ │  │
│          │          │                │          │     │  │
│          │          │                │   PCIe   │     │  │
│          │          │                │   pass   │     │  │
│          │ ┌────────┴─────────┐      │   thru   │     │  │
│          │ │                  │      │          │     │  │
│          │ │     bridge0      │      │          │     │  │
│  ┌────┐  │ │┌─────┐    ┌─────┐│   ┌──┴──┐    ┌──┴──┐  │  │
│  │IPMI├──┼─┼┤ ix0 │    │ ix1 ││   │ ix2 │    │ ix3 │  │  │
│  └────┘  │ │└──┬──┘    └──┬──┘│   └──┬──┘    └──┬──┘  │  │
│          │ └───┼──────────┼───┘      │          │     │  │
│          └─────┼──────────┼──────────┼──────────┼─────┘  │
│                │          │          │          │        │
│                ▼          └──────────┘          ▼        │
│                                                          │
│            to laptop                        to uplink    │
│                                                          │
│                                                          │
│  Mobile Lab                                              │
│  ----------                                              │
│  Supermicro A2SDi-4C-HLN4F                               │
│  Supermicro SC-101F                                      │
│                                                          │
└──────────────────────────────────────────────────────────┘

[patrick3000]

Thanks. The only issue is that one of my physical adapters on Truenas SCALE that I pass to OPNsense is a 10 GB adapter on the motherboard. It's not PCIe based. I'm not sure if an on-board adapter can be passed with PCIe pass-through.

[Patrick M. Hausen]

Of course it can - how do you think it's connected to the CPU? Your MB is full of PCIe and other buses, a slot is not necessary. All four of my interfaces in the diagram above are on board units.

What could pose a problem is the possibility that the interface shares an IOMMU group with some other device that is essential for TrueNAS itself. I would know how to check in FreeBSD, but not in Linux. Come over to that other forum if you need help.

[patrick3000]

Thanks.

[cookiemonster]

usual place to begin with to identify is in the docs https://pve.proxmox.com/wiki/PCI_Passthrough
That's proxmox though. Careful though, the Truenas implementation might only be based on kvm.