[Help] Bridging - Proxmox VM's to OPNsense

[knewknow]

Hello. I'm very new to OPNsense (Proxmox VM), and trying to bridge my Proxmox VM's to eliminate the additional hop through my physical switch (only 1gbe) and also allow for full speed through the OPNsense LAN port (10gbe). While attempting to create a bridge, it feels like I'm at a standstill and can't figure out what the proper configuration is here. Any help and guidance would be much appreciated! Here are some additional details:

Flat network 192.168.1.0/24

1. Proxmox server (192.168.1.4)
- 1gbe NIC (management). Connected to QNAP M408-2C 1gbe port
- X520-DA2 installed (passthrough to opnsense VM)
2. Ubuntu VM (192.168.1.3)
- running media servers and other systems
3. OPNsense VM (192.168.1.10)
- fully passed through X520-DA2
- WAN = ix1
- LAN = ix0
- 10gbe connected WAN
- 10gbe connected LAN to QNAP M408-2C sfp+ 10gbe port
4. ISP in advanced DMZ to opnsense WAN (ix1)

What I want to achieve:
- currently the proxmox and all VM's (except for opnsense) is running off of 1gbe onboard NIC
- Ubuntu server is handing downloads, so I would want it to get the full speed of the 10gbe card

Unfortunately I need the same subnet (due to some configuration that requires it) so bridging is the route I'm taking now.

Where I'm at now:
- I have added a vmbr1 as a bridge on Proxmox
- I have added vmbr1 to my Ubuntu VM and OPNsense

In OPNsense:
- I have created a new Interface called LAN_VM which is assigned to the net0 vmbr1 device
- I have created a bridge with LAN and LAN_VM members

Here's where I'm stuck. I'm thinking that I need to set LAN to the new bridge0, but I cannot add this due to the following error: "You cannot set device bridge0 to interface lan because it cannot be a member of itself.".

I'm sure that I'm thinking of this in the wrong way. Any help? Thanks so much.

[meyergru]

I suggest a bridged setup like described here: https://forum.opnsense.org/index.php?topic=44159.0

IDK if you can pass thru each of ix0 and ix1 individually. If so, then you could keep the passthru for WAN and only connect ix0 to the LAN bridge. Within OpnSense, you would then use net0 instead of ix0. If not, you would have to add another bridge for the WAN interface with ix1 and net1.

P.S.: I do not understand where bridge0 comes into play. You only need to create vmbrX on Proxmox with ix0 connected, then attach the WAN interface of your OpnSense VM to vmbr0. Of course that will cause an interface rename from ix0 to vtnetX inside OpnSense, such that you will have to re-assign the LAN interface.

[knewknow]

I suggest a bridged setup like described here: https://forum.opnsense.org/index.php?topic=44159.0

Thanks so much. I'll work on it tomorrow morning and will report back.

[knewknow]

P.S.: I do not understand where bridge0 comes into play. You only need to create vmbrX on Proxmox with ix0 connected, then attach the WAN interface of your OpnSense VM to vmbr0. Of course that will cause an interface rename from ix0 to vtnetX inside OpnSense, such that you will have to re-assign the LAN interface.

I took a peek, and the configuration that you're recommending is essentially removing the pci passthrough of the x520 to opnsense (ix0, ix1). From my limited understanding, bridge0 in my case is to allow OPNsense to handle the bridging between ix0 and vtnet0. I did spend a bit too much time this morning trying to get it working and just kept hitting roadblocks, so the sensible option might be to just have Proxmox manage it instead. If you have any more insight on my specific use case, I'll definitely take it!

[meyergru]

I think what you can / should do is either use a NIC as passthru or as a bridge member under Proxmox, you cannot have both. So if you have a specific NIC passed thru as ix0, you cannot use it as a bridge member in OpnSense, which is what bridge0 seems to imply.

So it is either ix0 as passthru, which creates the problem that you cannot use it for Proxmox itself or for other bridged VMs or ix0 used as a bridge member for vmbrX under Proxmox and then be able to use it for multiple VMs (including PVE itself and OpnSense LAN).

The only thing I do not know exactly is if you can passthru "half" of the X520-DA2 to use as WAN and use the other half as a vmbrX bridge member for LAN. I think it is impossible, since those two NICs are only PCIe functions, not full devices. Thus, you will be forced to use bot ix0 and ix1 as bridged devices if you want to use any one of them as such.

The guide suggests to use bridged devices, anyway. Partly, that is, because the NIC drivers under Linux support more devices than FreeBSD (and are better supported for some, like RealTek).

[knewknow]

I think what you can / should do is either use a NIC as passthru or as a bridge member under Proxmox, you cannot have both. So if you have a specific NIC passed thru as ix0, you cannot use it as a bridge member in OpnSense, which is what bridge0 seems to imply.

Thanks meyergru. I will go with your recommendation. I actually would have preferred to just create a subnet and use some routing, but I have a few finicky devices that are unreachable when tailscale is running when on a separate subnet.

Only reason that I went passthrough on OPNsense is that in real life tests, I get between 2.6-2.7gbe u/l as opposed to 2.3-2.4gbe max u/l when running through the linux driver. I'm not sure why that is, but I'm sure that I'll survive :).