Cannot get Bridging to work at all

[EasyGoing1]

I'm having a difficult time getting bridging to work at all with OPNSense 22.7

Using a VM, I have an install that I can play with, so here is the description of my current config and how I got there:

The ESXi server has 4 NICs.
1 - WAN
2 - LAN
3 - OPT1
4 - OPT2

After initial setup I verified that I can reach the Internet from the LAN interface (10.10.10.0/24) without any issues. My goal at this point was just to get bridging to work at all before including the NIC that is in the LAN interface so I'm only using the unused two NICS for the bridge (OPT1 and OPT2).

Here is what I did next:

• Interfaces / OPT1 & OPT2 and Enabled them and did nothing else.
• Interfaces / Other Types / Bridge, hit the PLUS button and selected both OPT1 and OPT2 for the interfaces, and set the description to BRIDGE
• Interfaces / Assignments and added a new interface using BRIDGE as the port and called it BRIDGE
• Interfaces / BRIDGE / Enabled and set a static IP address of 10.10.11.1/24
• System / Settings / Tunables and set net.link.bridge.pfil_member = 0 and net.link.bridge.pfil_bridge = 1
• Firewall / Rules / BRIDGE - added a new rule that allows all IP4 traffic unrestricted.
• Power / Reboot - rebooted the entire firewall (cold reboot of the VM)

Configured a NIC on my workstation with IP address 10.10.11.2/24 gateway 10.10.11.1 and plugged that nic into one of the ports that make up the bridge. No other NICs are active on my workstation, only that NIC.

I cannot ping 10.10.11.1

What am I doing wrong?

[Patrick M. Hausen]

How are you passing those NICs to the OPNsense VM? Are those virtual NICs or PCIe passthrough?

[EasyGoing1]

How are you passing those NICs to the OPNsense VM? Are those virtual NICs or PCIe passthrough?

Within ESXi, I have each NIC assigned to its own vSwitch, then each vSwitch assigned to its own port group and then in the virtual machine, I added four NICs each assigned to a different port group.

[Patrick M. Hausen]

Permit promiscuous mode for those port groups. Also if you cannot use PCIe passthrough (recommended) you might get better performance doing all the bridging and switching in ESXi. If this is just a test and you intend to deploy on hardware, eventually, go ahead.

[EasyGoing1]

Permit promiscuous mode for those port groups. Also if you cannot use PCIe passthrough (recommended) you might get better performance doing all the bridging and switching in ESXi. If this is just a test and you intend to deploy on hardware, eventually, go ahead.

I tried creating a bridge in ESXi so that I only added two NICs to the OPNSense VM, and that created some odd behavior...

When I plugged my workstation NIC into ANY one of the three bridged ESXi ports I could reach the LAN interface of OPNSense without any issues ... HOWEVER, as soon as I plugged a device into a second port in that bridge, the first connection lost its ability to talk to OPNSense.

The best I can figure out is that OPNSense is being given a virtual NIC with a single MAC address and even though that mac address exists in the ESXi bridge its still only a single MAC address within OPNSense so then it can only have a discussion with a single MAC address so that when I have two devices plugged into those bridged ports, it somehow just assigns the mac address to the second connection leaving the first connection flapping in the wind... but that's the best I can think of in terms of explaining the behavior.

[EasyGoing1]

Permit promiscuous mode for those port groups.

I did enable promiscuous mode in each vSwitch and the port groups are set to inherit those settings ... I also - just for good measure - enabled promiscuous mode in each of the two ports in the bridge within OPNSense as well as the BRIDGE interface ... same behavior. No love.

[Patrick M. Hausen]

You cannot do passthrough?

[EasyGoing1]

You cannot do passthrough?

I am unfamiliar with passthru in ESXi ... and I don't recall seeing that setting anywhere. Where might I find it?

[EasyGoing1]

You cannot do passthrough?

Looks like a negative on that option, trying to add a PCI device, that option is ghosted out.

I guess I should mention that the NICs are all integrated onto the motherboard ... its one of those "soft router" devices.

[Patrick M. Hausen]

You must enable that feature for individual cards, first. Host > Manage > Hardware ...

[EasyGoing1]

You must enable that feature for individual cards, first. Host > Manage > Hardware ...

Looks like it's not capable...?

[EasyGoing1]

You must enable that feature for individual cards, first. Host > Manage > Hardware ...

Looks like I was able to enable passthru once I removed the vSwitch that was assigned to the NIC. I'll get the rest of them setup like this and see if I cant get this thing bridged ... however, that does raise the question of whether or not I will be able to assign NICs to other VMs though Im assuming I would use passthru on those as well.

[Patrick M. Hausen]

Passthrough, not SR-IOV ...

Tick the small checkbox left to the interface, then click on "Toggle passthrough". Remove all connections to that interface, first.

[Patrick M. Hausen]

You can assign one PCIe device to exactly one VM with passthrough. That's the point. The VM gets full access to the hardware. That's recommended for a firewall, anyway.

You can designate a single interface and e.g. use VLANs for other VMs.

[EasyGoing1]

You can assign one PCIe device to exactly one VM with passthrough. That's the point. The VM gets full access to the hardware. That's recommended for a firewall, anyway.

You can designate a single interface and e.g. use VLANs for other VMs.

Can those other VMs share those NICs that are passed thru? I guess my only issue would be if I passthru three out of four NICs and that 4th one is dedicated to my Internet connection, then I'll lose the ability to have other VMs using the LAN interface of the firewall... the idea was to have one NIC dedicated to WAN and the other three dedicated to LAN with VMs also accessing LAN.