I'm having a difficult time getting bridging to work at all with OPNSense 22.7
Using a VM, I have an install that I can play with, so here is the description of my current config and how I got there:
The ESXi server has 4 NICs.
1 - WAN
2 - LAN
3 - OPT1
4 - OPT2
After initial setup I verified that I can reach the Internet from the LAN interface (
10.10.10.0/24) without any issues. My goal at this point was just to get bridging to work at all before including the NIC that is in the LAN interface so I'm only using the unused two NICS for the bridge (OPT1 and OPT2).
Here is what I did next:
- Interfaces / OPT1 & OPT2 and Enabled them and did nothing else.
- Interfaces / Other Types / Bridge, hit the PLUS button and selected both OPT1 and OPT2 for the interfaces, and set the description to BRIDGE
- Interfaces / Assignments and added a new interface using BRIDGE as the port and called it BRIDGE
- Interfaces / BRIDGE / Enabled and set a static IP address of 10.10.11.1/24
- System / Settings / Tunables and set net.link.bridge.pfil_member = 0 and net.link.bridge.pfil_bridge = 1
- Firewall / Rules / BRIDGE - added a new rule that allows all IP4 traffic unrestricted.
- Power / Reboot - rebooted the entire firewall (cold reboot of the VM)
Configured a NIC on my workstation with IP address
10.10.11.2/24 gateway
10.10.11.1 and plugged that nic into one of the ports that make up the bridge. No other NICs are active on my workstation, only that NIC.
I cannot ping 10.10.11.1
What am I doing wrong?
How are you passing those NICs to the OPNsense VM? Are those virtual NICs or PCIe passthrough?
Quote from: pmhausen on January 04, 2023, 09:25:08 AM
How are you passing those NICs to the OPNsense VM? Are those virtual NICs or PCIe passthrough?
Within ESXi, I have each NIC assigned to its own vSwitch, then each vSwitch assigned to its own port group and then in the virtual machine, I added four NICs each assigned to a different port group.
Permit promiscuous mode for those port groups. Also if you cannot use PCIe passthrough (recommended) you might get better performance doing all the bridging and switching in ESXi. If this is just a test and you intend to deploy on hardware, eventually, go ahead.
Quote from: pmhausen on January 04, 2023, 09:43:49 AM
Permit promiscuous mode for those port groups. Also if you cannot use PCIe passthrough (recommended) you might get better performance doing all the bridging and switching in ESXi. If this is just a test and you intend to deploy on hardware, eventually, go ahead.
I tried creating a bridge in ESXi so that I only added two NICs to the OPNSense VM, and that created some odd behavior...
When I plugged my workstation NIC into ANY one of the three bridged ESXi ports I could reach the LAN interface of OPNSense without any issues ... HOWEVER, as soon as I plugged a device into a second port in that bridge, the first connection lost its ability to talk to OPNSense.
The best I can figure out is that OPNSense is being given a virtual NIC with a single MAC address and even though that mac address exists in the ESXi bridge its still only a single MAC address within OPNSense so then it can only have a discussion with a single MAC address so that when I have two devices plugged into those bridged ports, it somehow just assigns the mac address to the second connection leaving the first connection flapping in the wind... but that's the best I can think of in terms of explaining the behavior.
Quote from: pmhausen on January 04, 2023, 09:43:49 AM
Permit promiscuous mode for those port groups.
I did enable promiscuous mode in each vSwitch and the port groups are set to inherit those settings ... I also - just for good measure - enabled promiscuous mode in each of the two ports in the bridge within OPNSense as well as the BRIDGE interface ... same behavior. No love.
You cannot do passthrough?
Quote from: pmhausen on January 04, 2023, 09:56:35 AM
You cannot do passthrough?
I am unfamiliar with passthru in ESXi ... and I don't recall seeing that setting anywhere. Where might I find it?
Quote from: pmhausen on January 04, 2023, 09:56:35 AM
You cannot do passthrough?
Looks like a negative on that option, trying to add a PCI device, that option is ghosted out.
I guess I should mention that the NICs are all integrated onto the motherboard ... its one of those "soft router" devices.
You must enable that feature for individual cards, first. Host > Manage > Hardware ...
Quote from: pmhausen on January 04, 2023, 11:05:51 AM
You must enable that feature for individual cards, first. Host > Manage > Hardware ...
Looks like it's not capable...?
Quote from: pmhausen on January 04, 2023, 11:05:51 AM
You must enable that feature for individual cards, first. Host > Manage > Hardware ...
Looks like I was able to enable passthru once I removed the vSwitch that was assigned to the NIC. I'll get the rest of them setup like this and see if I cant get this thing bridged ... however, that does raise the question of whether or not I will be able to assign NICs to other VMs though Im assuming I would use passthru on those as well.
Passthrough, not SR-IOV ...
Tick the small checkbox left to the interface, then click on "Toggle passthrough". Remove all connections to that interface, first.
You can assign one PCIe device to exactly one VM with passthrough. That's the point. The VM gets full access to the hardware. That's recommended for a firewall, anyway.
You can designate a single interface and e.g. use VLANs for other VMs.
Quote from: pmhausen on January 04, 2023, 11:33:39 AM
You can assign one PCIe device to exactly one VM with passthrough. That's the point. The VM gets full access to the hardware. That's recommended for a firewall, anyway.
You can designate a single interface and e.g. use VLANs for other VMs.
Can those other VMs share those NICs that are passed thru? I guess my only issue would be if I passthru three out of four NICs and that 4th one is dedicated to my Internet connection, then I'll lose the ability to have other VMs using the LAN interface of the firewall... the idea was to have one NIC dedicated to WAN and the other three dedicated to LAN with VMs also accessing LAN.
A passed through NIC is exclusive to that VM. The hardware ends up "inside" the VM. You cannot connect a vSwitch.
But why do you need three interfaces for LAN in a virtualised context? For your VMs it's all vSwitches, anyway.
Quote from: pmhausen on January 04, 2023, 12:06:13 PM
A passed through NIC is exclusive to that VM. The hardware ends up "inside" the VM. You cannot connect a vSwitch.
But why do you need three interfaces for LAN in a virtualised context? For your VMs it's all vSwitches, anyway.
Well, the idea was to essentially have a SOHO router that also hosted VMs and have three of the four NICs just get lumped together under one LAN side of the "router" with VMs being able to also use that LAN interface.
That would kind of ... mimic a traditional SOHO firewall with the added bonus of hosting virtual machines.
The bridge works of course with passthru ... but that begs the question of why can't OPNSense build bridges with virtual NICs? That doesn't make a lot of sense to me ... though now that I think about it, I THINK bridging happens at layer 2 and layer 2 would get managed within each NIC ... so if that is the case, then it does make sense.
So then it looks like my only option is to buy an external 2.5G switch (more than I wanted to spend on this setup) or just use those ports under vSwitches and assign a different subnet to each port ... which isn't all that bad but just not ideal.
There is no such thing as a free lunch :) Maybe your project is a bit too ambitious.
The "SOHO router" build definitely does work running on bare hardware. Why FreeBSD bridging doesn't with virtual interfaces I honestly have no idea. You could try to switch from VMXNET3 to E1000 if you have not done that already. This will not limit your speed to 1G although the OPNsense will report it as such. But since it's all emulated, it's just a question of which "API" OPNsense uses to talk to the virtual hardware.
HTH,
Patrick
Quote from: pmhausen on January 04, 2023, 12:30:46 PM
There is no such thing as a free lunch :) Maybe your project is a bit too ambitious.
One can hope ... thought it was the perfect solution ... got the hardware that has 4 2.5G NICS, with the N5105 Intel processor, then I added a 1TB NVMe drive and 32 gigs of ram ... all that at my door for just under $300. THAT would have been ambitious even two years ago.
In theory I had the right idea ... just didn't have a clue as to how I was going to bridge those three NICs ... just assumed it would somehow be possible.
Quote from: pmhausen on January 04, 2023, 12:30:46 PM
You could try to switch from VMXNET3 to E1000 if you have not done that already.
I have not tried that yet and I definitely will. But failing that, a different hypervisor might work, though I'm getting a bit burned out on this setup for the time being.
☺