Bridge / VMWare problems - physical port works, vmx1 does not?

[just_here]

I'm running OPNSense 23.1_6 on an N5105 with 6 Intel i225-V's. It's running VMWare ESXi 8.0 with NICs in PCI passthrough mode to the OPNSense VM.

There is also a VM running a clean Debian 11.6 install, I want this VM to join the "domotica" network on my OPNSense install.

So I have:
- an interface ETH5_Domotica (opt5, igc1) on a passthrough NIC. A RPi is connected to this port and functions perfect.
- an interface ESXI_Domotica (opt7, vmx0) which is a VMWare NIC.
- a bridge BG_Domotica (bridge1) between ETH5_Domotica and ESXI_Domotica
- an interface Domotica (opt8, bridge1) on the bridge

Both ETH5_Domotica and ESXI_Domotica are enabled but have no IP set or DHCP server running. They both have a single firewall rule that allows all IPv4 traffic (I do not know if this is necessary).

The Domotica interface (on the bridge) is configured with a static IP 172.16.30.1(/21) and has a DHCP service enabled, providing leases between 172.16.30.100 and 172.16.31.250. It has a single firewall rule that allows all IPv4 traffic (I know this is necessary).

The raspberry pi connected on the physical ETH5_Domotica port works great, it can release/renew DHCP address and reach the internet as well as other clients.

In VMWare ESXi the interface ESXI_Domotica (from the OPNSense VM) is connected to a dedicated Domotica port group and vSwitch. Both of these allow promiscuous mode and mac address changes (I do not know for sure this is required).

The Debian 11.6 VM also has a network adapter which is connected to the Domotica port group / vSwitch.
The Debian 11.6 VM cannot acquire an IP address. When I set it to a static IP in the subnet (172.16.30.123), with the correct subnet (255.255.248.0) and gateway (172.16.30.1) it cannot reach the gateway (which is OPNSense).

I only see 1 (pass) message in Firewall->Log files->Live view that pass/block packets coming from anything other than the Raspberry Pi. But it's IPv6, so, no IPv4 traffic seemingly.

Code: [Select]

ESXI_DOMOTICA 2023-02-14T11:07:30 fe80::20c:29ff:fe31:3487 ff02::2 ipv6-icmp IPv6 RFC4890 requirements (ICMP)

I have tried:
- Setting a static IP on the Debian install
  - Ping: "Destination Host Unreachable"
  - Telnet: "No route to host"
- Setting dhcp: a private IP address is assigned after a long wait (169.x.x.x)

I have no clue why the Debian VM is just completely unable to see OPNSense 

[just_here]

Okay, so my problem is "fixed".

Problem is that "Forged Transmits" are disabled by default in VMWare ESXi 8.0. When you create a bridge like I did, the "ESXI_Domotica" interface will have a mac address that ESXi sees (e.g. FF:B8). But there is a bridge on this interface, and that bridge also has a mac address (e.g. FF:A7).
The DHCP server runs on the bridge's interface, so responses will originate from _that_ mac address (FF:A7).
ESXi will see the DHCP response from this mac address (FF:A7), and conclude it's not from the mac address that "ESXI_Domotica" has (FF:B8), and thus categorise it as a forged transmit.

As soon as you allow forged transmits, everything works. 

I don't really like this "solution" though, so I think the conclusion is I'm going to learn more about either:
- Going toward VLANs
- Deleting the bridge and just setting up a seperate subnet on each interface so all routing has to go across the firewall and the DHCP responses originate from the correct MAC address.

[Patrick M. Hausen]

What is the exact reason you are using this bridged construct? You can use a passthrough interface in OPNsense (which I would recommend) and attach all VMs in ESXi to the same LAN/VLAN by the virtue of ESXi and your switch. There is rarely a reason your OPNsense would need a virtual interface if you have enough ports to go with passthrough.

[just_here]

Functionally: I want to setup a subnet/vlan for my IOT / Domotica devices.

Technically: I have a VM running HomeAssistant that needs to be in the same subnet/vlan as a physical device (RPi4) on a physical port.

I could choose to not passthrough one NIC to OPNSense, but just leave it in VMWare, add it to the switch and use that for my RPi. This would also function as a "bridge" (a switch).

However, I would like to have as much as possible control in OPNSense. the only reason I'm reaching out to ESXi's vSwitch's is because I have VM's running that also need to plugin to OPNSense.

Would you recommend against this setup?

[Patrick M. Hausen]

Passthrough one device to OPNsense, create VLANs as necessary in OPNsense, configure matching switch port as trunk port. Create VLANs and matching port groups in ESXi for ESXi main interface, configure matching switch port as trunk port. Configure switch ports with port based VLAN assignmnents as necessary for devices. Done.

Yes, in that scenario all traffic from VMs to OPNsense and further (Internet, inter VLAN) goes through the switch. Is that a problem with (probably) Gigabit? The software bridge in OPNsense will most probably perform worse.

HTH,
Patrick

[just_here]

Thanks a bunch for your answers.

Yeah, I suspected that what I wanted was a bit to specific / unusual. Your setup sounds a lot simpler (as long as ESXi works properly).

All (6) NIC's are Intel i225-V 2.5gbit, and the internet connection is a 1 gigabit up/down FttH connection. The firewall is running on a Intel Celeron N5105 (4 cores, 2GHz, 4mb cache). So it should be able to handle the gbit connection (this is a must for me). I don't have a managed / trunk capable switch since I'm trying to run a low energy setup (less devices = better). The OPNSense setup also replaces my FttH providers modem.

Physical ports:
1: ESXi Management
2: WAN (separate fiber to ethernet media converter, so just normal ethernet coming in)
3: LAN
4: LAN backup
5: Trunk port for Unifi access points with 4 vlans (LAN, Isolation, Domotica, Guest)
6: Domotica port for RPi next to firewall for power monitoring

Do you see any troubles there with the bridge setup vs the ESXi vSwitch setup there?

[Patrick M. Hausen]

Yes, your bridge setup doesn't work as well, does it? So use the setup that does work as intended. ESXi vSwitches are vastly superior in terms if features, performance, reliability, to the FreeBSD bridge.