Messages

1

23.1 Legacy Series / Re: Bridge / VMWare problems - physical port works, vmx1 does not?

« on: February 16, 2023, 07:35:52 am »

Thanks a bunch for your answers.

Yeah, I suspected that what I wanted was a bit to specific / unusual. Your setup sounds a lot simpler (as long as ESXi works properly).

All (6) NIC's are Intel i225-V 2.5gbit, and the internet connection is a 1 gigabit up/down FttH connection. The firewall is running on a Intel Celeron N5105 (4 cores, 2GHz, 4mb cache). So it should be able to handle the gbit connection (this is a must for me). I don't have a managed / trunk capable switch since I'm trying to run a low energy setup (less devices = better). The OPNSense setup also replaces my FttH providers modem.

Physical ports:
1: ESXi Management
2: WAN (separate fiber to ethernet media converter, so just normal ethernet coming in)
3: LAN
4: LAN backup
5: Trunk port for Unifi access points with 4 vlans (LAN, Isolation, Domotica, Guest)
6: Domotica port for RPi next to firewall for power monitoring

Do you see any troubles there with the bridge setup vs the ESXi vSwitch setup there?

2

23.1 Legacy Series / Re: Bridge / VMWare problems - physical port works, vmx1 does not?

« on: February 15, 2023, 11:12:59 pm »

Functionally: I want to setup a subnet/vlan for my IOT / Domotica devices.

Technically: I have a VM running HomeAssistant that needs to be in the same subnet/vlan as a physical device (RPi4) on a physical port.

I could choose to not passthrough one NIC to OPNSense, but just leave it in VMWare, add it to the switch and use that for my RPi. This would also function as a "bridge" (a switch).

However, I would like to have as much as possible control in OPNSense. the only reason I'm reaching out to ESXi's vSwitch's is because I have VM's running that also need to plugin to OPNSense.

Would you recommend against this setup?

3

23.1 Legacy Series / Re: Bridge / VMWare problems - physical port works, vmx1 does not?

« on: February 15, 2023, 10:33:26 pm »

Okay, so my problem is "fixed".

Problem is that "Forged Transmits" are disabled by default in VMWare ESXi 8.0. When you create a bridge like I did, the "ESXI_Domotica" interface will have a mac address that ESXi sees (e.g. FF:B8). But there is a bridge on this interface, and that bridge also has a mac address (e.g. FF:A7).
The DHCP server runs on the bridge's interface, so responses will originate from _that_ mac address (FF:A7).
ESXi will see the DHCP response from this mac address (FF:A7), and conclude it's not from the mac address that "ESXI_Domotica" has (FF:B8), and thus categorise it as a forged transmit.

As soon as you allow forged transmits, everything works. 

I don't really like this "solution" though, so I think the conclusion is I'm going to learn more about either:
- Going toward VLANs
- Deleting the bridge and just setting up a seperate subnet on each interface so all routing has to go across the firewall and the DHCP responses originate from the correct MAC address.

4

23.1 Legacy Series / Bridge / VMWare problems - physical port works, vmx1 does not?

« on: February 14, 2023, 11:41:17 am »

I'm running OPNSense 23.1_6 on an N5105 with 6 Intel i225-V's. It's running VMWare ESXi 8.0 with NICs in PCI passthrough mode to the OPNSense VM.

There is also a VM running a clean Debian 11.6 install, I want this VM to join the "domotica" network on my OPNSense install.

So I have:
- an interface ETH5_Domotica (opt5, igc1) on a passthrough NIC. A RPi is connected to this port and functions perfect.
- an interface ESXI_Domotica (opt7, vmx0) which is a VMWare NIC.
- a bridge BG_Domotica (bridge1) between ETH5_Domotica and ESXI_Domotica
- an interface Domotica (opt8, bridge1) on the bridge

Both ETH5_Domotica and ESXI_Domotica are enabled but have no IP set or DHCP server running. They both have a single firewall rule that allows all IPv4 traffic (I do not know if this is necessary).

The Domotica interface (on the bridge) is configured with a static IP 172.16.30.1(/21) and has a DHCP service enabled, providing leases between 172.16.30.100 and 172.16.31.250. It has a single firewall rule that allows all IPv4 traffic (I know this is necessary).

The raspberry pi connected on the physical ETH5_Domotica port works great, it can release/renew DHCP address and reach the internet as well as other clients.

In VMWare ESXi the interface ESXI_Domotica (from the OPNSense VM) is connected to a dedicated Domotica port group and vSwitch. Both of these allow promiscuous mode and mac address changes (I do not know for sure this is required).

The Debian 11.6 VM also has a network adapter which is connected to the Domotica port group / vSwitch.
The Debian 11.6 VM cannot acquire an IP address. When I set it to a static IP in the subnet (172.16.30.123), with the correct subnet (255.255.248.0) and gateway (172.16.30.1) it cannot reach the gateway (which is OPNSense).

I only see 1 (pass) message in Firewall->Log files->Live view that pass/block packets coming from anything other than the Raspberry Pi. But it's IPv6, so, no IPv4 traffic seemingly.

Code: [Select]

ESXI_DOMOTICA 2023-02-14T11:07:30 fe80::20c:29ff:fe31:3487 ff02::2 ipv6-icmp IPv6 RFC4890 requirements (ICMP)

I have tried:
- Setting a static IP on the Debian install
  - Ping: "Destination Host Unreachable"
  - Telnet: "No route to host"
- Setting dhcp: a private IP address is assigned after a long wait (169.x.x.x)

I have no clue why the Debian VM is just completely unable to see OPNSense 

5

22.7 Legacy Series / Re: N5105 - ESXI 8.0 - I225-V passthrough - OPNSense 22.7.6 - "no carrier"

« on: November 10, 2022, 09:29:54 pm »

Apparently the fibre <-> ethernet media converter provided by my ISP is of low quality and actually crashed. Turning it off and on again brought up the interface. I hope for longer than 30 minutes this time 

6

22.7 Legacy Series / N5105 - ESXI 8.0 - I225-V passthrough - OPNSense 22.7.6 - "no carrier"

« on: November 10, 2022, 09:03:10 pm »

I bought an Intel Celeron N5105 with 6x Intel I225-V NICs (Topton). I run a VMWare ESXi hypervisor with 1 virtual machine, OPNSense. Versions:

Code: [Select]

VMWare ESXI 8.0
OPNsense 22.7.6-amd64
FreeBSD 13.1-RELEASE-p2
OpenSSL 1.1.1q 5 Jul 202

5 of my NICs have been passthroughed to the OPNSense VM. The first (not passthroughed) NIC is in use as ESXI management interface.

I have FttH internet, using a media converter. So my first port (igc2 in this case) is connected to a fibre -> ethernet media converter proven to work. My ISP sends internet through on VLAN 300, so I made the following setup:

vlan01 (tag 300) on igc2 as wan.
bg01 (bridge) on igc0 en igc4 (NICs of physical port 2 and 3)
bg01 set as lan.
ETH1_WAN on igc2, turned on but everything default/off as it's in a vlan.
ETH2_LAN (igc0) + ETH3_LAN (igc4), turned on, but everything default/off as it's in a bridge.

Everything works, everybody happy.
After 20-30 minutes: poof. WAN gone.

OPSense: both igc2 and vlan01 -> "no carrier".

I've tried:
OPNSense restart
ESXI restart
Actual shutdown and startup of the physical machine.

DHCP server and ipv4 adres are only on the LAN interface (bg0), not on igc0 and igc4.
DHCP client is only on vlan01 for ISP / internet.

Any pointers?