OPNsense on Proxmox > Can't get DHCP over physical NIC to work

[quqide]

Hi all,

I recently decided to migrate from a UniFi Dream Machine to OPNsense running as a VM on Proxmox.

OPNsense itself is working fine, and I've already moved several VMs onto the new OPNsense LAN bridge in Proxmox. Those VMs receive IP addresses via DHCP without any issues.

However, I'm running into a problem when I try to connect a physical device directly to the NIC that is attached to the OPNsense LAN bridge:

• DHCP does not work on the physical NIC
• If I configure a static IP on the physical device, I can reach OPNsense and the rest of the network just fine
• This makes it appear to be a DHCP-only issue

My setup in short with 4 NICs total

• SFP+ #1 and #2: bonded and used for VMs (still connected to UniFi)
• RJ45 #1: OPNsense WAN (currently connected to an unused VLAN on UniFi; when connected directly to my ISP WAN, DHCP works fine)
• RJ45 #2: OPNsense LAN
• 10.27.10.0/24 old Subnet (UniFi)
• 10.27.11.0/24 (New LAN Subnet; currently Proxmox has no IP assigned to the vmbr but I can reach it via static IP on the Client)

VMs connected to the corresponding vmbr receive DHCP leases correctly
Physical devices connected to this NIC do not receive DHCP leases

Proxmox Nework config:

Code Select Expand

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto enp87s0
iface enp87s0 inet manual

auto enp90s0
iface enp90s0 inet manual

auto enp2s0f0np0
iface enp2s0f0np0 inet manual

auto enp2s0f1np1
iface enp2s0f1np1 inet manual

iface wlp91s0 inet manual

auto bond0
iface bond0 inet manual
bond-slaves enp2s0f0np0 enp2s0f1np1
bond-miimon 100
bond-mode balance-rr

auto vmbr1
iface vmbr1 inet static
address 10.27.10.20/24
gateway 10.27.10.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
bridge-mcsnoop 0
#VMs

auto vmbr3
iface vmbr3 inet manual
bridge-ports enp90s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
bridge-mcsnoop 0
#OPNsense LAN

auto vmbr0
iface vmbr0 inet manual
bridge-ports enp87s0
bridge-stp off
bridge-fd 0
bridge-mcsnoop 0
#OPNsense WAN

source /etc/network/interfaces.d/*

As DHCP works on VMs connected to the bridge I don't think it's a OPNsense configuration issue but a Proxmox NIC issue.
In the OPNsense log I can see it handing out DHCP offers. On Proxmox the Firewall on those to bridges is off.

When doing a TCP dump on the NIC on Proxmox:

Code Select Expand

root@proxmox:~# tcpdump -i enp90s0 -n port 67 or 68
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp90s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:05:39.455550 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:e0:4c:8f:b7:91, length 300
09:05:42.742892 IP 10.27.11.1.67 > 10.27.11.133.68: BOOTP/DHCP, Reply, length 306
09:05:44.211604 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:e0:4c:8f:b7:91, length 300
09:05:44.212508 IP 10.27.11.1.67 > 10.27.11.133.68: BOOTP/DHCP, Reply, length 306
09:05:52.962341 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:e0:4c:8f:b7:91, length 300
09:05:52.963488 IP 10.27.11.1.67 > 10.27.11.133.68: BOOTP/DHCP, Reply, length 306
09:06:09.456204 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:e0:4c:8f:b7:91, length 300
09:06:09.457166 IP 10.27.11.1.67 > 10.27.11.133.68: BOOTP/DHCP, Reply, length 306
09:06:41.201306 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:e0:4c:8f:b7:91, length 300
09:06:44.495939 IP 10.27.11.1.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 306
09:06:45.700361 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:e0:4c:8f:b7:91, length 300
09:06:45.701570 IP 10.27.11.1.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 306
09:06:53.944040 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:e0:4c:8f:b7:91, length 300
09:06:53.944912 IP 10.27.11.1.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 306
09:07:09.453246 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:e0:4c:8f:b7:91, length 300
09:07:09.454095 IP 10.27.11.1.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 306

I tried with Google and ChatGPT but can't get it to work. Current thinking is that something gets blocked by Proxmox - thats why I added

Code Select Expand

bridge-mcsnoop 0 to the config - but nothing helps.
I tried multiple different Clients. Also, on a Windows Client, it succesfully set the DHCP local domain name like home.arpa but nothing else.

Thank you!

[meyergru]

Two basic things:

1. In order for passthru to work, you must have exclusive access in the VM, so configuring the NIC on the PVE host is a no-go. There are lots of prerequisites to do this. It cannot be successful if you do not see the NIC in the OpnSense VM.

2. This worked under Proxmox with a virtio adapter? Then Linux is fine handling your Realtek device. How do I know it is Realtek? Because your MAC OUI tells me so. See this, point 6.

I generally do not recommend using passthru unless strictly neccessary, see: https://forum.opnsense.org/index.php?topic=44159.0. bridge-mcsnoop is explained there, too. And as a general rule: Do not use AI or YT videos to learn about OpnSense. There is plenty of ggod documentation and also the tutorial section of this forum.

[quqide]

Thanks, I took a look at your guide as well whilst troubleshooting, whats where I got mcsnoop from. I hope I didn't misinterpret your guide.

I am doing bridging with the LAN and WAN Interface I am using virtio for the VM, so in OPNsense I have my two vtnet adapters which I assigned.
So I should not be using PCI passthrough.

The NIC controller of my two RJ45 is an Intel i226-lm according to Proxmox hardware infos. But that should not mater as I am bridging both NICs from the Proxmox host. At least that is what I wanted to do to not use PCI Passthrogh.

[meyergru]

So your problem is that your OpnSense has LAN on vmbr3 aka enp90s0 and it should act as a DHCP server (which it does according to your packet traces) but your clients attached to that NIC (via a switch?) do not get DHCP IPs assigned?

If the NIC is not attached directly to a client, but via a Unifi switch or the dream machine itself, you should know that there are settings in Unifi to block non-legit DHCP servers. Maybe your Dream Machine is the only allowed DHCP server. In order to verify, try to attach a client directly to that NIC (make sure to use the correct one).

Also, since your PVE host is already attached via some bonded SFP+ NICs, I would rather use those instead of another NIC. You can distribute VLANs over that NIC(s) to do that. Besides that, I am not a huge fan of bonding except for reliability. With Unifi equipment, you will gain no more throughput via bonding, because most Unifi hardware does not support "real" load-balancing. I think that balance-rr would not work in the way you think it does.

[quqide]

Exactly, vmbr3 is my current OPNsense LAN which correlates with enp90s0. Theres no UniFi gear attached to this NIC; just my MacBook directly (also tried Windows Laptop).

In the OPNsense log I can see it offering IPs to the clients but they don't use them, or their answer packet gets missing or something else.

The bond is currently still connected to UniFi as I am in "productive use" and I get killed by my wife if I unplug it for longer periods because of our Home Assistant VM and DNS Server running over that :P.
Thats why I created vmbr3 to test everything. My plan was to switch to the bond, as you suggested, once everything works and unplug the dream machine completely.

But as of now my OPNsense LAN on vmbr3 has absolutely no connection to any UniFi gear and should be handing out IP-addresses.
And the real funny thing is that when I connect vmbr3 to any VM ON Proxmox for testing purposes it DOES GET an IP-address. It's just the physical NIC thats annoying me.

Another fun fact: My Vodafone Cable ISP is handing out public IP-addresses over DHCP. My WAN bridge (vmbr0 / enp87s0) is configured like the LAN bridge. Only difference is the DHCP packets come IN to the WAN interface from Vodafone to OPNsense and that works (tested it while wife is out of the house). Just DHCP OUT gets caught up somewhere.