[SOLVED]Weird ARP entry originating from opnsense

[i.schmidt]

So, this seems really weird and i hope it's just some misunderstanding on my side but here is the story:

For some time we had weird troubles with DHCP and DHCP relaying not working really well. Sometimes the replys didn't reach the requesting device. Very frustrating.
I started a little digging and found, that the dhcp replys were not directed at the relay server on the opnsense, but at an unknown MAC Address, 00:00:5E:00:01:0B
Digging further i found that MAC Address on the switchport that is connected to the opnsense FW, BUT none of the interfaces on the FW own that MAC Address.
When i list all MAC addresses on the FW, there is no match, see attachment 1
So I tested if i could "reach" that MAC address in the hopes that it was just some stuck address on the switches ARP Table. To my surprise something really weird came up. I used arping to send ARP requests for an IP address on the FW, on a Net that my PC is directly connected to. The firewall responded. The TCP Header in the response shows SRC MAC address is the correct interface on the FW, but the ARP reply part of that packet lists the wrong, offending MAC address as destination for that IP. See attachment 2

I have absolutely NO IDEA WTF is happening here. Rebooting the FW didn't help. Next i will try HA failover to the other FW, to see if that phenomenon persists. If not, it might be a faulty Adapter. If it persists i am truly lost.

For some more understanding of the architecture of our Network:
The FW is a virtualised system on Proxmox. The LAN facing interface is a Mellanox 10G Adapter, directly connected to the VM via PCI passthrough.
On this interface, every LAN is a VLAN interface as a child of that Mellanox Card.
We use a HA setup. Both FW consist of the same Hardware.
Every IP address on the FW is either a CARP address or a n IP Alias (if we need more than one adress on that interface) except for the IP by which we reach the Web GUI, which is a fixed address, one for master, one for slave.

Please feel free to ask any questions if something is unclear.

[i.schmidt]

I checked if the weird MAC address switches to the other FW by doing a manual Failover and yes, now the other FW answers with this MAC address.

Maybe CARP uses this range of MAC addresses for decoupling the CARP address from the physical interface to enable instant migration to another host?

If so, the question remains, why sometime (a lot of times) DHCPOFFER packets don't arrive at the client, when the DHCP Server clearly sends them out.
There are no blocked UDP packets on UDP Port 67/68 on the interfaces.
Additionally: Why can't I see that MAC address on any interface?

[i.schmidt]

I found the cause of the issue.
There seems to be an issue with unreliable paket routing somewhere in the network stack of the virtualization Layer or somethin like that.
It seems like, sometimes VLAN Tags are not respected somewhere along the path between the client and the DHCP Server.
I fixed it by changing the way the DHCP Server is connected to the different VLANs. Now it has an Interface on every VLAN and we don't rely on relaying anymore. Works like a charm at the cost of some IP Addresses and configuring a bunch of interfaces.