More than 3 vNICs results in duplicate MACs

[MrGRID]

Hi

Opnsense newbie here, posting for the first time :-)

First up, environment details as follows:

Hypervisor: VMware 8.0U3 (fully patched to date)
Physical Server Hardware: AMD EPYC CPUs / NVIDIA ConnectX-7 & Intel X710 / NVME
Opnsense Version: 24.7.12-amd64 (fully patched to date)
FreeBSD Version: 14.1-RELEASE-p6

The Problem:

I've been running physical Fortigates for a few years, and after countless CVEs, licensing requirements and hardware becoming out of date restricting which OS I can upgrade to, I've finally had enough, and wanted to move to a virtualised environment where I'm in control of the hardware and can patch to the latest versions when needed. I've been running Opnsense in my home environment, and have gotten on with it quite well hence wanting to use it in a more complex and performant environment, but my home environment is very basic, and doesn't require vLANs or multiple NICs.

Having built the Opnsense VM, everything is fine with 3 vNICs, but when adding a 4th or greater vNIC, Opnsense starts duplicating MAC addresses (see attached "Duplicate MACs" image) and the GUI stops working. On the first 3 vNICs, I've manually assigned the MACs both in VMware on the VM settings, and within the Opnsense GUI, and allocated them to each interface, but this doesn't help at all.

The vNICs are using the VMXNET3 driver, and I have the "1.5_1 os-vmware tools" installed in Opnsense. If I add a 4th or greater vNIC using the E1000E driver, then the issue goes away, and I can add as many vNICs as I want and everything works with no duplicate MACs (see attached "No Duplicate MACs" image).

Could this be a problem with the vm-tools driver available to Opnsense / FreeBSD? It's normally vm-tools that adds the VMXNET3 functionality / capability.

I've seen similar posts where the solution was to use an ethernet pinning package that runs at startup (something like that, I can't find the exact reference now), but I'm not sure this is the same issue.

If anyone has experience of this and is able to offer some guidance, it would be greatly appreciated, and if it is a bug in the vm-tools package, how would we get that resolved?

Many thanks
MG

[bartjsmit]

I have seven VMXNET3 vNICs on my OPNsense vSphere guest, all with (different) automatic MAC addresses.

Did you assign different portgroups to them? Are they trunked through a single physical NIC?

[MrGRID]

Thanks for quick reply!

Yes, each vNIC belongs to a different PortGroup attached to the same Distributed vSwitch, trunked through a single physical NIC.

Is there a best practice for this type of configuration to avoid this kind of issue?

I just thought it odd that VMXNET3 exhibits this issue, but changing to E1000E resolves it.

Thanks
MG

[MrGRID]

With all vNICs using automatic MAC allocation, and all but 1 vNIC using the VMXNET3 driver, I still get a duplicate MAC on the VMXNET3 (see attached image), and the GUI (running on interface vmx1) stops responding.

I can't see any other way to configure this, and the only working solution I currently have is to use the E1000E driver, which I'd rather not do, but currently have no choice, and it isn't something I can just swap out later for VMXNET3 without manually allocating the MAC addresses, otherwise they'll change when I assign a VMXNET3 NIC.

Any suggestions welcome at this point.

Thanks
MG

[bimbar]

VMWare support?

[MrGRID]

Thanks, but I don't think it's a VMware issue. It's an Opnsense / FreeBSD issue.

The MAC addresses allocation in VMware is working as it should (individual MACs for each vNIC, there's nothing else for it to do). Even with manually allocated MACs in VMware for each vNIC, Opnsense duplicates a random MAC when using VMXNET3.

I'm running out of time as I've been looking at this since last week, and need to move forwards. I'm just going to build it with E1000E now that I know that works, then at some point if this ever gets resolved in an future update, I'll build another firewall using VMXNET3, and migrate across.

Thanks
MG

[MrGRID]

Hi

Just a quick follow up from my previous post ... I'm always happy to be proven wrong - even by myself! ...

The issue WAS in fact with VMware!

The built-in FreeBSD template is incorrectly configured! By default, it uses the (legacy) BIOS. If you change it to EFI, you can use VMXNET3 with multiple vNICs, and the MAC address issues do not occur (see attached images).

This was the last thing I could think of to try and it was really a shot in the dark. Glad it worked, wish by brain worked a little faster :-)

Hopefully that will save someone else a bit of time troubleshooting!

Thanks
MG

[franco]

WRT duplicated MAC addresses across VMs:

check that /etc/hostid is different for each VM otherwise it will generate the same MAC addresses for the same interface names. This is a bit of an odd behaviour in FreeBSD as we have previously seen.


Cheers,
Franco

[MrGRID]

Hi Franco

Thanks for the response.

Yes, I've checked the hostid and that's all fine. I've not experienced this issue with any other VMs before, but I've never run FreeBSD, so I think there must have just been something funky with the legacy "BIOS" option and running above a certain number of VMXNET3 interfaces.

This is the first time I've seen this issue, but I don't normally run this many vNICs on a single VM. Happy it's sorted though, and everything is now working as needed, so all good :-)

Thanks
MG