OPNSense VM on Proxmox shuffles network mac addresses

[JL]

For the n-th time I found Proxmox+OPNSense to get into trouble.

Each time it takes a long time to get to the issue which is essentially 'rewriting' interface names.
Yes, I did not bother to go look for configuration files, yet, there was no indication the config was wrong.
Neither for Proxmox neither for OPNSense the config changed in recent months.

What appears to happen and is observed as a progressive issue is ... first some disruptions, eventually in Proxmox the OPNSense VM shows normal config on the outside but on the inside the interface MAC from the VM is assigned to a different interface or as a secondary MAC (yeah, i know)

I'm now swithching the WAN from virtio to e1000 to see if that makes any difference.

To me this is a mind numbing issue. At one point the OPNSense admin interface was no longer reachable while this is over a dedicated interface with a direct connection over UTP.

The fix was to use a Proxmox feature to rename the interfaces, last time I did something similar manually, this also worked.

Specific to OPNSense I'm worried Proxmox can simply reassign MAC and this doesn't revert, for OPNSense to allow for multiple MAC or a MAC which is not matching the configured vtnetX interface is concerning.

Hoping for your feedback, available to work on analysis if possible.

[meyergru]

First, Proxmox changes its scheme to name network interfaces once in a while - this is mainly because of Debian / systemd releases changing it.

If you want to have predictable names, you can follow the advice given here:

https://systemd.io/PREDICTABLE_INTERFACE_NAMES/

I always use net.ifnames=0 on the kernel commandline. With a Proxmox server, you rarely get into trouble with that, unless your network devices really change (like with USB adapters or when you switch hardware). Then, I only have to set eth0-X in /etc/network/interfaces. This has no effect on the VMs, because their MACs are assigned in the VM configuration and the device type is determined by the NIC emulation type, which does not change when you do that.


Second: When you use OpnSense as a VM, the virtual NICs are contained in the VM definition - actually, upon creation, a random MAC is being assigned to the interface. These MACs may change when you backup & restore a VM, especially, if you restore a "unique" instance. This is true regardless on which emulation you use for the NICs. It is different with PCIE passthru, since then, the MAC address is the physical adapter's MAC.

Besides: IDK if it is possible to use the "change MAC" feature from within an OpnSense VM - I would not even try that, but leave the control to Proxmox.


Also, FreeBSD sometimes renames drivers for NICs with a new release. That may mess up the interface assigments for the logical interfaces (say if em0 becomes igb0). The assignment will not switch without a new OpnSense release, however.


So, the chain is:

1. physical NIC. This has a name which depends on Proxmox / Debian / systemd versions. Best to fixate it.
2. Mapping of VM virtual NIC to Proxmox bridge interface - this determines what MAC is used and which emulation type of adapter is presented to the VM.
3. "physical" NIC in OpnSense: its name is dependend on the type of emulation and thus, the selected driver. The MAC is determined from outside and should not be changed from within OpnSense.

As for your "secondary" MAC question: As I said, I never tried to change a VM MAC from within. I do not even get in which situation you would have multiple MACs on the same interface (unless it is a bridge, but that does not directly map to a "physical" NIC).

[JL]

Thanks @meyerguru for the informative response, none of that is on topic in this case.

The Proxmox cluster was not given any changes, neither was OPNSense given any chances.
I'm familiar enough with both and with networking, OS and such to consider this a possible bug lurking.

On investigation of the outage I could only conclude the interfaces were shifted, meaning vtnet0 had two mac address (one mac and one ethernet address) and vtnet1 had the mac address of vtnet0, vnet2 had the mac address of vtnet1 and so on.

I remember seeing an unusual error on Promox networking some time ago but this did not result in any notable issue, that was weeks if not one or two months ago.

The reason I mention the renaming as a fix was because runinng the below command, resolved the issue.

Code Select Expand

pve-network-interface-pinning generate
This command renamed the interfaces from enpNNsNfN to nicN
That's it.

No other changes were required to fix an outage taking over 12 hours to (casually) analyse and attempt to pinpoint.
The fact the OPNSense VM interface were scrambled/shift concerns me as this suggests an OPNSense VM to be really weak.

I'm now preparing to shifting away from use of virtio to see if this offers more stability in the long run.
To memory this is the 2nd time this kind of workaround is required and I never see any cause in the logs.
No errors, no warnings etc.

[meyergru]

All of that was on topic, trust me ;-)

It will not change a bit unless you use passthru (and even then, you will find that a kernel change on Proxmox often renumbers the PCIe numbers). Using other emulation types will only give you yet other NIC names in OpnSense. It will not heal any Proxmox-related renumbering if the kernel or QEMU package changes. And yes, there are no warnings about that. New kernels just name interfaces another way (it has to do with how they enumerate PCI buses and interfaces).

Your Proxmox interface names have changed - for whatever reason, I listed a few - and those names are referenced in /etc/network/interfaces. Thus, if the names of the physical interfaces changed, the resulting assigments for the vmbrX bridges changed, too.

OpnSense has nothing to do with that, the VM accepts whatever interfaces are given to it from Proxmox and uses them in order as vtnetX or igbX, as decided by the emulation type. The fact that your MACs have changed number in order only illustrates this.

And yes, Proxmox is aware of the problem, so they have provided a script which does the heavy lifting and assigns fixed interface names to the MACs found at the time the script runs. This is one of approaches that are given in the systemd documentation I linked to. It certainly solves your problem until you change hardware.

[meyergru]

P.S.: To add another potential cause of renumbering: USB NICs. They are known to be detected dynamically, so the order of detection may change even without any software updates, purely for timing reasons.

(This USB-related problem is also the reason why ZFS does not rely on /dev/sdX, but chooses serial numbers or GPT UUIDs for disk identification. It is also problematic for USB serial adapter naming.)

The Proxmox pve-network-interface-pinning tool would help with USB NICs, too.