Strange shuffle of interfaces / PCIe-device-assignements after adding new HW

[Layer8]

Dear forum,

i am not sure if the reason for this issue is opnsense, freebsd, the current ryzen/amd plattform or maybe my ESXi hypervisor. It could also be a bit of everthing... i dont know.

I am running a VMware ESXi 7 (free licence) on a ryzen based system. 

Im struggling with this issue since days. The main problem is, that assignements between a interfaces / devicenames / pcie adresses seems to get shuffled after adding new NICs. I will tell you the possible reason for this behavour at the end of this post, but first of all i try to explain the problem.

First step.

I installed a fresh and clean opnsense 20.7.7 as a guest into a VMware ESXi (free) VM and initially added four vNICs which are configured as follows:



(1) physical NIC = (2) Standard vSwitch = (3) Portgroup = (4) VM Netzwork Adaptor Number = (5) PCIe ID = (6) FreeBSD device name  = (7) opnsense Interface = ( Interface config

vmnic3 = WAN = WAN = Network Adaptor 1 = unknown = vmx0 = WAN = PPPoE
vmnic2 = vSwitch0 (default) = LAN = Network Adaptor 2 = unknown = vmx1 = LAN = Static IPv4 .1.234/24 with DHCPserver
vmnic2 = vSwitch0 (default) = WiFi privat with VLAN ID 101 = Network Adaptor 3 = unknown = vmx2 = Static IPv4 .101.234/24 with DHCPserver
vmnic2 = vSwitch0 (default) = WiFi guest with VLAN ID 102 = Network Adaptor 4 = unknown = vmx3 = Static IPv4 .102.234/24 with DHCPserver

This initial setup is running without any problems (internet access, ip assignement through dhcp and so on). This setting is also working after shutdowns or reboots.

Second step.

I shuted down the opnsense and added a fifth and sixth vNIC like pictured




vmnic2 = vSwitch0 (default) = WiFi smart with VLAN ID 103 = Network Adaptor 5 = unknown = should be vmx4 = not yet configured
vmnic2 = vSwitch0 (default) = WiFi smart with VLAN ID 104 = Network Adaptor 6 = unknown = should be vmx5 = not yet configured

When i power on the opnsense guest VM, i can see through the console windows, that its booting fine till login screen,
i also can login through console and
i also can see the four intially assigned devices listed,
i also can see that PPPoE gets a Internet IP from my ISP and
i also can ping any internet IPs like 1.1.1.1,
but i cant open the webui and cant ping the ip over *.1.234  through my LAN anymore.

Observations:

I tried a lot of things to understand whats going and because its reproduceable i figured out the following.

When i remove the fifth and sixth Network Adaptor from the VM while it is running, this is printed in the console window:



So, if you compare my intial setup with only four attached NICs, how can it be that Network Adapter five is vmx1 and Network Adapter six is vmx3? This device names were initially assigned to Network Adaptor 2 and 4!

I booted opnsense with the intial setup from step one (with only four Network Adaptors) to reach the webui again. Then i attached Network Adapter five and six again while opnsense was running. This is the result:



As you can see, the PCIe-addresses and the devices names are completely different to the addresses from the detach-test. After attaching the devices to the running opnsense, its still possible to reach the webui. Also its possible to assign interfaces to vmx4 and vmx5 through webui. BUT: After a reboot, i can not reach opnsense again - same issue like in second step.


So, here is my theory:

I think, that PCIe-addresses can be assigned from the guest OS and its not given by the hypervisor/ESXi-environment or the VM-config file. After adding new hardware, the guest OS shuffles the PCIe-addresses with a to me unknown algorythm.


I am not 100% sure but i think this is a ryzen based issue (dont know if intel also supports a feature like this). On ryzen based systems, there is mostly a UEFI-setting which is called "PCIe ARI support" and/or "PCIe ARI enumeration". This options seems to allow the OS to address PCIe-devices itself. I think, after adding new hardware to the system, FreeBSD starts to resort the PCIe-adresses, which leads to different assignements between PCIe-address and PCIe device names.

By pure chance, a friend told me today, that he also has wired assignementproblems with his brand new ryzen based router hardware with opnsense as baremetal installation and without any hypervisor. Hes initial setup was with two onboard NICs and for the first days, the system was running fine. Some days ago, he added a four-port Intel NIC PCIe card. Since then, he struggled with opnsense in the same way as described above. I told him to disable  PCIe ARI option in the UEFI. Since then, he no longer has this wired NIC assignement problem.

Because my opnsense guest is already in production mode (switched from "add several vNICs" to "add only two NICs with VLAN assignment), i had no chance till now to see, if i can disable PCIe ARI in the guests UEFI or if this is a global setting in the mainboard UEFI, which can be used from guest OSs through AMD-V.



What do FreeBSD and OPNsense devs think about this issue?

Is there a mechanism in FreeBSD/OPNsense to keep the PCIe-address-assignements even after a user adds new hardware?

[SpawnY]

Hey Layer8,

iam so happy to see this topic!

I have already tested a few different scenarios with my new hardware. But in the end it seems to be the same/similar problem.
I have freshly installed OPNsense version 27.7 on the Ryzen hardware and noticed how the freshly declared interfaces change.
In the first setup LAN was set to my Intel Quad em3 port. However, after the first reboot, LAN was set to re0 (motherboard port).

Currently it seems to stay stable with the bios function "Disable PCIe ARI".

The next days will show if it works like this.
However, thanks for the hint / workaround!

Cheers Chris!

[Gauss23]

Known problem with vxnet drivers in BSD. Has nothing to do with Ryzen. Some weeks ago there was a thread about this topic where some was editing some file in ESXi and was able to keep the order this way.

[Layer8]

Thanks for this hint. Searched the forum for ESXi and found some topics. You mean this thread, right?

https://forum.opnsense.org/index.php?topic=19378

Additonal / simmilar topics i found:
https://forum.opnsense.org/index.php?topic=20049
https://forum.opnsense.org/index.php?topic=19159
https://forum.opnsense.org/index.php?topic=9058


I assume that this issue is NOT (only) a ESXi problem, but its a problem which seems to mainly occur in combination with ESXi:

In conclusion, under ESXi this issue occurs with both, em# and vmx# devices. em# devices are based on Intel E1000 1Gbit NICs, which means these are emulated NICs based on physical NICs, which means that you can use original Intel drivers for this kind of NICs (the same drivers which can be used for physical E1000 based NICs). vmx# devices are based on vmnet virtual devices delevoped by vmware. vmnet is designed as a virtual device from scrach. Both devices are completely different, so in my opinion, drivers or the devices itself cant be the reason for this issue (alone).

I also assume not any longer that it might be a ryzen only problem, but also a intel problem, because in the german thread OP has a Xeon E3-1240v6 based system.

The theory, that this should be a problem with ESXi also dont explain why my friend has/had this problems. Remember, he is running opnsense on a bare metal system, not in a VMware guest system.

I still think this might be a problem with adressing techniques of devices in virtual environments. Its obvious, that systems with hypervisors support these techniques in nearly all cases, while bare metal systems do not support these techniques necessarily.

@SpawnY : Whats your opnsense based on? Virtualised or bare metal?


Edit: I just found this reddit thread, where OP has the same problems bare metal:

https://www.reddit.com/r/PFSENSE/comments/6v9meo/nic_ordering_confusion/

[SpawnY]

Hey Layer8,

system is still stable.

Whats your opnsense based on? Virtualised or bare metal?

Its bare metal based.


I just saw the following item on the OPNsense roadmap:

• Fix stability and reliability issues with regard to vmx(4), vtnet(4), ixl(4), ix(4) and em(4) ethernet drivers.

Maybe this is related?

[Layer8]

I found more sources with informations about this topic:

https://communities.vmware.com/t5/Virtual-Machine-Guest-OS-and-VM/Rhel-6-nic-order-with-4-network-adapter-or-more/m-p/1793932
https://vnetwise.wordpress.com/2013/03/22/howto-force-vmware-esxi-to-re-order-nic-ports-to-maintain-standards/

https://kb.vmware.com/s/article/2019871

The  KB-article makes clear that this is not only a problem of virtual NICs (= the NICs used inside of a guest system), it is also a problem of the hypervisor/ESXi itself, and consequently also a problem of physical hardware in combination with FreeBSD (and maybe other OS?).

I also got the response that this is a problem of pfsense in a hyper-v setting too.

@SpawnY: Thanks for response and pointing us to this road map entry for january 2021!  This is very interesting. Hope one of the officials will tell us more about it.