ALL physical interfaces no longer detected by OS on reboot

[JdeFalconr]

Preface: this situation is very likely less bug and more my own fault. I am not laying blame on anyone but myself. I wanted to bring this up, though, as it does seem like odd behavior.

I am making my first foray into virtualization (ESXi 6.7) and I'm working to virtualize my Opnsense install. I did a fresh install, imported my configuration just fine from my former physical install, and was up and running. I had an extraneous interface that I wanted to remove and so I powered off the VM and removed the extra virtual NIC, then powered back on.

Lo and behold, when it booted I was greeted with a message that no interfaces were assigned. I could not log in as root and my normal admin credentials got me in but the familiar console/SSH menu did not come up and I couldn't figure out how to bring it up. My prompt was just a dollar sign and I couldn't sudo either. I was completely locked out and had no means to reassign my interfaces and get up and running again. Even the password reset tool on the install image threw errors and failed to run.

So I get that maybe I missed steps and didn't cleanly remove the interface in Opnsense. But I find it exceedingly odd that the aforementioned actions would affect ALL interfaces and cause them to become unassigned, their configurations wiped from existence. Like I said this is all very likely my fault but it still seems quite bizarre.

EDIT: WOW. This happened again except this time I'm fairly sure I did things right. I went into Interfaces / Assignments and deleted the interface there. I did not remove it from the VM. On the next reboot BAM, no interfaces assigned. In fact it flat-out doesn't even look list my interfaces like the VM thinks the hardware isn't there at all. Restoring from one of Opnsense's backups (menu option 13) doesn't help (this time I gave myself SSH/Console access). Time to revert to a snapshot. Thank God for virtual machines.

[mimugmail]

This is a general BSD problem as it doesnt use UUIDs like Linux. If you add/remove adapters you have to reassign or manually edit the config.xml

[JdeFalconr]

Thanks for the reply. That is definitely a problem. So what are the steps to remove an adapter without hosing my system? They aren't even detected by the OS after reboot so I can't reassign. Is it simply not possible?

[mimugmail]

So ifconfig doesnt detect any Interface anymore?

[JdeFalconr]

I didn't think to try that (not much of a Linux guy) though I should have. At boot the console shows only a single interface that isn't one of the original ones and says no interfaces are assigned. If this happens again I'll do an IFConfig. I appreciate the suggestion.

[JdeFalconr]

So ifconfig doesnt detect any Interface anymore?

This problem happened again, this time without any interface modifications. No idea what's going on here but clearly this install is somehow unstable after a reboot. An IFConfig shows no physical interfaces whatsoever, only Lo0, ENC0, pflog0, pfsync0 and ovpns1. That last one looks to have retained its config somehow. My two physical interfaces are em0 and em1.

Could I have my VM config messed up somehow? I've gotta get this fixed, I can't have my router be so unreliable.

[JdeFalconr]

Hah I may have found the problem. Based on something I saw about PCI interrupts not being available from the boot process I Google and found https://forums.freebsd.org/threads/vmxnet3-nic-disappeared-after-upgrade-to-freebsd-11-and-pci-e-errors-in-dmesg.58037/. Sure enough in /boot/loader.conf the value hw.pci.enable_msi was set to zero. Changing it to 1 brought back my NICs. I looked and sure enough in my config I had imported from my physical box that value was present under tunables and set to zero. I remedied that promptly.

Is there any harm in clearing out the Tunables section entirely of my config before I import it, just for good measure?