ESXi 6.5.0 U1 booting goes very wrong

[weust]

I'm messing around with ESXi 6.5.0 U1 at home, and created a VM based on VM version 13 (6.5 and later) using mostly basic settings, except I removed the USB controller and added a second NIC (vmxnet3).
OS set to Other, FreeBSD (64-bit).
Left the SCSI controller at LSI Logic Parallel, because FreeBSD still does not support the VMware Paravirtual controller.

Booting the OPNsense-17.7.5-OpenSSL-dvd-amd64.iso, and installing from it using GPT/UEFI is fine too, but the first reboot messes up the VM in such a way that the ESXi web console shows the VM as powered off, but you can't power it on in any way.
Even the console can't reach it ("esxcli vm process list" only shows running VM's), so a host reboot is the only way to reach it again.

Maybe I've searched wrong, but I can't find anything related to FreeBSD 11 and this. I did read something about FreeBSD 10 on a VMware page, but EFI is almost fully supported in 11...

Anyone tried this yet?
Going to try a BIOS/MBR installation next. See if that works.

[weust]

Same issues with a BIOS/MBR install, but the ESXi events mentioned doing an unregister and register if the config was alright. Weird.
But doing that did make the VM boot normally...

Going to try that for EFI/GPT/UEFI now as well.

[franco]

Hi weust,

I've heard about this, could be a vt(4) driver issue... Try unsetting the vt(4) driver usage from System: Settings: Administration and see if that survives the first reboot.

A fix to try could be here: https://reviews.freebsd.org/D9709


Cheers,
Franco

[BertM]

weust,

I have over a dozen virtual OPNsense instances running on several VMware hosts (6.0 and 6.5) without any problems, but.......I tend to play safe.
All these OPNsense instances run Rock Solid.
The largest OPNsense instance has 9 Network Cards, runs 84 IPsec Site-to-Site VPNs and provides network services for 25 Windows servers, 12 Linux servers and over 300 workstations.

When I setup an OPNsense virtual machine, I use the following settings:
Compatibility "ESXi 6.0 and later (VM version 11)"
SCSI controller "LSI Logic Parallel"
Network Adapters "E1000"
Guest OS "Other ==> FreeBSD (64-bit)".

During the installation of OPNsense, I just take all the defaults.

Setting things up this way never gives me problems.
In general, I try to stay away from the vmxnet3 network cards, because they also gave me issues in several other Guest OS-es (like messing-up the network after VMware tools update, etc.)

If you setup like this, does it still give you problems?

Kind regards,
Bert

[weust]

@franco, that would mean I could actually boot so far that I can access the webpage.
It doesn't seem to boot far enough that I can even ping it.

@BertM, I will give that a try later today.

At work we use vmxnet3 for all servers and VDI's. About 2000 VM's in total, across ESXi 5, 5.5 and 6.5 (we started with 6.0 for the VDI's, but moved on to 6.5 U1 during a POC).
Servers are a mix of 2003 (almost gone, finally), 2008 R2 and 2012 R2. Plus Windows 7 and Windows 10 VM's.
Never a problem so far.
Also 300 or so Linux VM's.

Odd thing is, the ISO boots fine with vmxnet3, install is fine too.

[weust]

Finally got around to test with a new VM.
ESXi 6.0, LSI Logic Parallel, E1000.

One first boot, same issues.
Makes me believe something is wrong with the network card I use.
It's a Qlogic QLE3442-CU-CK for which there are no 6.5 drivers, but there are 6.0 drivers.

Going to try a Intel card I have laying around. See what that thing does.

Also, will try Debian first. See if that has any issues too.

[weust]

Just noticed that when I create a new VM and set to Linux and Debian 9 64 bit, it will use the VMXnet3 NIC.
Guess even VMware thinks FreeBSD isn't ready for VMXnet3 :-)

[phoenix]

Guess even VMware thinks FreeBSD isn't ready for VMXnet3 :-)

Vsphere/ESXi 6.5 supports VMXNET3 for OPNsense.

[weust]

Yeah, but not on my hardware :-)

Just installed Debian 9.
Same issue.
Replacing NIC now.

[weust]

That didn't help either.
At least the Intel NIC was recognizes out of the box by ESXi 6.5.

No clue what the problem is.
Doing one more test with the onboard NIC (simple 1GbE by Intel) and then I'll mess with something else.

[BertM]

weust,

I have no experience at all with the Q-Logic cards.
When I run OPNsense under VMware, I mostly do that on HP Proliant hardware with the following network cards:

• Broadcom NetXtreme BCM5719
• Broadcom NC382i Integrated Multi Port PCI Express
• Intel I350
• Intel 82571EB

In two cases, for desaster recovery purposes, I have been running OPNsense in VMware on Apple Mac Mini server, using both the internal Broadcom NetXtreme BCM57766 and the Thunderbolt connected Broadcom NetXtreme BCM57762

So at least I know that these cards work OK.

Kind regards,
Bert

[weust]

I used this Qlogic card model in lots of servers when they were new.
But all in Windows Server 2008 R2 and 2012 (non-R2).

Great cards, because unlike the Intel cards you can setup the vNIC in the BIOS and make them permanent from there.
Just don't get the HP branded models, because they stripped that part out. Even from the driver!

My main (home use) OPNsense runs in a Gen2 VM on Hyper-V 2016. Rock solid.