Messages

I also reached an uptime of 16 days so far and still going, which clearly shows that the microcode update has either fixed or at least very much reduced the issue.

In similar vein to @misery's post, some noteworthy and recent additions to the Proxmox forum: https://forum.proxmox.com/threads/vm-freezes-irregularly.111494/post-550684

@Neuer_User, AdriftAtlas' experience might inform you (unless you are AdriftAtlas ;))...

Per R1CH's post immediately below AdriftAtlas' post, it is certainly somewhat risky to flash the BIOS (especially if not available from the supplier), but I don't agree with R1CH's assertion that loading the microcode following POST, i.e. in the OS, is of no difference in this case. Per my last post here, it is a mechanism to be independent of, and unaffected by, OS updates that do not include the needed microcode. However, R1CH's comments do likely apply to Debian based distros like Proxmox which typically include the whole microcode release package.

Do pay attention to the ChangWang descriptions. I didn't download the ISO to have a look as I didn't want to unblock port 8888, but by the description it seems to be for V3, V4 and V5 mainboards... The link @misery supplied through Google Translate: https://www-changwang-com.translate.goog/down/84.html?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en-US&_x_tr_pto=wapp, reveals it is the "chicken blood version" and it is "still an open version with full blood and full power consumption (please pay attention to heat dissipation by yourself)", but that it does fix the "crash restart problem in the case of a virtual machine"... YMMV...!

AdriftAtlas seems to have successfully updated his BIOS. That are good news. I am still testing, if the microcode update really solves the problem here. At the moment opnsense is running for about 3,5 days without reboot (and with tools installed). That is looking good, but to be sure the machine must run at least 10 days (better 20). If the problem is solved, I will probably try the BIOS update (but not before setting up a backup solution based on an older APU2 I have still laying around).

@Neuer_User, I'm curious what your original revision might be...

I have exactly the same revision (but a slightly different CPU, the N5105).

I had 8.0.1 running since yesterday, but to be honest, I had not much faith in it because I have seen instability on 6.7, 7.x and 8.0 already.

I don't know if the microcode is going to help but I just managed to update my cpu microcode under ESXi 8.0.1:

Steps on an ESXi host:

Download the correct microcode binary specific for your CPU (06-9c-00 binary for a Intel N6005 Pentium)

mv 06-9c-00 uc_intel
gzip uc_intel
cp uc_intel.gz /bootbank/uc_intel.b00
overwrite the file, reboot and:

Code Select Expand

[root@fanless-N6005:~] vsish -e cat /hardware/cpu/cpuList/0 | grep -i -E 'family|model|stepping|microcode|revision'
   Family:0x06
   Model:0x9c
   Stepping:0x00
   Number of microcode updates:1
   Original Revision:0x0000001d
   Current Revision:0x24000024

I have the same problem. I will try the microcode update and see, if it helps! (Thanks for posting the steps!)

Short Update: After about three days I had the same reset. Same entries in vmware.log. I was even working on the PC when it happened.

So, that clarifies a couple of things: 1.) It is NOT the vmware-tools. Altough with vmware-tools it seems to happen more frequently. 2.) It could indeed be the CPU microcode.

So, as a next test I added the new firmware to esxi 8.0 (not yet updated to 8.0.1). I also reactivate vmware-tools. Let's see how long it runs...  ::)

Wow, indeed! That is exactly my configuration and a VERY similar experience. Haven't thought that a microcode issue could result in VM-only freezes.  :o

I will definitely update the microcode and also esxi to version 8.0.1. I might also give the BIOS update a try. But the first thing is to complete the current test, i.e. running without vmware-tools. At the moment opnsense is running for 2 days and 5 hours without issues. That's definitely much better than my last tests with tools installed, but it is not guaranteed, as I also had a test with tools installed running for 6 days once.

But I will wait for at least two weeks if the machine freezes or shows any other irregularities. And then I'm gonna do the updates.

And, thanks really a lot for your help. That is so much appreciated!!!

Just having a look over your logs, I'm still seeing references to em0, an intel adapter.

Are you sure? I think you see the "em0_vlan" interfaces. They kept their old names, but they are just logical vlan interfaces reassigned to the "vm-physical" VMXNET3 nics.

References to "unsupported partitions" are likely red herrings.

These "unsupported partitions" are the tmpfs RAM-disks used by opnsense for /tmp and /var/log and others. I also saw these references. It is possible to exclude them from the vmware-tools access by some config options. But then that problem should be with every opnsense in vmware install. (opnsense does not install a vmware-tool.conf file.)

I would recommend a fresh install per my last post.

Yeah, I might give that a try.

EDIT: You may also need to exclude your tinc0 interface. I would setup any required VLANs as new VMXNET3 NICs and do the tagging at the vSwitch. My thoughts here are that RPC heartbeats are going out one of these interfaces, but I could well be clutching at straws...

Well, I guess with a new install I would probably not configure it at all. Just to see, if the resets also happen. But in the end, I, of course, need the tinc interface (as that is my vpn). For the heartbeats going out on a wrong interface: Everything may be possible, but it does indeed sound a bit as clutching at straws, as there are clear network routes in the system, none of which overlaps in any way with each other. Additionally, it is strange that only a few heartbeats would go the wrong way after some undefined timespan.

Now, let`s see how the system behaves without the os-vmware package.  ???

Thanks for the approach. I will give it a try and see what is the result. I guess, as a very first test, I will now try to let opnsense run without the vmware-tools. While it looks as if vmware-tools are the culprit, it is possible that the "heartbeat" messages are not the cause but only a side effect of the hard reset. So far, I haven't let the machine run without the tools for more than six days. So, first test is really to let it run for a longer time. If no reset happens, I can try to setup a parallel test vm like you suggested.

In case the vm also resets without the tools, I guess, I only have limited possibilities. Yes, I could setup a new opnsense installation, just to see if the installation could be somewhat damaged (which I currently do not believe), and if that doesn't help, I probably go the proxmox route. 

I'll bet you its CPU related.

Do you have option to run a XEON instead??

What do you mean? The "virtual CPU"? The host is running without any issues and is not restarting or doing anything weird. So, there does not seem to be any CPU issue for ESXi. Only the VM is being hard reset by ESXi pretty often.

Or do you mean that the opnsense kernel in the VM doesn't work nicely with the intel N5105 cpu and crashes or something like that? There is nothing in any log, but I'm not sure that one would see any entry. In such a case, I guess it should be possible to change the presented CPU features to the guest system somewhere?

Concerning other HW: No, I exactly purchased this appliance for this usage. I do have three small mini servers (running proxmox), but they only have two nics each and are actually strongly in use.

So, more bad news: After turing on debugging and having a hard reset again this night, there is no usable information in any of the created log files. All just show that the reset came out of the blue. ESXi again claims a missing heartbeat and therefore resets the vm.

I will first go back to a snapshot taken a few days ago (all these hard reset have apparently corrupted the rrd db). Then I will remove the vmware-plugin and see if the machine then runs fine or shows any other issues.

 :( :( :(

Ok, that was faster than expected. Hard reset happened already, now even with fully "recommended" HW settings. And a second hard reset only 7 min later. Same vmware.log entries. Here is the first reset:

Code Select Expand

2023-04-17T06:47:50.338Z In(05) vcpu-0 - VMXNET3 user: Ethernet1 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T06:47:50.338Z In(05) vcpu-0 - VMXNET3 user: Activate request succeeds for device 1.
2023-04-17T12:24:44.966Z In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.
2023-04-17T12:24:44.966Z In(05) vmx - Tools: [AppStatus] Last heartbeat value 90444 (last received 8s ago)
2023-04-17T12:24:44.966Z In(05) vmx - TOOLS: appName=toolbox, oldStatus=1, status=2, guestInitiated=0.
2023-04-17T12:24:45.209Z In(05) vcpu-0 - CPU reset: soft (mode Emulation)
2023-04-17T12:24:45.209Z In(05) vcpu-1 - CPU reset: soft (mode HV)
2023-04-17T12:24:45.209Z In(05) vcpu-3 - CPU reset: soft (mode HV)
2023-04-17T12:24:45.209Z In(05) vcpu-2 - CPU reset: soft (mode HV)
2023-04-17T12:24:45.211Z In(05) vcpu-0 - SVGA: Unregistering IOSpace at 0x1070
2023-04-17T12:24:45.211Z In(05) vcpu-0 - SVGA: Unregistering MemSpace at 0xe8000000(0xe8000000) and 0xfe000000(0xfe000000)
2023-04-17T12:24:45.211Z In(05) vcpu-0 - SVGA: FIFO is already mapped
...
2023-04-17T12:24:45.692Z In(05) vcpu-0 - Tools: ToolsRunningStatus_Reset, delayedRequest is 0x0
2023-04-17T12:24:45.692Z In(05) vcpu-0 - Tools: Changing running status: 1 => 0.
2023-04-17T12:24:45.692Z In(05) vcpu-0 - Tools: [RunningStatus] Last heartbeat value 90444 (last received 9s ago)
2023-04-17T12:24:45.692Z In(05) vcpu-0 - GuestLib Generated SessionId 15800366821342947488
2023-04-17T12:24:45.693Z In(05) vcpu-0 - GuestRpc: Reinitializing Channel 0(toolbox)
2023-04-17T12:24:45.693Z In(05) vcpu-0 - GuestMsg: Channel 0, Cannot unpost because the previous post is already completed
2023-04-17T12:24:45.693Z In(05) vcpu-0 - Tools: [AppStatus] Last heartbeat value 90444 (last received 9s ago)
2023-04-17T12:24:45.693Z In(05) vcpu-0 - TOOLS: appName=toolbox, oldStatus=2, status=0, guestInitiated=0.
2023-04-17T12:24:45.693Z In(05) vcpu-0 - TOOLS: appName=toolbox-dnd, oldStatus=0, status=0, guestInitiated=0.
2023-04-17T12:24:45.693Z In(05) vcpu-0 - VMXNET3 user: UPT support is not requested
2023-04-17T12:24:45.694Z In(05) vcpu-0 - Ethernet2 MAC Address: 00:0c:29:fe:1a:84
2023-04-17T12:24:45.694Z In(05) vcpu-0 - VMXNET3 user: Ethernet2 RSS fields requested by vmx: 3
2023-04-17T12:24:45.694Z In(05) vcpu-0 - VMXNET3 user: UPT support is not requested
2023-04-17T12:24:45.694Z In(05) vcpu-0 - Ethernet1 MAC Address: 00:0c:29:fe:1a:7a
2023-04-17T12:24:45.694Z In(05) vcpu-0 - VMXNET3 user: Ethernet1 RSS fields requested by vmx: 3
2023-04-17T12:24:45.694Z In(05) vcpu-0 - VMXNET3 user: UPT support is not requested
2023-04-17T12:24:45.694Z In(05) vcpu-0 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:24:45.694Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:24:45.694Z In(05) vcpu-0 - UHCI: HCReset
2023-04-17T12:24:45.694Z In(05) vcpu-0 - Checkpoint_Unstun: vm stopped for 458828 us
2023-04-17T12:24:45.695Z In(05) vcpu-0 - CPU reset: hard (mode Emulation)
2023-04-17T12:24:45.695Z In(05) vcpu-1 - CPU reset: hard (mode Emulation)
2023-04-17T12:24:45.695Z In(05) vcpu-3 - CPU reset: hard (mode Emulation)
2023-04-17T12:24:45.695Z In(05) vcpu-2 - CPU reset: hard (mode Emulation)
2023-04-17T12:24:45.696Z In(05) vcpu-0 - PVSCSI: scsi0: switching to async
2023-04-17T12:24:45.696Z In(05) vcpu-0 - Creating virtual dev for 'scsi0:0'.
2023-04-17T12:24:45.696Z In(05) vcpu-0 - DumpDiskInfo: scsi0:0 createType=11, capacity = 16777216, numLinks = 4
2023-04-17T12:24:45.696Z In(05) vcpu-0 - , allocationType = 2
2023-04-17T12:24:45.696Z In(05) vcpu-0 -
2023-04-17T12:24:45.696Z In(05) vcpu-0 - SCSIDiskESXPopulateVDevDesc: Using FS backend
2023-04-17T12:24:45.696Z In(05) vcpu-0 - DISKUTIL: scsi0:0 : geometry=1044/255/63
2023-04-17T12:24:45.696Z In(05) vcpu-0 - SCSIFilterESXAttachCBRCInt: CBRC not enabled or opened without filters,skipping CBRC           filter attach.
2023-04-17T12:24:45.696Z In(05) vcpu-0 - SCSIFilterESXAttachCBRCInt: Invalidating all hashes for this digest
2023-04-17T12:24:45.697Z In(05) vcpu-0 - SCSIFilterSBDAttachCBRC: device scsi0:0 is not SBD. Skipping CBRC attach SBD way.
2023-04-17T12:24:45.697Z In(05) vcpu-0 - DataSetsMgrSync: Reconnect dataSets store after checkpoint.
2023-04-17T12:24:45.697Z In(05) vcpu-0 - CPT: vm was stunned for 461410 us
2023-04-17T12:24:45.700Z In(05) vcpu-0 - OpNotificationMgr_RemoveAllApps: Attempting to remove registered apps during warm or cold reboot
...
2023-04-17T12:24:46.032Z In(05) vcpu-1 - CPU reset: soft (mode Emulation)
2023-04-17T12:24:46.033Z In(05) vcpu-2 - CPU reset: soft (mode Emulation)
2023-04-17T12:24:46.034Z In(05) vcpu-3 - CPU reset: soft (mode Emulation)
...
2023-04-17T12:24:52.526Z In(05) vcpu-2 - UHCI: HCReset
2023-04-17T12:24:52.626Z In(05) vcpu-2 - VMXNET3 user: UPT support is not requested
2023-04-17T12:24:52.626Z In(05) vcpu-2 - Ethernet2 MAC Address: 00:0c:29:fe:1a:84
2023-04-17T12:24:52.626Z In(05) vcpu-2 - VMXNET3 user: Ethernet2 RSS fields requested by vmx: 3
2023-04-17T12:24:52.626Z In(05) vcpu-2 - VMXNET3 user: Supported set 0x000000ff, configured set 0x00000001
2023-04-17T12:24:52.626Z In(05) vcpu-2 - Ethernet2 MAC Address: 00:0c:29:fe:1a:84
2023-04-17T12:24:52.626Z In(05) vcpu-2 - Ethernet2 MAC Address: 00:0c:29:fe:1a:84
2023-04-17T12:24:52.695Z In(05) vcpu-2 - PVSCSI: ReqRing: 1024 entries, eSz=128, 32 pages
2023-04-17T12:24:52.695Z In(05) vcpu-2 - PVSCSI: CmpRing: 4096 entries, eSz=32, 32 pages
2023-04-17T12:24:52.695Z In(05) vcpu-2 - PVSCSI: MsgRing: 32 entries, eSz=128, 1 pages
2023-04-17T12:24:52.695Z In(05) vcpu-2 - PVSCSI: scsi0: switching to sync
2023-04-17T12:24:52.695Z In(05) vcpu-2 - PVSCSI: scsi0: init reqCallThresholdCapable 1
2023-04-17T12:24:52.740Z In(05) vcpu-2 - VMXNET3 user: UPT support is not requested
2023-04-17T12:24:52.740Z In(05) vcpu-2 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:24:52.740Z In(05) vcpu-2 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:24:52.740Z In(05) vcpu-2 - VMXNET3 user: Supported set 0x000000ff, configured set 0x00000001
2023-04-17T12:24:52.740Z In(05) vcpu-2 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:24:52.740Z In(05) vcpu-2 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:24:52.798Z In(05) vcpu-1 - VMXNET3 user: UPT support is not requested
2023-04-17T12:24:52.798Z In(05) vcpu-1 - Ethernet1 MAC Address: 00:0c:29:fe:1a:7a
2023-04-17T12:24:52.799Z In(05) vcpu-1 - VMXNET3 user: Ethernet1 RSS fields requested by vmx: 3
2023-04-17T12:24:52.799Z In(05) vcpu-1 - VMXNET3 user: Supported set 0x000000ff, configured set 0x00000001
2023-04-17T12:24:52.799Z In(05) vcpu-1 - Ethernet1 MAC Address: 00:0c:29:fe:1a:7a
2023-04-17T12:24:52.799Z In(05) vcpu-1 - Ethernet1 MAC Address: 00:0c:29:fe:1a:7a
2023-04-17T12:24:56.307Z In(05) vcpu-0 - Tools: Tools heartbeat timeout.
2023-04-17T12:24:56.307Z In(05) vcpu-0 - Tools: Running status rpc handler: 0 => 0.
2023-04-17T12:24:58.391Z Wa(03) vmx - USBGL: DevID(1000000105572008): Failed to reset device. Error(0x16): Invalid argument.
2023-04-17T12:24:58.391Z Wa(03) vmx - USBGL: DevID(1000000105572008): Failed to reset port.
2023-04-17T12:24:58.565Z In(05) vmx - USBGL: DevID(1000000105572008): Claimed device interface(0) successfully.
2023-04-17T12:25:08.283Z Wa(03) vcpu-0 - Balloon: Start: vmmemctl reset balloon
2023-04-17T12:25:11.793Z In(05) vcpu-3 - VMXNET3 user: UPT support is not requested
2023-04-17T12:25:11.793Z In(05) vcpu-3 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:25:11.793Z In(05) vcpu-3 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:25:11.799Z In(05) vcpu-3 - VMXNET3 user: Activate device 0.
2023-04-17T12:25:11.799Z In(05) vcpu-3 - VMXNET3 user: Ethernet0 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:25:11.799Z In(05) vcpu-3 - VMXNET3 user: Activate request succeeds for device 0.
2023-04-17T12:25:11.823Z In(05) vcpu-0 - VMXNET3 user: Quiesce device 0.
2023-04-17T12:25:11.823Z In(05) vcpu-0 - VMXNET3 user: UPT support is not requested
2023-04-17T12:25:11.823Z In(05) vcpu-0 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:25:11.823Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:25:11.827Z In(05) vcpu-0 - VMXNET3 user: Activate device 0.
2023-04-17T12:25:11.831Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:25:11.831Z In(05) vcpu-0 - VMXNET3 user: Activate request succeeds for device 0.
2023-04-17T12:25:11.832Z In(05) vcpu-0 - VMXNET3 user: Quiesce device 0.
2023-04-17T12:25:11.832Z In(05) vcpu-0 - VMXNET3 user: UPT support is not requested
2023-04-17T12:25:11.832Z In(05) vcpu-0 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:25:11.833Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:25:11.837Z In(05) vcpu-0 - VMXNET3 user: Activate device 0.
2023-04-17T12:25:11.864Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:25:11.864Z In(05) vcpu-0 - VMXNET3 user: Activate request succeeds for device 0.
2023-04-17T12:25:11.894Z In(05) vcpu-0 - VMXNET3 user: Quiesce device 0.
2023-04-17T12:25:11.894Z In(05) vcpu-0 - VMXNET3 user: UPT support is not requested
2023-04-17T12:25:11.894Z In(05) vcpu-0 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:25:11.894Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:25:11.898Z In(05) vcpu-0 - VMXNET3 user: Activate device 0.
2023-04-17T12:25:11.899Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:25:11.899Z In(05) vcpu-0 - VMXNET3 user: Activate request succeeds for device 0.
2023-04-17T12:25:11.900Z In(05) vcpu-0 - VMXNET3 user: Quiesce device 0.
2023-04-17T12:25:11.900Z In(05) vcpu-0 - VMXNET3 user: UPT support is not requested
2023-04-17T12:25:11.900Z In(05) vcpu-0 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:25:11.900Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:25:11.908Z In(05) vcpu-0 - VMXNET3 user: Activate device 0.
2023-04-17T12:25:11.931Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:25:11.931Z In(05) vcpu-0 - VMXNET3 user: Activate request succeeds for device 0.
2023-04-17T12:25:11.951Z In(05) vcpu-3 - VMXNET3 user: UPT support is not requested
2023-04-17T12:25:11.951Z In(05) vcpu-3 - Ethernet2 MAC Address: 00:0c:29:fe:1a:84
2023-04-17T12:25:11.951Z In(05) vcpu-3 - VMXNET3 user: Ethernet2 RSS fields requested by vmx: 3
2023-04-17T12:25:11.954Z In(05) vcpu-3 - VMXNET3 user: Activate device 2.
2023-04-17T12:25:11.954Z In(05) vcpu-3 - VMXNET3 user: Ethernet2 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:25:11.954Z In(05) vcpu-3 - VMXNET3 user: Activate request succeeds for device 2.
2023-04-17T12:25:12.032Z In(05) vcpu-3 - VMXNET3 user: UPT support is not requested
2023-04-17T12:25:12.032Z In(05) vcpu-3 - Ethernet1 MAC Address: 00:0c:29:fe:1a:7a
2023-04-17T12:25:12.032Z In(05) vcpu-3 - VMXNET3 user: Ethernet1 RSS fields requested by vmx: 3
2023-04-17T12:25:12.037Z In(05) vcpu-3 - VMXNET3 user: Activate device 1.
2023-04-17T12:25:12.037Z In(05) vcpu-3 - VMXNET3 user: Ethernet1 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:25:12.037Z In(05) vcpu-3 - VMXNET3 user: Activate request succeeds for device 1.
2023-04-17T12:25:30.205Z In(05) vcpu-1 - GuestStore: gstored access is disabled.
2023-04-17T12:25:30.206Z In(05) vcpu-1 - ToolsGetAppInfoEnabledFromConfigStore: Returning the cached value: '1'.
2023-04-17T12:25:30.208Z In(05) vcpu-1 - DEVSWAP: GuestOS does not require LSI adapter swap.
2023-04-17T12:25:30.208Z In(05) vcpu-1 - Guest: toolbox: Version: 12.2.0.41219 (build-21223074)
2023-04-17T12:25:30.208Z Wa(03) vcpu-1 - GuestRpc: application toolbox, changing channel 65535 -> 0
2023-04-17T12:25:30.208Z In(05) vcpu-1 - GuestRpc: Channel 0, guest application toolbox.
2023-04-17T12:25:30.208Z In(05) vcpu-1 - Tools: [AppStatus] Last heartbeat value 90444 (last received 53s ago)
2023-04-17T12:25:30.208Z In(05) vcpu-1 - TOOLS: appName=toolbox, oldStatus=0, status=1, guestInitiated=0.
2023-04-17T12:25:30.214Z In(05) vcpu-3 - TOOLS soft reset detected.
2023-04-17T12:25:30.214Z In(05) vcpu-3 - Vix: [mainDispatch.c:4129]: VMAutomationReportPowerStateChange: Reporting power state change (opcode=2, err=0).
2023-04-17T12:25:30.215Z In(05) vcpu-3 - TOOLS Received tools.set.version rpc call, version = TOOLS_VERSION_UNMANAGED, type is unknown
2023-04-17T12:25:30.215Z In(05) vcpu-3 - TOOLS Setting toolsVersionStatus = TOOLS_STATUS_UNMANAGED
2023-04-17T12:25:30.215Z In(05) vcpu-3 - Tools_SetVersionAndType did nothing; new tools version (2147483647) and type (0) match old Tools version and type
2023-04-17T12:25:30.215Z In(05) vcpu-3 - TOOLS Setting autoupgrade-checked FALSE.
2023-04-17T12:25:30.308Z In(05) vcpu-0 - Tools: Running status rpc handler: 0 => 1.
2023-04-17T12:25:30.308Z In(05) vcpu-0 - Tools: Changing running status: 0 => 1.
2023-04-17T12:25:30.308Z In(05) vcpu-0 - Tools: [RunningStatus] Last heartbeat value 90445 (last received 0s ago)
2023-04-17T12:25:30.370Z In(05) vcpu-2 - Vix: [mainDispatch.c:4129]: VMAutomationReportPowerStateChange: Reporting power state change (opcode=2, err=0).
2023-04-17T12:25:30.370Z In(05) vcpu-2 - TOOLS Received tools.set.version rpc call, version = TOOLS_VERSION_UNMANAGED, type is unknown
2023-04-17T12:25:30.370Z In(05) vcpu-2 - TOOLS Setting toolsVersionStatus = TOOLS_STATUS_UNMANAGED
2023-04-17T12:25:30.370Z In(05) vcpu-2 - Tools_SetVersionAndType did nothing; new tools version (2147483647) and type (0) match old Tools version and type
2023-04-17T12:26:00.205Z In(05) vcpu-0 - GuestInfo: HostinfoDetailedDataHeader version: 1
2023-04-17T12:26:00.206Z No(00) vcpu-0 - ConfigDB: Setting guestInfo.detailed.data = <not printed>


And here the second one seven minutes later:

Code Select Expand

2023-04-17T12:32:46.727Z In(05) vcpu-0 - CPU reset: soft (mode Emulation)
2023-04-17T12:32:46.727Z In(05) vcpu-1 - CPU reset: soft (mode HV)
2023-04-17T12:32:46.727Z In(05) vcpu-3 - CPU reset: soft (mode HV)
2023-04-17T12:32:46.727Z In(05) vcpu-2 - CPU reset: soft (mode HV)
2023-04-17T12:32:46.728Z In(05) vcpu-0 - SVGA: Unregistering IOSpace at 0x1070
2023-04-17T12:32:46.728Z In(05) vcpu-0 - SVGA: Unregistering MemSpace at 0xe8000000(0xe8000000) and 0xfe000000(0xfe000000)
2023-04-17T12:32:46.728Z In(05) vcpu-0 - SVGA: FIFO is already mapped
...
2023-04-17T12:32:46.753Z In(05) vcpu-0 - Chipset: The guest has requested that the virtual machine be hard reset.
2023-04-17T12:32:46.754Z In(05) vcpu-0 - Destroying virtual dev for scsi0:0 vscsi=1589730605015058
2023-04-17T12:32:46.754Z In(05) vcpu-0 - VMMon_VSCSIStopVports: No such target on adapter
2023-04-17T12:32:46.755Z In(05) vcpu-0 - DEVICE: Resetting device 'ALL'.
2023-04-17T12:32:46.755Z In(05) vcpu-0 - USB: Per-Device Resetting device 0x1000000105572008
2023-04-17T12:32:47.213Z Wa(03) vcpu-0 - USBGL: DevID(1000000105572008): Failed to reset device. Error(0x16): Invalid argument.
2023-04-17T12:32:47.213Z Wa(03) vcpu-0 - USBGL: DevID(1000000105572008): Failed to reset port.
2023-04-17T12:32:47.213Z In(05) vcpu-0 - USB: Per-Device Resetting device 0x200000050e0f0003
2023-04-17T12:32:47.213Z In(05) vcpu-0 - Tools: ToolsRunningStatus_Reset, delayedRequest is 0x0
2023-04-17T12:32:47.213Z In(05) vcpu-0 - Tools: Changing running status: 1 => 0.
2023-04-17T12:32:47.213Z In(05) vcpu-0 - Tools: [RunningStatus] Last heartbeat value 90869 (last received 12s ago)
2023-04-17T12:32:47.213Z In(05) vcpu-0 - GuestLib Generated SessionId 576637647794113200
2023-04-17T12:32:47.213Z In(05) vcpu-0 - GuestRpc: Reinitializing Channel 0(toolbox)
2023-04-17T12:32:47.213Z In(05) vcpu-0 - GuestMsg: Channel 0, Cannot unpost because the previous post is already completed
2023-04-17T12:32:47.213Z In(05) vcpu-0 - Tools: [AppStatus] Last heartbeat value 90869 (last received 12s ago)
2023-04-17T12:32:47.213Z In(05) vcpu-0 - TOOLS: appName=toolbox, oldStatus=1, status=0, guestInitiated=0.
2023-04-17T12:32:47.214Z In(05) vcpu-0 - VMXNET3 user: UPT support is not requested
2023-04-17T12:32:47.214Z In(05) vcpu-0 - Ethernet2 MAC Address: 00:0c:29:fe:1a:84
2023-04-17T12:32:47.214Z In(05) vcpu-0 - VMXNET3 user: Ethernet2 RSS fields requested by vmx: 3
2023-04-17T12:32:47.214Z In(05) vcpu-0 - VMXNET3 user: UPT support is not requested
2023-04-17T12:32:47.214Z In(05) vcpu-0 - Ethernet1 MAC Address: 00:0c:29:fe:1a:7a
2023-04-17T12:32:47.214Z In(05) vcpu-0 - VMXNET3 user: Ethernet1 RSS fields requested by vmx: 3
2023-04-17T12:32:47.214Z In(05) vcpu-0 - VMXNET3 user: UPT support is not requested
2023-04-17T12:32:47.214Z In(05) vcpu-0 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:32:47.214Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:32:47.214Z In(05) vcpu-0 - UHCI: HCReset
2023-04-17T12:32:47.214Z In(05) vcpu-0 - Checkpoint_Unstun: vm stopped for 461772 us
2023-04-17T12:32:47.215Z In(05) vcpu-0 - CPU reset: hard (mode Emulation)
2023-04-17T12:32:47.215Z In(05) vcpu-1 - CPU reset: hard (mode Emulation)
2023-04-17T12:32:47.215Z In(05) vcpu-2 - CPU reset: hard (mode Emulation)
2023-04-17T12:32:47.215Z In(05) vcpu-3 - CPU reset: hard (mode Emulation)
2023-04-17T12:32:47.217Z In(05) vcpu-0 - PVSCSI: scsi0: switching to async
2023-04-17T12:32:47.217Z In(05) vcpu-0 - Creating virtual dev for 'scsi0:0'.
2023-04-17T12:32:47.217Z In(05) vcpu-0 - DumpDiskInfo: scsi0:0 createType=11, capacity = 16777216, numLinks = 4
2023-04-17T12:32:47.217Z In(05) vcpu-0 - , allocationType = 2
2023-04-17T12:32:47.217Z In(05) vcpu-0 -
2023-04-17T12:32:47.217Z In(05) vcpu-0 - SCSIDiskESXPopulateVDevDesc: Using FS backend
2023-04-17T12:32:47.217Z In(05) vcpu-0 - DISKUTIL: scsi0:0 : geometry=1044/255/63
2023-04-17T12:32:47.217Z In(05) vcpu-0 - SCSIFilterESXAttachCBRCInt: CBRC not enabled or opened without filters,skipping CBRC           filter attach.
2023-04-17T12:32:47.217Z In(05) vcpu-0 - SCSIFilterESXAttachCBRCInt: Invalidating all hashes for this digest
2023-04-17T12:32:47.217Z In(05) vcpu-0 - SCSIFilterSBDAttachCBRC: device scsi0:0 is not SBD. Skipping CBRC attach SBD way.
2023-04-17T12:32:47.217Z In(05) vcpu-0 - DataSetsMgrSync: Reconnect dataSets store after checkpoint.
2023-04-17T12:32:47.217Z In(05) vcpu-0 - CPT: vm was stunned for 464645 us
2023-04-17T12:32:47.222Z In(05) vcpu-0 - OpNotificationMgr_RemoveAllApps: Attempting to remove registered apps during warm or cold reboot
...
2023-04-17T12:32:47.709Z In(05) vcpu-0 - DISKUTIL: scsi0:0 : geometry=1044/255/63
2023-04-17T12:32:47.709Z In(05) vcpu-0 - DISKUTIL: scsi0:0 : capacity=16777216 logical sector size=512
2023-04-17T12:32:47.775Z In(05) vcpu-1 - CPU reset: soft (mode Emulation)
2023-04-17T12:32:47.776Z In(05) vcpu-2 - CPU reset: soft (mode Emulation)
2023-04-17T12:32:47.777Z In(05) vcpu-3 - CPU reset: soft (mode Emulation)
2023-04-17T12:32:47.784Z In(05) vcpu-0 - BIOS-UUID is 56 4d 78 4d e5 4c 52 06-67 da 1c ed 57 fe 1a 70
2023-04-17T12:32:54.206Z In(05) vcpu-1 - CPU reset: soft (mode Emulation)
2023-04-17T12:32:54.216Z In(05) vcpu-2 - CPU reset: soft (mode Emulation)
2023-04-17T12:32:54.226Z In(05) vcpu-3 - CPU reset: soft (mode Emulation)
2023-04-17T12:32:54.317Z In(05) vcpu-0 - Tools: Tools heartbeat timeout.
2023-04-17T12:32:54.318Z In(05) vcpu-0 - Tools: Running status rpc handler: 0 => 0.
...
2023-04-17T12:32:54.440Z In(05) vcpu-2 - UHCI: Global Reset
2023-04-17T12:32:54.551Z In(05) vcpu-2 - UHCI: HCReset
2023-04-17T12:32:54.654Z In(05) vcpu-2 - VMXNET3 user: UPT support is not requested
2023-04-17T12:32:54.654Z In(05) vcpu-2 - Ethernet2 MAC Address: 00:0c:29:fe:1a:84
2023-04-17T12:32:54.654Z In(05) vcpu-2 - VMXNET3 user: Ethernet2 RSS fields requested by vmx: 3
2023-04-17T12:32:54.654Z In(05) vcpu-2 - VMXNET3 user: Supported set 0x000000ff, configured set 0x00000001
2023-04-17T12:32:54.654Z In(05) vcpu-2 - Ethernet2 MAC Address: 00:0c:29:fe:1a:84
2023-04-17T12:32:54.654Z In(05) vcpu-2 - Ethernet2 MAC Address: 00:0c:29:fe:1a:84
2023-04-17T12:32:54.723Z In(05) vcpu-2 - PVSCSI: ReqRing: 1024 entries, eSz=128, 32 pages
2023-04-17T12:32:54.723Z In(05) vcpu-2 - PVSCSI: CmpRing: 4096 entries, eSz=32, 32 pages
2023-04-17T12:32:54.723Z In(05) vcpu-2 - PVSCSI: MsgRing: 32 entries, eSz=128, 1 pages
2023-04-17T12:32:54.723Z In(05) vcpu-2 - PVSCSI: scsi0: switching to sync
2023-04-17T12:32:54.723Z In(05) vcpu-2 - PVSCSI: scsi0: init reqCallThresholdCapable 1
2023-04-17T12:32:54.768Z In(05) vcpu-2 - VMXNET3 user: UPT support is not requested
2023-04-17T12:32:54.768Z In(05) vcpu-2 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:32:54.768Z In(05) vcpu-2 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:32:54.768Z In(05) vcpu-2 - VMXNET3 user: Supported set 0x000000ff, configured set 0x00000001
2023-04-17T12:32:54.768Z In(05) vcpu-2 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:32:54.768Z In(05) vcpu-2 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:32:54.842Z In(05) vcpu-1 - VMXNET3 user: UPT support is not requested
2023-04-17T12:32:54.842Z In(05) vcpu-1 - Ethernet1 MAC Address: 00:0c:29:fe:1a:7a
2023-04-17T12:32:54.842Z In(05) vcpu-1 - VMXNET3 user: Ethernet1 RSS fields requested by vmx: 3
2023-04-17T12:32:54.842Z In(05) vcpu-1 - VMXNET3 user: Supported set 0x000000ff, configured set 0x00000001
2023-04-17T12:32:54.842Z In(05) vcpu-1 - Ethernet1 MAC Address: 00:0c:29:fe:1a:7a
2023-04-17T12:32:54.842Z In(05) vcpu-1 - Ethernet1 MAC Address: 00:0c:29:fe:1a:7a
2023-04-17T12:33:00.417Z Wa(03) vmx - USBGL: DevID(1000000105572008): Failed to reset device. Error(0x16): Invalid argument.
2023-04-17T12:33:00.417Z Wa(03) vmx - USBGL: DevID(1000000105572008): Failed to reset port.
2023-04-17T12:33:00.585Z In(05) vmx - USBGL: DevID(1000000105572008): Claimed device interface(0) successfully.
2023-04-17T12:33:10.352Z Wa(03) vcpu-0 - Balloon: Start: vmmemctl reset balloon
2023-04-17T12:33:13.814Z In(05) vcpu-1 - VMXNET3 user: UPT support is not requested
2023-04-17T12:33:13.814Z In(05) vcpu-1 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:33:13.815Z In(05) vcpu-1 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:33:13.819Z In(05) vcpu-1 - VMXNET3 user: Activate device 0.
2023-04-17T12:33:13.819Z In(05) vcpu-1 - VMXNET3 user: Ethernet0 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:33:13.819Z In(05) vcpu-1 - VMXNET3 user: Activate request succeeds for device 0.
2023-04-17T12:33:13.839Z In(05) vcpu-2 - VMXNET3 user: Quiesce device 0.
2023-04-17T12:33:13.839Z In(05) vcpu-2 - VMXNET3 user: UPT support is not requested
2023-04-17T12:33:13.839Z In(05) vcpu-2 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:33:13.839Z In(05) vcpu-2 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:33:13.843Z In(05) vcpu-2 - VMXNET3 user: Activate device 0.
2023-04-17T12:33:13.851Z In(05) vcpu-2 - VMXNET3 user: Ethernet0 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:33:13.851Z In(05) vcpu-2 - VMXNET3 user: Activate request succeeds for device 0.
2023-04-17T12:33:13.853Z In(05) vcpu-2 - VMXNET3 user: Quiesce device 0.
2023-04-17T12:33:13.853Z In(05) vcpu-2 - VMXNET3 user: UPT support is not requested
2023-04-17T12:33:13.853Z In(05) vcpu-2 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:33:13.853Z In(05) vcpu-2 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:33:13.859Z In(05) vcpu-2 - VMXNET3 user: Activate device 0.
2023-04-17T12:33:13.884Z In(05) vcpu-2 - VMXNET3 user: Ethernet0 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:33:13.884Z In(05) vcpu-2 - VMXNET3 user: Activate request succeeds for device 0.
2023-04-17T12:33:13.913Z In(05) vcpu-0 - VMXNET3 user: Quiesce device 0.
2023-04-17T12:33:13.913Z In(05) vcpu-0 - VMXNET3 user: UPT support is not requested
2023-04-17T12:33:13.913Z In(05) vcpu-0 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:33:13.914Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:33:13.917Z In(05) vcpu-0 - VMXNET3 user: Activate device 0.
2023-04-17T12:33:13.918Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:33:13.918Z In(05) vcpu-0 - VMXNET3 user: Activate request succeeds for device 0.
2023-04-17T12:33:13.919Z In(05) vcpu-0 - VMXNET3 user: Quiesce device 0.
2023-04-17T12:33:13.920Z In(05) vcpu-0 - VMXNET3 user: UPT support is not requested
2023-04-17T12:33:13.920Z In(05) vcpu-0 - Ethernet0 MAC Address: 00:0c:29:fe:1a:70
2023-04-17T12:33:13.920Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 RSS fields requested by vmx: 3
2023-04-17T12:33:13.923Z In(05) vcpu-0 - VMXNET3 user: Activate device 0.
2023-04-17T12:33:13.950Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:33:13.950Z In(05) vcpu-0 - VMXNET3 user: Activate request succeeds for device 0.
2023-04-17T12:33:13.969Z In(05) vcpu-2 - VMXNET3 user: UPT support is not requested
2023-04-17T12:33:13.969Z In(05) vcpu-2 - Ethernet2 MAC Address: 00:0c:29:fe:1a:84
2023-04-17T12:33:13.969Z In(05) vcpu-2 - VMXNET3 user: Ethernet2 RSS fields requested by vmx: 3
2023-04-17T12:33:13.972Z In(05) vcpu-2 - VMXNET3 user: Activate device 2.
2023-04-17T12:33:13.972Z In(05) vcpu-2 - VMXNET3 user: Ethernet2 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:33:13.972Z In(05) vcpu-2 - VMXNET3 user: Activate request succeeds for device 2.
2023-04-17T12:33:14.058Z In(05) vcpu-1 - VMXNET3 user: UPT support is not requested
2023-04-17T12:33:14.058Z In(05) vcpu-1 - Ethernet1 MAC Address: 00:0c:29:fe:1a:7a
2023-04-17T12:33:14.058Z In(05) vcpu-1 - VMXNET3 user: Ethernet1 RSS fields requested by vmx: 3
2023-04-17T12:33:14.065Z In(05) vcpu-1 - VMXNET3 user: Activate device 1.
2023-04-17T12:33:14.065Z In(05) vcpu-1 - VMXNET3 user: Ethernet1 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0
2023-04-17T12:33:14.065Z In(05) vcpu-1 - VMXNET3 user: Activate request succeeds for device 1.
2023-04-17T12:33:32.434Z In(05) vcpu-3 - GuestStore: gstored access is disabled.
2023-04-17T12:33:32.434Z In(05) vcpu-3 - ToolsGetAppInfoEnabledFromConfigStore: Returning the cached value: '1'.
2023-04-17T12:33:32.434Z In(05) vcpu-3 - DEVSWAP: GuestOS does not require LSI adapter swap.
2023-04-17T12:33:32.434Z In(05) vcpu-3 - Guest: toolbox: Version: 12.2.0.41219 (build-21223074)
2023-04-17T12:33:32.434Z Wa(03) vcpu-3 - GuestRpc: application toolbox, changing channel 65535 -> 0
2023-04-17T12:33:32.434Z In(05) vcpu-3 - GuestRpc: Channel 0, guest application toolbox.
2023-04-17T12:33:32.434Z In(05) vcpu-3 - Tools: [AppStatus] Last heartbeat value 90869 (last received 58s ago)
2023-04-17T12:33:32.434Z In(05) vcpu-3 - TOOLS: appName=toolbox, oldStatus=0, status=1, guestInitiated=0.
2023-04-17T12:33:32.439Z In(05) vcpu-3 - TOOLS soft reset detected.
2023-04-17T12:33:32.439Z In(05) vcpu-3 - Vix: [mainDispatch.c:4129]: VMAutomationReportPowerStateChange: Reporting power state change (opcode=2, err=0).
2023-04-17T12:33:32.440Z In(05) vcpu-3 - TOOLS Received tools.set.version rpc call, version = TOOLS_VERSION_UNMANAGED, type is unknown
2023-04-17T12:33:32.440Z In(05) vcpu-3 - TOOLS Setting toolsVersionStatus = TOOLS_STATUS_UNMANAGED
2023-04-17T12:33:32.440Z In(05) vcpu-3 - Tools_SetVersionAndType did nothing; new tools version (2147483647) and type (0) match old Tools version and type
2023-04-17T12:33:32.440Z In(05) vcpu-3 - TOOLS Setting autoupgrade-checked FALSE.
2023-04-17T12:33:33.318Z In(05) vcpu-0 - Tools: Running status rpc handler: 0 => 1.
2023-04-17T12:33:33.318Z In(05) vcpu-0 - Tools: Changing running status: 0 => 1.
2023-04-17T12:33:33.318Z In(05) vcpu-0 - Tools: [RunningStatus] Last heartbeat value 90870 (last received 0s ago)
2023-04-17T12:33:33.423Z In(05) vcpu-0 - Vix: [mainDispatch.c:4129]: VMAutomationReportPowerStateChange: Reporting power state change (opcode=2, err=0).
2023-04-17T12:33:33.424Z In(05) vcpu-0 - TOOLS Received tools.set.version rpc call, version = TOOLS_VERSION_UNMANAGED, type is unknown
2023-04-17T12:33:33.424Z In(05) vcpu-0 - TOOLS Setting toolsVersionStatus = TOOLS_STATUS_UNMANAGED
2023-04-17T12:33:33.424Z In(05) vcpu-0 - Tools_SetVersionAndType did nothing; new tools version (2147483647) and type (0) match old Tools version and type
2023-04-17T12:34:02.439Z In(05) vcpu-1 - GuestInfo: HostinfoDetailedDataHeader version: 1
2023-04-17T12:34:02.440Z No(00) vcpu-1 - ConfigDB: Setting guestInfo.detailed.data = <not printed>


I will see, if I can enable debig logging for the vmware tools. But to me, it seems that this vmware-tools is somewhat buggy and maybe it stops sending the heartbeat, so esxi resets the machine.

Thanks for the lots of good information. For me as a total newbie for esxi that is very helpful.

I modified the virtual hardware now so to use PVSCSI and VMXNET3 drivers. (The switch to the different nic drivers actually was a pain in the ..., as simply switching to them on the esxi interface made them completely dysfunctional and a switching back to e1000e just gave an "invalid config" message. I needed to remove the interfaces and then create new ones. Now it is working.) Guest OS Family was already set to "FreeBSD 13 (64bit)".

The performance seems to be somewhat worse now with the vmxnet drivers than with the e1000 drivers but still sufficient. I will now give it some time to see if I get hard resets again. If yes, I will then add the vmware.conf file (which is missing on opnsense) so that it starts logging debug info.

Thanks for your help, Ben.

I am very new to esxi, so I am not really much familiar with it yet. (Been a proxmox user before.)

The physical hardware is a HUNSN RJ03 with a N5105 celeron CPU, 16GB RAM and a 256 GB M2. SSD. The virtual HW uses a four core processor with four threads (so identical to the host CPU), 8GB RAM, a LSI Logic Parallel SCSI controller and three e1000 virtual NICs.

I still have no idea why these hard resets happen. This night there was another one. Again there is no message in remote syslog. The vmware log has again the same entries (so a tools timeout). I don't know, if there really is a problem in the virtual machine or if that is a software bug in the vmware-tools package.

I guess the next test should be without the vmware-tools package and see, if there will be some real issues after a couple of days (which would have initiated the vmware-tools hard reset) or if the machine continues working without problems.

OK, so with the updated software the opnsense router worked for nearly 6 days. Then the same problem again. This is from VMWARE log:

Code Select Expand

2023-04-06T14:32:50.256Z In(05) vcpu-3 - E1000: e1000e-- tx queue 1 is enabled.
2023-04-06T14:32:50.256Z In(05) vcpu-3 - E1000: e1000e-- tx queue 1 is enabled.
2023-04-11T16:53:11.740Z In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.
2023-04-11T16:53:11.740Z In(05) vmx - Tools: [AppStatus] Last heartbeat value 626418 (last received 11s ago)
2023-04-11T16:53:11.741Z In(05) vmx - TOOLS: appName=toolbox, oldStatus=1, status=2, guestInitiated=0.
2023-04-11T16:53:12.230Z In(05) vcpu-0 - CPU reset: soft (mode Emulation)
2023-04-11T16:53:12.230Z In(05) vcpu-1 - CPU reset: soft (mode HV)
2023-04-11T16:53:12.230Z In(05) vcpu-3 - CPU reset: soft (mode HV)
2023-04-11T16:53:12.230Z In(05) vcpu-2 - CPU reset: soft (mode HV)
2023-04-11T16:53:12.231Z In(05) vcpu-0 - SVGA: Unregistering IOSpace at 0x1070
2023-04-11T16:53:12.231Z In(05) vcpu-0 - SVGA: Unregistering MemSpace at 0xe8000000(0xe8000000) and 0xfe000000(0xfe000000)
2023-04-11T16:53:12.231Z In(05) vcpu-0 - SVGA: FIFO is already mapped
2023-04-11T16:53:12.242Z In(05) vcpu-0 - PCIBridge4: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.242Z In(05) vcpu-0 - pciBridge4:1: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.242Z In(05) vcpu-0 - pciBridge4:2: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.242Z In(05) vcpu-0 - pciBridge4:3: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.242Z In(05) vcpu-0 - pciBridge4:4: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.242Z In(05) vcpu-0 - pciBridge4:5: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.243Z In(05) vcpu-0 - pciBridge4:6: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.243Z In(05) vcpu-0 - pciBridge4:7: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.243Z In(05) vcpu-0 - PCIBridge5: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.243Z In(05) vcpu-0 - pciBridge5:1: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.243Z In(05) vcpu-0 - pciBridge5:2: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.243Z In(05) vcpu-0 - pciBridge5:3: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.243Z In(05) vcpu-0 - pciBridge5:4: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.243Z In(05) vcpu-0 - pciBridge5:5: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.243Z In(05) vcpu-0 - pciBridge5:6: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.243Z In(05) vcpu-0 - pciBridge5:7: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.243Z In(05) vcpu-0 - PCIBridge6: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.244Z In(05) vcpu-0 - pciBridge6:1: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.244Z In(05) vcpu-0 - pciBridge6:2: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.244Z In(05) vcpu-0 - pciBridge6:3: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.244Z In(05) vcpu-0 - pciBridge6:4: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.244Z In(05) vcpu-0 - pciBridge6:5: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.244Z In(05) vcpu-0 - pciBridge6:6: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.244Z In(05) vcpu-0 - pciBridge6:7: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.244Z In(05) vcpu-0 - PCIBridge7: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.244Z In(05) vcpu-0 - pciBridge7:1: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.244Z In(05) vcpu-0 - pciBridge7:2: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.244Z In(05) vcpu-0 - pciBridge7:3: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.245Z In(05) vcpu-0 - pciBridge7:4: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.245Z In(05) vcpu-0 - pciBridge7:5: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.245Z In(05) vcpu-0 - pciBridge7:6: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.245Z In(05) vcpu-0 - pciBridge7:7: ISA/VGA decoding enabled (ctrl 0004)
2023-04-11T16:53:12.256Z In(05) vcpu-0 - Chipset: The guest has requested that the virtual machine be hard reset.
2023-04-11T16:53:12.258Z In(05) vcpu-0 - Destroying virtual dev for scsi0:0 vscsi=1136783353978881


Again the exact same message "The guest has requested that the virtual machine be hard reset."

I also logged all messages from opnsense on a remote syslog server. There is nothing in there. The reset came out of the blue for opnsense.

If there is no fix for that, I will 1.) observe the behaviour for another couple of days/weeks (depending on what happens). then 2.) will deactivate os-vmware again and see if that fixes the problem once and forever, and 3.) if nothing helps, then install Proxmox instead of esxi.  :(

I will give it another try. I updated to 23.1.5-4 and also ESXi to 8.0c. I'll report back if I see again reboots.