No clean shutdown for OPNsense as guest in Hyper-V Server 2022

[iorx]

Hi,

This one is really taking a toll on me. OPNsense has been no trouble before (other installs, not sure if they are on the latest version of OPNsense though...), has cleanly shutdown/rebooted before when host is rebooted. Now I can't figure out why it goes down like a ton of bricks.

I've got a squeaky clean (just renstalled for the second time) Windows Server 2022 Standard with the Hyper-V role.
Got two Linux guests (DietPi and Home Asstant) and two Windows guest (Server 2022 and Windows 10).
Everything is updated as it should and on the latest.

OPNsense installed from the latest iso (DVD) and updated.

The setting are the same for the guest.
In the event of a host reboot the guests have the setting to do a Save State and resume from there when host is back up.
The Windows and Linux guest do as expected. But, as mentioned, OPNsense is "plug pulled" on.

Integrated Services:

Code: [Select]

VMName Name                    Enabled PrimaryStatusDescription
------ ----                    ------- ------------------------
fwa    Guest Service Interface False   OK
fwa    Heartbeat               True    OK
fwa    Key-Value Pair Exchange True    OK
fwa    Shutdown                True    OK
fwa    Time Synchronization    True    OK
fwa    VSS                     True    OK


Network config (don't know if it matters). WAN is 0.0.0.0 because it was not connected to WAN while testing.

Code: [Select]

Name          IsManagementOs VMName SwitchName MacAddress   Status IPAddresses
----          -------------- ------ ---------- ----------   ------ -----------
nicLAN        False          fwa    sw01       00155D201400 {Ok}   {192.168.32.3}
nicWAN-VLAN20 False          fwa    sw01       000DB9562155 {Ok}   {0.0.0.0}
nicGST-VLAN50 False          fwa    sw01       00155D201401 {Ok}   {192.168.99.2}
I've made clean installs of OPNsense both in gen 1 and gen 2 guests. Same behaviour.

I've attached output of "dmesg -a" from a normal reboot and from the host reboot.
(please tell if I should clean out any "private" information in these, I couldn't identify any from a glance at them)

So maybe someone here has clue what's up here?

From the "hyper-v reboot.txt"

Code: [Select]

cd0: Attempt to query device size failed: NOT READY, Medium not present
WARNING: /mnt was not properly dismounted
WARNING: /mnt: mount pending error: blocks 24 files 0
Mounting filesystems...
tunefs: / is not clean - run fsck.

camcontrol: ATA ATA_IDENTIFY via pass_16 failed
camcontrol: ATA ATAPI_IDENTIFY via pass_16 failed
** /dev/gpt/rootfs
** SU+J Recovering /dev/gpt/rootfs

USE JOURNAL? yes

** Reading 33554432 byte journal from inode 4.

RECOVER? yes

** Building recovery table.
** Resolving unreferenced inode list.
** Processing journal entries.

WRITE CHANGES? yes


***** FILE SYSTEM IS CLEAN *****
** 8 journal records in 2048 bytes for 12.50% utilization
** Freed 1 inodes (0 dirs) 0 blocks, and 2 frags.

***** FILE SYSTEM MARKED CLEAN *****
And some start parameters from Hyper-V

Code: [Select]

Name    AutomaticStopAction AutomaticStartDelay
----    ------------------- -------------------
dpa                    Save                  25
fwa                    Save                   0
haa                    Save                  20
srv2022                Save                  15
w10                    Save                  20

[iorx]

Hi again!

Probably nothing wrong with OPNsense. FreeBSD, Windows 2022/10, DietPi was also dirty shutdown. Argh!

This looks like it was really random. Sometimes the machines did a clean shut down of the guest and some time they didn't.
I was wrong that it was a specific guest OS which was the problem, many reboot later I figured that all guest were randomly killed.

Is Server 2022 "borken" out of the box? This is clean installs of Server 2022 with the Hyper-V role only. Strange! The hardware is a intel NUC 7, i7 with nvme for storage. Ok this is not server grade hardware but Server 2022 shouldn't behave broken anyway I think. I going forth trying to figure out why VM shut down i totally broken, I'll post my findings.

So, I scripted the same function with GPO startup/shutdown powershell scripts (attached).

Probably overcomplicated as this is a bit of cut & paste powershell knowledge putting this together, bit it works and honours the parameter set on the VMs.

Attached for your viewing pleasure / cringe PS experience.

[franco]

With OpenBSD on UFS with Hyper-V in another company we had strange reports of firmware updates disappearing after the reboot of the Hyper-V VM...  Here being on FreeBSD I would recommend to try ZFS instead to see if that helps.


Cheers,
Franco

[iorx]

Hi thanks!

Did you have the possibility to check if the system cleanly shutdown OPNsense?

ZFS is more resilient and self repairing but running that in an vhdx, virtual disk, becomes strange. A dynamic disk balloons up to it maximum size even though it's no full in anyway. I've had ZFS 127GB (lazy default choice for a new VM in Hyper-V) virtual disk taking up 127GB of host space but they reports just a fraction of this as allocated from within OPNsense.

OPNsense is fine. Server 2022 Hyper-V is crap, at least on my hardware. It just blindly killing the guests at shutdown. Solved for now with some woodoo, powershell and GPO startup/shutdown scripts that take care of saving and restoring machines in an event of reboot. Script attached in previous post.

I manage many systems with Hyper-V and OPNsense virtualized in stand alone/HA, though on server grade hardware. This was a hobby project to have a machine, jack of al trades, in the summer house. A NUC running everything instead the two Rasp 3 and an AMD APU running OPNsense. Having to possibility to pop in more vm and functions is good and fun! 

Brgs,

[mnaim]

Hi, Im running several OPNsenses on 2022 Hyper-V, all have clean shutdowns.

Year ago, they were UFS, now ZFS. In Hyper-V no issues, on physical HW I had some SSD corruption with UFS, thats why I switched to ZFS.

Did you saw shutdown sequence in Hyper-V console?

[Patrick M. Hausen]

ZFS is more resilient and self repairing but running that in an vhdx, virtual disk, becomes strange. A dynamic disk balloons up to it maximum size even though it's no full in anyway. I've had ZFS 127GB (lazy default choice for a new VM in Hyper-V) virtual disk taking up 127GB of host space but they reports just a fraction of this as allocated from within OPNsense.

ZFS is copy-on-write so the guest OS will eventually write nearly every single sector of the virtual disk even when it only uses e.g. 20% of them.

Therefore thin provisioning plain does not work. You should use a thick provisioned disk. Since 30-60 GB are frequently more than enough for an OPNsense installation, that is IMHO not as bad as it looks at a first glance.

HTH,
Patrick