OPNsense Forum

Archive => 22.1 Legacy Series => Topic started by: Dunuin on February 10, 2022, 11:22:15 am

Title: shutdown not working after update from 21.7.8 to 22.1
Post by: Dunuin on February 10, 2022, 11:22:15 am

Hi,

My OPNSenses are virtualized using KVM (one hypervised by bhyve on TrueNAS, two on ProxmoxVE).
With 21.7.8 I shut them down using the qemu-guest-agent plugin.

Now I upgraded one of the OPNsenses on the Proxmox Server to 22.1 and I'm no longer able to shutdown OPNsense. I tried it from the hypervisor using the qemu guest agent, from the hypervisor using ACPI and also directly from the OPNsense WebUI. In all cases shutting down gets stuck with 100% CPU utilization and I need to forcefully stop the VM.

Can someone point me in the right direction what could cause this or what to look for?
Looks like the logs are only stored in RAM and are lost after I kill the VM so not sure where to look for.

The attachment shows what the console is showing when shutting down.

Title: Re: shutdown not working after update from 21.7.8 to 22.1
Post by: Dunuin on February 12, 2022, 12:14:18 pm

I disabled "/var RAM disk" to get logs. But this doesn't seem to help either as the shutdown is stopping syslog-ng service so there are no logs after that could tell me why the VM gets stuck wuth 100% CPU load.
I asked other people running OPNsense 22.1 in a proxmox VM and there the shutdown is working.

Any ideas?

Title: Re: shutdown not working after update from 21.7.8 to 22.1
Post by: Dunuin on February 14, 2022, 12:25:49 pm

I also tried to stop every plugin (dnscrypt-proxy, zabbix_agentd) before stutting down but then the shutdown also hangs.

Title: Re: shutdown not working after update from 21.7.8 to 22.1
Post by: binaryanomaly on February 14, 2022, 12:32:48 pm

I had a similar problem with 21.7.8, some upgrade to a version before had introduced it suddenly.
I even had to reboot the vmhost to make the OPNsense VM restart it got stuck so bad.
Also KVM and proxmox here but I'm not using qemu guest.

Fortunately the upgrade to 22.1 fixed it.

I was thinking of exporting the config and do a clean setup from scratch in parallel to see if that fixes it.
Maybe that could be an approach for you as well?

Title: Re: shutdown not working after update from 21.7.8 to 22.1
Post by: Dunuin on February 15, 2022, 01:30:34 pm

Quote from: binaryanomaly on February 14, 2022, 12:32:48 pm

I was thinking of exporting the config and do a clean setup from scratch in parallel to see if that fixes it.
Maybe that could be an approach for you as well?

Will test that.

What I also found out:
With 1 vCPU the VM will never shutdown (waited up to 12 hour for the shutdown to finish where the CPU utulization of the VM always stayed at 100%).
With 2 vCPUs the VM will sometimes shutdown and sometimes get stuck. CPU Utilization is then 50% until I forcefully stop the VM.
With 3 or 4 vCPUs shutting down always works.

Someone knows why a shutdown can get stuck with less than 3 vCPUs? Tested it like 30 times now and only thing that changed was the amout of vCPUs.

Increasing it to 3 or 4 vCPUs is also not a great workaround as this will increas the hosts CPU queue and the VM never utilized two full vCPUs. Was working perfectly fine and fast enough until now with just 1 vCPU.

Edit:
Shutdown also hangs after upgrading the OPNsense VM to 22.1 on the TrueNAS server with bhyve as hypervisor. And there the number of vCPUs is 3 so increasing the amount of vCPUs might not fix it always.

Title: Re: shutdown not working after update from 21.7.8 to 22.1
Post by: fhloston on March 04, 2022, 11:29:52 am

Same issue here after the 22.1 upgrade. OPNsense VMs hang on shutdown. After hard stop/start they then boot/run normally.

Title: Re: shutdown not working after update from 21.7.8 to 22.1
Post by: alexpacio on March 04, 2022, 04:16:23 pm

I'm using Fedora 34 with libvirt and qemu/kvm, I tried to upgrade an older one major version Opnsense VM and also to fresh install 22.1, but in both cases, if I try to reboot/shutdown programmatically or using ACPI, I get stuck after the partition synchronization messages.

Title: Re: shutdown not working after update from 21.7.8 to 22.1
Post by: nzkiwi68 on March 06, 2022, 06:57:05 pm

I had a similar problem a while ago on older versions.

I fixed it by:
System > Settings > Miscellaneous

And then disable all of the periodic backups:
Periodic RRD Backup > DISABLED
Periodic DHCP Leases Backup > DISABLED
Periodic NetFlow Backup > DISBALED
Periodic Captive Portal Backup > DISABLED

Reboot and the now shutdown and reboot worked great.

Title: Re: shutdown not working after update from 21.7.8 to 22.1
Post by: keropiko on April 17, 2022, 12:06:02 pm

Hello @nzkiwi68, i can confirm that your solution works. Now i can reboot normally my opnsense vm.
Strange this thing happens though.

Title: Re: shutdown not working after update from 21.7.8 to 22.1
Post by: Aerowinder on April 18, 2022, 01:52:07 pm

Quote from: nzkiwi68 on March 06, 2022, 06:57:05 pm

I had a similar problem a while ago on older versions.

I fixed it by:
System > Settings > Miscellaneous

And then disable all of the periodic backups:
Periodic RRD Backup > DISABLED
Periodic DHCP Leases Backup > DISABLED
Periodic NetFlow Backup > DISBALED
Periodic Captive Portal Backup > DISABLED

Reboot and the now shutdown and reboot worked great.

This fixed the issue for me as well. I tried to disable all except DHCP lease backup, rebooted, and the lockup still happened. My DHCP server(s) are disabled on OPN. Perhaps that is the cause of the issue? OPN is attempting to backup something that does not exist and just hangs instead of timing out?