OPNsense on Proxmox Freezing Every Few Nights

[gregg098]

Running 22.1.10 as a VM in Proxmox on a new 4 port i225 mini PC with a Linux Bridge for WAN and a Linux Bridge for LAN. No pass through.  I have Xfinity internet, a handful of VLANs, and I do not run IPS/IDS.

Over the last two weeks, every few mornings, I wake up to no internet. When I connect to Proxmox, I see the OPNsense VM sitting with a yellow pause symbol saying "Internal Error."  If I try to goto a console view, all I see is similar to the screenshot below. The last few log entries before I restart it are all netflow maintenance items. Then nothing.

2022-07-26T00:00:30-07:00   Notice   flowd_aggregate.py   vacuum done   
2022-07-26T00:00:30-07:00   Notice   flowd_aggregate.py   vacuum interface_086400.sqlite   
2022-07-26T00:00:30-07:00   Notice   flowd_aggregate.py   vacuum interface_003600.sqlite

The weird part is that the error messages in the screenshot are always showing vlan50. This vlan is identical in every way to other vlans I have setup. I've verified firewall rules, RAs, interface settings, etc.

If I restart OPNsense, everything comes back. I'm struggling to get more information on the crash.

I *think* this started with the most recent release, but I'm not entirely sure. This setup was running great the last few months.

There are some other forum posts about flapping WAN connections in recent builds, but I'm not sure if they're related or not.

Anyone have any ideas?

[hongkongkiwi]

Same issue here, did you ever figure out what the problem was?

[valantur]

My setup is similar to yours and I'm having the same issue.
I don't see any messages when I check the OPNsense console through Proxmox though, it's just frozen at the logon screen.
I'm running 10 other things some VMs, some Linux containers, and they all work fine... it's just OPNsense that freezes.
So far I tested with versions 22.7 and 22.1.10 and they both fail after a few days.

Any ideas how to troubleshoot this?

Thanks!

Edit:

I found some Proxmox syslog messages related to the VM crashing:


Aug 11 23:40:40 host1 QEMU[19185]: KVM internal error. Suberror: 3
Aug 11 23:40:40 host1 QEMU[19185]: extra data[0]: 0x0000000080000b0e
Aug 11 23:40:40 host1 QEMU[19185]: extra data[1]: 0x0000000000000031
Aug 11 23:40:40 host1 QEMU[19185]: extra data[2]: 0x0000000000000083
Aug 11 23:40:40 host1 QEMU[19185]: extra data[3]: 0x0000000800b4bff8
Aug 11 23:40:40 host1 QEMU[19185]: extra data[4]: 0x0000000000000002
Aug 11 23:40:40 host1 QEMU[19185]: RAX=00000000000000e8 RBX=0000000000000000 RCX=0000000000462c01 RDX=00007ffffffff610
Aug 11 23:40:40 host1 QEMU[19185]: RSI=00007fffdf9face0 RDI=0000000000000004 RBP=00007fffdf9facf8 RSP=ffffffffffffff4c
Aug 11 23:40:40 host1 QEMU[19185]: R8 =0000000000000000 R9 =0000000000000000 R10=000000000065dac8 R11=0000000000000216
Aug 11 23:40:40 host1 QEMU[19185]: R12=00007fffdf9fae60 R13=0000000000000000 R14=000000c0000821a0 R15=0000000000000000
Aug 11 23:40:40 host1 QEMU[19185]: RIP=ffffffff811257ec RFL=00010083 [--S---C] CPL=0 II=0 A20=1 SMM=0 HLT=0
Aug 11 23:40:40 host1 QEMU[19185]: ES =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
Aug 11 23:40:40 host1 QEMU[19185]: CS =0020 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
Aug 11 23:40:40 host1 QEMU[19185]: SS =0028 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
Aug 11 23:40:40 host1 QEMU[19185]: DS =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
Aug 11 23:40:40 host1 QEMU[19185]: FS =0013 0000000800b32130 ffffffff 00c0f300 DPL=3 DS   [-WA]
Aug 11 23:40:40 host1 QEMU[19185]: GS =001b ffffffff82c11000 ffffffff 00c0f300 DPL=3 DS   [-WA]
Aug 11 23:40:40 host1 QEMU[19185]: LDT=0000 0000000000000000 ffffffff 00c00000
Aug 11 23:40:40 host1 QEMU[19185]: TR =0048 ffffffff82c11384 00002068 00008b00 DPL=0 TSS64-busy
Aug 11 23:40:40 host1 QEMU[19185]: GDT=     ffffffff82c113ec 00000067
Aug 11 23:40:40 host1 QEMU[19185]: IDT=     ffffffff81f5ba10 00000fff
Aug 11 23:40:40 host1 QEMU[19185]: CR0=80050033 CR2=ffffffff811257ec CR3=0000000800b4bfd8 CR4=003506e0
Aug 11 23:40:40 host1 QEMU[19185]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
Aug 11 23:40:40 host1 QEMU[19185]: DR6=00000000ffff0ff0 DR7=0000000000000400
Aug 11 23:40:40 host1 QEMU[19185]: EFER=0000000000000d01
Aug 11 23:40:40 host1 QEMU[19185]: Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??

WTF?!

[kopfdreher]

I have the same problem that sporadically freezes Opnsense. This only happens with Opnsense, other containers run without problems. Is there already a solution?

[misery]

Same problem here with OPNsense 22.7.10 and 22.7.9. I don't use Proxmox but AlpineLinux with QEMU/KVM and libvirt as host and OPNSense as guest with PCI-passthrough ethernet.

Code: [Select]

KVM internal error. Suberror: 3
extra data[0]: 0x0000000080000b0e
extra data[1]: 0x0000000000000031
extra data[2]: 0x0000000000000083
extra data[3]: 0x0000000800af2ff8
extra data[4]: 0x0000000000000003
RAX=0000000000000227 RBX=0000000800e2a140 RCX=000000080078c17a RDX=0000000000000000
RSI=00007fffffffda70 RDI=0000000000000004 RBP=00007fffffffdb80 RSP=0000000800e83f40
R8 =0000000000000000 R9 =0000000000000000 R10=52fce69e45e1f63b R11=0000000000000202
R12=00007fffffffda70 R13=0000000000000004 R14=0000000800841f60 R15=0000000000000000
RIP=ffffffff81133eec RFL=00010002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
CS =0020 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0028 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
FS =0013 0000000800a69120 ffffffff 00c0f300 DPL=3 DS   [-WA]
GS =001b ffffffff82610000 ffffffff 00c0f300 DPL=3 DS   [-WA]
LDT=0000 0000000000000000 ffffffff 00c00000
TR =0048 ffffffff82610384 00002068 00008b00 DPL=0 TSS64-busy
GDT=     ffffffff826103ec 00000067
IDT=     ffffffff81f5d710 00000fff
CR0=80050033 CR2=ffffffff81133eec CR3=0000000800af2000 CR4=003506e0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2022-12-11T00:06:49.117840Z qemu-system-x86_64: terminating on signal 15 from pid 3087 (/usr/sbin/libvirtd)
2022-12-11 00:06:49.722+0000: shutting down, reason=destroyed
2022-12-11 00:07:22.381+0000: starting up libvirt version: 8.9.0, qemu version: 7.1.0, kernel: 5.15.81-0-lts,


Code: [Select]

KVM internal error. Suberror: 3
extra data[0]: 0x0000000080000b0e
extra data[1]: 0x0000000000000031
extra data[2]: 0x0000000000000083
extra data[3]: 0x0000000800a68fe0
extra data[4]: 0x0000000000000000
RAX=0000000800a68120 RBX=fffffe000916a090 RCX=00000000c0000101 RDX=00000000ffffffff
RSI=0000000000000015 RDI=fffffe000916a090 RBP=fffffe000916a080 RSP=fffffe0009169fb0
R8 =0000000000000000 R9 =00000000ffffffff R10=0000000000000000 R11=0000000800a61b40
R12=00007fffffff6540 R13=0000000000000000 R14=0000000800bb60a0 R15=0000000800a68120
RIP=ffffffff811338c1 RFL=00010082 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
CS =0020 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0000 0000000000000000 ffffffff 00c00000
DS =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
FS =0013 0000000800288120 ffffffff 00c0f300 DPL=3 DS   [-WA]
GS =001b ffffffff82612000 ffffffff 00c0f300 DPL=3 DS   [-WA]
LDT=0000 0000000000000000 ffffffff 00c00000
TR =0048 ffffffff82612384 00002068 00008b00 DPL=0 TSS64-busy
GDT=     ffffffff826123ec 00000067
IDT=     ffffffff81f5d710 00000fff
CR0=80050033 CR2=ffffffff811338c1 CR3=0000000800a68120 CR4=003506e0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2022-12-28T10:44:48.612670Z qemu-system-x86_64: terminating on signal 15 from pid 2968 (/usr/sbin/libvirtd)
2022-12-28 10:44:49.229+0000: shutting down, reason=destroyed
2022-12-28 10:44:52.406+0000: starting up libvirt version: 8.9.0, qemu version: 7.1.0, kernel: 5.15.85-0-lts,

Some people disable C-States in BIOS to solve this. I will try that.
https://forums.servethehome.com/index.php?threads/topton-jasper-lake-quad-i225v-mini-pc-report.36699/page-100#post-358735

OPNSense is the only VM that crashes like that.

[guenti_r]

We have several OPNSense on top of Proxmox and no issues at all.
Which NIC are in use?

EDIT: maybe provide the related VM settings

[misery]

Which hardware do you use? I have Intel I226-V NICs with Intel Celeron N5105 CPU.

Here are my settings.

Code: [Select]

<domain type="kvm">
  <name>opnsense</name>
  <uuid>79cc93ac-2f7e-4d6d-b99e-910fb93c8296</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://freebsd.org/freebsd/13.1"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">4194304</memory>
  <currentMemory unit="KiB">4194304</currentMemory>
  <vcpu placement="static">4</vcpu>
  <os>
    <type arch="x86_64" machine="pc-i440fx-7.1">hvm</type>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on"/>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" discard="unmap"/>
      <source file="/var/lib/libvirt/images/opnsense.qcow2"/>
      <target dev="vda" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
    </disk>
    <controller type="usb" index="0" model="ich9-ehci1">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x7"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci1">
      <master startport="0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0" multifunction="on"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci2">
      <master startport="2"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x1"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci3">
      <master startport="4"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pci-root"/>
    <controller type="ide" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/>
    </controller>
    <interface type="bridge">
      <mac address="52:54:00:40:22:7c"/>
      <source bridge="br0"/>
      <model type="virtio"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </interface>
    <interface type="network">
      <mac address="52:54:00:92:13:be"/>
      <source network="dmz"/>
      <model type="virtio"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x09" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <channel type="unix">
      <target type="virtio" name="org.qemu.guest_agent.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="vnc" port="-1" autoport="yes">
      <listen type="address"/>
    </graphics>
    <audio id="1" type="none"/>
    <video>
      <model type="vga" vram="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0c" function="0x0"/>
    </hostdev>
    <watchdog model="i6300esb" action="reset">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0a" function="0x0"/>
    </watchdog>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
    </memballoon>
    <rng model="virtio">
      <backend model="random">/dev/random</backend>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0b" function="0x0"/>
    </rng>
  </devices>
</domain>



[guenti_r]

I mean in VM-Config (virtio?)
EDIT: read the codebox...

[guenti_r]

Which hardware do you use? I have Intel I226-V NICs with Intel Celeron N5105 CPU.

Here are my settings.

Uff, a very slow machine.....not enough RAM...

Why pc-i440fx instead of q35?

[misery]

It is fast enough... I never have more than 10 % CPU in usual mode. Memory can be increased up to 16 GB here. But OPNSense uses 1 GB at the moment. I don't think that this will be the reason for the crash.

> Why pc-i440fx instead of q35?
It was the default for FreeBSD. I didn't changed it manually.

[guenti_r]

Fast enough...ok 
I think this is more Alpinelinux or Hardware related. Maybe there is a Forum out there?

[misery]

I don't think that it is Alpine related. I read a lot of Proxmox users that has the problem, too. (like above or the link to the other forum)
What do you think how slow a N5105 is? :-D But ok... that's not the crashing problem.

[guenti_r]

I read a lot of Proxmox users that has the problem, too.

Not really true. "a lot" means many....
If you read carefully, these all have similar Hardware or CPU.
I think this is the way to go (and solve).
We have only Xeon´s or AMD Epyc´s and Server-grade hardware.
No issues at all with OPNsense in KVM.

[misery]

I read a lot of Proxmox users that has the problem, too.

Not really true. "a lot" means many....
If you read carefully, these all have similar Hardware or CPU.

I didn't say they all have "my" hardware but they have Proxmox.
Whatever... I will ask in servethehome forum. Bye! :-)


Edit:
Best help: https://forum.proxmox.com/threads/vm-freezes-irregularly.111494/

[guenti_r]

Sure, all the same crappy China-Hardware.
But it's always the other's fault?