Running in "boot loop" with KVM/Proxmox

[KiX]

Hi OPNsense guys!
I'm new to OPNsense but I'm trying to get it running on my old "ProLiant Microserver N40L" (AMD Turion II Neo N40L Dual-Core Processor) which is running the latest Proxmox VE (5.4-6) but it's driving me crazy because I can't get the OPNsense installer (19.1.4 DVD ISO, checksum correct) running!
I have tried many many different virtualization settings, other BIOS, different CPU types, VirtIO SCSI etc etc. but the bootloader is "looping" and I don't know why.
Attached is the screenshot with the last message I see, then the Proxmox VE BIOS image is back again and OPNsense is loading...
Can anyone please tell me, how I can debug why the installer isn't booting? Maybe my baremetal hardware isn't compatible? 

Thank you so much!

[KiX]

Today I tried the older version 18.7 and it worked without any problems.
Nevertheless, how can I debug the issue with 19.1 so we can find the issue?
Thanks! 

[TheGrandWazoo]

Hi,

I am running OPNSense 19.1.x on Proxmox VE 5.4-x without issues...but I am using Intel-based processors on the hosts. Possible you can screenshot the Hardware and Options and post them.

I have a variety of setups using SeaBIOS/OVMF, SCSI/VirtiIO but all use the Para Virt for network. I have used E1000 but opted for the VirtIO (Paravirt) network adapter.

[KiX]

Hi,
I tried many configurations in the last days but nothing worked, except the 32bit version (OPNsense-19.1.4-OpenSSL-dvd-i386.iso) which is running now with the latest version 19.1.9 with kvm32 and the "hw.mca.enabled=0" hook in loader.conf.local. (according to https://pve.proxmox.com/wiki/PfSense_Guest_Notes)
But it's not the 64bit version
BUT, when I disable "KVM hardware virtualization", the 64bit version with kvm64 is booting & working, but extremly slow and not production ready of course...

Can anyone please give me some advice to get the 64bit version running?

I recorded a video from the VM which is trying to boot and I have here the full specs from my bare metal server:
HP ProLiant Microserver N40L

Video:


CPU:

Code: [Select]

root@pve01:~# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             AuthenticAMD
CPU family:            16
Model:                 6
Model name:            AMD Turion(tm) II Neo N40L Dual-Core Processor
Stepping:              3
CPU MHz:               1500.000
CPU max MHz:           1500.0000
CPU min MHz:           800.0000
BogoMIPS:              2995.12
Virtualization:        AMD-V
L1d cache:             64K
L1i cache:             64K
L2 cache:              1024K
NUMA node0 CPU(s):     0,1
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save
PCI:

Code: [Select]

root@pve01:~# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] RS880 Host Bridge
00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] RS780 PCI to PCI bridge (ext gfx port 0)
00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] RS780/RS880 PCI to PCI bridge (PCIE port 0)
00:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] RS780 PCI to PCI bridge (PCIE port 2)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42)
00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor HyperTransport Configuration
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Address Map
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Miscellaneous Control
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Link Control
01:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
01:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
02:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 02)
03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 10)
04:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5723 Gigabit Ethernet PCIe (rev 10)
Memory:

Code: [Select]

root@pve01:~# dmidecode -t memory
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.6 present.

Handle 0x0014, DMI type 16, 15 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: Single-bit ECC
Maximum Capacity: 8 GB
Error Information Handle: Not Provided
Number Of Devices: 2

Handle 0x0016, DMI type 17, 28 bytes
Memory Device
Array Handle: 0x0014
Error Information Handle: Not Provided
Total Width: 72 bits
Data Width: 64 bits
Size: 8192 MB
Form Factor: DIMM
Set: None
Locator: DIMM0
Bank Locator: BANK0
Type: Other
Type Detail: Synchronous
Speed: 1333 MHz
Manufacturer: Manufacturer00
Serial Number: SerNum00
Asset Tag: Not Specified
Part Number: ModulePartNumber00
Rank: Unknown

Handle 0x0018, DMI type 17, 28 bytes
Memory Device
Array Handle: 0x0014
Error Information Handle: Not Provided
Total Width: 72 bits
Data Width: 64 bits
Size: 8192 MB
Form Factor: DIMM
Set: None
Locator: DIMM1
Bank Locator: BANK1
Type: Other
Type Detail: Synchronous
Speed: 1333 MHz
Manufacturer: Manufacturer01
Serial Number: SerNum01
Asset Tag: Not Specified
Part Number: ModulePartNumber01
Rank: Unknown

[KiX]

Has anyone some ideas? 

[ruffy91]

Try setting the Machine to q35 and the CPU to host.
I have it running with these settings on a similar CPU (although I use OVMF UEFI, not SeaBIOS)

[KiX]

Try setting the Machine to q35 and the CPU to host.
I have it running with these settings on a similar CPU (although I use OVMF UEFI, not SeaBIOS)

Hi @ruffy91, thanks for the idea!
But it's also looping like in the video 

Is there anyway to debug this issue deeper?

[eugenmayer]

Also running on Proxmox 5.x, on a lot of boxes.

This is my VM config (maybe this helps):

Code: [Select]

boot: dcn
bootdisk: scsi0
cores: 2
cpu: host
ide2: none,media=cdrom
memory: 2400
name: gateway
net0: virtio=BYYYYYYYYYYYYYYYYYYY=vmbr30
net1: virtio=ZZZZZZZZZZZZZZZZZZbridge=vmbr0
net2: virtio=BBBBBBBBBBBBBBBBBB,bridge=vmbr0
numa: 0
onboot: 1
ostype: other
protection: 1
scsi0: local:500/vm-500-disk-1.qcow2,size=10G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=XXXXXXXXXXXXXXXXXXXXXXXXX
sockets: 1
startup: order=1
tablet: 0
vga: serial0

Host ist a: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz

I got this KVM setup on about 7 boxes (on different hardware), never encountered what you experienced. Generally as

[KiX]

Thanks @eugenmayer for your config!
But I checked my kvm with your config and my kvm is also running with these settings.
I also tried today to install proxmox from scratch with debian, but It's also not working, same boot loop as in my video.

I don't know if qemu or proxmox is making this issues.

@OPNsense devs, please give me some hints how I can debug this issue deeper so I can help you better.

Thanks!

[KiX]

Hi guys,

I've figured out, that it's an issue from the BSD/HardenedBSD kernel, an erratum/bug from the AMD 10h CPU :

Code: [Select]

pve01 kernel: [  284.573818] SVM: KVM: Guest triggered AMD Erratum 383
Snipped from the pmap.c file from the kernel:

Code: [Select]

/*
* If the kernel is running on a virtual machine, then it must assume
* that MCA is enabled by the hypervisor.  Moreover, the kernel must
* be prepared for the hypervisor changing the vendor and family that
* are reported by CPUID.  Consequently, the workaround for AMD Family
* 10h Erratum 383 is enabled if the processor's feature set does not
* include at least one feature that is only supported by older Intel
* or newer AMD processors.
*/
if (vm_guest != VM_GUEST_NO && (cpu_feature & CPUID_SS) == 0 &&
    (cpu_feature2 & (CPUID2_SSSE3 | CPUID2_SSE41 | CPUID2_AESNI |
    CPUID2_AVX | CPUID2_XSAVE)) == 0 && (amd_feature2 & (AMDID2_XOP |
    AMDID2_FMA4)) == 0)
workaround_erratum383 = 1;
Has anyone some experience with an secure workaround for this? Or have I really switch CPU/Server to get an VM running for OPNsense

[bunchofreeds]

Can you confirm your Proxmox host has the latest BIOS running and any storage controllers are also up to date.
OPNsense from 19.1 onwards uses HardenedBSD 11.2, which in turn is running an updated boot method.
It will require quite a recent BIOS to be operational - usually something released late 2018 to 2019.
Sorry I forget the exact reason for this but I ran into something similar when trying to run OPNsense on Hyper-V.
I believe it is to do with how UEFI is invoked.
A bit of research around HardenedBSD and its boot method from 11.2 onwards should provide some insights.

Hopefully this helps...

[iMx]

Proxmox also does not include the amd64-microcode (nor intel) by default either, so this might be another option.

https://forum.proxmox.com/threads/cpu-microcode-update.44296/

[cj_duke]

Hi guys,

I've figured out, that it's an issue from the BSD/HardenedBSD kernel, an erratum/bug from the AMD 10h CPU :

Code: [Select]

pve01 kernel: [  284.573818] SVM: KVM: Guest triggered AMD Erratum 383
Has anyone some experience with an secure workaround for this? Or have I really switch CPU/Server to get an VM running for OPNsense

Had same issue running into boot-loop with AMD Phenom II X6 1090T cpu. Found a workaround in the following post:
https://forum.opnsense.org/index.php?topic=11419.msg52375#msg52375