Hi Community,
I need help diagnosing why my OPNsense instance keeps crashing. I am running OPNsense on Proxmox 7.2-3. VM config attached. Crash dump attached.
My first instance kept crashing every day so I decided to install a new VM instance inside Proxmox and have kept my install relatively vanilla with the exception of a handful of port forwards and one theme plugin. After 2 stable days my instance once again crashed and I have the message in the web GUI - "A problem was detected. Click here for more information." I haven't had any issues with Proxmox crashing so the host OS seems to be running just fine. I am still new to OPNsense and don't really know how to debug or troubleshoot. I have spent a lot of time looking through the logs and I don't see anything or don't know where to look. Any help would be most appreciated.
I connected to my firewall and went to /var/carash/textdump.tar.0. I downloaded that file expanded it and I can see in panic.txt that there is a "page fault." I have 16GB of ram in the machine and have allocated 8GB to the VM.
I also found this in msgbuf.txt
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x103
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff81136002
stack pointer = 0x0:0xfffffe00c28dff18
frame pointer = 0x0:0x0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 30927 (python3.9)
trap number = 12
panic: page fault
cpuid = 0
time = 1660675440
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c28dfcd0
vpanic() at vpanic+0x17f/frame 0xfffffe00c28dfd20
panic() at panic+0x43/frame 0xfffffe00c28dfd80
trap_fatal() at trap_fatal+0x385/frame 0xfffffe00c28dfde0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00c28dfe40
calltrap() at calltrap+0x8/frame 0xfffffe00c28dfe40
--- trap 0xc, rip = 0xffffffff81136002, rsp = 0xfffffe00c28dff18, rbp = 0 ---
doreti_exit() at doreti_exit+0x17
KDB: enter: panicBackground about my system:
- Hardware Topton N6005 - from https://www.aliexpress.com/item/3256804173890686.html?spm=a2g0o.order_list.0.0.620e1802vX8kyB (https://www.aliexpress.com/item/3256804173890686.html?spm=a2g0o.order_list.0.0.620e1802vX8kyB)
- Passing through 2 of the NICs for OPNsense - https://www.servethehome.com/how-to-pass-through-pcie-nics-with-proxmox-ve-on-intel-and-amd/ (https://www.servethehome.com/how-to-pass-through-pcie-nics-with-proxmox-ve-on-intel-and-amd/)
- Memory G.Skill RipJaws SO-DIMM Series 16GB - from https://www.amazon.com/gp/product/B08LQG2SDS/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 (https://www.amazon.com/gp/product/B08LQG2SDS/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1)
- Storage Samsung SSD 860 EVO
If this keeps happening I think I may try to scrap Proxmox and run the firewall directly on the hardware to see if that helps although I was really hoping to run a few other VMs on the machine.
hi,
I have this one from Topon https://www.aliexpress.com/item/1005003993468819.html
I'm running proxmox 7.2.7 and i have no issue running opnsense.
I think two cores are enough for vm.
Best regards
Hi,
for me 2 cores in Proxmox for OPNsense were not enough. With 2 core I had crashes. More cores solved it for me. But I do have an one generation older CPU, J4105, with ~30% less performance per core.
You uploaded one of my crash dumps, not yours.
Quote from: milkywaygoodfellas on August 17, 2022, 03:49:43 PM
You uploaded one of my crash dumps, not yours.
🤦♂️ Thanks for the catch! I updated my post...
Have you tried using bridge interfaces instead of PCI passthrough? Just to clarify what is the culprit.
If that works, you can keep on trying to optimize the PCI passthrough, which has some pitfalls as far as I have read (like IOMMU grouping, interrupt mapping and blacklisting so that the linux host does not accidentally use the devices). Some instructions tell that UEFI boot works better and AFAIK, you use BIOS boot.
The topton device does not seem to have a very advanced BIOS, so keeping that out of the equation might help as well.
On a side note: My Proxmox is on 7.2-7, yours is still on 7.2-3?
Ok I did take a look at my boot order in the Bios and it looks like first priority was a listing for Proxmox and the second listing was a UEFI boot. I adjusted my boot order to UEFI as the priority. My machine starts up and then loads the Grub bootloader to start Proxmox. I'll see if that makes any difference...
I'm hitting the same issue on the same device (Topton N6005 fro AliExpress) with 32GB ram and a 1TB PCIE NVME.
https://forum.opnsense.org/index.php?topic=29837.0 (https://forum.opnsense.org/index.php?topic=29837.0)
I did try to run OPNSense in UEFI mode (q35 machine) or the default, with the same results.
when i tried the bridge mode approach I couldn't figure out how exactly to set the bridges for WAN/LAN without going through the management bridge - any ideas to avoid problems ?
Thanks.
@chenganir not using Topton, but currently running OPNsense as Proxmox vm with uefi/q35 and Linux bridges for wan/lan as it's just easier.
Not sure about your question. The short version would be To pick two unused physical ports and create Linux bridges with them. Use VirtIO. Give these bridges to OPNsense. Give Proxmox a cidr in the OPNsense lan subnet on the vmbr used for lan. Connect your VM's there. If you have a switch plug it into the physical port/bridge port used for lan.
Not sure what you mean by going through the management bridge?
@rmlinnovator
Hi, I am running a very similar hardware and no crashes here yet. I am still testing though.
About your configuration I noticed two things you should investigate on:
1) According to Proxmox documentation you should set "machine" option to "q35" if one wants to pass through PCIe hardware.
2) To optimize communication between the host and the guest it is also suggested to install and enable the qemu guest agent
Let me know if this helps.
Best regards
Thanks Vesalius. I did just that, and now i'm facing another issue - my speeds are half of my actual ISP speed. I have one OPNSense running native on a protectli box getting full speed, and when I move the lan/wan over to the OPNSense running in Proxmox I get half of that speed. Same config (xml) on both. I have assigned 4 cores (host type) + AES enabled in the VM, 8GB RAM and 64GB storage.
I'ver tried to apply some tunables I found online, and also disabled all HW offloading. So far no crashing.
I was also wondering what options I need to select for the PCI Passthrough of the NICs ? All-Functions? ROM-Bar? PCIExpress ?
Quote from: fgerardi on August 18, 2022, 09:00:34 AM
@rmlinnovator
Hi, I am running a very similar hardware and no crashes here yet. I am still testing though.
About your configuration I noticed two things you should investigate on:
1) According to Proxmox documentation you should set "machine" option to "q35" if one wants to pass through PCIe hardware.
2) To optimize communication between the host and the guest it is also suggested to install and enable the qemu guest agent
Let me know if this helps.
Best regards
Thanks for these two tips. I appreciate the insight. I have implemented both of them...Let's see...
I had the same issue. There are a bunch of threads on other forums about instability with Proxmox & pfsense/opnsense and these N6005 and N5105 units from Topton/Changwang/King Novy/etc. Proxmox ran just fine, but my OPNsense VM would just stop and say "internal-error." I gave up after trying every tweak in the book and moved on to something else. These machines are not reliable.
Some additional information: I originally migrated the VM from another machine that ran perfect for a long time. Had tons of crashes. Finally did a complete reinstall with a few config changes, imported my config, and things were OK for a few days. Then crashes again.
I'm running OPNSense bare metal on my KingNovy N5105 and it's been solid as a rock.
Quick update. After making the recommended changes. I haven't had any issues as of yet.
🤞🏻
Is your machine running stable now and which changes were you referring to, Reply #9?
I'm about to setup Proxmox on a Topton N5105...
I ended up installing on bare metal and it's been rock solid. I couldn't get Proxmox to be stable enough for my needs. I'm much happier running Opensense on bare metal.
ok thanks, good to know. I might give VM a try and if it's too much trouble do bare metal.
Are you doing backups/snapshots somehow?