Upgrade from 23.1->23.1.3 kernel panic/crashing

[develishh]

It looks like that may be our only option...
The only problem is that we haven't even figured out what is causing the crash.
Let's start by finding out if we are actually getting the same page fault.

This command will give out the most relevant information:

Code Select Expand

tar -xf /var/crash/textdump.tar.last -O | grep Fatal -A 21

It would be great if everyone who is experiencing this or something similar could paste the output of this command (or a screenshot) here. I attached mine in this post:
[Screenshot-2023-04-08.png]
(In order to avoid having to take my router apart I passed the SSD through to a VM)

[myradon]

I've got same problem. I bought a Loksing X86-P2 mini PC 5; CPU Intel J4125 with NICS Intel I226-V with Samsung NVME running (latest) OPNsense 23.1.5_4. I've configured VLAN, IPS in Promiscious Mode.

Monitor just goes blank or see a shutdown with various services shutting down and even speaker beep. It happens after 10 minutes, 3 hours or within couple of seconds.

As suggested I've done a rollback to previous kernel. It doesn't make any difference.

[neonknight]

Same issue here. Fresh install on pcengines apu2c4 and a brand new apu6b4 (which I only bought because I thought that old apu2 was faulty ::) ). I updated the bios of said apu6b4 to the latest release. Opnsense was updated to 23.1.5_4 (mandatory to update, else os-igmp-proxy cannot be installed).

Crash is perfectly reproducible with this command:

Code Select Expand

curl https://3mdeb.com/open-source-firmware/pcengines/apu6/apu6_v4.17.0.3.rom > /dev/null

Upon crash I get loads of serial console output. I attached the beginning of it.

Using the old kernel (FreeBSD 13.1-RELEASE-p5 stable/23.1-n250372-c4ad069e50a) helps, thanks @develishh for the suggestion.

I'm unable to extract crash output, as the file does not exists. My system probably crashes before it can be written.

Code Select Expand

tar -xf /var/crash/textdump.tar.last -O | grep Fatal -A 21
tar: Error opening archive: Failed to open '/var/crash/textdump.tar.last'

[dcdata]

Thank you everyone. I'm relieved to have found this post.
The exact same issue is being experienced on my side using OPNsense in a KVM virtual machine.
Booting the old kernel resolves the problem. (13.1-RELEASE-p5 FreeBSD 13.1-RELEASE-p5 stable/23.1-n250372-c4ad069e50a SMP amd64)

[marcoj90]

I´m not alone...   :D

I have the exact same problem on 2 completly differend systems.

1. System:
An VM running on unRAID with direct access to to the hardware Intel + Broadcom NICs.

This system was running for about 4 years now. But after an updates i got these reboot problems. At 1st i thought it was a dead NIC so i bought a couple of different Broadcom NICs and also tried to use a virtual Network interface ... without succsess.

2. System: Mini PC Celeron J4125 4 x I225-V (bare-metal installation)

Here i got the same reboot problem. So i was using my whole weekend to analyse it a bit more and made about 18 fresh installations only on that device.

As far as i can say, the fresh downloaded Version 23.1 is absolutly stable. It runs without any problems. But also without extenstions (updates required). As soon as the updates are installed, right after the boot completed, the system crashes. But only, if the WAN interface gets connected. If the ethernet cable on the WAN gets removed, the system runs stable ... for some time (a few minutes in my case maximum was about 20min). Trying to switch between the interfaces doesn`t change anything.

This even happen, if im not recover any configuration. Just fresh install, interface assignment and static ip on LAN. After the update to the newest version the problems keep going.

For me it looks like some sort of I/O Problem.

If i just connect one PC to the LAN Interface, connect WAN and don´t use any internet connection, it seems to work till i try to configure something or using the internet connection. Sometimes it worked for a few MB till it crashes.

And this happen on both systems the same way.

Additional Information:
Because i was waiting for the mini pc, i was using a different router as gateway and the OPNsense VM with unplugged WAN Cable for openVPN, DHCP, DNS, DDClient for about 3 weeks without any crashes. Just the WAN ethernet cable disconnected.

[neonknight]

I updated my spare hardware (which crashed too) to 23.1.6. A few simple tests indicate the bug might be gone. I haven't been able to test thoroughly  and haven't tried with my production system (which I don't dare to update yet and risk losing the "good old working kernel").
Has anyone else updated and can confirm?

[develishh]

I have updated about a week ago and the problem seems to be resolved.  :D

[bikemike]

I am having the same issue with random reboots.  There are many, many threads on this in the forums with no real solution dating back to previous versions up to current.  I posted this thread, but not a lot of movement:

https://forum.opnsense.org/index.php?topic=33583.msg162367#msg162367

Was going to try to update the BIOS on my APU1D4, but not hopeful since others in this thread have not seen improvement afterwards.  Wish the OPNsense developers would put some attention on this.  I did however reach the four day mark until this morning when things crashed (twice).  Looks very similar to the output in the YT video someone posted in a previous comment but not related update checks.  Coming from pfSense where I had literally zero stability issues to random reboots nearly every other day is super frustrating.