Messages

@franco note that all the links at https://opnsense.org/download/ still point to the now retired 19.1.0 images. I am sure most of the people trying to download them will know how to work around the issue, but it's not good for the n00bs  ;)

@franco can you please confirm what the direct upgrade from 18.7.10 to 19.1.4 you mentioned should look like in the UI?

I just tried to kick it off on a firewall I had kept back on 18.7, and while 19.1.4 is indeed shown as available, it displays the release notes for the initial 19.1, not 19.1.4 (screenshot attached).

Is this the expected behaviour?

from the 19.1.4 release notes:

An UEFI boot panic scenario was debugged last week with the help of the community. This update includes a fix that will allow the ones affected by this 19.1 issue to upgrade or install (and boot of course) correctly. We are also including the IPsec VTI support and the latest Suricata 4.1.3 with stability and compatibility fixes.

Due to the severity of the UEFI boot panic 19.1.4 will be the new initial release for all upgrades from 18.7 within a day or two depending on additional testing and confirmation. Last but not least there will be new images some time next week to put this fully behind us. Thank you for your patience and understanding. :)

So if you want to be back up quickly you will have to reinstall 18.7, do all upgrades up to and excluding 19.1, and restore/redo your config, then sit tight until the upgrade path is released.

I have just done some tests on our lab DELL Poweredge R340; it's not currently available for me to use, but as long as I don't touch the HDDs, and do it out of hours, I can reboot it as many times as I want, at least for now.

Given the constraint above I focussed on trying to narrow down the conditions for the kernel trap using just a bootable iso.

Franco says the problem is upstream, so instead of booting the OPNsense iso (which for some reason takes half an hour to get to the point of the crash when mounted as virtual ISO via the iDRAC) I used unmodified OS ISOs.

Here are the results so far:

UEFI ENABLED
HardenedBSD-11-STABLE-v1100056.13-amd64-bootonly.iso
doesn't even manage to boot from the iso

UEFI DISABLED
HardenedBSD-11-STABLE-v1100056.13-amd64-bootonly.iso <-
kernel trap 12

FreeBSD-11.1-RELEASE-amd64-bootonly.iso
boots all the way to the installer

HardenedBSD-12-STABLE-v1200058.3-amd64-bootonly.iso
boots all the way to the installer

I then tried to disable virtualization support in the CPU based on some comments about Meltdown/Spectre fixes in the other thread, but I still get kernel trap 12, so that's not a viable workaround.

So at least in the case of Dell bare metal it seems that UEFI is not the culprit, as disabling it doesn't stop the kernel traps. It also seems not to be a FreeBSD 11 problem, as the vanilla FBSD iso works. This leaves changes between FreeBSD 11 and HardenedBSD 11 as the most likely cause for the kernel trap, but looking at the repo it seems the classical needle in a haystack. The interesting result from this testing is that HardenedBSD 12 works, so maybe an easier investigation path could be to look at the changes between HBSD 11 and 12 that are not merged from FBSD? Shawn what do you think?

Another option to collect more data could be to have a 19.1 debug iso (ie one with DDB enabled in the kernel) so we can actually collect core dumps for these crashes. I am sure that given enough time many of us, me included, could set up a HBSD dev environment and build the image myself, but if this can be a useful investigation avenue it seems better if one of the lead devs could just run the existing build workflow with the kernel option set.

I see further up the thread that a number of people complained about the same crashes on ESXi; our own production firewalls run on ESXi 6 but upgraded to 19.1 successfully, I can do some tests in that sense tomorrow, as this seems to imply there might be a simple workaround in the VM settings for the people running ESXi. This could also be a way forward for the people who have kernel traps on overspecced bare metal, at least up to the point the upstream issue is fixed, or OPNSense has moved to HBSD 12 (but that's at least a year away).

Franco,

as someone who has done a lot of support of the stuff he designed and implemented, I can understand your frustration when a lot of people start screaming "it's broken, fix it!" without constructive content; on the other hand while the community can help solving this issue (by providing test resources, reports, patches, or even just the time to test various builds on the hardware we own), this is such a fundamental issue with the end product functionality that I think you should take the lead and help us help you, even if it turns out that the fix will not be found in OPNSense but the underlyig OS..

There are multiple threads about kernel crashes in the forum, and they are a mix of general discussion about the issue, pure complaints and reports from people trying various things. Maybe you should close these and start a new pinned thread, summarising what you know and think of the issue so far in the opening post, and asking people who are unable to boot 19.1 to list hardware/configuration combinations so we can work towards reproduction.