OPNsense Forum
Archive => 19.1 Legacy Series => Topic started by: bunchofreeds on February 01, 2019, 12:16:07 am
-
Hi,
Creating a new VM of OPNsense 19.1 ISO under Hyper-V Server 2019 fails on booting.
OPNsense 18.7 ISO mounted against the exact same VM boots and installs successfully
The failure if soon after the OPNsense boot menu displays
Attached is a screenshot from Hyper-V Console of the OPNsense VM
Running Hyper-V under Windows Server 2019 Standard
Generation 2 (No Secure Boot)
Config Version 9.0 of VM
2 vCPU
4 GB Ram
127 GB Dynamic Disk
Hopefully this is enough to help out
-
What's the output of the `bt` command (without the backticks, of course)?
-
The boot is frozen on this output.
Can't enter any commands unfortunately
-
Image downloaded recently that isn't completely booting
OPNsense-19.1-OpenSSL-dvd-amd64.iso
Working image is
OPNsense-18.7-OpenSSL-dvd-amd64.iso
Boot hangs/freezes shortly after OPNsense boot menu
-
I haven't tried to install the release version but the beta was not installing properly on hyper-v. I'll give it a try on my WS 2012 hyper-v server and report back.
-
@Franco - did the last loader you sent out for test make it into the release version?
That one failed completely, at least with the older version I could get it to install using the CTRL-C method.
-
Hi,
I had similar Issues in my test environment. What helped was to configure the VM to be a Gen 1 VM which does not use UEFI boot.
-
Hopefully 19.1 can work using Gen 2 though, as the VM's can be built with only the virtual hardware that is required (No Comm, Serial and IDE etc.). This is good for minimising the virtual hardware and drivers for network appliances like firewalls and routers.
I can currently boot then install 18.7 using the Ctrl+C method for when the install hangs. So my next solution is to do this then upgrade to 19.1
Unfortunately with booting directly with the 19.1 ISO, it freezes entirely shortly after the boot menu for me?
-
Just confirmed that Gen 1 boots successfully with 19.1
-
After successfully installing 18.7 on a Gen 2 Hyper-V VM, then upgrading to 19.1. It fails on boot with the same error as attached in my earlier post.
So this seems to be the same error as others have posted after upgrading. It is possible this is not specific to Hyper-V as others have had the same 'Fatal trap 12: page fault while in kernel mode' when upgrading on a bare metal platform.
Hopefully this helps
-
bunchofreeds, could you also try booting with vm.pmap.pti set to 0?
-
@lattera I have tried to update configuration in /boot/loader.conf but it seems to get reset after a reboot?
I have to boot OPNsense using the old kernel (option 5 from the boot menu)
Then I'm using WinSCP to SSH to OPNsense and update /boot/loader.conf.
The update commits and I can open the file again and it has updated, but resets after a reboot.
Have also tried using Vi
Sorry I'm not great with FreeBSD and mostly use Windows.
-
OK sorry I did some reading
Pressed ESC at the boot prompt
Typed set vm.pmap.pti=0
Typed boot (hopefully this is correct)
The OS still has the error unfortunately
-
Typed show at the OK prompt
Attached screenshot showing option set
Unfortunately boot still fails
This is a test VM so not important. Thankyou for your help so far in understanding this
-
I tried updating my 18.7 system to 19.1 and it would not boot. This is on windows server 2012R2 hyper-v. A screen capture is attached. It's not halted. It will accept commands. If anyone has a suggestion to try something, let me know.
-
I'm getting the same kernel panic on ESXi 6.7 (https://forum.opnsense.org/index.php?topic=11419.0).
-
I halted the VM, then rebooted it. The update completed and afterwards, the system booted properly.
-
After rebooting, I was going to try rebooting again from the gui and I'm getting this error:
OPNsense\Core\HaltController handler class cannot be loaded.
It halted okay from the console.
Reboot seems to be okay.
I rebooted again the problem remained. Seems to be a bug or maybe a left-over from the error during the upgrade.
-
I also came across this problem on hyper-v and on 2016 and on windows10 help to unblock.
I tried to set it to 0 when loading an error and an eran virtual machine goes out how to bypass the error.
Thank .
-
I have the same exact issue using KVM. I've booted to kernel.old in the meantime until this gets resolved.
-
Is there an update yet from the OPNSense team? Easy to reproduce even on a Windows 10
hyper-V box. Would the next update cover this issue?
-
There's no solution that satisfies everyone yet. The OS kernel version changed and we are stuck with going thorough thousands of changes to find the issue while still fixing bugs and adding features for OPNsense.
Some issues seem to be resolved or worked around easily, but there are already multiple ones for multiple scenarios and we usually start at "won't boot" which makes it harder to see which is which...
Sorry,
Franco
-
After reaching the Boot Loader screen,
Pressing ESC to enter the loader Prompt
Enter 'start' which I believe loads the kernel and modules
Enter 'lsmod' which lists all loaded files for verification
Enter 'boot' then fails immediately with 'kernel trap 12 with interrupts disabled' ... etc. error
Maybe this will help someone more knowledgeable than me as to what is causing this?
-
19.1.2 released, I didn't see any changes in this regard in the changelog. However I tried to (after taking snapshots) my test-environment from 18.x to 19.1.2. However, as far as the logs go, I see it first tries to 19.1:
***GOT REQUEST TO UPGRADE: maj***
Fetching packages-19.1-OpenSSL-amd64.tar: ...............................pgrep: Cannot open pidfile `/tmp/opnsense-fetch.pid.oUd52D': No such file or directory
done
Fetching base-19.1-amd64.txz: ........ done
Fetching kernel-19.1-amd64.txz: .... done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Extracting packages-19.1-OpenSSL-amd64.tar... done
Extracting base-19.1-amd64.txz... done
Installing kernel-19.1-amd64.txz... done
Please reboot.
***REBOOT***
19.1.2 update was available in the updates-list though. Would this mean we will have a hard time updating these affected machines to a 19.x version that has this issue fixed later on, as it'll always first update to this non-working version? So maybe offtopic here, but how do we update to a 19.1.x version in this case, skipping the faulty 19.1 release?
-
If the price is right, I will be looking to acquire a budget-friendly system on which to run Hyper-V. HardenedBSD's budget for this kind of thing would be $500 USD.
Just a quick reminder that HardenedBSD accepts donations, both monetary and hardware. We appreciate all contributions of any kind.
-
A hosted system with full access via VPN wouldnt help?
-
Well, as stated by multiple people, any somehow Windows 10 Pro machine will do, as you can just enable Hyper-V and installing a Gen2 VM with OPNSense shows the same issue. While I expect the issue for Gen2 VM's the same as with UEFI enabled bare-metal hardware, I can't of course not sure of that. HAving said that, it's extremely easy to create an environment that manifestates the issue.
-
Well, as stated by multiple people, any somehow Windows 10 Pro machine will do, as you can just enable Hyper-V and installing a Gen2 VM with OPNSense shows the same issue. While I expect the issue for Gen2 VM's the same as with UEFI enabled bare-metal hardware, I can't of course not sure of that. HAving said that, it's extremely easy to create an environment that manifestates the issue.
The problem is that all my systems run HardenedBSD. ;P
My employer has lent me a laptop on which I can install the 180 day Windows Server 2019 eval. I'll be working on that today. I hope to have some results to report back within the next week or two.
For full Hyper-V debugging support, I'd need a permanent system. I've set up an Amazon Wishlist for HardenedBSD: https://smile.amazon.com/registry/wishlist/2AKXCIOXYO28N/ref=cm_sw_r_cp_ep_ws_MuDECbATM7CVZ
-
So you need a 550 €-NUC with an i5 for testing? Really?
-
So you need a 550 €-NUC with an i5 for testing? Really?
All Core i* CPUs have the SLAT instruction, which Hyper-V requires. The Intel NUC uses Core i* processors.
-
NUC's are awesome, but expensive.
Any consideration for second hand like an HP 8300 Elite with an i7, or an HP Z230 with i7 or E3-1200 which is a little closer to a server with ECC.
Not as cool obviously, but cheap, reliable and quiet with built in PSU etc.
Still no luck with HardenedBSD? Did you get to work with your colleague using Hyper-V?
-
NUC's are awesome, but expensive.
Any consideration for second hand like an HP 8300 Elite with an i7, or an HP Z230 with i7 or E3-1200 which is a little closer to a server with ECC.
Not as cool obviously, but cheap, reliable and quiet with built in PSU etc.
Still no luck with HardenedBSD? Did you get to work with your colleague using Hyper-V?
HardenedBSD's definitely open to hardware donations of any kind, especially those that fit our needs. :)
My employer has lent me a laptop on which I installed an evaluation version of Windows Server 2019. I've got Hyper-V on it. As of a few hours ago, I was able to reproduce the issue. This is gonna be a tough bug to figure out, especially since the kernel debugger freezes and doesn't accept input. But, I'll do my best to figure this out and hopefully provide a patch.
-
I/we Really appreciate your efforts Lattera.
Like you say, this will be difficult to diagnose
This might possibly get you closer to the issue??
After reaching the Boot Loader screen,
Pressing ESC to enter the loader Prompt
Enter 'start' which I believe loads the kernel and modules
Enter 'lsmod' which lists all loaded files for verification
Enter 'boot' then fails immediately with 'kernel trap 12 with interrupts disabled' ... etc. error
Maybe this will help someone more knowledgeable than me as to what is causing this?
-
I managed to capture an output which I believe is just before the system halts
OK boot
Booting...
Start @ 0xffffffff80316000 ...
EFI framebuffer information:
addr, size 0xf8000000, 0x800000
dimensions 1024 x 768
stride 1024
masks 0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
This is 'after' the above 'start' and then 'boot' options are entered and 'before' the later Trap 12 message is displayed.
I got this by pressing 'enter' to commit the 'boot' command, then immediately pressing the 'pause' icon for the Hyper-VM from it's console window...
-
I'm seeing the same issue. I've also reproduced on vanilla HardenedBSD 11-STABLE. I'm now building a custom build of HardenedBSD 11-STABLE with extra special debugging stuffs. :)
-
Good luck lattera!
Let me know if you create any images you would like me to test against my setup
Just out of interest and definitely for another day, I cannot boot at all using HardenedBSD-12-STABLE-v1200058.3-amd64-disc1
Hyper-V cannot detect the ISO as bootable (I'm downloading the ISO again in case its corrupt)
Edit: Downloaded again and still not bootable with 12
-
I've uploaded the custom, debug build of HardenedBSD 11-STABLE/amd64 here: https://hardenedbsd.org/~shawn/opnsense/2019-03-07_hbsd_11-stable_disc1.iso
-
So, I figured out what's wrong. Now to figure out how to fix it.
What's happening is that the FPU (Floating Point Unit) boot-time initialization code is trying to patch another part of executable kernel code. However, this executable code was not marked as writable. The reason HardenedBSD fails to boot is because HardenedBSD enforces the lack of the write permission whereas FreeBSD does not.
I'll talk with Oliver Pinter, my cofounder with HardenedBSD, who wrote our W^X implementation. I may also bring in some FreeBSD developers, the ones who work on the UEFI loader.
I believe there's a discrepancy in how the BIOS loader and the UEFI loader load the kernel into memory. The BIOS loader may very well set the writable permission for the kernel whereas the UEFI loader may not.
Here's how I reached this conclusion (links valid as of 07 Mar 2019 11:33 EST):
The failure happens here: https://github.com/HardenedBSD/hardenedBSD/blob/hardened/11-stable/master/sys/amd64/amd64/fpu.c#L241
The `ctx_switch_xsave` symbol is a function, defined here:
https://github.com/HardenedBSD/hardenedBSD/blob/hardened/11-stable/master/sys/amd64/amd64/cpu_switch.S#L141
So the kernel is trying to modify the third byte in the `ctx_switch_xsave` function, doing a bitwise OR with value 0x10 (16 decimal).
The kernel panic states that the memory cannot be modified, which means that the write permission is disabled for that location in memory.
I hope this makes sense. If it doesn't, please let me know. I'll try to explain differently.
-
Stunning work lattera!
I have downloaded the ISO, but will need guidance on what to do you help from here.
I also understand what you have said and see that it will require some input from other members to correctly resolve it.
Again... brilliant work!
Will wait and check back in to for any progress or if I can test any fixes.
-
Stunning work lattera!
I have downloaded the ISO, but will need guidance on what to do you help from here.
I also understand what you have said and see that it will require some input from other members to correctly resolve it.
Again... brilliant work!
Will wait and check back in to for any progress or if I can test any fixes.
If you have Hyper-V with Generation 2 support available, or have an affected physical system, test booting the ISO. If you're booting a physical system, go ahead and burn that ISO to a DVD and boot it. If you can, try booting in both BIOS and UEFI mode. There's no need to complete an installation, simply booting the ISO will suffice.
Funny enough, pfSense's Jim Thompson hinted to me that we might be missing a couple or a few commits in our 11.2-RELEASE branch: https://reviews.freebsd.org/D14768
I'll MFC those commits, rebuild a new ISO, and retest on my end. I can share the ISO just like I did in my previous post.
-
lattera,
Booted the ISO and received the panic on Proxmox VE.
Copied the kernel from your tarball from the "other" thread and it worked without issues.
Thanks for your continuous work.
Kev a.k.a. TheGrandWazoo
-
Downloaded and booted a Gen 2 VM with the ISO
2019-03-07_hbsd_11-stable_disc1.iso.
Same Kernel panic unfortunately.
Gen 1 with the same ISO boots to the installer
-
Yup. Kernel panic with that ISO is to be expected. There's only two modifications from vanilla HardenedBSD 11-STABLE in that build:
1. Debug mode for "ALL THE THINGS!" (CFLAGS+="-g -O0", removal of -O2 CFLAG, STRIP="")
2. Enable WITNESS
I'm building a new version of 11-STABLE with the proper commits from upstream MFC'd. I hope to have this new build tested on my systems before I snore loudly in my wife's right ear tonight. If all goes well, I'll push the commits up to GitHub and publish the new ISO for wider testing.
-
Thanks lattera,
I know I keep saying thanks, but it really is awesome the effort you guys are putting in to progress this excellent product.
-
Getting closer! I'm still getting a kernel panic, but this time in a different place.There's probably more commits to MFC. I've posted the ISO in case anyone wants to test: https://hardenedbsd.org/~shawn/opnsense/2019-03-07_hbsd_11-stable_disc1-02.iso
For those not running Hyper-V (bare metal systems especially) and getting the fpuinit crash, please test this new ISO and report back.
Screenshot attached.
-
Hyper-V Gen 2
Kernel Panic but at different Address
-
Hi,
tested on a bare metal system with uefi boot.
This board has no legacy boot option anymore. :-\
My CPU is an i3-8100T on Fujitsu Mainboard D3633-S.
I also get the "new" kernel panic.
-
2019-03-07_hbsd_11-stable_disc1-02.iso
Server 2019 Hyper-V:
(https://i.postimg.cc/wxSwQRg8/w2019.png)
W10 1809 Hyper-V (so pretty much same Windows base, but it does show different addresses):
(https://i.postimg.cc/LXxVPhD7/w10.png)
spare HPe DL360 gen9 (Xeon E5-2630 v4 for what is matters) - UEFI boot (though iLO):
(https://i.postimg.cc/t45TzP0B/gen9.png)
loads extremely slow for some reason, but boots perfectly fine.
If you want me to do specific tests on any of them, just ask.
-
So, I figured out what's wrong. Now to figure out how to fix it.
[...]
I hope this makes sense. If it doesn't, please let me know. I'll try to explain differently.
Nice troubleshooting. Well done.
-
Hi,
I'm having the same issue on Hyper-V with Gen2 machines. During the installer phase, I get as far as selecting a virtual hard drive, but the installer interface then hangs when selecting the drive. I'm still able to switch to other VTYs but wasn't able to actually install the OS.
I built a Gen1 machine with the exact same specifications and was able to install OPNsense that way - selecting GPT/UEFI for the disk, so as to boot it with the Gen2 VM (which shared the same VHD). I got the kernel panic described in this post, however escaping at the loader prompt and setting vm.pmap.pti=0 did actually allow me to boot OPNsense 19.1.4! I updated /boot/loader.conf afterwards, changing vm.pmap.pti="1" to vm.pmap.pti="0", and am able to reliably reboot the system without a panic. Hopefully that helps someone else with the same issue.
I actually ran into this issue some time ago when upgrading the OPNsense appliances we have at work to 19.1 - they are all Gen2 VMs, however I initially dismissed it as an oddity with the upgrade process and simply reverted to a snapshot of the installation prior to upgrade (18.7) - wish I'd reported it then as I may have been able to contribute to lattera's fantastic diagnosis work!