VM of OPNsense on Hyper-V Win Svr 2019 Std - Fails

[bunchofreeds]

NUC's are awesome, but expensive.
Any consideration for second hand like an HP 8300 Elite with an i7, or an HP Z230 with i7 or E3-1200 which is a little closer to a server with ECC.
Not as cool obviously, but cheap, reliable and quiet with built in PSU etc.

Still no luck with HardenedBSD? Did you get to work with your colleague using Hyper-V?

[lattera]

NUC's are awesome, but expensive.
Any consideration for second hand like an HP 8300 Elite with an i7, or an HP Z230 with i7 or E3-1200 which is a little closer to a server with ECC.
Not as cool obviously, but cheap, reliable and quiet with built in PSU etc.

Still no luck with HardenedBSD? Did you get to work with your colleague using Hyper-V?

HardenedBSD's definitely open to hardware donations of any kind, especially those that fit our needs. :)

My employer has lent me a laptop on which I installed an evaluation version of Windows Server 2019. I've got Hyper-V on it. As of a few hours ago, I was able to reproduce the issue. This is gonna be a tough bug to figure out, especially since the kernel debugger freezes and doesn't accept input. But, I'll do my best to figure this out and hopefully provide a patch.

[bunchofreeds]

I/we Really appreciate your efforts Lattera.

Like you say, this will be difficult to diagnose

This might possibly get you closer to the issue??

After reaching the Boot Loader screen,
Pressing ESC to enter the loader Prompt
Enter 'start' which I believe loads the kernel and modules
Enter 'lsmod' which lists all loaded files for verification

Enter 'boot' then fails immediately with 'kernel trap 12 with interrupts disabled' ... etc. error

Maybe this will help someone more knowledgeable than me as to what is causing this?

[bunchofreeds]

I managed to capture an output which I believe is just before the system halts

OK boot
Booting...
Start @ 0xffffffff80316000 ...
EFI framebuffer information:
addr, size   0xf8000000, 0x800000
dimensions   1024 x 768
stride      1024
masks      0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000

This is 'after' the above 'start' and then 'boot' options are entered and 'before' the later Trap 12 message is displayed.

I got this by pressing 'enter' to commit the 'boot' command, then immediately pressing the 'pause' icon for the Hyper-VM from it's console window...

[lattera]

I'm seeing the same issue. I've also reproduced on vanilla HardenedBSD 11-STABLE. I'm now building a custom build of HardenedBSD 11-STABLE with extra special debugging stuffs. :)

[bunchofreeds]

Good luck lattera!

Let me know if you create any images you would like me to test against my setup

Just out of interest and definitely for another day, I cannot boot at all using HardenedBSD-12-STABLE-v1200058.3-amd64-disc1
Hyper-V cannot detect the ISO as bootable (I'm downloading the ISO again in case its corrupt)

Edit: Downloaded again and still not bootable with 12

[lattera]

I've uploaded the custom, debug build of HardenedBSD 11-STABLE/amd64 here: https://hardenedbsd.org/~shawn/opnsense/2019-03-07_hbsd_11-stable_disc1.iso

[lattera]

So, I figured out what's wrong. Now to figure out how to fix it.

What's happening is that the FPU (Floating Point Unit) boot-time initialization code is trying to patch another part of executable kernel code. However, this executable code was not marked as writable. The reason HardenedBSD fails to boot is because HardenedBSD enforces the lack of the write permission whereas FreeBSD does not.

I'll talk with Oliver Pinter, my cofounder with HardenedBSD, who wrote our W^X implementation. I may also bring in some FreeBSD developers, the ones who work on the UEFI loader.

I believe there's a discrepancy in how the BIOS loader and the UEFI loader load the kernel into memory. The BIOS loader may very well set the writable permission for the kernel whereas the UEFI loader may not.

Here's how I reached this conclusion (links valid as of 07 Mar 2019 11:33 EST):

The failure happens here: https://github.com/HardenedBSD/hardenedBSD/blob/hardened/11-stable/master/sys/amd64/amd64/fpu.c#L241

The `ctx_switch_xsave` symbol is a function, defined here:

https://github.com/HardenedBSD/hardenedBSD/blob/hardened/11-stable/master/sys/amd64/amd64/cpu_switch.S#L141

So the kernel is trying to modify the third byte in the `ctx_switch_xsave` function, doing a bitwise OR with value 0x10 (16 decimal).

The kernel panic states that the memory cannot be modified, which means that the write permission is disabled for that location in memory.

I hope this makes sense. If it doesn't, please let me know. I'll try to explain differently.

[bunchofreeds]

Stunning work lattera!

I have downloaded the ISO, but will need guidance on what to do you help from here.

I also understand what you have said and see that it will require some input from other members to correctly resolve it.

Again... brilliant work!

Will wait and check back in to for any progress or if I can test any fixes.

[lattera]

Stunning work lattera!

I have downloaded the ISO, but will need guidance on what to do you help from here.

I also understand what you have said and see that it will require some input from other members to correctly resolve it.

Again... brilliant work!

Will wait and check back in to for any progress or if I can test any fixes.

If you have Hyper-V with Generation 2 support available, or have an affected physical system, test booting the ISO. If you're booting a physical system, go ahead and burn that ISO to a DVD and boot it. If you can, try booting in both BIOS and UEFI mode. There's no need to complete an installation, simply booting the ISO will suffice.

Funny enough, pfSense's Jim Thompson hinted to me that we might be missing a couple or a few commits in our 11.2-RELEASE branch: https://reviews.freebsd.org/D14768

I'll MFC those commits, rebuild a new ISO, and retest on my end. I can share the ISO just like I did in my previous post.

[TheGrandWazoo]

lattera,

Booted the ISO and received the panic on Proxmox VE.

Copied the kernel from your tarball from the "other" thread and it worked without issues.

Thanks for your continuous work.

Kev a.k.a. TheGrandWazoo

[bunchofreeds]

Downloaded and booted a Gen 2 VM with the ISO
2019-03-07_hbsd_11-stable_disc1.iso.

Same Kernel panic unfortunately.

Gen 1 with the same ISO boots to the installer

[lattera]

Yup. Kernel panic with that ISO is to be expected. There's only two modifications from vanilla HardenedBSD 11-STABLE in that build:

1. Debug mode for "ALL THE THINGS!" (CFLAGS+="-g -O0", removal of -O2 CFLAG, STRIP="")
2. Enable WITNESS

I'm building a new version of 11-STABLE with the proper commits from upstream MFC'd. I hope to have this new build tested on my systems before I snore loudly in my wife's right ear tonight. If all goes well, I'll push the commits up to GitHub and publish the new ISO for wider testing.

[bunchofreeds]

Thanks lattera,

I know I keep saying thanks, but it really is awesome the effort you guys are putting in to progress this excellent product.

[lattera]

Getting closer! I'm still getting a kernel panic, but this time in a different place.There's probably more commits to MFC. I've posted the ISO in case anyone wants to test: https://hardenedbsd.org/~shawn/opnsense/2019-03-07_hbsd_11-stable_disc1-02.iso

For those not running Hyper-V (bare metal systems especially) and getting the fpuinit crash, please test this new ISO and report back.

Screenshot attached.