OPNsense Forum

Archive => 19.1 Legacy Series => Topic started by: tamer on February 01, 2019, 09:51:22 pm

Title: Kernel panic after upgrade
Post by: tamer on February 01, 2019, 09:51:22 pm
After updating to 19.1, rebooting will cause kernel panic:


Fatal trap 12: page fault while in kenl mode
cpuid = 0; apic id = 00
fault virtual address    = 0xd5
fault code               = supervisor read data, page not present
instruction pointer      = 0x20:0xffffffff80f3c7b0
stack pointer            = 0x20:0xffffffff8259e6b0
frame pointer            = 0x20:0xffffffff8259e7a0
code segment             = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres1, long 1, def32 0, gran 1
processor eflags         = interrupt enabled, resume, IOPL = 0
current process          = 0 ()
[ thread pid 0 tid 0 ]
Stopped at        vm_map_lookup+0x70:      cmpb     $0,0xd5(%rbx)


running bt:

Tracing pid 0 tid 0 td 0xffffffff8202d260
vm_map_lookup() at vm_map_lookup+0x70/frame 0xffffffff8259e7a0
vm_fault_hold() at vm_faulthold+0x66/frame 0xffffffff8259e8e0
vm_fault() at vm_faul+0x75/frame 0xffffffff8259ea920
trap_pfault() at trap_pfault+0x14c/frame 0xffffffff8259e980
trap() at trap+0x2c7/frame0xffffffff8259ea90
calltrap() at calltrap+0x8/frame 0xffffffff8259ea90
--- trap 0xc, rip = 0xffffffff80bc0896, rsp = 0xffffffff8259eb60, rbp = 0xffffffff8259eb70 ---
mi_startup() at mi_startup+0xc6/frame 0xffffffff8259eb70
btext() at btext+0x2c


I'm running in ESXi 6.7, this issue wasn't tested on 6.5 though. Other upgrades do not cause panic (tested 18.1->18.7).

Let me know if there is anything in particular I need to check. I have downgraded to 18.7.10_4 (using a snapshot) for now.

Edit: I forgot to mention that using kernel.old after upgrade boots without issue.
Title: Re: Kernel panic after upgrade
Post by: dragon2611 on February 01, 2019, 11:12:06 pm
Dell r415 also kernel panics and reboots after tying to upgrade it to 19.1

Managed to get into it's idrac and boot the old kernel.
 >:(
Title: Re: Kernel panic after upgrade
Post by: somy on February 05, 2019, 03:47:07 pm
Adding HP 290 G1 sff to the list of kernel panics after upgrade....
kernel.old working...
Title: Re: Kernel panic after upgrade
Post by: shermdog on February 05, 2019, 10:39:22 pm
Also seeing this w/ Intel i3-8100 (Gigabyte B360M DS3H mobo).

(https://i.imgur.com/0hvmQ5Jl.jpg)
Title: Re: Kernel panic after upgrade
Post by: greY on February 06, 2019, 06:57:24 am
Have the same issue here on Hyper-V 9.0 after updating via GUI.
Title: Re: Kernel panic after upgrade
Post by: velvet on February 06, 2019, 07:43:48 am
I wanted to start a fresh installation - kernel panic!
on Dell PowerEdge R340!
No workaround helps! does anyone have a tip or any other help?
Title: Re: Kernel panic after upgrade
Post by: cguilford on February 06, 2019, 01:40:04 pm
I'm running this on my Dell Poweredge R210.  Have you all made sure that your Bios and firmware are up to date?  I made sure mine where all up to date before I built the box, as I just put this together last week and seems to be running well. 
Title: Re: Kernel panic after upgrade
Post by: velvet on February 06, 2019, 05:00:41 pm
I'm running this on my Dell Poweredge R210.  Have you all made sure that your Bios and firmware are up to date?  I made sure mine where all up to date before I built the box, as I just put this together last week and seems to be running well.

bios is up2date!
it's frustrating! I'm not the only one with the kernel panic!
even under vmware there are these problems! that's a joke!
Title: Re: Kernel panic after upgrade
Post by: golebole on February 07, 2019, 07:02:41 pm
Same problem, also i3 8100 and MSI board. Downgrade now until this is fixed.
Title: Re: Kernel panic after upgrade
Post by: lattera on February 07, 2019, 07:27:52 pm
What happens when you set:

hw.ibrs_disable=1
vm.pmap.pti=0
Title: Re: Kernel panic after upgrade
Post by: bunchofreeds on February 08, 2019, 03:08:16 am
This error is the same as what I and others have experienced in this thread

https://forum.opnsense.org/index.php?topic=11403.0

Hopefully there is a solution soon.

Edit: My particular error message is the same (Verbatim) as what Shremdog and velvet have posted.
Title: Re: Kernel panic after upgrade
Post by: neovs on February 08, 2019, 09:35:22 am
After entering
hw.ibrs_disable = 1
vm.pmap.pti = 0
Download successful.
What you need to change so as not to constantly enter these commands.
Title: Re: Kernel panic after upgrade
Post by: cguilford on February 08, 2019, 01:33:49 pm
Would putting them under System -> Settings -> Tunables work?
Title: Re: Kernel panic after upgrade
Post by: RNHurt on February 08, 2019, 03:56:10 pm
I just upgraded my Acer Aspire AMD machine from 18.7.10 --> 19.1.1 and it threw a kernel panic.  I was able to get back up and running by choosing the old kernel at the boot menu but I don't feel like this is a good solution.

Here's a screenshot of my panic:
Title: Re: Kernel panic after upgrade
Post by: buddwm on February 08, 2019, 04:07:00 pm
What happens when you set:

hw.ibrs_disable=1
vm.pmap.pti=0

Doesn't setting those bypass/disable Spectre and Meltdown mitigations?
Title: Re: Kernel panic after upgrade
Post by: myksto on February 08, 2019, 05:50:07 pm
What happens when you set:

hw.ibrs_disable=1
vm.pmap.pti=0

Where do we have to set those parameters?
Title: Re: Kernel panic after upgrade
Post by: greY on February 08, 2019, 07:06:41 pm
Those parameters are at System > Settings > Tunables
...you have to modify them there.
Title: Re: Kernel panic after upgrade
Post by: myksto on February 08, 2019, 07:15:06 pm
Those parameters are at System > Settings > Tunables
...you have to modify them there.
Ok but, wait a minute please.
I understand that those parameters are to be used in case of kernel panic.
They can be set in the GUI.
In case of kernel panic system doesn't boot so I can I change those parameters in the GUI?
Maybe they have to be changed before the ugrade?
I'm on 18.7 yet and I can't find those parameters in System -> Settings -> Tunables: maybe thare are to be addedd?
Can you please try to explain exactly what to do?

Thanks anche cheers,
Michele.
Title: Re: Kernel panic after upgrade
Post by: lattera on February 08, 2019, 07:41:51 pm
You can also set them for only the current boot by escaping to the loader prompt in the bootloader. So when you see the OPNsense boot menu, hit select option 3, then type:

set vm.pmap.pti="0"
set hw.ibrs_disable="1"
boot

Doing this does disable Meltdown/Spectre mitigations. But only for that one boot just to see if that's the problem.
Title: Re: Kernel panic after upgrade
Post by: greY on February 08, 2019, 11:00:34 pm
how ever this settings did not helped me  :o  (see attachment)
Title: Re: Kernel panic after upgrade
Post by: bunchofreeds on February 09, 2019, 12:13:44 am
These switches do not work for me either unfortunately. Running them at boot after confirming they are set with 'show'

For me this is what works and doesn't - All under Hyper-V 2019

19.1 Gen 2 - Fails (Have tried all manner of boot time switches offered here so far)
19.1 Gen 1 - Works
18.7 Gen 2 - Works
18.7 Gen 1 - Works

From what I understand, Gen 1 Hyper-V Hardware virtualisation boots with BIOS and IDE. Gen 2 Boots with UEFI and SCSI

https://www.serverwatch.com/server-tutorials/hyper-v-2012-r2-pros-and-cons-of-generation-1-vs.-generation-2-vms.html

Title: Re: Kernel panic after upgrade
Post by: newsense on February 09, 2019, 05:52:13 am
how ever this settings did not helped me  :o  (see attachment)

Would this apply to a Virtualbox 6.0.4 VM which ran the 18.x series last year without issues on a Win10 host ? The system is powered by a rather old Athlon 2 with 3 cores + HT and trying to boot 19.1 following the 18.7.10_4 reboot completely freezes the host machine.
Title: Re: Kernel panic after upgrade
Post by: Aloist on February 09, 2019, 05:42:21 pm
Did anyone succeed with booting 19.1 on a Dell Poweredge R410?
I am reluctant to upgrade not knowing that it will work on my hardware.

A question in that context: is there a way to make a full backup of all files, before an upgrade attempt, and go back in case the upgrade fails?

I assume just booting an older kernel might not work, as a lot of system files, besides the kernel, will have been replace by the upgrade, and they may no longer work with the older kernel.
Title: Re: Kernel panic after upgrade
Post by: newsense on February 09, 2019, 07:06:10 pm
You can always boot 19.1 from a stick in live mode and test things out without changing anything on the box.

If things look good you can even do a new install importing the config in the process.
Title: Re: Kernel panic after upgrade
Post by: myksto on February 09, 2019, 08:04:25 pm
I've two Dell Power Edge 1950 and I am reluctant to upgrade not knowing that it will work on my hardware too.

Is there any way to avoid this kernel panic?

I saw that version 19.1.1 has been released: will it solve this problem also?

Cheers,
Michele.
Title: Re: Kernel panic after upgrade
Post by: newsense on February 09, 2019, 08:08:32 pm
The post above yours would have been a good starting point... ::)
Title: Re: Kernel panic after upgrade
Post by: myksto on February 09, 2019, 08:17:08 pm
The post above yours would have been a good starting point... ::)

You mean your advice about using a pen drive to test the hardware or even reinstall the server by zero?
Unfortunately my servers need to work 7/24 and I can't do test with them.

I'm hoping some guru will find a "better" solution.  ;)

Cheers,
Michele.
Title: Re: Kernel panic after upgrade
Post by: newsense on February 09, 2019, 08:29:39 pm
The better solution is a proper testing environment. If all your options are "Testing in Production" then it will always be a gamble and no 'guru' will ever be able to help.

The most appropriate way to do it if production is the only option would be to engage the vendor support team - probably Deciso in this case ? - to assist with the upgrade process.
Title: Re: Kernel panic after upgrade
Post by: Aloist on February 10, 2019, 11:08:14 am
You can always boot 19.1 from a stick in live mode and test things out without changing anything on the box.

If things look good you can even do a new install importing the config in the process.

How do I get my current configuration onto this boot stick?

And what if the boot problems of the new FreeBSD release are related to the Raid controller or Raid configuration in the Dell hardware? That will not become apparent by booting from a stick, when the disks are not involved.
Title: Re: Kernel panic after upgrade
Post by: myksto on February 10, 2019, 07:14:23 pm
You can always boot 19.1 from a stick in live mode and test things out without changing anything on the box.

If things look good you can even do a new install importing the config in the process.

How do I get my current configuration onto this boot stick?

And what if the boot problems of the new FreeBSD release are related to the Raid controller or Raid configuration in the Dell hardware? That will not become apparent by booting from a stick, when the disks are not involved.

Infact booting with a usb stick not involving server hardware is almost no sense and of no use.
Title: Re: Kernel panic after upgrade
Post by: newsense on February 10, 2019, 10:09:12 pm

You can also set them for only the current boot by escaping to the loader prompt in the bootloader. So when you see the OPNsense boot menu, hit select option 3, then type:

set vm.pmap.pti="0"
set hw.ibrs_disable="1"
boot

Doing this does disable Meltdown/Spectre mitigations. But only for that one boot just to see if that's the problem.

This was required on the following CPU running VBox 6.0.4. Opnsense upgrade would otherwise freeze the Win10 host on reboot into the 19.1 kernel and the upgrade wouldn't continue. By applying the commands on each reboot the upgrade completed without issues.

For referrence this in the host CPU info:

Code: [Select]
Cores 6
Threads 6
Name AMD Phenom II X6 1070T
Code Name Thuban
Package Socket AM3 (938)
Technology 45nm
Specification AMD Phenom II X6 1075T Processor
Family F
Extended Family 10
Model A
Extended Model A
Stepping 0
Revision PH-E0
Instructions MMX (+), 3DNow! (+), SSE, SSE2, SSE3, SSE4A, AMD 64, NX, VMX
Title: Re: Kernel panic after upgrade
Post by: newsense on February 10, 2019, 11:11:38 pm
Wrong quote above, fixed now to reflect the correct one by laterra
Title: Re: Kernel panic after upgrade
Post by: Aloist on February 11, 2019, 02:09:00 pm
I have given up, as it is not possible to get support for upgrading Opnsense on the Dell Poweredge R410 which I am running.

I have a paid support subscription, but only got this answer:
"We do not have a Dell R410 to test. We have tested it on the hardware we sell and on these the upgrade works well. "

I do not consider this professional support and had actually expected better, when I decided to move from Cisco to Opnsense.

Now I have bitten into the bitter apple and ordered "hardware Deciso sells" to run Opnsense on it, as this seems to be the only safe way of have a system which can be kept up-to-date.




Title: Re: Kernel panic after upgrade
Post by: myksto on February 11, 2019, 03:21:50 pm
[...]

I have a paid support subscription, but only got this answer:
"We do not have a Dell R410 to test. We have tested it on the hardware we sell and on these the upgrade works well. "

[...]
I perfectly agree with you: not a professional answer. I can understand they can not have any hardware of the world but paid support must give more than that!
Title: Re: Kernel panic after upgrade
Post by: lattera on February 11, 2019, 03:23:30 pm
I have a spare R410 lying around. The RAM in it is dead. If you can hold out until I can buy new RAM (within a week or so), I'd be happy to test out on my (currently dead) R410.
Title: Re: Kernel panic after upgrade
Post by: Aloist on February 11, 2019, 04:10:36 pm
I have a spare R410 lying around. The RAM in it is dead. If you can hold out until I can buy new RAM (within a week or so), I'd be happy to test out on my (currently dead) R410.

That is kind of you, thank you.
But as I have now ordered the appliance from Deciso, I will move from the Dell R410 (we have a lot of older Dell Poweredges) to the supported hardware.

I would not be able to rely on your test, because originally, when I first installed Opnsense 18.7 on the R410, kernel crashes also happened, most likely due to Raid controller issues. I documented what I did to fix this:
Code: [Select]
Have trouble installing opensense on w99, it crashes, most likely due
to driver issues for the Dell PERC
I can catch the USB boot process and enter menu option 3, to set boot
parameters.
There, I add
set hw.mfi.mrsas_enabled=1
and boot.

It may work. Afterwards, I must login as installer, pw= opnsense

It worked. After reboot, I can enter shell and see with
dmesg|more
that the Megaraid SAS driver is active.
With command
mfiutil show config
I can see then the raid configuration is properly recognized.

It is a pity to give up the R410, as it has completely new disks and would have run for many years to come. Still, I have other work to do and cannot spend time with unstable support situations for an essential piece of hardware.

The Deciso hardware has no RAID and no dual power supply, is therefore inferior on that side.

I may keep the R410 as a backup. Once it is no longer a critical component, I can afford to risk version upgrades on it.


Title: Re: Kernel panic after upgrade
Post by: namezero111111 on February 12, 2019, 08:16:35 am
@Aloist

Maybe if it is such an ultra critical device for you you should invest in a CARP cluster; possibly with the Dell as slave for the new device for example.

Even in case of updates; if the primary gets messed up the secondary can take over until the first gets fixed.

For 99% of workloads it would be best to maybe virtualize the device for quick BMR backup. For failed updates, snapshot restore is an extra bonus.

Virtualization would also make a test environment for major changes and updates feasible, especially with spare server laying around as you said.
Title: Re: Kernel panic after upgrade
Post by: Aloist on February 12, 2019, 03:24:15 pm
@Aloist

Maybe if it is such an ultra critical device for you you should invest in a CARP cluster; possibly with the Dell as slave for the new device for example.

It is only the office firewall. But I work a lot outside of the office.

I like to trust that if I do a software update in a device in the office, it reboots and after a few minutes it will be up again.

If the system software update is so bad that at a reboot it ends up in a kernel panic, it comes never up again. I would have to be physically there, and reinstall from cold. This is what I fear, not a hardware failure.

We use RAID disks on all essential systems, because disk failure is the most frequent hardware failure. Typically after several years of 7/24 use. All else, i.e. power supply, RAM, CPU fails much more rarely, in my 40+ years IT experience.
Title: Re: Kernel panic after upgrade
Post by: Ralf Kirmis on February 13, 2019, 09:01:47 am
Hello,

i've the same  Problem with the Kernel Panic.
see attached Screenshot.
My CPU is an i3-8100T on Fujitsu Mainboard D3633-S
The two kernel parameters i have tried, but then Panic stays the same.
The USB Keyboard is dead after that, so the command bt doesn't work.

Does anyone have a solution?
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on February 13, 2019, 05:44:17 pm
I have also encountered this problem when going from 18.7.x to 19.1.x.

I am running the OPNsense in a VM under Proxmox VE 5.3-9 with UEFI for the first time and have encountered this error. I have NOT encountered this with regular BIOS (SeaBIOS) with 18.7.x to 19.1.x or new 19.1.x installs.
Changed the VM from UEFI to BIOS (SeaBIOS) and the OPNSense 19.1.x VM booted from HDD or ISO CDROM with no issues.

Also the snip below is in the release notes which maybe the cause of this issue:

Migration notes and minor incompatibilities to look out for:

o Gateway health graphs may need a manual reset due to the Apinger to Dpinger migration.  Apinger is no longer available.
o Intrusion detection GeoIP rules are automatically deactivated and need to be manually migrated to firewall alias GeoIP.
o Quagga plugin has been superseded by FRR plugin.  A binary quagga package has been conserved for the time being.
o Please read the FRR documentation with regard to the required system tunables[8].
o Bhyve UEFI boot may fail as a guest.  The problem is being investigated.
o SNMP plugin has been superseded by Net-SNMP plugin.


So it seems to be a general UEFI issue from my eyes.
Title: Re: Kernel panic after upgrade
Post by: lattera on February 13, 2019, 08:46:32 pm
I thought the bhyve UEFI boot issue was worked around by passing the -w flag to bhyve. Are there other issues related to booting OPNsense 19.1 in bhyve?
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on February 13, 2019, 09:56:24 pm
Lattera:

I do not know about the bhyve UEFI issue. I just noticed it in the release notes. And I am having an issue with UEFI and from past experience the Dell RXXX series of servers used UEFI by default. So I assumed the UEFI might be an issue in general not just in the bhyve implementation.
Title: Re: Kernel panic after upgrade
Post by: bunchofreeds on February 13, 2019, 11:51:49 pm
@TheGrandWazoo

I think it is UEFI specific also. Which would make sense considering the broad spectrum of hardware and virtual platforms being impacted.

Looking at the contents of the ISO and comparing between 18.7 and 19.1. The files supporting EFI boot have been updated. Being /BOOT/BOOT1.EFI and /BOOT/LOADER.EFI

https://wiki.freebsd.org/UEFI
https://www.freebsdfoundation.org/freebsd-uefi-secure-boot/     Bit easier to understand but is looking at Secure Boot


Looks to be a FreeBSD or HardenedBSD issue?

Edit:  Tried a Hyper-V install using HardenedBSD-11-STABLE-v1100056.13-amd64-bootonly
Kernel Panic at the same point. Similar output, non responsive.

Title: Clean installation boot problem and fix for QOTOM Micro PC
Post by: ianc on February 14, 2019, 08:12:32 am
Hi all.

I have just completed a clean install of OPNsense v19.1, but had to work around a problem that appears to be similar to that discussed above. However the workaround for me doesn't seem to fit well with some of the suggestions above, laying the blame at the feet of the UEFI Boot process.

So, for the record... I have just installed v19.1 (vga) on a QOTOM-Q190G4 box (physical hardware).

Installation attempts failed by hanging after the kernel loaded its collection of kernel modules and the "Booting" line of text appeared. Nothing further was shown on the screen although USB drive activity continued for some time. I have seen kernel panic messages on Linux consoles, but I don't know whether a BSD kernel panic message would be expected on the display at this point, if there was such an event here. So, with nothing on the display (and nothing yet configured, so no network testing possible) this could conceivably be the same kernel panic referenced in this forum thread.

The fix:  So far, this box's BIOS had been set to traditional MBR BIOS boot (non-UEFI). Somewhat unexpectedly after the comments above, once the system BIOS setting was changed to use UEFI boot, the boot process (and subsequent live operation and then installation) of OPNsense v19.1 went through without a hitch.

Whether this is related to the kernel panics discussed above or not, I can't be sure. However it does confirm a surprising and similar boot problem, and the corresponding solution, at least for this pairing of hardware and OPNsense release. Hopefully that may assist some others.
Title: Re: Kernel panic after upgrade
Post by: marjohn56 on February 14, 2019, 10:09:14 am
There is a little bit of strangeness here, it affects some devices and not others. I use Qotom's too, i5 versions, but what you say about the installation appearing to continue is true. If you had connected the serial port and looked there you may have seen the console output appear there. This is something I do by default now when doing clean installs, connect both HDMI and also have a serial link to my PC!
Title: Re: Kernel panic after upgrade
Post by: Aloist on February 14, 2019, 01:49:34 pm
I am happy to announce that I upgraded without problem from 18.7.10 to 19.1.1 through the normal upgrade process
on my Dell Poweredge R410 system.

I had feared upgrade problems due to report from others with similar Dell machines, but everything went fine.
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on February 14, 2019, 04:07:57 pm
Could it be a console issue with BIOS using VGA or UEFI using VGA instead of UEFI? I am trying many different combos to see if it might work (I bit time consuming).

I will see if I can copy the 18.7.x EFI files to 19.1.x and see if the issue disappears (or an Earth shattering Kaboom). It only a VM so I'll test anything right now.
Title: Re: Kernel panic after upgrade
Post by: lattera on February 15, 2019, 01:59:59 am
@TheGrandWazoo

I think it is UEFI specific also. Which would make sense considering the broad spectrum of hardware and virtual platforms being impacted.

Looking at the contents of the ISO and comparing between 18.7 and 19.1. The files supporting EFI boot have been updated. Being /BOOT/BOOT1.EFI and /BOOT/LOADER.EFI

https://wiki.freebsd.org/UEFI
https://www.freebsdfoundation.org/freebsd-uefi-secure-boot/     Bit easier to understand but is looking at Secure Boot


Looks to be a FreeBSD or HardenedBSD issue?

Edit:  Tried a Hyper-V install using HardenedBSD-11-STABLE-v1100056.13-amd64-bootonly
Kernel Panic at the same point. Similar output, non responsive.

FYI: I'll be looking at the Hyper-V boot issue soon. I'm teaming up with a member of the HardenedBSD Foundation who relies on Hyper-V in parts of his infrastructure. As soon as our schedules match up, we'll be working this out together. :)

I'll report back as soon as I have more information. Thanks for your continued patience and support!
Title: Re: Kernel panic after upgrade
Post by: peter008 on February 15, 2019, 07:19:42 pm
I can confirm that - with opnsense running as a VM under Hyper-V - is has something to to do with the bios of the server running the hypervisor: one old Siemens-Server of ours which just has MBR was upgrading the opnsense-VM to 19.x without problems, another Server with UEFI let the opnsense-VM freeze during updating as reported.

edit: forgot to mention that it made no difference to install the VM under Hyper-V as type 1 or 2
Title: Re: Kernel panic after upgrade
Post by: senseivita on February 16, 2019, 02:22:41 am
So it isn't safe to upgrade yet? I just went through this for the first time, and several boots where it says "cd0" is waiting for something or the other way around, at that point I have to hard-restart the system.

The first time it showed me the kernel error I tried booting the old kernel and it showed me the cd0 error. I begged for it to come around if I tried again and luckily it did. Right away after booting it mentioned something about upgrading the system, but it actually went to 18.7.10_4 which in a way *is* upgrading because I think I was on 18.1, I pulled the image from a local file server to get started faster.

I took two snapshops, one with memory and backed up immediately after the system came up. I'm also getting a lot of this, even before the upgrade, is this normal?
(http://fetch.vitanetworks.link/posting/opnsense/unresponsive.png)
Title: Re: Kernel panic after upgrade
Post by: newsense on February 16, 2019, 05:20:48 pm
That is a known issue that can happen sometimes.

Take a snapshot of the VM and update to 19.1.1 which is currently supported, 18.7 is now legacy
Title: Re: Kernel panic after upgrade
Post by: bunchofreeds on February 17, 2019, 05:41:07 am
@lattera

Thanks for looking into this and let me know if I can help in any way. I run Hyper-V and can download and test OS installs and configurations easily. I'm not much good with FreeBSD or HardenedBSD though sorry, but will help where I can.

Title: Re: Kernel panic after upgrade
Post by: bellyacres on February 18, 2019, 05:51:20 pm
I've tried to update two machines and get a kernel panic on both.  Main machine is an AMD 3600 chip, single core, Athlon if memory serves.  Other is a Intel Duo Core.  When I set
hw.ibrs_disable=1
vm.pmap.pti=0
The Duo Core boots fine.

Have been unable to test the AMD box as it's our main box here.  Hardly mission critical, but with a teenager and friends depending on Internet for their lifeblood it becomes critical.

Posted mostly just to be able to follow this post, and see what the final solution is.

Thanks to all the devs for a great solution.  Your work doesn't go unappreciated!

Mike
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on February 19, 2019, 03:13:07 pm
@TheGrandWazoo

I think it is UEFI specific also. Which would make sense considering the broad spectrum of hardware and virtual platforms being impacted.

Looking at the contents of the ISO and comparing between 18.7 and 19.1. The files supporting EFI boot have been updated. Being /BOOT/BOOT1.EFI and /BOOT/LOADER.EFI

https://wiki.freebsd.org/UEFI
https://www.freebsdfoundation.org/freebsd-uefi-secure-boot/     Bit easier to understand but is looking at Secure Boot


Looks to be a FreeBSD or HardenedBSD issue?

Edit:  Tried a Hyper-V install using HardenedBSD-11-STABLE-v1100056.13-amd64-bootonly
Kernel Panic at the same point. Similar output, non responsive.

FYI: I'll be looking at the Hyper-V boot issue soon. I'm teaming up with a member of the HardenedBSD Foundation who relies on Hyper-V in parts of his infrastructure. As soon as our schedules match up, we'll be working this out together. :)

I'll report back as soon as I have more information. Thanks for your continued patience and support!

Thank you for looking into it.

I did try the 18.7.x efi files in the /boot and it did not help (didn't think it would) but worth a shot.
Title: Re: Kernel panic after upgrade
Post by: buddwm on February 26, 2019, 04:44:59 am
Has any progress been made on this issue?  I'm on KVM and still can't seem to upgrade to the latest Opnsense version without getting the kernel panic.
Title: Re: Kernel panic after upgrade
Post by: akron on March 01, 2019, 06:13:20 pm
We hold off upgrade on production Instances for now.

is definitely an issue with HardenedBSD / OPN 19.1 UEFI only, BIOS works well


hopefully there will be a solution soon

at the moment several hardware, VMware and Hyper-V UEFI instances get the failed trap error
Title: Re: Kernel panic after upgrade
Post by: schnipp on March 01, 2019, 07:05:06 pm
I have migrated from 18.7.10_4 to 19.1.2. The migration process worked without any problems.

My system:

Many thanks to all the developers and contributors who made this version possible  :D
Title: Re: Kernel panic after upgrade
Post by: astoklas on March 01, 2019, 07:54:50 pm
Is there an update on this?
I try a fresh install with a i7-8700 CPU/Shuttle XV310 and see the same kernel panic.
The suggested boot options did not help

Edit:
This is a fresh install on a pyhsical device - no hypervisor involved.
But it is the exact same kernel panic.

Any advise?
Title: Re: Kernel panic after upgrade
Post by: RNHurt on March 02, 2019, 12:01:24 am
The latest update is 19.1.2 (Feb. 28th) but the release notes don't mention anything about fixing a kernel panic.  :(
Title: Re: Kernel panic after upgrade
Post by: lattera on March 02, 2019, 12:49:07 am
If the price is right, I will be looking to acquire a budget-friendly system on which to run Hyper-V. HardenedBSD's budget for this kind of thing would be $500 USD.

Just a quick reminder that HardenedBSD accepts donations, both monetary and hardware. We appreciate all contributions of any kind.
Title: Re: Kernel panic after upgrade
Post by: RGijsen on March 02, 2019, 10:07:51 am
If the price is right, I will be looking to acquire a budget-friendly system on which to run Hyper-V. HardenedBSD's budget for this kind of thing would be $500 USD.

Just a quick reminder that HardenedBSD accepts donations, both monetary and hardware. We appreciate all contributions of any kind.

While you guys are probably running a *nix variant, any Windows 10 Pro machine (with a fairly recent Intel CPU) will do for testing. My old Intel 5i5RYH NUC, with a i5 5250u cpu suffers the same, with an OPNsense test-VM on Hyper-V, which is available in 10 pro. Server 2016 / 2019 more or less equals Windows 10 (resp. 1607 and 1809 builds). So you wouldn't need an expensive box to test.
Title: Re: Kernel panic after upgrade
Post by: akron on March 02, 2019, 02:59:37 pm
I don't think is necessary, a simple laptop with windows 10 would do the job, however for a more professional environment I can provide a remote LAB with Hyper-V and ESXi where you can spin as many VMs as you wish to help test.

this UEFI bug plagues some baremetal, Hyper-V Gen 2 UEFI and ESXi UEFI VMs and me as well other dozens of users have interest to help get this fixed as we use OPNsense in productions environments.

Cheers
Title: Re: Kernel panic after upgrade
Post by: golebole on March 03, 2019, 12:04:10 pm
Hi,

is the problem solved with the new update or still the same?

Topic has 3000 Views, so I think the most important topic at the moment :).

Thanks
Christian
Title: Re: Kernel panic after upgrade
Post by: jmeyer on March 05, 2019, 01:53:24 pm
Hi,

I have the same Issue with a clean install of OPNsense 19.1 on a new DELL PowerEdge R340 Server.

Using the Boot Options and trying the different Images (nano, dvd,vga) did not help. Also tried enabling and disabling UEFI in the BIOS does not helped.

Quote
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
Stopped at fpuinit+0x179: orb $0x10,ctx_switch_xsave+0x3
Title: Re: Kernel panic after upgrade
Post by: RGijsen on March 06, 2019, 11:23:59 am
For shits and giggles I created a Hyper-V gen1 VM, installed 19.1 and updated to the latest-as-of-yet 19.1.2, ran fine under gen1. Mounted the disk under a Gen2, and *poof*, still crash. So no, 19.1.2 didn't fix it, although we would have already known that.

While I totally understand the limited resources of the team (all respect for them!), it's getting hard for us to rely on this given that 18.x is now EOL (ie not secure in my book) but 19.x doesn't run at all.
Title: Re: Kernel panic after upgrade
Post by: franco on March 06, 2019, 12:23:50 pm
Please sponsor the project if you feel we're not doing enough. <3
Title: Re: Kernel panic after upgrade
Post by: peter008 on March 06, 2019, 12:55:58 pm
No offense, but I think you should not be so sensitive.

It is very annoying indeed if you recommend updating to a version (19.x) which constantly crashes the total system of the users, while on the other site you take out a running (18.x) of support.

Besides you did not make it clear for days that the freezes were based on the update, so maybe I was not the only one who tried to find the mistake for hours on his own system.

After all it was quite a challenge to eliminate these mistakes and get back to running systems.

We all appreciate your work, but, sorry, those kind of bugs suck.
Title: Re: Kernel panic after upgrade
Post by: franco on March 06, 2019, 12:58:41 pm
I know. Everybody expects work here and there. And we get constantly side-tracked by fixing operating system and third-party related issues while pulling off progress for the software that we actually do write:

https://github.com/opnsense/core/pulse/monthly

I have to be sensitive, because people ask for a lot without realising that not helping us move forward doesn't work well for them.


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: franco on March 06, 2019, 01:09:09 pm
I have actually said this before. Users that do not like this particular issue ask us to:

Never change the operating system version.

We cannot make that choice. FreeBSD very quickly and almost deliberately destroys usability of 11.x not officially supported.

So we have no choice but to upgrade and when users don't like the changes we ship but did not want to have in the first place the concerns are voiced in the wrong direction, because we're simply caught in the middle.

So between not upgrading the operating system and upgrading it we will always have to bite the bullet and upgrade until somebody comes along and provides an OS that will:

* Push long term security updates and third-party ports updates
* Provides timely updates for new hardware
* Provided 100% hardware compatibility

Maybe it's called Linux, but we're not there. Sorry. In conclusion:

1. *Everybody* knows what they get in an open source software. That also means things may not work but most things eventually get fixed.

2. You can use any old version for as long as you want. There are no restrictions here.

When you ask for 1. and 2. at the same time and steady security updates on top you're pushing too far for a free product, maybe even for a paid one.
Title: Re: Kernel panic after upgrade
Post by: RGijsen on March 06, 2019, 03:42:46 pm
All fair enough, but I have to agree with peter008 here; this is quite some bug, killing opnSense for weeks on end now, which had quite easiliy be tracked or at least found before eol-ing 18.x. I'm just wondering (trying to be constructive again), what hardware do you test the builds on before putting them out on the street? I assume there's multiple devices tested, right? Of course it's obvious not all hardware can be tested, there's practically an unlimited number of different configurations out there. Still, this bug as far as I could distill from the fora is with UEFI machines, and maybe slightly related to Spectre / Meltdown patches (although disabling them didn't help me). That means pretty much any Intel based pc / laptop from the last.. 5 or 6 years? Testing on an ESX machine or Hyper-V which ships in every Windows 10 Pro on ots own is trivial. So really I'm trying to understand your point here.

You keep telling us to donate, which I might if I have enough confidence (we are still evaluating OPNSense in favour of our current pfSense setup) but I'm not sure what you'd want. You want my money? You want the most generic Windows 10 box or laptop? I've read a thread this morning were people offer a VPS to test this on as well. Sure, might be not ideal, but WHAT do you exactly need that you don't have right now? It's hard to believe you don't have a laptop running Windows 10? But please prove me wrong and tell me what you DO need.
Title: Re: Kernel panic after upgrade
Post by: franco on March 06, 2019, 03:52:51 pm
Feel free to take sides. Here is the hard truth:

It may take 6 months to fix this. It may be fixed in 11.4. Or it may never be fixed.

I'm not asking for money. I'm asking for incentive, motivation and acceptance. I can see that some are not willing to give it at all. That's fine. I can understand. But such people will not change OPNsense or the community behind it.

Best to use the product that works for you and be done with it.


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: RGijsen on March 06, 2019, 04:01:19 pm
Not sure what you are trying to achieve with your attitude. A few post back you replied to me that I should sponsor the project. I ask you how, and this is your response?

People offered machines (even if VPS) but that doesn't seem to be what you want either. No offence, but I said I tried to be constructive, and asked how I / we can help. Not what we can do NOT to help you. But alas. Take out the popcorn.
Title: Re: Kernel panic after upgrade
Post by: franco on March 06, 2019, 04:05:28 pm
No need for popcorn, we simply don't seem to "click" and I can't get that across very well.


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: franco on March 06, 2019, 04:10:44 pm
PS: To be more clear, we seem to want different things from OPNsense. You want fixes, I want others to take responsibility for providing fixes or at least a basis for them. If we solely rely on others to get what we want the whole house of cards falls apart and nothing will get done and ideally that should be avoided, should it not?
Title: Re: Kernel panic after upgrade
Post by: RGijsen on March 06, 2019, 04:14:54 pm
Honestely, I don't care how you feel about me. IIf you've made your image on me based on the 14 (count 'em!) posts I've made so far, that's tells me more about you than about me.

As stated, I tried to be constructive, as do other people. But hey, we don't click. So whatever I offer is probably not any good. Too bad, I can live with that. I can't contribute code-wise if its <> .NET or gwbasic. But if my car's engine blows out, I can't fix it either. That doesn't mean I shouldn't drive one.

Anyway, it seems I can't help at all fixing this issue. Sorry community, my bad, I tried.
Title: Re: Kernel panic after upgrade
Post by: franco on March 06, 2019, 04:19:00 pm
It's a start. Thank you for listening to what I said.


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: lattera on March 06, 2019, 05:43:02 pm
For shits and giggles I created a Hyper-V gen1 VM, installed 19.1 and updated to the latest-as-of-yet 19.1.2, ran fine under gen1. Mounted the disk under a Gen2, and *poof*, still crash. So no, 19.1.2 didn't fix it, although we would have already known that.

While I totally understand the limited resources of the team (all respect for them!), it's getting hard for us to rely on this given that 18.x is now EOL (ie not secure in my book) but 19.x doesn't run at all.

Microsoft's documentation shows that FreeBSD isn't supported in gen2: https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/plan/should-i-create-a-generation-1-or-2-virtual-machine-in-hyper-v#BKMK_FreeBSD

However, the documentation linked to above shows the 10.x line, not the 11.2 version that OPNsense is on.

There's also this document: https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/plan/should-i-create-a-generation-1-or-2-virtual-machine-in-hyper-v#use-uefi-firmware

I've started work on debugging Hyper-V regressions. My employer lent me some hardware to test on. I've got it for a period of one to three weeks. I hope to report back soon with results.
Title: Re: Kernel panic after upgrade
Post by: RGijsen on March 06, 2019, 06:48:25 pm
Please check this updated document, yours is from 2016: https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/supported-freebsd-virtual-machines-on-hyper-v
States gen2 is supported, as long you disable secure-boot. That's what I find with other FreeBSD installs on Hyper-V as well.
Title: Re: Kernel panic after upgrade
Post by: bitwolf on March 06, 2019, 07:15:07 pm
Franco,

as someone who has done a lot of support of the stuff he designed and implemented, I can understand your frustration when a lot of people start screaming "it's broken, fix it!" without constructive content; on the other hand while the community can help solving this issue (by providing test resources, reports, patches, or even just the time to test various builds on the hardware we own), this is such a fundamental issue with the end product functionality that I think you should take the lead and help us help you, even if it turns out that the fix will not be found in OPNSense but the underlyig OS..

There are multiple threads about kernel crashes in the forum, and they are a mix of general discussion about the issue, pure complaints and reports from people trying various things. Maybe you should close these and start a new pinned thread, summarising what you know and think of the issue so far in the opening post, and asking people who are unable to boot 19.1 to list hardware/configuration combinations so we can work towards reproduction.
Title: Re: Kernel panic after upgrade
Post by: franco on March 06, 2019, 07:31:26 pm
Hi bitwolf,

You are certainly right.

What I can tell you is I cannot take the lead on this without jeopardising my day job, my private life and areas of the OPNsense project that do work without worrying about them from a user perspective. This is where my frustration comes from.

It's too much to ask for me personally.


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: lattera on March 06, 2019, 07:48:23 pm
Never fear, for lattera is here!

I'm at least looking into the Hyper-V regression(s). I, too, am doing this in my spare time, but it's worth it. :)
Title: Re: Kernel panic after upgrade
Post by: peter008 on March 06, 2019, 08:51:35 pm
Honestely, I don't care how you feel about me. IIf you've made your image on me based on the 14 (count 'em!) posts I've made so far, that's tells me more about you than about me.

As stated, I tried to be constructive, as do other people. But hey, we don't click. So whatever I offer is probably not any good. Too bad, I can live with that. I can't contribute code-wise if its <> .NET or gwbasic. But if my car's engine blows out, I can't fix it either. That doesn't mean I shouldn't drive one.

Anyway, it seems I can't help at all fixing this issue. Sorry community, my bad, I tried.

All good points, and I do see that you still try to stay matter-of-fact.

My advice to franco is not to blame the users for being upset, the serious people do not blame you, but the fact that there route to the WWW has been cut and - as you know - that´s a real show stopper today  ;)

Which actually amazes me is your statement about if this all can be fixed in OPNsense. That sounds really frustrated and makes me doubt if you still want to continue the project.

If that's how it is, please tell us, because, come on, not only you but we as the users also do not want to waste our time if it is useless.
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on March 06, 2019, 11:03:51 pm
Ok, let's all just settle down. Grab a beer or a Crown Royal or both and sit back...relax.

First, I started with m0n0wall, then was going to develop something with pf (because I liked it better) then found pfSense. It was great but the development of it seem to went downhill since the main developer left and pfSense was purchased. Then I found OPNsense (via m0n0wall website) and I will not go back to pfSense...just a lot more activity here from a development standpoint and filled in a lot of holes.

Being a Software Developer, SysOps, Network Engineer, DB Admin and/or 'what other IT people don't want to do', I can understand BOTH sides of the fence, but lets work together. Communications on both sides is key to resolving any issue; venting helps only one party and usually fuels the other party (Look at the thread)

I came upon this issue due to reading about how BIOS was going to be dead in the 2k20's and EFI was the way to go...so here I am because of a change I made in December to start using EFI.

I am using Proxmox VE 5.3-11 or KVM/QEMU for the most part. I have not used any bare metal OPNsense yet.

We know that 18.7.x used FreeBSD 11.1 and 19.1.x is using FreeBSD 11.2 both with HardenedBSD. We know that using it with a BIOS works and EFI does not. So, is there something WE all can do to provide back traces or more information on the boot up process that might signal a "Well there it is!!!".

We know the EFI files got changed in 19.1.x (size change) so does something in the EFI partition need to change for the two to play nice? The ISO image also does the exact same thing when just trying to install a VM with EFI, just not an upgrade.

I wish I knew move about the EFI system to be more help but I am willing to provide data to help us all get through this.

Kev a.k.a. The Grand Wazoo
I don't usually make changes, but when I do it is in production...stay on call my friends.

Title: Re: Kernel panic after upgrade
Post by: bitwolf on March 07, 2019, 12:21:37 am
I have just done some tests on our lab DELL Poweredge R340; it's not currently available for me to use, but as long as I don't touch the HDDs, and do it out of hours, I can reboot it as many times as I want, at least for now.

Given the constraint above I focussed on trying to narrow down the conditions for the kernel trap using just a bootable iso.

Franco says the problem is upstream, so instead of booting the OPNsense iso (which for some reason takes half an hour to get to the point of the crash when mounted as virtual ISO via the iDRAC) I used unmodified OS ISOs.

Here are the results so far:

UEFI ENABLED
HardenedBSD-11-STABLE-v1100056.13-amd64-bootonly.iso
doesn't even manage to boot from the iso

UEFI DISABLED
HardenedBSD-11-STABLE-v1100056.13-amd64-bootonly.iso <-
kernel trap 12

FreeBSD-11.1-RELEASE-amd64-bootonly.iso
boots all the way to the installer

HardenedBSD-12-STABLE-v1200058.3-amd64-bootonly.iso
boots all the way to the installer

I then tried to disable virtualization support in the CPU based on some comments about Meltdown/Spectre fixes in the other thread, but I still get kernel trap 12, so that's not a viable workaround.

So at least in the case of Dell bare metal it seems that UEFI is not the culprit, as disabling it doesn't stop the kernel traps. It also seems not to be a FreeBSD 11 problem, as the vanilla FBSD iso works. This leaves changes between FreeBSD 11 and HardenedBSD 11 as the most likely cause for the kernel trap, but looking at the repo it seems the classical needle in a haystack. The interesting result from this testing is that HardenedBSD 12 works, so maybe an easier investigation path could be to look at the changes between HBSD 11 and 12 that are not merged from FBSD? Shawn what do you think?

Another option to collect more data could be to have a 19.1 debug iso (ie one with DDB enabled in the kernel) so we can actually collect core dumps for these crashes. I am sure that given enough time many of us, me included, could set up a HBSD dev environment and build the image myself, but if this can be a useful investigation avenue it seems better if one of the lead devs could just run the existing build workflow with the kernel option set.

I see further up the thread that a number of people complained about the same crashes on ESXi; our own production firewalls run on ESXi 6 but upgraded to 19.1 successfully, I can do some tests in that sense tomorrow, as this seems to imply there might be a simple workaround in the VM settings for the people running ESXi. This could also be a way forward for the people who have kernel traps on overspecced bare metal, at least up to the point the upstream issue is fixed, or OPNSense has moved to HBSD 12 (but that's at least a year away).
Title: Re: Kernel panic after upgrade
Post by: lattera on March 07, 2019, 02:38:26 am
I have just done some tests on our lab DELL Poweredge R340; it's not currently available for me to use, but as long as I don't touch the HDDs, and do it out of hours, I can reboot it as many times as I want, at least for now.

Given the constraint above I focussed on trying to narrow down the conditions for the kernel trap using just a bootable iso.

Franco says the problem is upstream, so instead of booting the OPNsense iso (which for some reason takes half an hour to get to the point of the crash when mounted as virtual ISO via the iDRAC) I used unmodified OS ISOs.

Here are the results so far:

UEFI ENABLED
HardenedBSD-11-STABLE-v1100056.13-amd64-bootonly.iso
doesn't even manage to boot from the iso

UEFI DISABLED
HardenedBSD-11-STABLE-v1100056.13-amd64-bootonly.iso <-
kernel trap 12

FreeBSD-11.1-RELEASE-amd64-bootonly.iso
boots all the way to the installer

HardenedBSD-12-STABLE-v1200058.3-amd64-bootonly.iso
boots all the way to the installer

I'm seeing the same type of results in Hyper-V as well. However, it's with UEFI enabled due to being Generation 2. Generation 1 works fine for me.

It's possible that the issue with the Dell systems is related to the issue with the Hyper-V systems.

ISo at least in the case of Dell bare metal it seems that UEFI is not the culprit, as disabling it doesn't stop the kernel traps. It also seems not to be a FreeBSD 11 problem, as the vanilla FBSD iso works. This leaves changes between FreeBSD 11 and HardenedBSD 11 as the most likely cause for the kernel trap, but looking at the repo it seems the classical needle in a haystack. The interesting result from this testing is that HardenedBSD 12 works, so maybe an easier investigation path could be to look at the changes between HBSD 11 and 12 that are not merged from FBSD? Shawn what do you think?

Another option to collect more data could be to have a 19.1 debug iso (ie one with DDB enabled in the kernel) so we can actually collect core dumps for these crashes. I am sure that given enough time many of us, me included, could set up a HBSD dev environment and build the image myself, but if this can be a useful investigation avenue it seems better if one of the lead devs could just run the existing build workflow with the kernel option set.

I'm building a custom version of HardenedBSD 11-STABLE/amd64 with DDB/KDB and remote KGDB along with CFLAGS="-g -O0" for "ALL THE THINGS!" I can upload the installation media once they're built.

As far as attempting to see what needs to be backported from 12-STABLE to 11-STABLE, that would entail _A LOT_ of work. More work than I have time for. However, if someone in the community wants to take that on, I'm definitely not going to stop him/her and would love to review patches. ;)

I see further up the thread that a number of people complained about the same crashes on ESXi; our own production firewalls run on ESXi 6 but upgraded to 19.1 successfully, I can do some tests in that sense tomorrow, as this seems to imply there might be a simple workaround in the VM settings for the people running ESXi. This could also be a way forward for the people who have kernel traps on overspecced bare metal, at least up to the point the upstream issue is fixed, or OPNSense has moved to HBSD 12 (but that's at least a year away).

OPNsense's move to HardenedBSD 12 is eight months away, assuming Franco does the initial import of the source code soon. :)
Title: Re: Kernel panic after upgrade
Post by: peter008 on March 07, 2019, 09:19:40 am
Ok, let's all just settle down. Grab a beer or a Crown Royal or both and sit back...relax.

Sorry you got something wrong (same as franco did), but to criticize something factually does not mean to be vicious.

We all want to have running systems, that´s why (most of us) just report errors hoping they might be fixed soon.

franco complained about not being payed enough for his work, the admin wants an Intel NUC from the community for 550 € just to test Hyper-V.

franco also stated that the freeze-problems in 19.x with virtualized and some other bare metal systems might never be solved.

To me, no offence, this does not look very respectable.

I am afraid I have to look for alternatives again concerning our firewalls.
Title: Re: Kernel panic after upgrade
Post by: lattera on March 07, 2019, 04:09:04 pm
franco complained about not being payed enough for his work, the admin wants an Intel NUC from the community for 550 € just to test Hyper-V.

FYI: it takes resources to debug issues. No resources means no debugging. My employer is awesome and lent me a laptop on which I can do the necessary debugging. That's what happens when one looks for potential solutions rather than griping with feelings of entitlement. ;P

If you have a better suggestion, rather than a gripe, I'm all ears.
Title: Re: Kernel panic after upgrade
Post by: AdSchellevis on March 07, 2019, 06:34:02 pm
Hi,

I've spend more than a day trying to replicate the issue and tracking it's origin, since it doesn't occur on all EUFI boot systems.
Virtualbox for example boots without issues in UEFI mode, on Parallels (osx) I was able to find the crash as well .

Code: [Select]
fpuinit_bsp1 () at /usr/src/sys/amd64/amd64/fpu.c:241
fpuinit () at /usr/src/sys/amd64/amd64/fpu.c:277
0xffffffff810adb3b in hammer_time (modulep=<optimized out>, physfree=<optimized out>) at /usr/src/sys/amd64/amd64/machdep.c:1801
0xffffffff80316024 in btext () at /usr/src/sys/amd64/amd64/locore.S:79

Let me make one thing very clear, none of our systems suffer from this issue, a lot of people where actively involved during the beta stages up to 19.1 using all kinds of hardware.

I've seen a couple of people complaining, nagging, not being to **any** help to anyone.
I understand you have an issue, we all do, but... there are always alternatives, using other types of setups, being involved earlier and actively helping improving the system.
Don't forget, if your setup fails and you have done nothing to prevent that from happening, it's still your issue.... nobody got paid to solve it for you.

The patch [1] available might not be the final fix, nor will it fix all issues in the world, but it looks promising.

I would like to thank Franco, Shawn and anybody involved in actually pinning this issue down.

A kernel with debug options enabled is available on our website [2], but if Franco has some time available he can probably move it to a better spot, maybe build some iso with kernel.


Best regards,

Ad


Title: Re: Kernel panic after upgrade
Post by: lattera on March 07, 2019, 07:02:53 pm
Code: [Select]
fpuinit_bsp1 () at /usr/src/sys/amd64/amd64/fpu.c:241
fpuinit () at /usr/src/sys/amd64/amd64/fpu.c:277
0xffffffff810adb3b in hammer_time (modulep=<optimized out>, physfree=<optimized out>) at /usr/src/sys/amd64/amd64/machdep.c:1801
0xffffffff80316024 in btext () at /usr/src/sys/amd64/amd64/locore.S:79

I would like to thank Franco, Shawn and anybody involved in actually pinning this issue down.

A kernel with debug options enabled is available on our website [2], but if Franco has some time available he can probably move it to a better spot, maybe build some iso with kernel.


Best regards,

Ad

  • https://github.com/HardenedBSD/hardenedBSD-stable/commit/77a10c68ac3bf9cd0da50bd537dab858462c07f7
  • https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/sets/kernel-19.7.a_25-amd64.txz

Hey Ad,

I've been working on this for the past few days. Put in around 20 hours so far tracking down the issue. :)

We effectively have two forum topics for the same problem. I've documented the issue here: https://forum.opnsense.org/index.php?topic=11403.msg54432#msg54432

So, I've figured out the root cause. I need to do more research in order to write a patch. I'm hoping to have a patch ready within the next week or two.
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on March 07, 2019, 07:22:22 pm
I know you guys are going to think I am "BAT SHIT INSANE" but I was able to get the system to boot with EFI...but before everyone goes nuts let me tell you want I did and maybe it might make sense or just confuse the crap out of you.

I downloaded the 11.2_bootonly of FreeBSD and copied the EFI files to the /boot dir and the loader.efi to the /efi/boot/BOOTx64.efi partition (You have to mount the partition 'mount -t msdosfs /dev/<your efi partition> /mnt').
This did NOT work. I received the same error.

I then proceeded to download 12.0_bootonly of FreeBSD and copied the EFI file like mentioned above. This did NOT work.

Here's the "Bad Shit Insane" part...I copied the /boot/kernel/kernel from the 12.0_bootonly to the /boot/kernel/kernel of the OPNsense and 'Holy Shit' I am up and running using EFI as a firmware to boot OPNsense.

Now I know you are saying "This does not do Squat for me" but it might give someone in FreeBSD and/or HardenedBSD land a "Light Bulb" over their head to maybe think of a change that is different between the two trains. Or could help OPNsense talk to the developers of the FreeBSD/HardenedBSD kernels and give them some insight.

Trying my best guys. I hope this helps.

Guides I used to help me with this...
https://wiki.freebsd.org/UEFI
https://www.happyassassin.net/2014/01/25/uefi-boot-how-does-that-actually-work-then/
https://www.freebsd.org/doc/en_US.ISO8859-1/books/arch-handbook/boot-kernel.html - because the panic is actually in the mi_startup().
https://forums.freebsd.org/threads/linuxkpi-kernel-panic-in-freebsd-11-2-prerelease-4-r333170-intel-skylake-hd-graphics.65848/

Kev a.k.a. The Grand Wazoo.
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on March 07, 2019, 07:57:20 pm
Hi,

I've spend more than a day trying to replicate the issue and tracking it's origin, since it doesn't occur on all EUFI boot systems.
Virtualbox for example boots without issues in UEFI mode, on Parallels (osx) I was able to find the crash as well .

Code: [Select]
fpuinit_bsp1 () at /usr/src/sys/amd64/amd64/fpu.c:241
fpuinit () at /usr/src/sys/amd64/amd64/fpu.c:277
0xffffffff810adb3b in hammer_time (modulep=<optimized out>, physfree=<optimized out>) at /usr/src/sys/amd64/amd64/machdep.c:1801
0xffffffff80316024 in btext () at /usr/src/sys/amd64/amd64/locore.S:79

Let me make one thing very clear, none of our systems suffer from this issue, a lot of people where actively involved during the beta stages up to 19.1 using all kinds of hardware.

I've seen a couple of people complaining, nagging, not being to **any** help to anyone.
I understand you have an issue, we all do, but... there are always alternatives, using other types of setups, being involved earlier and actively helping improving the system.
Don't forget, if your setup fails and you have done nothing to prevent that from happening, it's still your issue.... nobody got paid to solve it for you.

The patch [1] available might not be the final fix, nor will it fix all issues in the world, but it looks promising.

I would like to thank Franco, Shawn and anybody involved in actually pinning this issue down.

A kernel with debug options enabled is available on our website [2], but if Franco has some time available he can probably move it to a better spot, maybe build some iso with kernel.


Best regards,

Ad

  • https://github.com/HardenedBSD/hardenedBSD-stable/commit/77a10c68ac3bf9cd0da50bd537dab858462c07f7
  • https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/sets/kernel-19.7.a_25-amd64.txz

UPDATE: I copied the 19.7 kernel mentioned above in the #2 link to the /boot/kernel/kernel of the 19.1.2 install (well mine it 19.1-netmap) and all is well in the world. Boots with NO issues with EFI or BIOS firmware.

Thank you guys. Let me know if I can continue to help.

Kev
Title: Re: Kernel panic after upgrade
Post by: franco on March 07, 2019, 08:20:51 pm
franco complained about not being payed enough for his work, the admin wants an Intel NUC from the community for 550 € just to test Hyper-V.

I have never been paid for *any* work on OPNsense. You're at best misleading, at worst deliberately trying to "get back at" me for telling you to stop imposing your personal beneficial view on the project. You can apologise for your toxic comment or leave for good.


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: bunchofreeds on March 07, 2019, 08:45:02 pm
This post is becoming a bit annoying.

Lots of helpful information to resolve the issue, but peppered with some pretty nasty feedback and grudges.

I would really like to have this resolved, pretty sure we all do.

Can we please focus on the fix and respect those who are helping to achieve that

 
Title: Re: Kernel panic after upgrade
Post by: peter008 on March 07, 2019, 09:39:46 pm
You can apologise for your toxic comment ...


Good joke !
Title: Re: Kernel panic after upgrade
Post by: franco on March 07, 2019, 09:43:31 pm
Not a joke. End of the line for you.


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: peter008 on March 07, 2019, 09:46:27 pm
Not a joke. End of the line for you.


...said the king !
Title: Re: Kernel panic after upgrade
Post by: franco on March 07, 2019, 09:46:49 pm
You're still here. I'm asking you to leave.
Title: Re: Kernel panic after upgrade
Post by: peter008 on March 07, 2019, 09:50:34 pm
ok, i just wanted to go back to planet earth anyway.
Title: Re: Kernel panic after upgrade
Post by: franco on March 07, 2019, 09:55:13 pm
Please leave it. This is your last warning.
Title: Re: Kernel panic after upgrade
Post by: peter008 on March 07, 2019, 09:56:20 pm
 :'( :'( :'( :'( :'( :'( :'( :'(
Title: Re: Kernel panic after upgrade
Post by: franco on March 07, 2019, 10:00:03 pm
I've disabled your ability to post. You can ask for an unlock via PM. The reason given was:

> Continued attacks on forum members and ignoring moderator requests


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: akron on March 08, 2019, 01:24:22 am
I'm a peaceful person however as an enterprise user it amazes me the quantity of bashers and shills in this forum. (Almost same rethoric as pfsense decadent times)

First of all everyone is free to choose what is necessary to deliver the job.

Coming from Cisco and Dell background, spending 15000€ per device in useless hardware for many years because of the BIG brand and BIG things and the BIG names, still useless for the price TAG.

Second this is an open source, community fuelled project, actually one of the best projects in terms of performance and features globally available.

If one is hurt about these bugs, please open your wallet and buy the iPhone of firewalls, CISCO, PALO ALTO, Fortigate and so on and be happy.

No one can demand or snow flake around because there is a bug, I've been running enterprise hardware from Deciso moving away from Cisco and never once been disappointed. Franco helped many times fixed bugs, and as an active user contributed many times monetarily to the project and will continue to do so, because it pays the bills around here.

Also running dozens of virtual workloads, from time to time there is a bug here and there, however one is responsible for the R&D and as AD mentioned, also planning proper upgrade procedures and regressions is the user responsibility.

No serious enterprise company sysadmin cries around because there is a bug in this or that open source project, maybe is fine crying around when Cisco fails, or palo alto but not open source.

To conclude, this projects saved thousands of pounds to many organizations across the scene and this types of toxic comments undermine the project vision, so no point escalating it going further.
Title: Re: Kernel panic after upgrade
Post by: marjohn56 on March 08, 2019, 07:09:40 am
Couldn't have put it better myself.... 👍👍👍



Title: Re: Kernel panic after upgrade
Post by: monstermania on March 08, 2019, 09:14:07 am
@akron
Well said!

BTW: My company uses a commercial UTM-Solution for many years now. The utm is fully supported only on her own hw! So, if you want full support you have to buy the appliance!
Of course, the utm had never had any boot problems.  ;)
Title: Re: Kernel panic after upgrade
Post by: golebole on March 08, 2019, 09:35:25 am

UPDATE: I copied the 19.7 kernel mentioned above in the #2 link to the /boot/kernel/kernel of the 19.1.2 install (well mine it 19.1-netmap) and all is well in the world. Boots with NO issues with EFI or BIOS firmware.


Is it possible to update the kerne to this one in the next 19.1 version :)?
Title: Re: Kernel panic after upgrade
Post by: RGijsen on March 08, 2019, 10:25:16 am
I'm a peaceful person however as an enterprise user it amazes me the quantity of bashers and shills in this forum. (Almost same rethoric as pfsense decadent times)

First of all everyone is free to choose what is necessary to deliver the job.

Coming from Cisco and Dell background, spending 15000€ per device in useless hardware for many years because of the BIG brand and BIG things and the BIG names, still useless for the price TAG.

Second this is an open source, community fuelled project, actually one of the best projects in terms of performance and features globally available.

If one is hurt about these bugs, please open your wallet and buy the iPhone of firewalls, CISCO, PALO ALTO, Fortigate and so on and be happy.

No one can demand or snow flake around because there is a bug, I've been running enterprise hardware from Deciso moving away from Cisco and never once been disappointed. Franco helped many times fixed bugs, and as an active user contributed many times monetarily to the project and will continue to do so, because it pays the bills around here.

Also running dozens of virtual workloads, from time to time there is a bug here and there, however one is responsible for the R&D and as AD mentioned, also planning proper upgrade procedures and regressions is the user responsibility.

No serious enterprise company sysadmin cries around because there is a bug in this or that open source project, maybe is fine crying around when Cisco fails, or palo alto but not open source.

To conclude, this projects saved thousands of pounds to many organizations across the scene and this types of toxic comments undermine the project vision, so no point escalating it going further.

I totally agree on that. But I still feel some of the devs could be more professional in their communication as well. Of course, it's their project, but if you disagree on something you are immediately labeled a pain in the @ss, bullied away, 'feel free to use another software', 'we don't click', we don't get paid, we don't have hardware, get banned, or whatever has passed in the last days. It's their project, but that doesn't mean they need to act like a God or something. People say things in the heat, probably we all do. But especially the admins / devs could be a bit more open minded as well, they are more or less an advocate for the atmosphere on the forum. It's what killed so many open source projects before. Don't let that happen again.

Just my 2 cents.
Title: Re: Kernel panic after upgrade
Post by: franco on March 08, 2019, 11:08:57 am
I disagree. Two users snapped because I said this:

Please sponsor the project if you feel we're not doing enough. <3

Everything went downwards from there orchestrated by the two users who snapped.

If you want to be treated professionally, treat others like you want to be treated. :)


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: Deku on March 08, 2019, 03:35:07 pm
I have this Kernel panic issue as well and watching the thread.  I'd be happy to try the debug version or whatever special release you want to provide that might help us with this issue.  My documentation on the panic thus far is at https://forum.opnsense.org/index.php?topic=11767.0 (https://forum.opnsense.org/index.php?topic=11767.0) if it helps.

I really appreciate franco's and AdSchellevis work on this project.  Even more so now as it sounds like it's voluntary.  As much time as franco puts into the project, I thought it was his day job working for Deciso and building Opnsense.  If not, it totally should be.  It takes a lot of love and passion to work on something like this for an extended period.  Thanks to all those involved and our friends, like lattera, working on HardenedBSD.   
Title: Re: Kernel panic after upgrade
Post by: franco on March 08, 2019, 04:14:45 pm
Here's an ISO snapshot based on the following commit: https://github.com/opnsense/src/commit/060d54597

https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-201903080927-OpenSSL-dvd-amd64.iso.bz2

All feedback is welcome. Other types of images can be requested if needed. The image is for testing, we don't recommend production use just yet.


Thank you,
Franco
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on March 08, 2019, 04:17:44 pm
Here's an ISO snapshot based on the following commit: https://github.com/opnsense/src/commit/060d54597

https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-201903080927-OpenSSL-dvd-amd64.iso.bz2

All feedback is welcome. Other types of images can be requested if needed. The image is for testing, we don't recommend production use just yet.


Thank you,
Franco

Downloading it now. Will let you know in a few minutes.
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on March 08, 2019, 04:26:05 pm
Here's an ISO snapshot based on the following commit: https://github.com/opnsense/src/commit/060d54597

https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-201903080927-OpenSSL-dvd-amd64.iso.bz2

All feedback is welcome. Other types of images can be requested if needed. The image is for testing, we don't recommend production use just yet.


Thank you,
Franco

Downloading it now. Will let you know in a few minutes.

 ;D Success  ;D

ISO image booted without issues.

Clean install and a boot from EFI firmware to HDD without issues.  ;D

Well done team!!!

- Waz

Title: Re: Kernel panic after upgrade
Post by: Ralf Kirmis on March 08, 2019, 05:20:53 pm
Hi,
also on a bare metal box with uefi only boot the image  works.
The machine is booting like a charm.  ;D

Good work guys.

Title: Re: Kernel panic after upgrade
Post by: lattera on March 08, 2019, 05:32:16 pm
I totally agree on that. But I still feel some of the devs could be more professional in their communication as well.

If I've made a mistake in my community interactions, please let me know. I'd like to learn from the experience in order to serve the community better. Life is a journey, mistakes are made, and hopefully learned from. :)
Title: Re: Kernel panic after upgrade
Post by: marjohn56 on March 08, 2019, 05:43:19 pm
I totally agree on that. But I still feel some of the devs could be more professional in their communication as well.

If I've made a mistake in my community interactions, please let me know. I'd like to learn from the experience in order to serve the community better. Life is a journey, mistakes are made, and hopefully learned from. :)


You've made no mistakes, neither has Franco.. Keep up the good work.
Title: Re: Kernel panic after upgrade
Post by: greY on March 08, 2019, 06:44:15 pm
Here's an ISO snapshot based on the following commit: https://github.com/opnsense/src/commit/060d54597

https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-201903080927-OpenSSL-dvd-amd64.iso.bz2

All feedback is welcome. Other types of images can be requested if needed. The image is for testing, we don't recommend production use just yet.


Thank you,
Franco
thx Franco
successfully booted on Hyper-V 9.0 (MS Server 2019) !


Title: Re: Kernel panic after upgrade
Post by: franco on March 08, 2019, 07:47:36 pm
Good, thank you for testing this so quickly. The core team discussed this internally and we have an accelerated plan of action for 19.1.4 next week:

1. Release the bad commit revert in a new kernel to restore the previous behaviour for everyone.
2. Change the upgrade paths to the new 19.1.4 for the 18.7 major upgrades one or two days after the 19.1.4 release.
3. Release new images based on 19.1.4 in the following week.
4. Rework the bad patch further to make it work for everyone and release it in a subsequent 19.1.x update together with the Netmap rework. This will require new test image runs with user participation. The time frame for this is very roughly April/May.

All further confirmations or new problem reports arising from the test image are welcome.

I hope this episode shows that we don't always get it 100% right but with a reasonable amount of patience and a level head we can move past almost anything together. :)


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: bunchofreeds on March 08, 2019, 08:24:38 pm
SUCCESS!!

Booted successfully a new Hyper-V VM Gen 2 with the updated ISO.

Fantastic work all round team. Thanks so much for continuously dedicating your time to solving these issues.
I can now continue with my plan of having OPNsense running as a virtual appliance, reducing my hardware footprint and applying more resource to OPNsense.

Really appreciate the way OPNsense worked with HardenedBSD on this one.

Thanks again all!!

EDIT:  Using Windows Server 2019 Std and as such creating Version 9 Hyper-V VM's
Title: Re: Kernel panic after upgrade
Post by: bimmerdriver on March 08, 2019, 09:12:41 pm
What version of windows server did you use? I have 2012R2. I can test tonight or tomorrow if required.
Title: Re: Kernel panic after upgrade
Post by: lattera on March 08, 2019, 09:37:34 pm
I hope this episode shows that we don't always get it 100% right but with a reasonable amount of patience and a level head we can move past almost anything together. :)

We're humans, we make mistakes occasionally. This also shows the importance of crowd testing beta and release candidates. :)
Title: Re: Kernel panic after upgrade
Post by: bimmerdriver on March 09, 2019, 02:16:36 am
The link to the snapshot is not working. The host pkg.opnsense.org is unreachable.
Title: Re: Kernel panic after upgrade
Post by: psycmos on March 09, 2019, 02:20:48 am
The link to the snapshot is not working. The host pkg.opnsense.org is unreachable.

Hi bimmerdriver.
You can download it from:
https://opnsense.c0urier.net/FreeBSD%3A11%3Aamd64/snapshots/

Somebody can help me?

I'm using unraid with NIC Intel 82575EB.
Version 18.7 intel NIC works well, but version OPNsense-201903080927-OpenSSL-dvd-amd64.iso.bz2 not recognize my intel. So There any any way to get it working on 19.1?

Thanks for this topic ;)
Best Regards, Nuno
Title: Re: Kernel panic after upgrade
Post by: bimmerdriver on March 09, 2019, 06:21:24 pm
The website is back up so I downloaded the iso. I installed it on windows server hyper-v 2012R2 using a generation 2 vm. It booted properly. Now, I'm back to the problem where the installer hangs at the point of selecting a guided installation or other installation. I interrupted it with <CTRL-C>, logged in again as installer and this time it proceeded to install. I did not complete the installation, but I think this confirms the fix on hyper-v 2012R2.
Title: Re: Kernel panic after upgrade
Post by: bimmerdriver on March 10, 2019, 02:57:18 am
I totally agree on that. But I still feel some of the devs could be more professional in their communication as well.

If I've made a mistake in my community interactions, please let me know. I'd like to learn from the experience in order to serve the community better. Life is a journey, mistakes are made, and hopefully learned from. :)


You've made no mistakes, neither has Franco.. Keep up the good work.
Agreed. Keep up the great work.
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on March 11, 2019, 03:47:06 am
Good, thank you for testing this so quickly. The core team discussed this internally and we have an accelerated plan of action for 19.1.4 next week:

1. Release the bad commit revert in a new kernel to restore the previous behaviour for everyone.
2. Change the upgrade paths to the new 19.1.4 for the 18.7 major upgrades one or two days after the 19.1.4 release.
3. Release new images based on 19.1.4 in the following week.
4. Rework the bad patch further to make it work for everyone and release it in a subsequent 19.1.x update together with the Netmap rework. This will require new test image runs with user participation. The time frame for this is very roughly April/May.

All further confirmations or new problem reports arising from the test image are welcome.

I hope this episode shows that we don't always get it 100% right but with a reasonable amount of patience and a level head we can move past almost anything together. :)


Cheers,
Franco

You tell me where the ISO's are or what to use to update for testing and I will test them on my VM's...well the best I can. I am using Proxmox VE 5.3 which is KVM/QEMU based on Debian distro with modified Ubunta LTS kernel. Also running ZFS over iSCSI to FreeNAS if you need something tested there.

I have two VM's in a HA configuration...one with UEFI and the other is BIOS with LibrsSSL, IPS and VIRTIO enabled.

I have two other VM's waiting to install from ISO with UEFI and BIOS.

-WAZ
Title: Re: Kernel panic after upgrade
Post by: franco on March 11, 2019, 03:30:28 pm
19.1.4 will be out tomorrow... can publish a test image the day after. Official images next week...


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: JDtheHutt on March 11, 2019, 08:37:00 pm
I had this issue too and it did cause breakage at home. Sure, I'd rather it hadn't happened, but I get this stuff for free and it's generally solid work, so I am appreciative of all the effort made by the Devs. I am consistently amused, puzzled, and annoyed by those who keep ranting on about how their essential production business system has been affected. I'm no expert, but I'm pretty sure that if I were running something like that I would a) not just mash the upgrade button the second a release is out b) not do it without testing on a standalone box first c) not do it without backups with a restore strategy I can revert to d) maybe consider paying for the proper support levels if I'm profiting from this and need the help urgently. But as I said, I'm no expert, maybe throwing my expensive business kit into the fire with no plan is how people roll these days.
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on March 12, 2019, 04:35:43 pm
19.1.4 will be out tomorrow... can publish a test image the day after. Official images next week...


Cheers,
Franco

Upgraded to 19.1.4 on a BIOS and UEFI firmware boot without issues.

Great job and thanks again, OPNsense Team.

-Waz
Title: Re: Kernel panic after upgrade
Post by: buddwm on March 12, 2019, 05:44:11 pm
Upgraded to 19.1.4 and can confirm all is well.  Great job on getting this out so quickly.
Title: Re: Kernel panic after upgrade
Post by: bimmerdriver on March 13, 2019, 02:16:15 am
I upgraded also on a windows server 2012r2 hyper-v with a generation 2 (uefi) vm. No problems.
Title: Re: Kernel panic after upgrade
Post by: newsense on March 13, 2019, 07:13:21 am
I upgraded also on a windows server 2012r2 hyper-v with a generation 2 (uefi) vm. No problems.

Quickly...it was not. Such a catastrophic bug should have been caught way early in testing and never make it to GA, let alone 4 releases later over many weeks. While I may be lacking all the information, it was a combination of multiple factors that led to 19.1 being shipped with it.

In the aftermath of this bug being fixed as a community effort, the most troublesome fact remains for the casual and perhaps not 100% informed viewer that the core team behind OPNsense didn't seem have the business' support it needed when it was high time for it - leaving the devs scrambling to find proper hardware to further troubleshoot and test regressions.


Here's hope lessons learned will be taken into consideration and the core team has the adequate support going forward -- as I don't see a future where the success of this entire project is ensured by testing it (mainly) of a handful of 'branded' appliances.
Title: Re: Kernel panic after upgrade
Post by: jschellevis on March 13, 2019, 08:55:47 am
@newsense, as said by others before an open source project is a community effort. This means that if you want to catch issues before a release it relies fully on your input and that of others.

The core team cannot test on any exotic piece of hardware, you will have to do that work yourself.

There are a lot of people participating in testing and running the development version long before it's release, they are reporting issues and help getting them fixed.

We are grateful for everyone putting in the work required to make this project as great as it is.

For those looking for professional support, just buy support from us.. we'll help you regardless of your hardware choice, but keep in mind that bug fixing is time consuming work.
Title: Re: Kernel panic after upgrade
Post by: golebole on March 13, 2019, 09:04:16 am
Hi,

Thanks for the good news. I saw yesterday the update and clicked update. I think I still get an older version, right? I had the same issue with the fatal error again. When do you think can I use the official update function?

Best regards
Christian
Title: Re: Kernel panic after upgrade
Post by: franco on March 13, 2019, 09:14:05 am
@newsense please don't start gaslighting after the fact.

I came here to announce that upgrades from 18.7 now land in 19.1.4 directly, but your comment takes all achievement away from everyone who contributed to the solution and the extra hours put in to make it happen on a tight schedule. It seems rather petty to me.

The discussion is already utterly absurd due to freaking out over indications that particular technical issues may never be fixed (can you fix all your problems in your real life, at work, for the planet?) and a general lack of understanding that 2-Clause BSD comes with no warranty or liability as others have pointed out.

The new 11.2 base has publicly called for testing since October 2018. If the community can't find a bug that we can't find we have to assume that it's not a big deal contrary to your "catastrophic" assessment. ;)

If people could get their facts straight before going into a discussion about how much one can coerce a project to do what one wants that does indeed already do *a lot* to move forward that would be great.


Thank you,
Franco
Title: Re: Kernel panic after upgrade
Post by: bitwolf on March 14, 2019, 05:09:15 pm
@franco can you please confirm what the direct upgrade from 18.7.10 to 19.1.4 you mentioned should look like in the UI?

I just tried to kick it off on a firewall I had kept back on 18.7, and while 19.1.4 is indeed shown as available, it displays the release notes for the initial 19.1, not 19.1.4 (screenshot attached).

Is this the expected behaviour?
Title: Re: Kernel panic after upgrade
Post by: lattera on March 14, 2019, 05:12:26 pm
I kinda wonder if it'd be best to show all changelogs from 19.1 to 19.1.4.
Title: Re: Kernel panic after upgrade
Post by: franco on March 14, 2019, 05:53:31 pm
This is expected. When 19.1 was released 18.7 was discontinued and points to 19.1. Normally that is fine, sometimes we have to move the "19.1" that 18.7 sees to a higher release, but that happens without 18.7 noticing and the only way to fix this is to build new 18.7 releases which is not possible anymore.

The pledge is 19.1 or later. It also depends on your mirror.


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: lattera on March 14, 2019, 06:00:40 pm
Franco,

Thanks for the clarification! Would you like me to file a feature request in github for teaching opnsense's updater how to display multiple changelogs?
Title: Re: Kernel panic after upgrade
Post by: franco on March 14, 2019, 06:02:14 pm
Not without a technical discussion about how this would work in practice (major upgrades). The upgrade sets are symlinks and there's no way to know where they point to from an updating system.


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: lattera on March 14, 2019, 06:06:41 pm
Gotcha. Once I'm finished with the Hyper-V issues, I'll take a look at how opnsense's updater works and take the discussion private with some potential solutions for the team to discuss.
Title: Re: Kernel panic after upgrade
Post by: franco on March 14, 2019, 06:08:03 pm
Sounds good, thanks! :)


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: bimmerdriver on March 15, 2019, 02:40:27 am
Gotcha. Once I'm finished with the Hyper-V issues, I'll take a look at how opnsense's updater works and take the discussion private with some potential solutions for the team to discuss.
If you need someone to test an installation on windows server 2012r2 hyper-v, let me know.
Title: Re: Kernel panic after upgrade
Post by: franco on March 15, 2019, 07:11:05 am
Oh, I posted this in the wrong thread, see below.

@bimmeldriver yes please, installer may hang but CTRL+C should work around it ok.

So this is the preliminary amd64 ISO for 19.1.4. It can be used in production and upgrades normally...

https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-19.1.4-OpenSSL-dvd-amd64.iso.bz2

All images, checksums and the announcement follow next week when we will have wrapped up testing image integrity for all other images (it takes about 2 days in total to do that).


Cheers,
Franco
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on March 15, 2019, 04:07:50 pm
Oh, I posted this in the wrong thread, see below.

@bimmeldriver yes please, installer may hang but CTRL+C should work around it ok.

So this is the preliminary amd64 ISO for 19.1.4. It can be used in production and upgrades normally...

https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-19.1.4-OpenSSL-dvd-amd64.iso.bz2

All images, checksums and the announcement follow next week when we will have wrapped up testing image integrity for all other images (it takes about 2 days in total to do that).


Cheers,
Franco

Franco/Lattera,

I will dedicate a few VM's on my side to test upgrades and installs for 18.7 to 19.1 and 19.1 to 19.7 on the QEMU/KVM platform. Willing to help out as much as I can.

Let me know if the upgrade images are ready for 18.7 to 19.1.4 and I will test that also.

Not sure I have any "baremetal" equipment to use in testing except some old Dell's.

Thanks again.

-Waz
Title: Re: Kernel panic after upgrade
Post by: lattera on March 15, 2019, 04:14:58 pm
Oh, I posted this in the wrong thread, see below.

@bimmeldriver yes please, installer may hang but CTRL+C should work around it ok.

So this is the preliminary amd64 ISO for 19.1.4. It can be used in production and upgrades normally...

https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-19.1.4-OpenSSL-dvd-amd64.iso.bz2

All images, checksums and the announcement follow next week when we will have wrapped up testing image integrity for all other images (it takes about 2 days in total to do that).


Cheers,
Franco

Franco/Lattera,

I will dedicate a few VM's on my side to test upgrades and installs for 18.7 to 19.1 and 19.1 to 19.7 on the QEMU/KVM platform. Willing to help out as much as I can.

Let me know if the upgrade images are ready for 18.7 to 19.1.4 and I will test that also.

Not sure I have any "baremetal" equipment to use in testing except some old Dell's.

Thanks again.

-Waz

Thank you very much! Your offer of assistance really means a lot to me. I've got a busy weekend as my wife has planned a few special events. I'll likely get back to tracking down the kernel NX bug seen mostly on Hyper-V systems and some Dell systems on Tuesday of next week. Each time I build a new HardenedBSD 11-stable ISO, I will let you know for your own testing. All I need to know is if it boots to the installer, no need to actually perform the install. I can also upload a memstick image so you don't have to dust off dvd discs. ;)
Title: Re: Kernel panic after upgrade
Post by: bimmerdriver on March 16, 2019, 01:40:57 am
Oh, I posted this in the wrong thread, see below.

@bimmeldriver yes please, installer may hang but CTRL+C should work around it ok.

So this is the preliminary amd64 ISO for 19.1.4. It can be used in production and upgrades normally...

https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-19.1.4-OpenSSL-dvd-amd64.iso.bz2

All images, checksums and the announcement follow next week when we will have wrapped up testing image integrity for all other images (it takes about 2 days in total to do that).


Cheers,
Franco
I created a new generation 2 vm from scratch using this iso. I had to interrupt the installer twice using <CTRL-C>. The first time was at the selection of guided installation. The second time was to accept the recommendation for the swap partition. I had previously saved the configuration from a different opnsense vm and it restored with no problems. Both IPv4 / dhcpv4 and IPv6 / dhcpv6 working. Looks good based on my testing.
Title: Re: Kernel panic after upgrade
Post by: Charles2019 on March 16, 2019, 04:01:11 pm
Oh, I posted this in the wrong thread, see below.

@bimmeldriver yes please, installer may hang but CTRL+C should work around it ok.

So this is the preliminary amd64 ISO for 19.1.4. It can be used in production and upgrades normally...

https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-19.1.4-OpenSSL-dvd-amd64.iso.bz2

All images, checksums and the announcement follow next week when we will have wrapped up testing image integrity for all other images (it takes about 2 days in total to do that).


Cheers,
Franco

First I want to say thanks for all the hard work put into this outstanding software!

I downloaded the ISO and burned it to a DVD.  When I boot from the DVD, it reboots over and over.  It does load the Kernel modules and then shows Booting.  The screen refreshes and looks like it is starting up and then that is when the computer restarts.  I even tried the Safe Mode and that didn't help.

The DVD Reader isn't that old while my Motherboard, Processor and RAM are.
Motherboard:  ASUS M4A785-M
Processor:  AMD Athlon II X2 245 Speed 2900Mhz
RAM: 8GB

The bios on the Motherboard is up to date.

Thanks,
Charles
Title: Re: Kernel panic after upgrade
Post by: lattera on March 16, 2019, 04:15:47 pm
Oh, I posted this in the wrong thread, see below.

@bimmeldriver yes please, installer may hang but CTRL+C should work around it ok.

So this is the preliminary amd64 ISO for 19.1.4. It can be used in production and upgrades normally...

https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-19.1.4-OpenSSL-dvd-amd64.iso.bz2

All images, checksums and the announcement follow next week when we will have wrapped up testing image integrity for all other images (it takes about 2 days in total to do that).


Cheers,
Franco

First I want to say thanks for all the hard work put into this outstanding software!

I downloaded the ISO and burned it to a DVD.  When I boot from the DVD, it reboots over and over.  It does load the Kernel modules and then shows Booting.  The screen refreshes and looks like it is starting up and then that is when the computer restarts.  I even tried the Safe Mode and that didn't help.

The DVD Reader isn't that old while my Motherboard, Processor and RAM are.
Motherboard:  ASUS M4A785-M
Processor:  AMD Athlon II X2 245 Speed 2900Mhz
RAM: 8GB

The bios on the Motherboard is up to date.

Thanks,
Charles

I believe setting `vm.pmap.pti=0` at the boot prompt will solve your boot loop issue. Once your system is booted, you can set that permanently via the GUI or via manual edit of /boot/loader.conf.local
Title: Re: Kernel panic after upgrade
Post by: Charles2019 on March 16, 2019, 04:38:18 pm
Oh, I posted this in the wrong thread, see below.

@bimmeldriver yes please, installer may hang but CTRL+C should work around it ok.

So this is the preliminary amd64 ISO for 19.1.4. It can be used in production and upgrades normally...

https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-19.1.4-OpenSSL-dvd-amd64.iso.bz2

All images, checksums and the announcement follow next week when we will have wrapped up testing image integrity for all other images (it takes about 2 days in total to do that).


Cheers,
Franco

First I want to say thanks for all the hard work put into this outstanding software!

I downloaded the ISO and burned it to a DVD.  When I boot from the DVD, it reboots over and over.  It does load the Kernel modules and then shows Booting.  The screen refreshes and looks like it is starting up and then that is when the computer restarts.  I even tried the Safe Mode and that didn't help.

The DVD Reader isn't that old while my Motherboard, Processor and RAM are.
Motherboard:  ASUS M4A785-M
Processor:  AMD Athlon II X2 245 Speed 2900Mhz
RAM: 8GB

The bios on the Motherboard is up to date.

Thanks,
Charles

I believe setting `vm.pmap.pti=0` at the boot prompt will solve your boot loop issue. Once your system is booted, you can set that permanently via the GUI or via manual edit of /boot/loader.conf.local

On the menu I pressed 3 for Escape to loader prompt and entered vm.pmap.pti=0 and it said not found.
I even tried menu 6 Configure Boot Options but nothing is there about vm.pmap.pti=0

Title: Re: Kernel panic after upgrade
Post by: lattera on March 16, 2019, 04:53:49 pm
When you're at the boot prompt (after selecting option 3), type this in:

set vm.pmap.pti="0"
boot
Title: Re: Kernel panic after upgrade
Post by: Charles2019 on March 17, 2019, 11:21:31 pm
When you're at the boot prompt (after selecting option 3), type this in:

set vm.pmap.pti="0"
boot

That worked perfectly!   What is that setting for?
Title: Re: Kernel panic after upgrade
Post by: bunchofreeds on March 18, 2019, 03:48:45 am
This should help @Charles2019

https://wiki.freebsd.org/SpeculativeExecutionVulnerabilities
Title: Re: Kernel panic after upgrade
Post by: Charles2019 on March 19, 2019, 01:32:06 am
This should help @Charles2019

https://wiki.freebsd.org/SpeculativeExecutionVulnerabilities

So from my understanding, I am turning off the protection for Meltdown. 

Has OPNSense been updated with this below?:
https://reviews.freebsd.org/rS329462

When can I enable the Meltdown protection again?

Title: Re: Kernel panic after upgrade
Post by: bunchofreeds on March 19, 2019, 02:17:54 am
I'm not an expert in this and it would be best for one of the OPNsense / HardenedBSD team to answer this to provide certainty... but I understand you have an AMD Athlon CPU, and putting that against the chart supplied in the link I sent, then your CPU is not vulnerable to meltdown, being variants 3 and 3a. So disabling the meltdown specific mitigation's should be OK in your case, and having it enabled is in fact causing issue with your AMD CPU.

I have an Intel CPU, so this does need to be enabled for me across the board unfortunately.
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on March 22, 2019, 06:07:35 pm
Tired a new OVMF Firmware on the Proxmox or QEMU/KVM to see if that might fix the kernel panics and it did NOT. I had high hopes that the panics might have been because of old OVMF firmware (UEFI).

The project is based at https://github.com/tianocore/tianocore.github.io/wiki/edk-ii and by https://www.tianocore.org.

Actual builds are based at https://www.kraxel.org/repos/  You can extract the running code on any RPM service and replace the OVMF_CODE.fd and OVMF_VARS.fd with the OVMF_CODE-pure-efi.fd and OVMF_VARS-pure-efi.fd to just have the newest code. Mine was dated on Nov 2016. The new code did give me the ability to PXE boot via HTTP which was a big bonus for me.

One thing note with the issue at hand from the FAQ...
https://github.com/tianocore/tianocore.github.io/wiki/OVMF-FAQ#are-ovmf-releases-fully-uefi-compliant
Are OVMF releases fully UEFI compliant?

While the goal is to be as fully UEFI compliant as possible, you should not assume that an OVMF release is fully UEFI compliant unless the particular release states full compliance.

For virtual machines, there are some challenging areas in achieving full UEFI compliance. For example, UEFI 'non-volatile' variables may be difficult to fully support in some virtual machine environments if a flash memory device is not emulated.

This could be an issue with the previous change to the kernel and VM based UEFI.

Hope this helps.

-Waz

Title: Re: Kernel panic after upgrade
Post by: lattera on March 22, 2019, 06:09:17 pm
Can you post a screenshot of the panic?
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on March 22, 2019, 07:39:12 pm
Can you post a screenshot of the panic?

I apologize, this does not effect 9.1.4 at this time. I was just trying to use the original 9.1 upgrade and ISO to see if a new version of UEFI would fix the issue.

Sorry for the confusion but if you still need it is 9.1-netmap version ok?

-Waz
Title: Re: Kernel panic after upgrade
Post by: lattera on March 26, 2019, 03:07:55 am
I'll just leave this here. :)

Test ISO coming soon.
Title: Re: Kernel panic after upgrade
Post by: lattera on March 26, 2019, 03:18:42 pm
The ISO has been uploaded here: https://hardenedbsd.org/~shawn/opnsense/2019-03-26_hbsd_11-stable_disc1.iso

Can anyone interesting in running OPNsense 19.1 in a Hyper-V Gen2 instance please test?
Title: Re: Kernel panic after upgrade
Post by: Meik on March 26, 2019, 04:44:31 pm
Hi lattera,

the Attached Image ist the boot Screen and the Settings (German)
Secure Boot is of.

Hope this helps - Meik

Hyper-V Host: Window Server 2019, Dual XEON E5-2620 v2, Supermicro X9DR3-F Board, 64GB-ECC-Reg-Ram, NVMe-Storage-Pool with Linux optimised vhdx (-BlockSizeBytes 1MB)
Title: Re: Kernel panic after upgrade
Post by: lattera on March 26, 2019, 05:17:18 pm
Hi lattera,

the Attached Image ist the boot Screen and the Settings (German)
Secure Boot is of.

Thanks for helping! What are the specs for your hardware? I'm primarily curious about CPU make and model.
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on March 26, 2019, 06:13:42 pm
Screenshots attached.
Title: Re: Kernel panic after upgrade
Post by: Meik on March 26, 2019, 06:39:28 pm
I've added the Hardware above.

i tested also on a Server 2012 R2, Gen2, Configuration Version 5.0, there installer works, Live-CD too
E3-1270 v3, Intel Board S1200RP, 32GB ECC RAM

On a Server 2016, Gen2, Configuration Version 8.0, it works like the Server 2012 R2
E3-1240 v5, Supermicro Board X11SSL-CF, 32GB ECC Ram

On the Server 2019 (above) i create a Gen2 Configuration Version 5.0, the same "error"

On a Windows 10 1809, Gen2 Configuration Version 9.0, the same error
i3-4150, Supermicro X10SLV-Q, 8GB RAM (a small PBX-Host)

Something that i can test? (for testing i removed the NIC from the configuration, set use other CPU-Version, error is the same)
Title: Re: Kernel panic after upgrade
Post by: lattera on March 26, 2019, 06:44:25 pm
I've added the Hardware above.

i tested also on a Server 2012 R2, Gen2, Configuration Version 5.0, there installer works, Live-CD too
E3-1270 v3, Intel Board S1200RP, 32GB ECC RAM

On a Server 2016, Gen2, Configuration Version 8.0, it works like the Server 2012 R2
E3-1240 v5, Supermicro Board X11SSL-CF, 32GB ECC Ram

On the Server 2019 (above) i create a Gen2 Configuration Version 5.0, the same "error"

On a Windows 10 1809, Gen2 Configuration Version 9.0, the same error
i3-4150, Supermicro X10SLV-Q, 8GB RAM (a small PBX-Host)

Something that i can test? (for testing i removed the NIC from the configuration, set use other CPU-Version, error is the same)

Is this your processor, then? https://ark.intel.com/content/www/us/en/ark/products/52276/intel-xeon-processor-e3-1270-8m-cache-3-40-ghz.html

No need for changes on your end. A basic Gen2 config (with secure boot disabled) should be enough for testing.
Title: Re: Kernel panic after upgrade
Post by: Meik on March 26, 2019, 08:05:26 pm
no, the Windows Server 2012 R2 Host has a v3:
https://ark.intel.com/content/www/de/de/ark/products/75056/intel-xeon-processor-e3-1270-v3-8m-cache-3-50-ghz.html

the Windows Server 2016 a:
https://ark.intel.com/content/www/us/en/ark/products/88176/intel-xeon-processor-e3-1240-v5-8m-cache-3-50-ghz.html

the Windows Server 2019 a:
https://ark.intel.com/content/www/us/en/ark/products/75789/intel-xeon-processor-e5-2620-v2-15m-cache-2-10-ghz.html


Title: Re: Kernel panic after upgrade
Post by: JDtheHutt on March 27, 2019, 02:21:25 pm
I need to apologise. Seems my error was not the kernel panic. It started after my upgrade and exhibited similar behaviour to other reports, and also happened if booting on a live USB, so I assumed was the same.

However, turns out my SSD had suddenly begun failing and died the other day. Having finally sourced a replacement today, everything is now perfect. Not sure why the SSD was also causing the live USB boot to fail, but it was as no such issue with the new one and I've not changed anything else.

Config imported and back up and running in a few minutes. Amazing work from all involved in the development. Thank you!
Title: Re: Kernel panic after upgrade
Post by: TheGrandWazoo on March 27, 2019, 03:40:18 pm
Yesterday, I also booted a Bare Metal Dell Optiplex 9020 using the HBSD-11-2019-03-26 ISO CD and was successful with the boot. I did a F12 for the boot menu and selected a UEFI boot from the CD.

This machine has a Intel Core i5-4680 @ 3.4GHz with 8Gb RAM. Firmware is at A24, 2018-10-24.

Also, had a successful boot with HBSD-11-2019-03-26 ISO CD on a Dell Latitude E5540. Intel Core i5-4310U @ 2.00GHz and 4Gb RAM. Firmware is A23, 2018-10-08.

-Waz
Title: Re: Kernel panic after upgrade
Post by: bimmerdriver on March 31, 2019, 05:01:58 am
The ISO has been uploaded here: https://hardenedbsd.org/~shawn/opnsense/2019-03-26_hbsd_11-stable_disc1.iso

Can anyone interesting in running OPNsense 19.1 in a Hyper-V Gen2 instance please test?
I created a gen 2 VM on a windows server 2012R2 hyper-v using the iso and there were no problems. I went all the way through the installation until the final step to reboot, but I didn't boot the installed image.
Title: Re: Kernel panic after upgrade
Post by: lattera on March 31, 2019, 11:15:04 pm
The ISO has been uploaded here: https://hardenedbsd.org/~shawn/opnsense/2019-03-26_hbsd_11-stable_disc1.iso

Can anyone interesting in running OPNsense 19.1 in a Hyper-V Gen2 instance please test?
I created a gen 2 VM on a windows server 2012R2 hyper-v using the iso and there were no problems. I went all the way through the installation until the final step to reboot, but I didn't boot the installed image.

Awesome! And the original 19.1 ISO caused crashes for you, correct?
Title: Re: Kernel panic after upgrade
Post by: bimmerdriver on April 01, 2019, 01:15:17 am
The ISO has been uploaded here: https://hardenedbsd.org/~shawn/opnsense/2019-03-26_hbsd_11-stable_disc1.iso

Can anyone interesting in running OPNsense 19.1 in a Hyper-V Gen2 instance please test?
I created a gen 2 VM on a windows server 2012R2 hyper-v using the iso and there were no problems. I went all the way through the installation until the final step to reboot, but I didn't boot the installed image.

Awesome! And the original 19.1 ISO caused crashes for you, correct?
The installer was hanging at one or two places, requiring <CTRL-C> and logging back in. That did not happen with the linked ISO. However, that is the first time I've tried to install HBSD directly. I've only tried installing OPNsense previously.
Title: Re: Kernel panic after upgrade
Post by: agh1701 on April 01, 2019, 03:44:40 pm
The ISO has been uploaded here: https://hardenedbsd.org/~shawn/opnsense/2019-03-26_hbsd_11-stable_disc1.iso

Can anyone interesting in running OPNsense 19.1 in a Hyper-V Gen2 instance please test?
Installed, booted and logged in.  Only limited testing then shutdown.  Works fine.
Title: Re: Kernel panic after upgrade
Post by: CaptainKaze on July 09, 2019, 02:56:42 pm
When you're at the boot prompt (after selecting option 3), type this in:

set vm.pmap.pti="0"
boot

Hi there ;) is there a way to set this global, so this setting survives a reboot?
Thank you very much.
Title: Re: Kernel panic after upgrade
Post by: hbc on July 09, 2019, 10:05:28 pm
Add it to /boot/loader.conf.local
Title: Re: Kernel panic after upgrade
Post by: franco on July 10, 2019, 05:57:22 pm
Or System: Settings: Tunables.
Title: Re: Kernel panic after upgrade
Post by: CaptainKaze on July 14, 2019, 10:32:10 pm
ah perfect. so easy ... thank you :)