OPNsense Forum

Hi all,
Running into a problem and I am not sure if I am missing something here. I downloaded opnsense and extracted it out of the .bz2. I normaly use RUFUS to make my usb bootable with the img files and this has always worked but when I try to do this with the opnsense.img it creates the usb but when I try to boot to it on the empty system it just reboots like it is not detecting anything img on the usb. I am pretty sure this system can run this as I have tested it with pfsense and some other FreeBSD programs. Any idea what I am missing?

Hi melfesto,

Can you try this with another OS like Unix and a plain dd command? Rufus has had problems in the past, and the ISO doesn't work in Rufus at all (but I think it says so in an error).


Cheers,
Franco

sure thing I will give it a try and let you know if it worked. Thanks.

I have yet to have physdiskwrite fail to create successful bootable USB sticks for opnsense. Even when all other tools fail. It is all I use now instead of wasting time with other options.

Hello
It is working https://etcher.io 

Hi there !

I've the same problem...

My hardware :

- Qotom Q190G4-S2 (http://amzn.to/2qdZJhq)
- 8Go RAM Kingston (http://amzn.to/2rbSdBe)
- 120Go SSD

What I've already test :

- Booting on Windows 10 (OLD install on the SSD) : OK
- Creating USB OPNSense Installer with Etcher : NOK
- Creating USB OPNSense Installer with RUFUS : NOK
- Creating USB PfSense Installer with Etcher / RUFUS : OK

Think it's an OPNSense issue and not a hardware issue because the hardware works with others OS (pfSense/Windows 10)

Someone have an idea to fix this problem ?

Use physdiskwrite

Quote from: csmall on May 14, 2017, 02:44:03 AM
Use physdiskwrite

Have tried. NOK. With several USB key of different models...

Installation of pfSense works fine... What the hell with OPNSense ? Some comments in this page talk about the same issue :

https://oitibs.com/opnsense-usb-install-guide-rufus/

LAST CHANCE AND IT WORKS !

The last thing i didn't try was making USB with Linux and the dd command. So, I put the USB key on my RaspberryPi and type these commands :

wget https://fleximus.org/mirror/opnsense/releases/mirror/OPNsense-17.1.4-OpenSSL-vga-amd64.img.bz2
bzip2 -d OPNsense-17.1.4-OpenSSL-vga-amd64.img.bz2
dd if=OPNsense-17.1.4-OpenSSL-vga-amd64.img of=/dev/sda bs=16k

and now, system is booting fine.

I think that there are some issues with Windows and tools like Etcher, UUI, DiskImager, RUFUS and others, and it could be great to put a warning on the documentation.

A word to the wise, and this comes from over 20 years experience in IT.

A product that doesn't start well, never ends well.

I was going to install OpnSense in lieu of Pfsense on a box I just built. I built the box with OpnSense in mind. Then when I went to create the install media, I ran into the same issues detailed above.

Y'know what that caused me to do? Install Pfsense.

Because if you're not bright enough to put out a standard image that any tool can put on a disk or a stick, this product cannot be trusted. That's practically a written guarantee that I will have to deal with weirdness later on that some mental child thought would be a better idea than what millions accept as a standard and best practice.

No, I'm not going to run some hokey command line tool that the dweeb responsible for the distribution thought would be a great idea. Thanks for sucking time out of my life I'll never get back, but no worries, it won't happen again.  C'ya!

Do you have any insight in how "better images" are built? I'll do anything you say if you tell me how to achieve this. :)


Cheers,
Franco

Let me explain what's going on.

The USB image is an image of a physical disk. In this case the disk image uses GUID Partition Table (GPT) partitioning. GPT was invented around 2001 to get around limitations of older Master Boot Record (MBR) style partitioning

On MBR partitioned drives, Sector 0 is the special boot sector. The BIOS knows to read Sector 0, copy the contents of the 512-byte sector into memory, and JUMP to the first byte - which is executable code.

On a GPT partitioned drive, it is sector 1 that contains the partition information. Your UEFI BIOSs know to ignore sector 0, and instead reads sector 1 for partition information.

QuoteNote: For compatibility with non-UEFU BIOSes, drives can still create a dummy MBR in sector zero, which provides just enough code to read sector 1, parse it, and transfer control to the active partition like a UEFI BIOS would

Part of the format of GPT partitioned disks is:

• Sector 1 contains the GPT Header
• GPT header gives the sectors that contain partition entries
• GPT header has a checksum of all the partition entries
• GPT header indicates the sector that contains the backup GPT Header
• GPT header has a checksum of itself
• The last sector of the disk contains a backup GPT Header

If for example your OpnSense USB image is only 900MB (950,943,744 bytes) long, it is sector 1,857,314 that is the last sector. That means that in your image it is sector 1,857,314 that contains the backup GPT Header:

• Sector 0: dummy MBR
• Sector 1: GPT Header
• ...
• Sector 1,857,314: Backup GPT Header

The important point here for people trying to blindly copy (i.e. DD) a GPT image to a new disk is: you just created an invalid disk. The last sector of your physical device doesn't contain the required GPT backup copy.

When you copy that to a real 32GB USB stick, you have a problem:

• Sector 0: dummy MBR
• Sector 1: GPT Header
• ...
• Sector 1,857,314: Backup GPT Header (from the image; not really the last sector of the drive)
• ...
• Sector 62,652,415: Empty (Last sector of the drive which should, but doesn't contain the backup GPT)

You've just written out an invalid GUID-Partition Table disk: the backup GPT Header in the last sector is missing.

In reality this normally doesn't cause any problems. The UEFI BIOS reads the GPT Header, and transfer boot control to the active partition. FreeBSD will actually check that the backup exists, and give you a warning if it doesn't. But other than that, it's not catastrophic.

But Windows 10 disagrees

When you (e.g.  Rufus, physdiskwrite, etcher, Win32DiskImager) finish writing your image and close the physical disk handle, something deep inside Windows checks that you didn't fuck up the disk. It detects that there is no backup GPT Header in the (empty) final sector, and creates one. When it does so, it needs to update the checksum of the GPT header (because we're changing the value that indicates the sector that contains the backup header):

• Sector 1 contains the GPT Header
• GPT header gives the sectors that contain partition entries
• GPT header has a checksum of all the partition entries (0xB6E18EEF)
• GPT header indicates the sector that contains the backup GPT Header (Sector 1,857,314 62,652,415)
• GPT header has a checksum of itself (0xDC779EC2 0xAE3C7980)

And that would have been fine. The Sector numbers are correct. The header checksum is correct. Except it's not.

When Windows 10 recalculates the CRC-32 of all the partition entries, it calculates it wrong.

Windows is expecting:

SizeOfPartitionEntry (usually 128 bytes) * NumberOfPartitionEntries (usually 128 entries)

to be a multiple of sector size (e.g. 512). On a standard GPT formatted disk, there are 128 bytes per partition entry (same is true in OpnSense).

On a GPT disk, formatted by Windows, there are 128 partition entries:

128 bytes * 128 entries = 16,384 bytes / 512 bytes per sector = 32 whole sectors

In OpnSense, the GPT image contains 3 partition entries:

128 bytes * 3 entries = 384 bytes / 512 bytes bytes per sector = not even one whole sector

Windows is calculating the CRC over the entire 512-byte sector, when it should be calculating on only the first 384 bytes of the sector. This is documented in the UEFI specification 2.6 - 5.3.1: (http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_6%20Errata%20A%20final.pdf)

Quote5.3.1 GPT overview

There is a 32-bit CRC of the GPT Partition Entry Array that is stored in the GPT Header in Partition Entry Array CRC32 field. The size of the GPT Partition Entry Array is Size Of Partition Entry multiplied by Number Of Partition Entries. If the size of the GUID Partition Entry Array is not an even multiple of the logical block size, then any space left over in the last logical block is Reserved and not
covered by the Partition Entry Array CRC32 field.

This causes the updated GPT Header (and the backup GPT Header) to contain the wrong CRC-32 checksums.

Normally this wouldn't be a problem. You can still boot the image.

But FreeBSD disagrees

Rather than booting based on what's in the GPT Header and partition entries, FreeBSD actually checks the checksums. And it detects that the primary GPT Header contains an invalid checksum. It also checks the backup GPT Header, and detects that it contains an invalid checksum.

So it gives you the error message:

Quotegptboot: primary GPT table checksum mismatch
gptboot: backup GPT table checksum mismatch
gptboot: unable to load GPT

And refuses to boot. It could if it just bothered to try. This is after it already did boot - the UEFI BIOS transferred control to gptboot. GptBoot just had to be a jerk and reset the machine rather than continuing.


So there's a chain of fail:

• USB image writers do not output a valid GPT disk
• FreeBSD doesn't use a number of partition entries that makes them a multiple of the sector size
• Windows outputs an invalid checksum (because it assumes everyone makes their partition entries a multiple of the sector size)
• FreeBSD's gptboot refuses to boot the disk as presented

Every image writer, writing a GPT disk without correcting the GPT backup sector, is doing it wrong. FreeBSD should simply say they have 4 partition entries rather than 3. Windows should calculate the CRC correctly. GptBoot should boot a perfectly valid disk (the UEFI said so).


So many fails. Lets see who stands up the point fingers.

What I did was use the free hex editor HxD to manually calculate the two CRC32 checksums. HxD lets you open a physical disk and hex edit it. It also has the ability to calculate CRC-32 of selected bytes.

My suggestion: have it be 4 GPT partitions rather than 3. That's something probably in OPNSense's control, solves the issues, and gets people up and running.

...very cool 8-)

+1 for "post of the month"

I would say post of the year 2017. :)

FlakStapper, many thanks for the explanations and suggestion. In light of numerous doubts and expressed frustrations, your contribution is the only thing that can actually bring us forward.

I added a mock swap partition of 512k in the UEFI case in front of the actual root partition... does that change the behaviour in a desired fashion?

https://pkg.opnsense.org/snapshots/OPNsense-201705231230-LibreSSL-vga-amd64.img.bz2


Thanks,
Franco

That workaround works around!

The recalcuated checksums are correct, and the image boots.


GPT Header from image

45 46 49 20 50 41 52 54	Signature	"EFI PART"
00 00 01 00	EFI Version	0x0001 0x0000 ==> 1.0
5C 00 00 00	Header size	0x0000005C ==> 92 bytes
CB 18 6E 09	CRC-32 checksum of all 92 bytes	0x096E18CB
00 00 00 00	Reserved. Must be zero
01 00 00 00 00 00 00 00	LBA of primary GPT header	0x0000000000000001 ==> Sector 1
92 5A 1D 00 00 00 00 00	LBA of backup GPT header	0x00000000001D5A92 ==> Sector 1,923,730
03 00 00 00 00 00 00 00	First usable LBA	0x0000000000000003 ==> Sector 3
90 5A 1D 00 00 00 00 00	Last usable LBA	0x00000000001D5A90 ==> Sector 1,923,728
F1 99 FF E9 A2 3F E7 11 90 3F 00 0C 29 54 E1 33	DiskGUID	{F199FFE9-A23F-E711-903F-000C2954E133}
02 00 00 00 00 00 00 00	Partition entry LBA	0x0000000000000002 ==> Sector 2
04 00 00 00	Number of partition entries	0x00000004 ==> 4 entries
80 00 00 00	Size of partition entries	0x00000080 ==> 128 bytes per entry
6A BD 1F 16	CRC-32 checksum of partition entries array	0x161FBD6A

GPT Header as it exists on disk after writing

45 46 49 20 50 41 52 54	Signature	"EFI PART"
00 00 01 00	EFI Version	0x0001 0x0000 ==> 1.0
5C 00 00 00	Header size	0x0000005C ==> 92 bytes
65 45 73 65	CRC-32 checksum of all 92 bytes	0x096E18CB
00 00 00 00	Reserved. Must be zero
01 00 00 00 00 00 00 00	LBA of primary GPT header	0x0000000000000001 ==> Sector 1
FF FF BB 03 00 00 00 00	LBA of backup GPT header	0x0000000003BBFFFF ==> Sector 62,652,415
03 00 00 00 00 00 00 00	First usable LBA	0x0000000000000003 ==> Sector 3
FD FF BB 03 00 00 00 00	Last usable LBA	0x0000000003BBFFFD ==> Sector 62,652,413
F1 99 FF E9 A2 3F E7 11 90 3F 00 0C 29 54 E1 33	DiskGUID	{F199FFE9-A23F-E711-903F-000C2954E133}
02 00 00 00 00 00 00 00	Partition entry LBA	0x0000000000000002 ==> Sector 2
04 00 00 00	Number of partition entries	0x00000004 ==> 4 entries
80 00 00 00	Size of partition entries	0x00000080 ==> 128 bytes per entry
6A BD 1F 16	CRC-32 checksum of partition entries array	0x161FBD6A

Now i'm just need to wait for my "default route not set" issue (https://github.com/opnsense/core/issues/850#issuecomment-301972040) before i can actually try OPNSense.

Commit is here for reference: https://github.com/opnsense/tools/commit/cccd73b4335

I'll publish a VGA image on top of this for 17.1.8 once that is out. Many thanks again.

Now on to the next one. :D


Cheers,
Franco

So, here's a VGA UEFI image for 17.1.8 that Windows likes, finally. Thanks again for the help.

https://pkg.opnsense.org/snapshots/OPNsense-17.1.8-OpenSSL-vga-amd64.img.bz2

This problem still exists with the current VGA.

Fixed it for you:

> A problem still exists with the current VGA.

There is no additional data here and it does not belong into a thread marked as solved not replied to for almost 2 years.


Cheers,
Franco

Quote from: franco on May 21, 2019, 06:33:02 PM
There is no additional data here and it does not belong into a thread marked as solved not replied to for almost 2 years.


Cheers,
Franco

There is if the problem still exists which it does.

FlakStapper gave a very good reason for this at https://forum.opnsense.org/index.php?topic=5121.0#msg21207

You realise you're still referencing an issue that was and remains fixed.


Cheers,
Franco

Sorry to bother, but it definitely has that problem for the i386 version of the img files (both 20.1 and 19.7).  The amd64 version works fine however.  Are only a couple of people are trying to use such antiquated hardware as I am?   While having a fully functional version for I386 systems is truly excellent, it seems the price to pay is using a DVD instead of a thumb drive.  Really not complaining,  but just FYI...

For major releases we test all images for proper function on different hardware, but I can't rule out the possibility that not all hardware acts the same way, especially older hardware with older and/or buggy BIOS versions.

Yet i386 is going away in a few months when 20.7 is arrived as we have been planning for the last couple of years. 20.1 already added a grace period for i386 in this regard, because the original goal was to finish the operating system migration from HBSD 11.2 to 12.1 with 20.1, but also said that as long as we have 11.2 there will be a matching i386 version available.


Cheers,
Franco

Thanks for your reply.  Given what you said, I guess it's kind of academic whether or not I can run the i386 installer from a thumb drive.   I appreciate you've kept my old hardware usable for this long anyway.   
Still just for kicks, I tried booting the same hardware with the amd64 version - it got well past the boot loader screen before complaining about the CPU.  By contrast, the I386 version didn't even make it to the boot loader.  Hmm....
I guess we'll never know why.   Oh well...  I'm just going to dig up some slightly newer old hardware to get ready for 20.7  :)

Thanks Again!

And Here we are today...

I have the same problem after an exhausting 5 hours, coming from pfsense, past 10 years...

Every other operating system boots fine from the USB, except opnsense...

I finally dug up this thread... and even tho four years ago..

Seeee ya!!