Messages

[GPT Header from image]

That workaround works around!

The recalcuated checksums are correct, and the image boots.


GPT Header from image

45 46 49 20 50 41 52 54	Signature	"EFI PART"
00 00 01 00	EFI Version	0x0001 0x0000 ==> 1.0
5C 00 00 00	Header size	0x0000005C ==> 92 bytes
CB 18 6E 09	CRC-32 checksum of all 92 bytes	0x096E18CB
00 00 00 00	Reserved. Must be zero
01 00 00 00 00 00 00 00	LBA of primary GPT header	0x0000000000000001 ==> Sector 1
92 5A 1D 00 00 00 00 00	LBA of backup GPT header	0x00000000001D5A92 ==> Sector 1,923,730
03 00 00 00 00 00 00 00	First usable LBA	0x0000000000000003 ==> Sector 3
90 5A 1D 00 00 00 00 00	Last usable LBA	0x00000000001D5A90 ==> Sector 1,923,728
F1 99 FF E9 A2 3F E7 11 90 3F 00 0C 29 54 E1 33	DiskGUID	{F199FFE9-A23F-E711-903F-000C2954E133}
02 00 00 00 00 00 00 00	Partition entry LBA	0x0000000000000002 ==> Sector 2
04 00 00 00	Number of partition entries	0x00000004 ==> 4 entries
80 00 00 00	Size of partition entries	0x00000080 ==> 128 bytes per entry
6A BD 1F 16	CRC-32 checksum of partition entries array	0x161FBD6A

GPT Header as it exists on disk after writing

45 46 49 20 50 41 52 54	Signature	"EFI PART"
00 00 01 00	EFI Version	0x0001 0x0000 ==> 1.0
5C 00 00 00	Header size	0x0000005C ==> 92 bytes
65 45 73 65	CRC-32 checksum of all 92 bytes	0x096E18CB
00 00 00 00	Reserved. Must be zero
01 00 00 00 00 00 00 00	LBA of primary GPT header	0x0000000000000001 ==> Sector 1
FF FF BB 03 00 00 00 00	LBA of backup GPT header	0x0000000003BBFFFF ==> Sector 62,652,415
03 00 00 00 00 00 00 00	First usable LBA	0x0000000000000003 ==> Sector 3
FD FF BB 03 00 00 00 00	Last usable LBA	0x0000000003BBFFFD ==> Sector 62,652,413
F1 99 FF E9 A2 3F E7 11 90 3F 00 0C 29 54 E1 33	DiskGUID	{F199FFE9-A23F-E711-903F-000C2954E133}
02 00 00 00 00 00 00 00	Partition entry LBA	0x0000000000000002 ==> Sector 2
04 00 00 00	Number of partition entries	0x00000004 ==> 4 entries
80 00 00 00	Size of partition entries	0x00000080 ==> 128 bytes per entry
6A BD 1F 16	CRC-32 checksum of partition entries array	0x161FBD6A

Now i'm just need to wait for my "default route not set" issue before i can actually try OPNSense.

[GPT]

Let me explain what's going on.

The USB image is an image of a physical disk. In this case the disk image uses GUID Partition Table (GPT) partitioning. GPT was invented around 2001 to get around limitations of older Master Boot Record (MBR) style partitioning

On MBR partitioned drives, Sector 0 is the special boot sector. The BIOS knows to read Sector 0, copy the contents of the 512-byte sector into memory, and JUMP to the first byte - which is executable code.

On a GPT partitioned drive, it is sector 1 that contains the partition information. Your UEFI BIOSs know to ignore sector 0, and instead reads sector 1 for partition information.

Note: For compatibility with non-UEFU BIOSes, drives can still create a dummy MBR in sector zero, which provides just enough code to read sector 1, parse it, and transfer control to the active partition like a UEFI BIOS would

Part of the format of GPT partitioned disks is:

• Sector 1 contains the GPT Header
• GPT header gives the sectors that contain partition entries
• GPT header has a checksum of all the partition entries
• GPT header indicates the sector that contains the backup GPT Header
• GPT header has a checksum of itself
• The last sector of the disk contains a backup GPT Header

If for example your OpnSense USB image is only 900MB (950,943,744 bytes) long, it is sector 1,857,314 that is the last sector. That means that in your image it is sector 1,857,314 that contains the backup GPT Header:

• Sector 0: dummy MBR
• Sector 1: GPT Header
• ...
• Sector 1,857,314: Backup GPT Header

The important point here for people trying to blindly copy (i.e. DD) a GPT image to a new disk is: you just created an invalid disk. The last sector of your physical device doesn't contain the required GPT backup copy.

When you copy that to a real 32GB USB stick, you have a problem:

• Sector 0: dummy MBR
• Sector 1: GPT Header
• ...
• Sector 1,857,314: Backup GPT Header (from the image; not really the last sector of the drive)
• ...
• Sector 62,652,415: Empty (Last sector of the drive which should, but doesn't contain the backup GPT)

You've just written out an invalid GUID-Partition Table disk: the backup GPT Header in the last sector is missing.

In reality this normally doesn't cause any problems. The UEFI BIOS reads the GPT Header, and transfer boot control to the active partition. FreeBSD will actually check that the backup exists, and give you a warning if it doesn't. But other than that, it's not catastrophic.

But Windows 10 disagrees

When you (e.g.  Rufus, physdiskwrite, etcher, Win32DiskImager) finish writing your image and close the physical disk handle, something deep inside Windows checks that you didn't fuck up the disk. It detects that there is no backup GPT Header in the (empty) final sector, and creates one. When it does so, it needs to update the checksum of the GPT header (because we're changing the value that indicates the sector that contains the backup header):

• Sector 1 contains the GPT Header
• GPT header gives the sectors that contain partition entries
• GPT header has a checksum of all the partition entries (0xB6E18EEF)
• GPT header indicates the sector that contains the backup GPT Header (Sector 1,857,314 62,652,415)
• GPT header has a checksum of itself (0xDC779EC2 0xAE3C7980)

And that would have been fine. The Sector numbers are correct. The header checksum is correct. Except it's not.

When Windows 10 recalculates the CRC-32 of all the partition entries, it calculates it wrong.

Windows is expecting:

Code Select Expand

SizeOfPartitionEntry (usually 128 bytes) * NumberOfPartitionEntries (usually 128 entries)

to be a multiple of sector size (e.g. 512). On a standard GPT formatted disk, there are 128 bytes per partition entry (same is true in OpnSense).

On a GPT disk, formatted by Windows, there are 128 partition entries:

Code Select Expand

128 bytes * 128 entries = 16,384 bytes / 512 bytes per sector = 32 whole sectors

In OpnSense, the GPT image contains 3 partition entries:

Code Select Expand

128 bytes * 3 entries = 384 bytes / 512 bytes bytes per sector = not even one whole sector

Windows is calculating the CRC over the entire 512-byte sector, when it should be calculating on only the first 384 bytes of the sector. This is documented in the UEFI specification 2.6 - 5.3.1:

5.3.1 GPT overview

There is a 32-bit CRC of the GPT Partition Entry Array that is stored in the GPT Header in Partition Entry Array CRC32 field. The size of the GPT Partition Entry Array is Size Of Partition Entry multiplied by Number Of Partition Entries. If the size of the GUID Partition Entry Array is not an even multiple of the logical block size, then any space left over in the last logical block is Reserved and not
covered by the Partition Entry Array CRC32 field.

This causes the updated GPT Header (and the backup GPT Header) to contain the wrong CRC-32 checksums.

Normally this wouldn't be a problem. You can still boot the image.

But FreeBSD disagrees

Rather than booting based on what's in the GPT Header and partition entries, FreeBSD actually checks the checksums. And it detects that the primary GPT Header contains an invalid checksum. It also checks the backup GPT Header, and detects that it contains an invalid checksum.

So it gives you the error message:

gptboot: primary GPT table checksum mismatch
gptboot: backup GPT table checksum mismatch
gptboot: unable to load GPT

And refuses to boot. It could if it just bothered to try. This is after it already did boot - the UEFI BIOS transferred control to gptboot. GptBoot just had to be a jerk and reset the machine rather than continuing.


So there's a chain of fail:

• USB image writers do not output a valid GPT disk
• FreeBSD doesn't use a number of partition entries that makes them a multiple of the sector size
• Windows outputs an invalid checksum (because it assumes everyone makes their partition entries a multiple of the sector size)
• FreeBSD's gptboot refuses to boot the disk as presented

Every image writer, writing a GPT disk without correcting the GPT backup sector, is doing it wrong. FreeBSD should simply say they have 4 partition entries rather than 3. Windows should calculate the CRC correctly. GptBoot should boot a perfectly valid disk (the UEFI said so).


So many fails. Lets see who stands up the point fingers.

What I did was use the free hex editor HxD to manually calculate the two CRC32 checksums. HxD lets you open a physical disk and hex edit it. It also has the ability to calculate CRC-32 of selected bytes.

My suggestion: have it be 4 GPT partitions rather than 3. That's something probably in OPNSense's control, solves the issues, and gets people up and running.