PSA: APU 2 -- BIOS/Firmware update required -- mountroot, error 19

[ciaduck]

I wanted to put this online somewhere, because my search results were almost entirely unhelpful. I hope this save someone some pain.

If you have a PC Engines APU 2, and you've had it for a while, you MUST update the firmware/BIOS on it to install OPNsense 22.1.

=== Some History (because I want to tell the story) ===
Long ago, in 2015, I tried OPNsense when it first forked, it wasn't for me. I had issues with my multiple VPN setup. I don't know if it was me or the software, but we weren't getting along. I decided to save my self some hours tinkering every week and go back to pfsense.

In December, I was unable to upgrade my existing pfSense from 2.4.5, it seems I encountered a pretty bad (yet common) bug, and the entire package system was corrupted with the wrong versions of things. I tried upgrading it "manually" via USB stick, and it left me with a non-functioning system. Thankfully a downgrade worked, and I had a router again. The corruption left me with a system I couldn't update, and would need to again manually re-install.

I finally had time again, and I've been wanting to try OPNsense again (thanks reddit!), what better time than now? I'm going to have to do a fresh install anyway.

=== Install Problems ===
My hardware is a PC Engines APU 2C4 I got quite some time ago. I think the firmware on it was dated 2016. I may have upgraded it once since I got it.

I downloaded the latest OPNsense. Used belenaEtcher to create a serial USB installer. Took screenshots and a manual backup of my current pfsense config.xml. And planned my migration over a day. I slept on it and felt good to go forward.

I popped the USB into my APU 2C4. Fired up TeraTerm using a spare windows craptop and my trusty USB RS232-Serial cable, and went to town. Powered off the unit, and booted the USB.

This is where the fun starts! After a few minutes of trying to boot and getting a bunch of "Waiting for CAM" messages, it gives up and says "Mounting from ufs:/dev/sdb failed with error 19". And dumps me out to a "mountroot>" command prompt.

It turns out that older versions of the APU firmware don't like using GPT, and the USB 3.0 drivers exacerbate the issue.

=== The Wrong Solution ===
At this point there was a lot of "advice" online to boot setting kern.cam.boot_delay="10000"
This is bad advice! It will make you wait forever for your device to figure out why it can't negotiate a USB 3.0 connection, and it doesn't solve the underlying cause/issue.

=== Problem Solving (or how I stubbornly used Windows) ===
After a bunch of research and looking into BSD and other help forums, I learned about the GPT issue. I finally figured out I really should upgrade the APU firmware, and that would (hopefully) solve the problem.

This is a very good resource on how to use flashrom via software you probably already have installed on your APU!
https://teklager.se/en/knowledge-base/apu-bios-upgrade/

The easiest option would have been to boot back into pfsense, and run flashrom via an SSH session. This didn't work for me because of the corrupt package versioning. I couldn't get the correct versions of packages, and this left me unable to run flashrom. I kept getting 'Shared object "libarchive.so.7" not found, required by "pkg"'. And after using the pkg-static to install flashrom, running it I would get a 'Undefined symbol "fstat@FBSD_1.5"'. These are due to the corrupt packages and versions in pfsense.

Another trip on the merry-go-round! I tried to create a USB 2.0 stick with MBR, TinyCore, and the APU firmware to flash. I couldn't figure out how to make a proper TinyCore USB using belenaEtcher. I'll spare you the details, I just used the PC Engines provided imager.

=== (my) SOLUTION ===
Ultimately, I created a USB 2.0 stick with MBR using Windows. I had to "undo" the GPT partition from a prior OPNsense attempt.
There was a bit of trial an error, the PC Engines imager doesn't partition the drive. I had to make sure it was MBR and FAT32 before starting.
https://docs.microsoft.com/en-us/windows-server/storage/disk-management/change-a-gpt-disk-into-an-mbr-disk
Then ran the PC Engines installer for TinyCore
https://pcengines.ch/howto.htm#TinyCoreLinux
Then loaded the latest APU firmware image onto it. (apu2 v4.15.0.3)
https://pcengines.github.io/

I booted, and ran the flashrom command with no issues.
flashrom -w /media/TINYCORE/apu2_v4.15.0.3.rom -p internal

After that, the etcher created USB with GPT for OPNsense 22.1 worked! No fancy kernel boot parameters required. It didn't take forever to boot either.

=== Conclusion ===
THOU SHALT UPDATE THINE FIRMWARE!

PC Engines is really awesome for continuing to support their systems after all these years. The fact that I could get a firmware that was released just 12 days ago, is mindblowing! I'm really happy with this hardware and the continued support. I'm also happy that it solved the problems. I will definitely be paying attention to firmware updates from now on.