APU2 installation trouble - some solutions

[kraileth]

Hi everyone,

A while ago I decided to build a home router and write about the process on my blog. As a person who uses FreeBSD as a daily driver even on the desktop, I of course wanted to use *BSD on there. I decided to get an APU2c4 and just start with it.

I had e a little bit of experience with pfSense in the past and I've been following OPNsense since it was first released, trying out each release in a VM. There's a lot of drama on the net and I didn't find a somewhat objective comparison pfSense vs. OPNsense and for that reason I'll try and write one up after testing both for a while.

I like OPNsense's interface a lot better and totally love that it's easily possible to switch to LibreSSL. The collaboration with HardenedBSD is also great. However after I found that pfSense works well on the hardware (also the FreeBSD 11.0-based 2.4 beta version), I was surprised to find out that installing OPNsense failed on me! I tried to provide some information on this here, offering to work with the devs if anybody was interested. Obviously I came along at the wrong time when everybody is busy polishing up 17.7 or so. For that reason I continued on my own and found several ways to accomplish an installation. I want to document those here so other people who might be in need find the info (and perhaps to give the devs some more clues on what is happening there).

• The trouble begun when I booted of a memstick that I had dd'd the serial version of 17.1 on: The kernel booted but I was hit by the infamous "error 19" mountroot screen. On the net there are several suggestions on what loader variables to set - none of them (and no combination) helped. I only ever got the mountroot screen and using ? confirmed that no da0 was present. (failure)
• On this forum a user reports that using the SD card instead of trying to install to the mSATA from USB works. I could not try that out because I have no SD card at hand. (not tested)
• My next attempt was to use a device that can mount ISO images and provides a virtual CD drive. I set the loader variables to use the serial console (the cdrom image obviously wasn't meant for serial) and that worked: The system booted all the way up to the login! However the TTY was not configured for serial as it seems and as soon as the login prompt appears, that is no longer forwarded over the serial connection, making it impossible to login and do something... (failure)
• Since everything before the login works, I thought that I might just configure the interfaces and then SSH in. I was able to select which  interfaces should assume what role. But it does not seem possible to configure a LAN address that way! So that also proved to not really help. (failure)
• I had read that the previous version 16.1 used to work, so I decided to give that a try. Unfortunately you guys seem to be extremely tidy when it comes to your mirrors... I've searched for a 16.1 image for hours with no success. I must say that despite that being rather frustrating, I'm kind of impressed. Several search engines, ftp search engines, torrent search engines etc. later, I still had... nothing. Even looking for old cruft from the 90's usually wields a result if you're patient enough, but here I tried every single result - but every link was dead. (failure)
• Then I tried 16.7 again. Didn't work despite the various loader variables (and combinations thereof) that I tried - except for one time when it would suddenly work! I tried to figure out what I had done differently that time. It took me quite a while to realize that I had just forgotten the double quotes when setting the boot delay! And really, it proved to be repeatable: set kern.cam.boot_delay="10000" doesn't work, set kern.cam.boot_delay=10000 does! I was able to install OPNsense and then go through the regular update process with multiple reboots. That's a bit cumbersome and time-consuming, but it works! (success)
• After I had a working OPNsense on my APU I made a backup of the configuration with an IP set for the LAN interface and DHCP enabled for it. Then I formated a USB stick with UFS and put that xml file on there and booted the cdrom ISO again, trying to load that configuration using the configuration importer. It found the drive (da0) but did not find the configuration on there. I tried to rename it to simply config.xml which that didn't help, either. I took a look at the documentation for the importer but that did not mention how to name it. So I looked at the source. Fortunately it's shell code so I could figure out the path and name to use. Booting from the virtual CD and loading the configuration worked. I didn't even need SSH: Now logging in worked over the console as well! (success)
• The previous method was more to my liking but not perfect. I read that such problems could probably be solved by disabling XHCI in the BIOS or use an USB2 port instead of a USB3 one. Disabling it in the BIOS is not possible with the APU2 as far as I can tell and it only has 2 USB3 ports. But I also suspected that it would come down to USB3/XHCI being the cause for error 19. I played with the idea of building a custom kernel without XHCI just to test it, but I'm not familiar with how the OPNsense project builds the ISOs. However that proved not to be necessary: The APU2 board has internal USB2, too! I attached a cable to it that provides a USB port, plugged in a memstick with OPNsense 17.1 serial - and it just worked like there had never been any problem! (success)

So this issue is definitely directly connected to USB3/XHCI. The only thing that's weird is that pfSense 2.4 (which is FreeBSD 11, too) and vanilla FreeBSD 11.0 seem to be able to mount the UFS filesystem using a memstick. Any idea what could be done to fix the images for successfully booting off of USB3?

Again I'd like to offer help in trying out any suggestions if I can. I'm not afraid of building custom stuff or anything. So if anybody has any ideas, please tell me.

[Rainmaker]

I registered here just to say thank you for this post. I too have an APU2C4, with an mSATA SSD on board. I have run IPFire via SD Card but currently have pfSense, which has been running on here for about a year or so. I don't care for politics and OPNSense seems to have a very nice UI, so having been playing with it in VMs on and off for a few months I wanted to give it a whirl. I hit the same issues you did, and found the boot time delay parameter on a BSD mailing list. It too included the double quotes (and so didn't work) but now I've found your post I can try again.

I did also try to use the nano / embedded image supplied by OPNSense, which promises to expand to fill the card on first boot and be capable of running as a fully-fledged install of its own right. It simply requires copying (I used dd) onto the card and booting. It booted perfectly, so I thought I had my answer for the APU. Unfortunately after taking time to configure it via the console and then rebooting it after an update, it bricked itself. Not a great first impression for a 'solid' BSD based distro!

I find it unfortunate that the staff/admin/developers haven't had time to respond to your concerns in all this time. As you say this bug is only present in OPNSense, not pfSense or upstream *BSD. It's especially sad as the APU is such a fantastic little unit for the purpose. A decent enough AES-NI capable CPU with a tiny TDP, onboard Intel NICs, great expansion and a small footprint. Aside from perhaps a higher clock per core, what more could you ask for? It's an underrated platform. Again, thank you for your helpful post.

[Rainmaker]

I spoke too soon. I tried to boot the latest release using the set boot delay parameter (yes, omitting the quote marks) and it still hit the same old error. 

Code: [Select]

uhub2: 4 ports with 4 removable, self powered
run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Error 5, Retries exhausted
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <KINGSTON SMS200S360G 603ABBF0> ATA8-ACS SATA 3.x device
ada0: Serial Number 50026B7267011FA8
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 57241MB (117231408 512 byte sectors)
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC" frequency 998148315 Hz quality 1000
Trying to mount root from ufs:/dev/ufs/OPNsense_Install [ro,noatime]...
mountroot: waiting for device /dev/ufs/OPNsense_Install...
Mounting from ufs:/dev/ufs/OPNsense_Install failed with error 19.

Loader variables:
  vfs.root.mountfrom=ufs:/dev/ufs/OPNsense_Install
  vfs.root.mountfrom.options=ro,noatime

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:tank
        cd9660:/dev/cd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>

As you say, there are just no copies of earlier versions on the net so I can't even try the 16.7 version you said works. The serial ISO doesn't work on USB, the SD card version craps itself and fails to boot... It looks like OPNSense just doesn't want to work for me. Hey ho, back to pfSense I go if I can't get this sorted. 

[andreamenchetti]

Hi, I'm sure it's not off topic... APU3 is like APU2 optimized for LTE sim accesses.
So... my APU3A4 (bios 4.0.11), mSATA 16GB:

Opnsense 17.7 (based on freebsd 11), no way to run by usb memstick, sd card, msata (ahci timeout).
Opnsense 17.7.5 (based on freebsd 11), no way to run by usb memstick, sd card, msata (ahci timeout),
update to 17.7.7 was bad too (ahci timeout in 5 minutes from boot time).
The tunable hint.ahci.0.msi=0 is a palliative... 15 minutes instead 5 minutes...

Opnsense 17.1.4 (based on freebsd 10), ok (with plugin os-boot-delay installed), update to 17.1.11_1 was ok too.

Try it yourself and enjoy.
Sorry for my English... 

AM

[dreamerns]

Hi there,

I registered to forum after spending half day yesterday trying to boot OPNSense on 20.1 on apu4d4. Same problem still persists and pfSense 2.4.5 works fine. Too bad there is still no solution after 3-4 years :-(

[aesth]

That's odd I had no trouble installing on an APU2. Maybe try with updated firmware or a different drive? https://pcengines.github.io/#mr-32

[franco]

Over the years a number of people have pointed out that BIOS updates are the best way to solve these kind of issues.


Cheers,
Franco