[SOLVED] OPNsense V17.7.5 installation on APU2

[mossi2000]

Hi,

I just want to share my experience on installing OPNsense on a APU2C4.
A while ago I bought the APU2 and installed pfSense 2.3.4 (memstick-serial-amd64).
Then I had no time to finalize the setup for production.
Since I will receive a Fiber-to-the-Home connection before christmas, I wanted to finish it and wanted to give  OPNsense a try, since the GUI and what I've read about it pleased me.

HW: APU2 with 4GB RAM, 16 GB m2-SSD. Latest BIOS 4.6. Serial connection to an USR TCP 232 302 to Ethernet via a Virtual COM on Windows.

Yesterday evening I tried to install the serial-amd64 version via USB stick. Tried different USB sticks, but when logged in as installer, copying of the system always hung at some cpdup command 45-64%. Then I read about the problem with USB3 ports and AHCI.
Using RUFUS and/or Win32Image is quite straight forward and TinyCore for flashing or MemStick preparation is easy.
If I would have had a smaller  USB 3 stick free (the ones I have are all 64 GB),I would have tried USB3 memory stick, but I think It would have helped much except for preparation speed..

Ok, retried this morning also with pfSense, same thing.
I searched for a cable to connect the internal USB2 ports to Type A conectors, but couldn't find one... Shit.
In the afternoon I tried to install the nano version via a TinyCore stick. Yep!! That worked.
I could open the GUI and do some settings.

But later on I see on the serial console I AHCI errors occurring:

(Just one example)

(aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich0:0:0:0): CAM status: Command timeout
(aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted
ahcich0: Timeout on slot 30 port 0
ahcich0: is 00000002 cs 00000000 ss 00000000 rs 40000000 tfd 50 serr 00000000 cm          d 00407e17

and I no longer can login on the serial console or open the GUI.

How can I get rid of these??

[Alphabet Soup]

Not much help, but I've had poor luck with FreeBSD, the external APU2C4 USB3 ports, and USB3 memory sticks unless I set that kern.cam.boot_delay variable to something non-zero.  I believe OPNsense sets this automatically now, so if that hasn't solved it for you maybe try:
a)  find/borrow a USB2 memory stick,
b)  try again to find that USB2 internal port adapter cable,
c)  transplant the m.2 drive to another host computer and write the image there, then move the m.2 drive back

[franco]

Install the os-boot-delay plugin prior to first reboot.

However, some devices seem to not reboot at all, something specific to APU but only happening for a limited number of users. Others can use their APU just fine.


Cheers,
Franco

[mossi2000]

I finally got it solved:
The root cause of the problem is that in FreeBSD 11.0 the AHCI driver has been "improved" in such a way, that it causes command timeouts and hangs on the APU2 (and as I read on several more devices).
It looks as if is dependent on the connected device (SSD..)
(I found a post where a guy got the AHCI sources from FreeBSD 10 and merged the AHCI driver back to 11, rebuilt the kernel to make it work)

The solution is to add the lines:
hint.ahci.0.msi="0"
hint.ahci.1.msi="0"

into /boot/device.hints
(Thanks to an italian guy whose post I found, and that I could understand it (my italian is not too bad))

Since I dont't know a way of how to access this file BEFORE installation, it' a bit tricky.


The installation failures are NOT related to USB2 / USB3. (I did the last trials using a USB3 stick, but just for the reason that writing to a USB 3 stick for preparing the bootable memstick  is 3 to 10 times faster...


Here's how I managed to install:
Using Rufus I created a boot stick with apu2-tinycore6.4  and put OPNsense-17.7.5-OpenSSL-nano-amd64.img.bz2 on it too.

Then I booted Tinycore.
I removed the partitions on the mSata SSD using fdisk. (I had a 24 MB partition left over from the last trial)

Then I wrote the OPNsense image to the SSD (Tinycore does not have the AHCI problem, so the write is safe)
bunzip2 -dc OPNsense-17.7.5-OpenSSL-nano-amd64.img.bz2 | dd of=/dev/sda bs=1M
Takes a while...

(If there's a way to mount the freshly written partition, one could edit the necessary file safely from here)

After that remove the stick and reboot.
OPNsense comes up and I logged into the shell ( on the serial console and edited the boot/device.hints file.
Using the GUI I added the os-bootdelay plugin (for safety) and then rebooted the system.

After the reboot I went to Update in the GUI and was able to update to OPNSense 17.7.7_1.
The trial before, without the hints produiced AHCI timeouts when downloading / installing the updates.

The serial version CANNOT be booted like this, because it can't install to where it's running from and besides
the install would have to have the hints already active for copying ... diabolic circle...

So I will live with the nanoBSD image and I've set up remote syslogging to my QNAP NAS, where there's lots of space on the HDDs.

Hope this helps somebody saving time....
Axel

[bigops]

I had also run into the same issue with APu2/3 installs.  But I have noticed that if i use a standard USB 2 device then the timeouts does not happen and the install goes through. 

One more thing to note is that you mentioned that you are looking at a fiber connection.  The APU is not able to handle 1 Gbps fiber.  The max I have seen it handle is around 650 M for download and 550 for upload.

[franco]

Hi Axel,

The solution is to add the lines:
hint.ahci.0.msi="0"
hint.ahci.1.msi="0"

Interesting, I'll try to get that into the next images (18.1-RC).


Cheers,
Franco

[mossi2000]

I had also run into the same issue with APu2/3 installs.  But I have noticed that if i use a standard USB 2 device then the timeouts does not happen and the install goes through. 

One more thing to note is that you mentioned that you are looking at a fiber connection.  The APU is not able to handle 1 Gbps fiber.  The max I have seen it handle is around 650 M for download and 550 for upload.

I'll ask my colleague from IT if he has a USB / mSATA adapter, so I can backup my SSD. When I find the adapter for the internal USB2 I will retry the installation.
Currently I cannot imagine how a pure USB2 source can avoid AHCI timeouts / hangs when copying.

Fiber: 1Gb/s won't be offered... I'll probably end up with 100Mb/ s or 200Mb/s which is really fast coming from a 1.5Mb/s DSL line. I'm looking for the one ISP that offers a modem and not a router :-)

[weust]

Could ask the ISO to put the router in bridge mode. You'll have a modem then.
I do that with the cable routers I've had so far.

[mossi2000]

Hi Franco,

The solution is to add the lines:
hint.ahci.0.msi="0"
hint.ahci.1.msi="0"

Interesting, I'll try to get that into the next images (18.1-RC).

my APU2C4 was running for about a week now, without traffic, I played around with some configurations - no probs so far.
Yesterday I upgraded to 17.7.8 via the GUI: Worked fine until - except that from some point on I neither had access to the GUI nor via serial console. I let it run for about an hour, no change - only that all 3 LEDs on the front of the board were ON. Just hang. I already thought that I would have to do the whole installation again, but fortunately the system came up and showed me version 17.7.8.

I looked a bit around in the GUI and wanted to make some settings... then the GUI hang and I saw the AHCI errors on the console. SHIT!! Switched off te box a little frustrated...
I rebooted half an hour ago and checked the boot/devices.hints.... and wonder, wonder my previous entries had disappeared after the update.....

Just re-added them and made some config changes..... nothing happened - no AHCI errors.

So maybe it could really be a good idea to include these...

Axel

[mossi2000]

I'll ask my colleague from IT if he has a USB / mSATA adapter, so I can backup my SSD. When I find the adapter for the internal USB2 I will retry the installation.

I borrowed such an adapter - unfortunately it is a m2.SATA adapter .... but yesterday I ordered something on amazon and just added the cable and a USB3 to MSATA adapter. So next week I'll be able to backup / restore the system and also try to install directly to the MSATA SSD.... both using USB2 and USB3.
I'm confident to find the root cause ..

[mossi2000]

I'm confident to find the root cause ..

Yeah!! Got it.

Proved way to install opnSense on APU2:
- You may use an USB 3.0 stick - the problem IS NOT USB 3.0
- Create memstick using e.g. Rufus (Windows) using -serial image
- Boot from memstick (press F10 in BIOS to select USB stick)
- No configuration of interfaces needed yet.
- DO NOT login as installer NOW - but as root
- select [8] shell
- cd /boot
- mount -o rw /
- vi devices.hints
add the two lines
hint.ahci.0.msi="0"
hint.ahci.1.msi="0"
at the end of the file
- exit vi with ESC :wq! to allow writing the readonly file
- exit  (from shell)
- select [6] to reboot

After the second boot login as installer and  the installation will go through without hang!

ATTENTION: If you perform the upgrade to 17.7.8 via the GUI, the hint settings will disappear again.
You will have to re-edit the file.

Before that I tried to install via USB 2.0 with both a USB 2.0 stick and a USB 3.0 stick  WITHOUT the hint written to the installer stick : The installation always hang at cpdup .... somewhere between 39% and 65%.
The otherwise unchanged USB 3.0 stick installed where it hang before on the USB2.0 port without the hint setting.

This is DEFINITIVELY only an AHCI problem!!


One thing I don't understand: Why  does the install memory stick need to boot the complete opnSense to start the installation? Booting the kernel, mounting the ufs partition from the stick and perhaps loading some basic things would IMHO be enough to run the installer.

[franco]

Use /boot/loader.conf.local to make these modifications persist through reboots and firmware updates.

This is DEFINITIVELY only an AHCI problem!!

I haven't added this to the installation yet, but this is definitely good news.

One thing I don't understand: Why  does the install memory stick need to boot the complete opnSense to start the installation? Booting the kernel, mounting the ufs partition from the stick and perhaps loading some basic things would IMHO be enough to run the installer.

Some find it annoying, but it has at least the following benefits:

1. An automated live system boot that even allows SSH access to the installer. Also useful for testing new major releases as it keeps the hard disk unmodified while still having access to the configuration. From there, you can even install the import to render your changes to the hard disk, which is less risky than an online update.

2. If the system boots and comes alive, the image/USB is not damaged and installation will yield less problems in general.

3. People build read only sticks with custom configurations to prevent data loss in some environments. You can take any installer image and put /conf/config.xml there to get a read-only, no write boot device with your preconfigured settings.


Cheers,
Franco

[franco]

Here is the commit: https://github.com/opnsense/core/commit/8e85edd41

[tillsense]

@mossi2000

can you confirm that repeatedly? Would be really good news. I've been targeting xHCI for a while.
> https://forum.opnsense.org/index.php?topic=6024.0

cheers till

[mossi2000]

Use /boot/loader.conf.local to make these modifications persist through reboots and firmware updates.

Yeah, thanks, 'll do this.

Some find it annoying, but it has at least the following benefits:

After the post I realized, that after the kernel is up, it's not so much more until the system is up.
And of course having aworking system from  boot from a stick before writing to the SSD is really cool.

People build read only sticks with custom configurations to prevent data loss in some environments. You can take any installer image and put /conf/config.xml there to get a read-only, no write boot device with your preconfigured settings.

Good idea, but from what I saw yesterday night, the config,xml is missing the added plugins. Without adding the plugins before rebooting after the configuration restore, the system is not complete...
Or did I miss something that should be obvious?