Messages

Part 4: Making things useful

Build cache

We've just installed ccache. Now we need to create a directory for it to use:

Code Select Expand

# mkdir -p /var/cache/ccache/poudriere

Ccache defaults to not use more than 5 GB of space for its cache. I doubt that it makes sense to increase it as repositories for OPNsense probably shouldn't be so big in the first place. On my FreeBSD systems I do increase it, though. If you are low on disk space, you can of course also lower the limit. I'm going to write out the default here for reference purposes:

Code Select Expand

# echo max_size = 5.0G > /var/cache/ccache/poudriere/ccache.conf

To make Poudriere use ccache, we have to edit its configuration again:

Code Select Expand

# vi /usr/local/etc/poudriere.conf

Find the "CCACHE_DIR=" line, comment it in and change it to:

Code Select Expand

CCACHE_DIR=/var/cache/ccache/poudriere

Now we're going to add Poudriere to the list of packages to build so that we will get updates in the future:

Code Select Expand

# echo ports-mgmt/poudriere >> /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-pkglist

Additional packages

Also add the programs that you actually wanted to build. To do so, simply put them into the package list file one per line and specify the programs as "category/name". My example here is the jail manager / virtualization framework cbsd.

You probably don't know the category if you're not working with ports regularly. You can use the "whereis" command to find out if you know the program's name:

Code Select Expand

# whereis cbsd
cbsd: /usr/ports/sysutils/cbsd


So it would be "sysutils/cbsd" that needs to be added to the package list file:

Code Select Expand

# echo sysutils/cbsd >> /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-pkglist

But what if you are looking for something but don't even know the exact name of the program? The easiest thing is probably to use this website:

https://www.freshports.org

To the right you will find a "search" box that you can use. Explore the site a little, it is very useful if you are working with FreeBSD ports.
When you are happy with your package list, simply invoke the bulk build command again to make Poudriere build the missing packages:

Code Select Expand

# poudriere bulk -j opnsense211 -p opnports -z customsense -f /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-pkglist

Package repositories

Last time we used "pkg add" to install the ccache package manually. While there's nothing really wrong with that, this method is less helpful if you want to keep your system up to date. This is where repositories come into play. A repository is a set of package files with additional metadata to e.g. make it searchable. Don't worry about the details - Poudriere takes care of all that for you. You only need to make the package manager aware of your additional repository. We'll do that now. Let's take a look at a configuration example:

Code Select Expand

# cat /usr/local/etc/pkg/repos/FreeBSD.conf
FreeBSD: { enabled: no }

Pretty simple, eh? This is an override config that disables the repository called "FreeBSD" which comes with the operating system and is otherwise enabled by default. Have a look at the file OPNsense.conf if you are curious.

Now create a new file:

Code Select Expand

# vi /usr/local/etc/pkg/repos/custom.conf

And paste the following into it:

Code Select Expand

Custom: {
  url      : file:///usr/local/poudriere/data/packages/opnsense211-opnports-customsense,
  priority : 20,
  enabled  : yes
}

This will configure a new repository called "Custom" for pkg to use. I'm setting a lower priority than the OPNsense repository has by default. Why? Because we want the official packages to take precedence to not break anything. If you plan on building your packages often, the cleanest solution would be add all required packages to your list and to disable the OPNsense repository completely.

When using multiple repositories you might run into the situation where pkg will not update to newer packages in your repository even if you give it a higher priority. This is due to the (default) option called "CONSERVATIVE_UPGRADE": pkg remembers which repository a package was originally installed from and will generally prefer that repository over others. Edit /usr/local/etc/pkg.conf and switch that option off if you want to go that route.

Now make the package manager read the repository information again:

Code Select Expand

# pkg update

After the previous step, I can simply install my jail manager:

Code Select Expand

# pkg install cbsd

There we go! It will ask me to confirm the actions that it is about to take and a couple of seconds later the program that is not packaged by OPNsense is installed cleanly on the system.

Build options

FreeBSD's ports framework allows for customizing build-time options for many packages. You can issue the following command to configure all of the ports in your list (AND their dependencies!):

Code Select Expand

# poudriere options -j opnsense211 -p opnports -z customsense -f /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-pkglist
===> Setting user-specified options for cbsd-13.0.0 and dependencies

Doing so is not much fun. HardenedBSD added hardening options to each and every port, even those that don't have any options on vanilla FreeBSD - and OPNsense inherited this. So for all the ports potentially involved this will display a configuration menu for you. Most of the time you will probably just hit Enter but you might want to actually customize a couple of ports. Seriously, this is pretty tedious. Cbsd is not an extreme example at all, but look how much configuration dialogs it caused:

Code Select Expand

# ls /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-options/| wc -l
     275

A better way might be to configure only the ports that you are interested in customizing. While cbsd doesn't in fact currenly have any interesting build-time options to tweak, we're going to do it anyway here for demonstration purposes. Here's how to do it:

Code Select Expand

# make -C /usr/ports/sysutils/cbsd config

This will bring up the configuration menu and if you confirm the selection, the options get saved. By default, the ports tree will save its options in /var/db/ports. There should be a "sysutils_cbsd" directory inside now. You can simply copy those over for use by Poudriere:

Code Select Expand

# mkdir -p /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-options
# cp -r /var/db/ports/* /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-options/

That way you can configure only the ports that you actually care about and not potentially hundreds of others as well.

Updating

Staying safe means regularly upgrading the system. How do you do it? The first step is updating the ports tree. Do this by issuing:

Code Select Expand

# cd /usr/ports && git pull && cd -

Step two is to simply start another bulk build the way you've done at least twice now. When Poudriere has finished you will have a repository with fresh packages in it. You can either update via the command line by issuing:

Code Select Expand

# pkg upgrade

Or you simply use the GUI as you're used to. It will show the updates from all configured repositories, so you don't even have to remember how to use pkg.

Have fun with your new even more useful OPNsense box(es) and enjoy a great Open Source project!

What now?

I still have a couple of things on my mind that I could put into another post - mostly ideas to make this whole procedure easier for users. But with these 4 parts people should be able to roll their own packages. So before putting more time and effort into this, I'll wait to see if this tutorial is actually useful to anybody.

Feel free to ask if you have questions about this. Of course comments or suggestions are welcome as well.

Part 3: Basic package building

Cloning Ports and installing Poudriere

OPNsense does not package Poudriere, so we have to build this one from ports. Luckily it's a self-contained program that doesn't require additional dependencies to be built for it. The first thing we have to do is getting the ports tree onto the machine. OPNsense provides a convenient script to do exactly that for you. Simply run the following command as root:

Code Select Expand

# opnsense-code ports

This will install Git and use it to download ("clone") the ports tree which will then be available in /usr/ports. Ports are organized in categories. Poudriere is in the ports-mgmt category which means that the actual port lives in /usr/ports/ports-mgmt/poudriere. Let's build and install the port now:

Code Select Expand

# make -C /usr/ports/ports-mgmt/poudriere install clean

This will fetch the source code for Poudriere, extract and configure it then build the program. When it's done it will install the package and clean up after itself.

Getting the distfiles in place

Before the package builder can be used, a build jail needs to be created. To set one up, Poudriere needs an archive of all the files that make up the base system. It also needs the source code for the OS (some packages like e.g. lsof require the source to be present as they are closely tied to the exact OS version). Unfortunately Poudriere makes some assumptions which are right only for vanilla FreeBSD. It does not support HardenedBSD's FTP structure for example. We are going to trick it by creating a local mirror for the relevant files. To do so we first create a directory structure that fits Poudriere's expectations:

Code Select Expand

# mkdir -p /usr/opnsense-dist/pub/FreeBSD/releases/amd64/amd64/21.1-RELEASE

Unfortunately OPNsense does not offer the complete distsets which we need. So we need to get the system source code next:

Code Select Expand

# opnsense-code src

This will clone the OPNsense source code repository to /usr/src. We want the code at the time of the 21.1 release so we make git switch to the corresponding tag:

Code Select Expand

# cd /usr/src && git checkout 21.1 && cd -

Then we build the required source tarball:

Code Select Expand

# tar -C / -cvJf /usr/opnsense-dist/pub/FreeBSD/releases/amd64/amd64/21.1-RELEASE/src.txz --exclude-vcs usr/src

Now we can either download the pre-built release tarball for the base system like this:

Code Select Expand

# fetch https://pkg.opnsense.org/FreeBSD:12:amd64/21.1/sets/base-21.1-amd64.txz -o /usr/opnsense-dist/pub/FreeBSD/releases/amd64/amd64/21.1-RELEASE/base.txz

Alternatively (no fun on a weak machine) we can build it from source ourselves. To do so we check out the OPNsense tools repository:

Code Select Expand

# opnsense-code tools

We need to define an environment variable because the build system will otherwise complain that we're trying to build "head". This is to prevent people from accidentally building development versions that will contain lightly tested code. In our case we're in "detached head" mode because we asked git to return to a tagged state (for the actual release version).

The "-j 5" parameter means that the building may use up to 5 threads. What is right for you depends on the number of threads that the CPU you are using offers. A general recommendation is to use "n + 1" for building if you want minimum compile time. My machine has 4 threads and does not do heavy package filtering over night, so I go with 5. You probably don't want to do this on your production firewall that needs to continue fulfilling its main purpose:

Code Select Expand

# env SRCBRANCH=HEAD make -C /usr/tools -j 5 base

When it's done (it *will* take a while!), copy off the resulting tarball into the directory holding our distsets:

Code Select Expand

# cp /usr/obj/usr/src/amd64.amd64/release/base.txz /usr/opnsense-dist/pub/FreeBSD/releases/amd64/amd64/21.1-RELEASE

Ok, we have both distfiles required by Poudriere. But as it will freak out if there's no accompanying MANIFEST file, let's create that, too:

Code Select Expand

# cd /usr/opnsense-dist/pub/FreeBSD/releases/amd64/amd64/21.1-RELEASE && /usr/src/release/scripts/make-manifest.sh base.txz src.txz > MANIFEST

Depending on the CPU of your machine this can take a while. The script will calculate the the SHA256 checksums of the distfiles and count the number of files that they contain. Poudriere uses the manifest to check whether the distfiles are good or if they probably were damaged.

Poudriere configuration

You need to know if your system uses ZFS and if it does, what the name of the ZFS pool is. Execute this command:

Code Select Expand

# zpool list

You will get output like this:

Code Select Expand

NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot   472G  2.65G   469G        -         -     0%     0%  1.00x  ONLINE  -

My system has a pool named "zroot" (FreeBSD's default name). If your output is missing the second line, you're on UFS.

The next step is to configure Poudriere. The program comes with a sample configuration that we can use as a template. To do so we copy the file and then edit it:

Code Select Expand

# cp /usr/local/etc/poudriere.conf.sample /usr/local/etc/poudriere.conf
# vi /usr/local/etc/poudriere.conf

If you are using ZFS, find the line that defines ZPOOL, remove the comment symbol ('#') and change the name if necessary. On a UFS system comment in the line "NO_ZFS=yes" instead.

Then look for the line "FREEBSD_HOST=_PROTO_://_CHANGE_THIS_". Change it to:

Code Select Expand

FREEBSD_HOST=file:///usr/opnsense-dist

Finally comment in the line "BUILD_AS_NON_ROOT=no" and save the file.

OPNsense 21.1 is based on HardenedBSD 12.1. Version 12.1 of FreeBSD is already EOL ("end of life") and no longer supported - and so from the perspective of the ports tree we're running on an obsolete release. We need to set a variable to make the build system stop complain and just do its thing:

Code Select Expand

# echo ALLOW_UNSUPPORTED_SYSTEM=yes > /usr/local/etc/poudriere.d/make.conf

OPNsense offers two different variants of the firmware: Using OpenSSL (default) or LibreSSL. After installation I immediately change to LibreSSL as it is the better one of the two IMO. Check what you are using! On the dashboard look in the "System Information" box under "Versions". If you are using LibreSSL, you need to tell the build system to adapt accordingly. Issue the following command (do NOT do this if you are using the standard OpenSSL!):

Code Select Expand

# echo DEFAULT_VERSIONS+= ssl=libressl >> /usr/local/etc/poudriere.d/make.conf

That's it for the configuration.

Build jail creation and initial ports build

We're good to prepare the jail now. Let's tell Poudriere to create one, to call it "opnsense211" (we cannot use periods in the name, hence 211 and not 21.1) and to use version "21.1-RELEASE" for which we've provided the distfiles:

Code Select Expand

# poudriere jail -c -j opnsense211 -v 21.1-RELEASE
[00:00:00] Creating opnsense211 fs at /usr/local/poudriere/jails/opnsense211... done
[00:00:00] Fetching MANIFEST for FreeBSD 21.1-RELEASE amd64
/usr/local/poudriere/jails/opnsense211/fromftp         222  B  307 kBps    00s
[00:00:00] Fetching base for FreeBSD 21.1-RELEASE amd64
/usr/local/poudriere/jails/opnsense211/fromftp         172 MB   79 MBps    02s
[00:00:09] Extracting base... done
[00:01:32] Fetching src for FreeBSD 21.1-RELEASE amd64
/usr/local/poudriere/jails/opnsense211/fromftp         157 MB   63 MBps    03s
[00:01:40] Extracting src... done
[00:03:45] Cleaning up... done
[00:03:45] Recording filesystem state for clean... done
[00:03:45] Upgrading using ftp
/etc/resolv.conf -> /usr/local/poudriere/jails/opnsense211/etc/resolv.conf
sed: /usr/local/poudriere/jails/opnsense211/usr/sbin/freebsd-update: No such file or directory
12.1--HBSD
[00:03:46] Recording filesystem state for clean... done
[00:03:46] Jail opnsense211 12.1--HBSD amd64 is ready to be used

We also want to hook up the ports tree with Poudriere and name it just "opnports":

Code Select Expand

# poudriere ports -c -m null -M /usr/ports -p opnports
[00:00:00] Imported ports tree "opnports" from /usr/ports

The one missing step is to define a list of ports to be built. Initially we're only building one port that we'll make use of in later builds: Ccache. It is a program that allows the caching of compilation artifacts for later re-use. If you are building packages regularly (and you should for the sake of security!) this will save you some time and resources in future builds.

Code Select Expand

# echo devel/ccache > /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-pkglist

Ok, all done! We are finally ready to begin the actual package building. We'll use the only jail that we have ("opnsense211") with the ports tree that we registered with Poudriere ("opnports") and call the package set "customsense":

Code Select Expand

# poudriere bulk -j opnsense211 -p opnports -z customsense -f /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-pkglist
[00:00:00] Creating the reference jail... done
[00:00:01] Mounting system devices for opnsense211-opnports-customsense
[00:00:01] Mounting ports/packages/distfiles
[...]
[00:00:19] Starting/Cloning builders
[00:00:19] Hit CTRL+t at any time to see build progress and stats
[00:00:20] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.15.10_3
[00:09:55] [01] [00:09:35] Finished ports-mgmt/pkg | pkg-1.15.10_3: Success
[00:09:56] [01] [00:00:00] Building devel/ccache | ccache-3.7.1_1
[00:11:12] [01] [00:01:16] Finished devel/ccache | ccache-3.7.1_1: Success
[00:11:14] Stopping 2 builders
[00:11:14] Creating pkg repository
[...]

We've created a pkglist with only one item, but Poudriere needs the pkg package manager, which is always built first. So when it's finished, we have two newly built packages on our system:

Code Select Expand

# ls /usr/local/poudriere/data/packages/opnsense211-opnports-customsense/All
ccache-3.7.1_1.txz      pkg-1.15.10_3.txz

It's actually a valid repository already, but we'll ignore that for a minute. Let's install the ccache package directly (i.e. without using the repository). This is done with the "add" subcommand of pkg:

Code Select Expand

# pkg add /usr/local/poudriere/data/packages/opnsense211-opnports-customsense/All/ccache-3.7.1_1.txz
Installing ccache-3.7.1_1...
Extracting ccache-3.7.1_1: 100%
[...]

And that's really it: We've just built our first own native package for OPNsense and installed it manually. It's a nice start! The next part will show how to make more out of it.

[#]

Part 2: Theory & Preparation

Jails

Part 1 spoke of clean "build environments". Poudriere leverages FreeBSD's jails to achieve that goal. But what is a jail?

Think of a "container" if you have no idea at all what this means. It's basically an isolated instance of the operating system programs (the so-called "userland"). Sounds like a virtual machine? Yes, it's closely related to that. Jails are in fact a light-weight means of virtualization. In contrast to full virtualization however it does not involve virtualized hardware. This "additional operating system instance" runs on the same machine as the host system and in fact uses the same kernel.

The latter is the limitation responsible for the fact that you cannot run a foreign operating system in a jail (yes, there are "Linux jails" but they make use of FreeBSD's abilities to translate Linux system calls to native ones, so technically it's still all FreeBSD even if it behaves like Linux).

Processes running in the jail are locked in (hence the name). In general they are not able to change anything related to the host operating system or other jails running besides them. There are ways to escape a jail, but that usually involves deliberate risky configuration of the jail or assistance from the host. The main take-away here is: Don't consider jails a perfectly secure feature under all circumstances but as a nice way to improve security in general.

However in our case we're using the jails system just for providing a second operating system instance where Poudriere controls which packages are installed. This way it can guarantee clean builds without harming the machines main tasks by messing with the packages installed on the host system.

Unix shell prompts and privilege

Unix by default follows a very simple rights model: You are either the almighty "root" user (think "Administrator") or one of the many nobodies on the system. There are multiple ways of elevating your privileges or "becoming root". In this tutorial we are going to use the "sudo" command for that (short for "do as superuser").

The shell prompt shows if you are working as the privileged root user. If so, it will end in the # character. For unprivileged users the symbol is normally a % sign (for derivatives of the c-shell) or a $ (in Bourne shell and derivatives).

In this tutorial all command lines that need to be executed begin with either a percent sign or the pound symbol. Do not include these if you type / copy the command lines. They just show you whether you need to execute the command line as root or as a regular user.

A word on editing files

In this tutorial I'm going to use the vi editor. If you are a Linux user you are likely to know it - or at least know what that means. Be warned though: This is FreeBSD. The "vi" command is NOT an alias for vim. It's nvi, a re-implementation of Joy's original vi. This means that quite some additional functionality you may be used to from vim is not available. For example it provides an exactly one level deep undo buffer. If you screw up more than your last action - too bad for you, consider starting over.

Vim users might consider to just install it like this:

Code Select Expand

# pkg install vim-console

People who do not know what vi is will have to either do a little research or use something else. It is an editor, yes, and in fact a very powerful one. But it will not be what you might be expecting. Vi is not for the uninitiated. There are countless jokes on the net about people who don't even know how to exit it again!

Here's the basics: When you open a file in vi, you cannot just start typing away. By default it is in so-called command mode. You need to press a key to change to insert mode first if you want to input text (e.g. pressing "i"). To exit insert mode, press ESC. In command mode, keys have a different function. "15G" for example means "goto line 15". "c3w" is for "replace the following three words with new input". And "dG" translates to "delete all content from the current line to the end of the document". Quitting the editor is ":q" and Enter, quitting if changes were made can be done with ":q!" + Enter and saving changes and quit either with ":wq" and Enter or with "ZZ".

Vi is not a intuitive editor. You have to learn it and actually practice. If you are getting started with it, you'll likely hate it the entire first week. You'll get along with it afterwards. And finally at some point you probably find that you've fallen in love with it. A lot of people swear by vi-derived editors!

Sounds great? It sure does, try it out! Vim is probably available for your operating system, too. It comes with "vimtutor" which will gently introduce you to powerful editing.

Sounds disgusting? Well, you only want to edit darn files and not "learn" an editor, right? Simply use something else. FreeBSD comes with "ee", the "easy editor", pre-installed to. Maybe it is for you? There are no modes and if you hit ESC, it provides a friendly menu window. Or install another editor like nano. The choice is yours.

ZFS

ZFS is a modern enterprise filesystem with great features. FreeBSD supports both the venerable UFS filesystem as well as ZFS. OPNsense has always supported ZFS on separate partitions. For a couple of versions now it also supports booting from ZFS. Unfortunately the installer that OPNsense still uses will just use UFS. So if you haven't deliberately taken action to install on ZFS the bad news for you is that you are using UFS.

Having ZFS is very much beneficial to package building as it allows e.g. for clones of so-called datasets as well as instant deletion of them. On classic filesystems whole directories need to be copied each time a clean jail is needed (i.e. for each package to build!) and cleaning up means deleting it again which also needs time.

While it's definitely good to have, ZFS is not a hard requirement, though.

Remote logins

The modern way for remote logins on machines running a Unix-like OS is via the Secure Shell (ssh). FreeBSD comes with the OpenSSH server daemon pre-installed as part of its base system (and so does OPNsense). It's disabled by default, though.

If your workstation is running any BSD or macOS you have the OpenSSH client available. The same is true for all common Linux distributions. Windows users will have to install an SSH client, though. One popular free choice is PuTTY. You could also use OpenSSH if you use WSL.

Without doing anything fancy, SSH allows for two type of authentication: password-based and key-based. Using passwords is a little easier to do but is generally considered much less secure. Seriously: Invest the little time to work with keys and forget pure password authentication.

In case you don't know much about all this, here's a very much simplified stab at "cryptography in a nutshell". The classical way of encrypting something is by using a "symmetric key", which basically means: The same key is used to encrypt and to decrypt. This has the downside that the sender and receiver of the encrypted message must posess the same "pre-shared secret". Imagine all kinds of scenarios where getting your key safely to the recipient goes sideways! This is not well fit for modern network technology.

The model known as "public key authentication" solves this problem by using asymetric keys. It makes use of higher math to achieve one-way functions; (luckily) the details are not important for us users. Imagine you got a set of very special keys: One can only be used to lock up something (but cannot open the lock again) while the other is able to unlock (but not lock) it. Now you can send out duplicate keys of the public key to everyone you know. They can encrypt something but as long as you keep the private key secret, only you can decrypt it.

This can also be used for authentication: If I have your public key, I can encrypt a secret message and send it to you. Now if you can decrypt it, encrypt it with my public key and send it back, I can decrypt it again and compare it. If it matches, you've proven to me that you hold the private key associated with your public key. If I trust the latter, you have authenticated to me.

If you don't have a keypair, yet, create one. On any reasonably new Unix-like system execute the following command on the console:

Code Select Expand

% ssh-keygen -t ed25519

It makes sense to protect your key with a passphrase. You can accept the other defaults offered you interactively.

The two files will be in ~/.ssh/id_ed25519 and .ssh/id_ed25519.pub respectively. The former is your private key - it should never leave your machine! The latter is the public key that can be freely copied everywhere.

If you are on Windows, consult the documentation for your SSH client. It shouldn't be hard to generate keys.

OPNsense configuration

Login via the WebUI using the root user for privileged access. Now check and possibly change some settings required for SSH access.

First go to System -> Access -> Users
If you don't have a user present that matches your username on your workstation, create it (you can SSH into the machine using a different account, but it's easier to just create the matching user). Do not input any password for the user, check the "Generate scrambled password" box instead.

For the shell pick "/bin/tcsh" if you don't prefer any of the other shells. Make sure that your user is a member of the "admins" group. Paste your SSH public key into the "Authorized keys" box. Then save your user.

Now go to Settings -> Administration
Check the box "Enable Secure Shell"
Unless your setup requires access from the outside, change the SSH Listen Interface to "LAN". Configure Sudo with "No password" and allow groups "wheel,admins". Save.

That's it, you should now be able to login to your OPNsense using SSH.

Root access to the console

Let's try it:

SSH into your OPNsense by executing

Code Select Expand

% ssh 192.168.1.1

If your OPNsense follows the default LAN IP scheme, this should be fine. Otherwise use the correct IP. You should be prompted for your SSH passphrase to unlock the key. If you typed the correct one, you will be greeted like this:

Code Select Expand


Enter passphrase for key '/home/kraileth/.ssh/id_ed25519':
----------------------------------------------
|      Hello, this is OPNsense 21.1          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website:      https://opnsense.org/        |         @@@\\\   ///@@@
| Handbook:     https://docs.opnsense.org/   |       ))))))))   ((((((((
| Forums:       https://forum.opnsense.org/  |         @@@///   \\\@@@
| Code:         https://github.com/opnsense  |        @@@@         @@@@
| Twitter:      https://twitter.com/opnsense |         @@@@@@@@@@@@@@@
----------------------------------------------

%

We have a shell prompt for an unprivileged user. You can now issue Unix commands.

Connecting to another machine via PuTTY works differently. Have a look at the documentation.

Execute the following command to gain root privileges (= administrator rights):

Code Select Expand

% sudo -i

You will then see a line like this:

Code Select Expand

*** OPNsense.localdomain: OPNsense 21.1.1 (amd64/LibreSSL) ***

As well as some information about your specific setup and the following menu:

Code Select Expand

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option:

Choose 8. This will give you a "#" prompt, meaning that you are now the administrative root user. Be careful with it. You now hold the power to harm the system!

Part 1: Introduction

Situation

FreeBSD offers over 30,000 ready-to-install binary packages. OPNsense provides only a couple hundred (just over 700 right now). This is because the FreeBSD project maintains a general purpose operating system that needs a wide range of software available to be most useful. OPNsense as a much smaller effort concentrates an staying light and focusing on what's common to do on the special-purpose system that it is.

For people who'd like to do a little more with their OPNsense, it suggests itself to extend it with additional packages. My router running it is an always-on device after all and perfectly fit to host my Git repositories for home use for example. Having a candidate that could provide services always available on my net while the system is running on embedded hardware to achieve low power consumption is very intriguing.

When I decided to do this project next after finishing the previous one (in early January), I did not know of the Community Repo (https://forum.opnsense.org/index.php?topic=20827.msg97031#msg97031). So this was not done deliberately as an action to harm that effort. In fact it is much easier to use pre-built packages than going down the route that I describe here. Still some people might be interested in doing it for themselves or have organizational rules that make trusting third party packages complicated (or impossible).

Why not use foreign or offer native packages?

So why not just install a couple of FreeBSD packages on the system? OPNsense is FreeBSD-based after all, right? Yes and no. Today OPNsense is an indirect FreeBSD-derivative system. The upstream for the OS is HardenedBSD, a security-focused close fork of FreeBSD that regularly pulls the latest changes from its mother.

But OPNsense is not just HardenedBSD with a couple of additional packages on top. There are some pretty invasive changes from system startup to the running OS. It's meant to be administrated entirely (or mostly) via its GUI whereas vanilla FreeBSD and HaredenedBSD are configured in classic Unix style on the command line and via editing text files.

Installing FreeBSD's packages on OPNsense is a particularly bad idea. But even using the HardenedBSD packages is not a good one. It's closer to OPNsense but incompatibilities probably do exist and in general native packages are the most sensible solution anyway.

Then why doesn't the OPNsense project simply ship most of or all the FreeBSD packages? Most of them could probably be built for OPNsense, too, after all. Well, maintaining FreeBSD's huge package repository is a daunting task. And even if most of the work is already done by FreeBSD and HardenedBSD, building all those packages is very demanding in resources. The FreeBSD project maintains build clusters of pretty beefy machines building packages all the time. Building an entire set of packages takes those servers a couple of days (when done with that, they begin the next build) for each architecture.

Even if the OPNsense project had the resources to do this, it would largely be wasted compute power. Who would ever run Chromium or LibreOffice on his or her firewall boxes?? Those packages (and their dependencies) alone take hours to build even on modern hardware. Considering to do this is not feasible at all. Therefore OPNsense dedicates its resources on the commonly used packages and is willing to add more if people request it (and the package is not too heavy).

OPNsense does offer easy access to FreeBSD's ports collection however. This "PFC" is a tree of directories which contains various ports. If you don't know what that is: Think "build recipes" for all the software available as packages (and more that e.g. may not be distributed in binary for license reasons). Technically these are Makefile fragments and additional files (patches, init scripts, ...). These ports can be used to build, package and install software directly on OPNsense.

There's a couple of issues with this, though. While it works perfectly well, this traditional way of installing software on FreeBSD is not ... clean. It needs to make available all of the build-time dependencies for your software, too, after all. Sure, you could manually clean up after it, but that's not much fun. Building software like this also means that what's already on the system can influence the result - which is not pretty. It also doesn't scale well: If you are responsible for more than a couple of OPNsense installations, building packages on all the devices (which are often not extremely fast) is time-consuming and wasting resources.

Solution: Building your own additional packages - properly

While you can still build all your software directly from ports on FreeBSD, the project has solved the above issues by providing a better way. There are two programs that cleanly build packages by creating pristine build environments for each package and tearing it down afterwards.

The newer tool, Synth, is more convenient to use but is in maintanence mode and is written in a less comon language (Ada) that needs a special compiler to build. I like it, but then I'm part of the successor project called Ravenports and thus not cannot claim to be neutral on this matter. The other is Poudriere. It's the officially supported build tool for FreeBSD and thus probably the better fit here despite being more complicated to work with.

If you are interested in making additional programs available on your OPNsense box, you must - at least for now! - be somewhat confident in your basic command line skills. What I'm describing here is not possible using the GUI - unless the community at some point decides to offer a proper plugin.

But enough pie in the sky already: Let's get our hands dirty! I'll try to be gentle and explain things along the way, but if you have no shell experience at all (I haven't forgotten you, Windows users. No guarantees though that I'll succeed in explaining everything sufficiently, but I'll try) this is probably not the kind of journey you'll enjoy. Better do at least a little preparation work beforehand.

Looks like I'm not really finding my way around on this forum. How can I mark this thread solved? Or is this a moderator thing? If it is, please mark it solved as the explanation that I was looking for was given. Thanks again!

And just as a quick note: The issue was solved upstream with a new release of Synth so I can begin writing my tutorial (as time permits).

Thanks for the very fast answer! That's some real dedication. :D
So it does indeed originate in HardenedBSD and it is fine the way it is. Good to know, "mystery" solved.

As far as I understood it, Synth needs to know what arch it is building for and checking a binary in a well-known place seemed to do the trick, I guess. Synth has been used on both FreeBSD and DragonFly for way over a year now, after all. Obviously nobody ever tried it on anything else.

But fortunately the author seems to care for issues like that enough to not close the issue with a comment like "nobody cares for OPNsense" - which is behavior that I've encountered far too often in Open Source (a lot of people only seem to care for their main OS, most often some Linux distro). So if this gets fixed upsteam I can write my tutorial. :)

[executable]

Hi there,

while preparing to write a little howto on using Jails with OPNsense, I noticed that the distribution only packages a small subset of software that people might want to use. That's perfectly fine for a firewall OS, but it's all FreeBSD under the hood, right? So we can just add missing stuff ourselves. I thus figured I'll write about building packages for OPNsense first.

I was pretty happy to see that the ports tree can be used with OPNsense (it doesn't work with pfSense due to some customizations they are doing!). So I checked out the HardenedBSD ports tree and was able to build Synth, my package builder of choice. But then Synth would fail to build any package.

After running out of ideas, I created an issue with the project on Github. After a little back and forth the cause for failure was found: Synth uses the binary /bin/sh to detect the platform that it's running on. And this is where OPNsense differs from FreeBSD - running file on the binary will return a different string (and one that puzzles me to be honest).

Here's the output on my FreeBSD workstation:

Code Select Expand

% file /bin/sh
/bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.1 (1101501), FreeBSD-style, stripped


And this is what OPNsense shows:

Code Select Expand

/bin/sh: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.0 (1100122), FreeBSD-style, stripped

Pretty close, but mind that the FreeBSD binary is correctly detected to be an executable whereas OPNsense's binary claims to be a shared object!

I found out that this is also the case on HardenedBSD, and that's likely where OPNsense inherited it from. But why is it like that? I suppose that it has something to do with hardening measures that file doesn't know about and thus mistakes HardenedBSD/OPNsense executables as shared objects.

As a quick and dirty "fix" I scp'd the statically linked /rescue executable from my workstation over to my test machine with OPNsense and put that in /bin/sh's place. Synth then works like a charm. But I wonder what exactly is happening here. Can anybody shed some light on this? Would be interesting to know if it needs to be fixed e.g. with file or if string detection with Synth has to consider another possible case.

Hi Ad and thanks for clarifying the status of priv separation for the GUI! I've edited my post accordingly to mention that it's currently a work in progress.

Hi everyone,

Having followed pfSense on and off for years, I was a little biased towards it when the fork happened. I took a look at both operating systems, though, but soon stopped due to a lack of time. Now I've revisited this case and decided to write a little series about it (I may link the relevant parts in the howto section, too). I've come to really like OPNsense and will definitly write more about it to keep spreading the word and make it more popular.

However I'm a newcomer and I'm not sure that I got everything right (I end up recommending OPNsense over pfSense in the end so it cannot be that bad, eh? ;)). Still I would appreciate some feedback, taking my first steps in the community. If you're interested, please have a look here: pfSense vs. OPNsense.

So far I have had little luck on the forums with the few posts that I made. I have a FreeBSD background and like to tinker with things. And I like to become part of the community of a project that I use and thus would love to find my place with OPNsense, too. The next topics that I intend to write about are jail management and building additional packages on more powerful hardware.

[(failure)]

Hi everyone,

A while ago I decided to build a home router and write about the process on my blog. As a person who uses FreeBSD as a daily driver even on the desktop, I of course wanted to use *BSD on there. I decided to get an APU2c4 and just start with it.

I had e a little bit of experience with pfSense in the past and I've been following OPNsense since it was first released, trying out each release in a VM. There's a lot of drama on the net and I didn't find a somewhat objective comparison pfSense vs. OPNsense and for that reason I'll try and write one up after testing both for a while.

I like OPNsense's interface a lot better and totally love that it's easily possible to switch to LibreSSL. The collaboration with HardenedBSD is also great. However after I found that pfSense works well on the hardware (also the FreeBSD 11.0-based 2.4 beta version), I was surprised to find out that installing OPNsense failed on me! I tried to provide some information on this here, offering to work with the devs if anybody was interested. Obviously I came along at the wrong time when everybody is busy polishing up 17.7 or so. For that reason I continued on my own and found several ways to accomplish an installation. I want to document those here so other people who might be in need find the info (and perhaps to give the devs some more clues on what is happening there).

• The trouble begun when I booted of a memstick that I had dd'd the serial version of 17.1 on: The kernel booted but I was hit by the infamous "error 19" mountroot screen. On the net there are several suggestions on what loader variables to set - none of them (and no combination) helped. I only ever got the mountroot screen and using ? confirmed that no da0 was present. (failure)


• On this forum a user reports that using the SD card instead of trying to install to the mSATA from USB works. I could not try that out because I have no SD card at hand. (not tested)


• My next attempt was to use a device that can mount ISO images and provides a virtual CD drive. I set the loader variables to use the serial console (the cdrom image obviously wasn't meant for serial) and that worked: The system booted all the way up to the login! However the TTY was not configured for serial as it seems and as soon as the login prompt appears, that is no longer forwarded over the serial connection, making it impossible to login and do something... (failure)


• Since everything before the login works, I thought that I might just configure the interfaces and then SSH in. I was able to select which  interfaces should assume what role. But it does not seem possible to configure a LAN address that way! So that also proved to not really help. (failure)


• I had read that the previous version 16.1 used to work, so I decided to give that a try. Unfortunately you guys seem to be extremely tidy when it comes to your mirrors... I've searched for a 16.1 image for hours with no success. I must say that despite that being rather frustrating, I'm kind of impressed. Several search engines, ftp search engines, torrent search engines etc. later, I still had... nothing. Even looking for old cruft from the 90's usually wields a result if you're patient enough, but here I tried every single result - but every link was dead. (failure)


• Then I tried 16.7 again. Didn't work despite the various loader variables (and combinations thereof) that I tried - except for one time when it would suddenly work! I tried to figure out what I had done differently that time. It took me quite a while to realize that I had just forgotten the double quotes when setting the boot delay! And really, it proved to be repeatable: set kern.cam.boot_delay="10000" doesn't work, set kern.cam.boot_delay=10000 does! I was able to install OPNsense and then go through the regular update process with multiple reboots. That's a bit cumbersome and time-consuming, but it works! (success)


• After I had a working OPNsense on my APU I made a backup of the configuration with an IP set for the LAN interface and DHCP enabled for it. Then I formated a USB stick with UFS and put that xml file on there and booted the cdrom ISO again, trying to load that configuration using the configuration importer. It found the drive (da0) but did not find the configuration on there. I tried to rename it to simply config.xml which that didn't help, either. I took a look at the documentation for the importer but that did not mention how to name it. So I looked at the source. Fortunately it's shell code so I could figure out the path and name to use. Booting from the virtual CD and loading the configuration worked. I didn't even need SSH: Now logging in worked over the console as well! (success)


• The previous method was more to my liking but not perfect. I read that such problems could probably be solved by disabling XHCI in the BIOS or use an USB2 port instead of a USB3 one. Disabling it in the BIOS is not possible with the APU2 as far as I can tell and it only has 2 USB3 ports. But I also suspected that it would come down to USB3/XHCI being the cause for error 19. I played with the idea of building a custom kernel without XHCI just to test it, but I'm not familiar with how the OPNsense project builds the ISOs. However that proved not to be necessary: The APU2 board has internal USB2, too! I attached a cable to it that provides a USB port, plugged in a memstick with OPNsense 17.1 serial - and it just worked like there had never been any problem! (success)

So this issue is definitely directly connected to USB3/XHCI. The only thing that's weird is that pfSense 2.4 (which is FreeBSD 11, too) and vanilla FreeBSD 11.0 seem to be able to mount the UFS filesystem using a memstick. Any idea what could be done to fix the images for successfully booting off of USB3?

Again I'd like to offer help in trying out any suggestions if I can. I'm not afraid of building custom stuff or anything. So if anybody has any ideas, please tell me.

And for comparison here's the currently installed pfSense 2.4BETA booting (had to split this off into a new post because it would exceed the length limitation otherwise):

Code Select Expand


# cu -l /dev/cuaU0 -s 115200                         
Connected                                                                             
PCEngines apu2                                                                                 
coreboot build 20170228                                                                                                                                 
4080 MB ECC DRAM                                                                         
                                                                                                   
SeaBIOS (version rel-1.10.0.1)                                               
                                                           
Press F10 key now for boot menu                                                     
                                                                                               
Booting from Hard Disk...                                                 
/boot/config: -S115200 -h                                                             
                                                                                               
Consoles: serial port                                                                                                                                   
BIOS drive C: is disk0                                                                   
BIOS 638kB/3668664kB available memory                                                             
                                                                             
FreeBSD/x86 ZFS enabled bootstrap loader, Revision 1.1
(root@buildbot2.netgate.com, Mon Mar 13 14:24:21 CDT 2017)
Loading /boot/defaults/loader.conf                                   
|                                                                               
          __ ____                                                       
   _ __  / _/ ___|  ___ _ __  ___  ___                                             
  | '_ \| |_\___ \ / _ \ '_ \/ __|/ _ \                     
  | |_) |  _|___) |  __/ | | \__ \  __/                                           
  | .__/|_| |____/ \___|_| |_|___/\___|                                                       
  |_|                                                                                                                                                                 
                                                                       
                                                                             
+============Welcome to pfSense===========+                             
|                                         |                 ______                                                         
|  1. Boot Multi User [Enter]             |                /      \                                         
|  2. Boot [S]ingle User                  |          _____/    f   \
|  3. [Esc]ape to loader prompt           |         /     \        /                                                     
|  4. Reboot                              |        /   p   \______/  Sense
|                                         |        \       /      \         
|  Options:                               |         \_____/        \
|  5. [K]ernel: kernel (1 of 2)           |               \        / 
|  6. Configure Boot [O]ptions...         |                \______/
|  7. Select Boot [E]nvironment...        |                                                                               
|                                         |                       
|                                         |                 
+=========================================+                       
                                                                     
                                                                                                                                     
/boot/kernel/kernel text=0x1753bf8 data=0xab8eb8+0x4c8c68 syms=[0x8+0x183bb8+0x8+0x183b31]
/boot/entropy size=0x1000                     
/boot/kernel/zfs.ko size 0x30b000 at 0x2ddf000   
loading required module 'opensolaris'
/boot/kernel/opensolaris.ko size 0xaca8 at 0x30ea000
Booting...
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p10 #81 51c8a24f312(RELENG_2_4): Tue May 16 13:17:06 CDT 2017
    root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
VT(vga): resolution 640x480
CPU: AMD GX-412TC SOC                                (998.15-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x730f01  Family=0x16  Model=0x30  Stepping=1
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x1d4037ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,Topology,PNXC,DBE,PTSC,PL2I>
  Structured Extended Features=0x8<BMI1>
  XSAVE Features=0x1<XSAVEOPT>
  SVM: NP,NRIP,AFlush,DAssist,NAsids=8
  TSC: P-state invariant, performance statistics
real memory  = 4815060992 (4592 MB)
avail memory = 4070526976 (3881 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <CORE   COREBOOT>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
ioapic1: Changing APIC ID to 5
ioapic0 <Version 2.1> irqs 0-23 on motherboard
ioapic1 <Version 2.1> irqs 24-55 on motherboard
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff806754c0, 0) error 1
random: entropy device external interface
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80675570, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80675620, 0) error 1
wlan: mac acl policy registered
iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8069ea50, 0) error 1
iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8069eb00, 0) error 1
iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8069ebb0, 0) error 1
kbd0 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff8122cb10, 0) error 19
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
padlock0: No ACE support.
acpi0: <CORE COREBOOT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x818-0x81b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.2 on pci0
pcib1: failed to allocate initial I/O port window: 0x1000-0x1fff
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xfe600000-0xfe61ffff,0xfe620000-0xfe623fff at device 0.0 on pci1
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 00:0d:b9:45:c8:9c
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
pcib2: <ACPI PCI-PCI bridge> at device 2.3 on pci0
pci2: <ACPI PCI bus> on pcib2
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x2000-0x201f mem 0xfe700000-0xfe71ffff,0xfe720000-0xfe723fff at device 0.0 on pci2
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: 00:0d:b9:45:c8:9d
igb1: Bound queue 0 to cpu 0
igb1: Bound queue 1 to cpu 1
igb1: Bound queue 2 to cpu 2
igb1: Bound queue 3 to cpu 3
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
pcib3: <ACPI PCI-PCI bridge> at device 2.4 on pci0
pci3: <ACPI PCI bus> on pcib3
igb2: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x3000-0x301f mem 0xfe800000-0xfe81ffff,0xfe820000-0xfe823fff at device 0.0 on pci3
igb2: Using MSIX interrupts with 5 vectors
igb2: Ethernet address: 00:0d:b9:45:c8:9e
igb2: Bound queue 0 to cpu 0
igb2: Bound queue 1 to cpu 1
igb2: Bound queue 2 to cpu 2
igb2: Bound queue 3 to cpu 3
igb2: netmap queues/slots: TX 4/1024, RX 4/1024
pci0: <encrypt/decrypt> at device 8.0 (no driver attached)
xhci0: <AMD FCH USB 3.0 controller> mem 0xfeb22000-0xfeb23fff at device 16.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Unable to map MSI-X table
usbus0 on xhci0
ahci0: <AMD Hudson-2 AHCI SATA controller> port 0x4010-0x4017,0x4020-0x4023,0x4018-0x401f,0x4024-0x4027,0x4000-0x400f mem 0xfeb25000-0xfeb253ff at device 17.0 on pci0
ahci0: AHCI v1.30 with 2 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ehci0: <AMD FCH USB 2.0 controller> mem 0xfeb25400-0xfeb254ff at device 19.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
sdhci_pci0: <Generic SD HCI> mem 0xfeb25500-0xfeb255ff at device 20.7 on pci0
sdhci_pci0: 1 slot(s) allocated
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
orm0: <ISA Option ROM> at iomem 0xef000-0xeffff on isa0
ppc0: cannot reserve I/O port range
uart1: <16550 or compatible> at port 0x2f8 irq 3 on isa0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
usbus0: 5.0Gbps Super Speed USB v3.0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
nvme cam probe device init
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <0x1022> at usbus0
uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ugen1.1: <AMD> at usbus1
uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <SATA SSD SBFM01.0> ACS-4 ATA SATA 3.x device
ada0: Serial Number 99C00773071300080542
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 15272MB (31277232 512 byte sectors)
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
Timecounter "TSC" frequency 998146682 Hz quality 1000
Trying to mount root from zfs:zroot/ROOT/default []...
Root mount waiting for: usbus1 usbus0
uhub0: 4 ports with 4 removable, self powered
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1
ugen1.2: <vendor 0x0438> at usbus1
uhub2: <vendor 0x0438 product 0x7900, class 9/0, rev 2.00/0.18, addr 2> on usbus1
uhub2: 4 ports with 4 removable, self powered
Configuring crash dumps...
No suitable dump device was found.
Filesystems are clean, continuing...
Mounting filesystems...

     ___
___/ f \
/ p \___/ Sense
\___/   \
    \___/

Welcome to pfSense 2.4.0-BETA...

GEOM_ELI: Device ada0p2.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: software
Dump device does not exist.  Savecore not run.
...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.24/mach/CORE
32-bit compatibility ldconfig path:
done.
External config loader 1.0 is now starting...
Launching the init system....... done.
Initializing.................. done.
Starting device manager (devd)...done.
Loading configuration......done.
Updating configuration...done.
Cleaning backup cache................done.
Setting up extended sysctls...done.
Setting timezone...done.
Configuring loopback interface...done.
Starting syslog...done.
Starting Secure Shell Services...done.
Setting up interfaces microcode...done.
Configuring loopback interface...done.
Creating wireless clone interfaces...done.
Configuring LAGG interfaces...done.
Configuring VLAN interfaces...done.
Configuring QinQ interfaces...done.
Configuring WAN interface...done.
Configuring LAN interface...done.
Configuring CARP settings...done.
Syncing OpenVPN settings...done.
Configuring firewall......done.
Starting PFLOG...done.
Setting up gateway monitors...done.
Starting DNS Resolver...done.
Synchronizing user settings...done.
Starting webConfigurator...done.
Configuring CRON...done.
Starting NTP time client...done.
Starting DHCP service...done.
Starting DHCPv6 service...done.
Configuring firewall......done.
Generating RRD graphs...done.
Starting syslog...done.
Starting CRON... done.
pfSense 2.4.0-BETA amd64 Tue May 16 13:10:27 CDT 2017
Bootup complete


Thanks roro for your post!

I've run into the same issue. After finally getting hardware (APU2C4)  to move OPNsense from the "play around" type of VM that I had it in to a production setting, I failed with the error described above. I don't have an SD card to try the suggested solution. But in general I think it would make sense to investigate and fix this. Since my APU is not running as I want it just yet, it's available for tinkering. Time is not extremely critical and so I volunteer to test things and provide more info if any developer wants to look into this.

The problem is definitely OPNsense specific - I was able to install pfSense without any problems. First I thought that it might be a weird issue with FreeBSD 11 but pfSense 2.4BETA can be installed as well without hitting this. My system is currently working but I would prefer to have OPNsense on there (I like LibreSSL, collaboration with HardednedBSD, etc. and think that the way some pfSense people agitate against OPNsense is disgusting and childish which is why I'd rather become part of this community). So I'd gladly delete the current OS and give any proposed fix a try. I'm familiar with FreeBSD (as a user, though, not as a developer!) and am not afraid of compiling code or something.

Here's some more information, hoping that it might be helpful. Attaching the serial console and booting OPNsense 17.1 from USB:

Code Select Expand


# cu -l /dev/cuaU0 -s 115200                             
Connected                                                                                 
PCEngines apu2                                                           
coreboot build 20170228                             
4080 MB ECC DRAM                                         
                                                                         
SeaBIOS (version rel-1.10.0.1)                   
                                                                           
Press F10 key now for boot menu                                           
                                                 
Select boot device:                                       
C o n s o l e s :   i n t e r n a l   v i d e o / k e y b o a r d     s e r i a l   p o r t
. USB MSC Drive  USB DISK 2.0 PMAP             
B I O S   d r i v e   C :   i s   d i s k 0 0iBytes)                                         
. Payload [memtest]                                                     
B I O S   d r i v e   D :   i s   d i s k 1 1                       
                                               
B I O S   6 3 8 k B / 3 6 6 8 6 6 0 k B   a v a i l a b l e   m e m o r y y   
/bboooott//ccoonnffiigg::  --SS111155220000  --DD                       
                                                                                                                           
|                                                                                                           
F r e e B S D / x 8 6   b o o t s t r a p   l o a d e r ,   R e v i s i o n   1 . 1 1
                                                               
( r o o t @ s e n s e y 6 4 ,   S u n   M a r   2 6   1 5 : 2 0 : 4 1   C E S T   2 0 1 7 ) )
                                                               
LLo a d i n g   / b o o t / d e f a u l t s / l o a d e r . c o n f f
                                                                             
/|\                                                                         
                                                                                                 
                  ______  _____  _____                                       
                 /  __  |/ ___ |/ __  |                                     
                 | |  | | |__/ | |  | |___  ___ _ __  ___  ___               
                 | |  | |  ___/| |  | / __|/ _ \ '_ \/ __|/ _ \             
                 | |__| | |    | |  | \__ \  __/ | | \__ \  __/             
                 |_____/|_|    |_| /__|___/\___|_| |_|___/\___|             
                                                                             
+=========================================+     @@@@@@@@@@@@@@@@@@@@@@@@@@@@
|                                         |   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|  1. Boot Multi User [Enter]             |   @@@@@                    @@@@@                     
|  2. Boot [S]ingle User                  |       @@@@@            @@@@@   
|  3. [Esc]ape to loader prompt           |    @@@@@@@@@@@       @@@@@@@@@@@
|  4. Reboot                              |         \\\\\         /////     
|                                         |   ))))))))))))       (((((((((((
|  Options:                               |         /////         \\\\\     
|  5. [K]ernel: kernel (1 of 2)           |    @@@@@@@@@@@       @@@@@@@@@@@
|  6. Configure Boot [O]ptions...         |       @@@@@            @@@@@   
|                                         |   @@@@@                    @@@@@
|                                         |   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|                                         |   @@@@@@@@@@@@@@@@@@@@@@@@@@@@ 
+=========================================+                             
                                                  17.1  ``Eclectic Eagle''
                                                                     
/boot/kernel/kernel text=0x164e4a0 data=0x13f218+0x4c5ee8 syms=[0x8+0x170d00+0x8+0x188c92]
/boot/entropy size=0x1000                                               
/boot/kernel/if_gre.ko size 0x6058 at 0x234e000             
/boot/kernel/if_tap.ko size 0x75a8 at 0x2355000                                 
/boot/kernel/pf.ko size 0x51298 at 0x235d000   
/boot/kernel/carp.ko size 0xe778 at 0x23af000                                   
/boot/kernel/if_bridge.ko size 0xe1e0 at 0x23be000
loading required module 'bridgestp'
/boot/kernel/bridgestp.ko size 0x6e40 at 0x23cd000
/boot/kernel/if_lagg.ko size 0x11078 at 0x23d4000
/boot/kernel/ng_UI.ko size 0x1638 at 0x23e6000
loading required module 'netgraph'
/boot/kernel/netgraph.ko size 0x160f8 at 0x23e8000
/boot/kernel/ng_async.ko size 0x3718 at 0x23ff000
/boot/kernel/ng_bpf.ko size 0x46b0 at 0x2403000
/boot/kernel/ng_bridge.ko size 0x4f30 at 0x2408000
/boot/kernel/ng_cisco.ko size 0x2ea0 at 0x240d000
/boot/kernel/ng_echo.ko size 0xf20 at 0x2410000
/boot/kernel/ng_eiface.ko size 0x33b0 at 0x2411000
/boot/kernel/ng_ether.ko size 0x46f8 at 0x2415000
/boot/kernel/ng_frame_relay.ko size 0x1d90 at 0x241a000
/boot/kernel/ng_hole.ko size 0x17e8 at 0x241c000
/boot/kernel/ng_iface.ko size 0x3988 at 0x241e000
/boot/kernel/ng_ksocket.ko size 0x6168 at 0x2422000
/boot/kernel/ng_l2tp.ko size 0x6648 at 0x2429000
/boot/kernel/ng_lmi.ko size 0x4000 at 0x2430000
/boot/kernel/ng_mppc.ko size 0x5ca8 at 0x2434000
loading required module 'rc4'
/boot/kernel/rc4.ko size 0xb38 at 0x243a000
/boot/kernel/ng_one2many.ko size 0x2ac8 at 0x243b000
/boot/kernel/ng_ppp.ko size 0x95c0 at 0x243e000
/boot/kernel/ng_pppoe.ko size 0x7548 at 0x2448000
/boot/kernel/ng_pptpgre.ko size 0x47a0 at 0x2450000
/boot/kernel/ng_rfc1490.ko size 0x25a0 at 0x2455000
/boot/kernel/ng_socket.ko size 0x60e8 at 0x2458000
/boot/kernel/ng_tee.ko size 0x21a0 at 0x245f000
/boot/kernel/ng_tty.ko size 0x2e98 at 0x2462000
/boot/kernel/ng_vjc.ko size 0x4498 at 0x2465000
/boot/kernel/ng_vlan.ko size 0x2fc0 at 0x246a000
/boot/kernel/if_enc.ko size 0x30a8 at 0x246d000
/boot/kernel/pflog.ko size 0x28d0 at 0x2471000
/boot/kernel/pfsync.ko size 0xcfe0 at 0x2474000
/boot/kernel/ng_car.ko size 0x33b0 at 0x2481000
/boot/kernel/ng_deflate.ko size 0x3718 at 0x2485000
/boot/kernel/ng_pipe.ko size 0x4f18 at 0x2489000
/boot/kernel/ng_pred1.ko size 0x3400 at 0x248e000
/boot/kernel/ng_tcpmss.ko size 0x2048 at 0x2492000
Booting...
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p8 #0 e84bb9532(stable/17.1): Sun Mar 26 15:34:40 CEST 2017
    root@sensey64:/usr/obj/usr/src/sys/SMP amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
VT(vga): resolution 640x480
[HBSD ASLR] status: opt-out
[HBSD ASLR] mmap: 30 bit
[HBSD ASLR] exec base: 30 bit
[HBSD ASLR] stack: 42 bit
[HBSD ASLR] vdso: 28 bit
[HBSD ASLR] map32bit: 18 bit
[HBSD ASLR] disallow MAP_32BIT mode mmap: opt-out
[HBSD ASLR (compat)] status: opt-out
[HBSD ASLR (compat)] mmap: 14 bit
[HBSD ASLR (compat)] exec base: 14 bit
[HBSD ASLR (compat)] stack: 14 bit
[HBSD ASLR (compat)] vdso: 8 bit
[HBSD HARDENING] procfs hardening: enabled
[HBSD LOG] logging to system: enabled
[HBSD LOG] logging to user: disabled
[HBSD SEGVGUARD] status: opt-out
[HBSD SEGVGUARD] expiry: 120 sec
[HBSD SEGVGUARD] suspension: 600 sec
[HBSD SEGVGUARD] maxcrashes: 5
CPU: AMD GX-412TC SOC                                (998.15-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x730f01  Family=0x16  Model=0x30  Stepping=1
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C>
  AMD Features=0x2e500800<SYSCAacpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
orm0: <ISA Option ROM> at iomem 0xef000-0xeffff on isa0
ppc0: cannot reserve I/O port range
uart1: <16550 or compatible> at port 0x2f8 irq 3 on isa0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
nvme cam probe device init
usbus0: 5.0Gbps Super Speed USB v3.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <0x1022> at usbus0
uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ugen1.1: <AMD> at usbus1
uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub0: 4 ports with 4 removable, self powered
uhub1: 2 ports with 2 removable, self powered
ugen1.2: <vendor 0x0438> at usbus1
uhub2: <vendor 0x0438 product 0x7900, class 9/0, rev 2.00/0.18, addr 2> on usbus1
ugen0.2: <vendor 0x13fe> at usbus0
umass0: <vendor 0x13fe USB DISK 2.0, class 0/0, rev 2.00/1.00, addr 1> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0xc100
umass0:2:0: Attached to scbus2
uhub2: 4 ports with 4 removable, self powered
run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Error 5, Retries exhausted
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <SATA SSD SBFM01.0> ACS-4 ATA SATA 3.x device
ada0: Serial Number 99C00773071300080542
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 15272MB (31277232 512 byte sectors)
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC" frequency 998149249 Hz quality 1000
Trying to mount root from ufs:/dev/ufs/OPNsense_Install [ro,noatime]...
mountroot: waiting for device /dev/ufs/OPNsense_Install...
Mounting from ufs:/dev/ufs/OPNsense_Install failed with error 19.

Loader variables:
  vfs.root.mountfrom=ufs:/dev/ufs/OPNsense_Install
  vfs.root.mountfrom.options=ro,noatime

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:tank
        cd9660:/dev/cd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>