Making OPNsense more useful with custom-built packages

[kraileth]

Part 1: Introduction

Situation

FreeBSD offers over 30,000 ready-to-install binary packages. OPNsense provides only a couple hundred (just over 700 right now). This is because the FreeBSD project maintains a general purpose operating system that needs a wide range of software available to be most useful. OPNsense as a much smaller effort concentrates an staying light and focusing on what's common to do on the special-purpose system that it is.

For people who'd like to do a little more with their OPNsense, it suggests itself to extend it with additional packages. My router running it is an always-on device after all and perfectly fit to host my Git repositories for home use for example. Having a candidate that could provide services always available on my net while the system is running on embedded hardware to achieve low power consumption is very intriguing.

When I decided to do this project next after finishing the previous one (in early January), I did not know of the Community Repo (https://forum.opnsense.org/index.php?topic=20827.msg97031#msg97031). So this was not done deliberately as an action to harm that effort. In fact it is much easier to use pre-built packages than going down the route that I describe here. Still some people might be interested in doing it for themselves or have organizational rules that make trusting third party packages complicated (or impossible).

Why not use foreign or offer native packages?

So why not just install a couple of FreeBSD packages on the system? OPNsense is FreeBSD-based after all, right? Yes and no. Today OPNsense is an indirect FreeBSD-derivative system. The upstream for the OS is HardenedBSD, a security-focused close fork of FreeBSD that regularly pulls the latest changes from its mother.

But OPNsense is not just HardenedBSD with a couple of additional packages on top. There are some pretty invasive changes from system startup to the running OS. It's meant to be administrated entirely (or mostly) via its GUI whereas vanilla FreeBSD and HaredenedBSD are configured in classic Unix style on the command line and via editing text files.

Installing FreeBSD's packages on OPNsense is a particularly bad idea. But even using the HardenedBSD packages is not a good one. It's closer to OPNsense but incompatibilities probably do exist and in general native packages are the most sensible solution anyway.

Then why doesn't the OPNsense project simply ship most of or all the FreeBSD packages? Most of them could probably be built for OPNsense, too, after all. Well, maintaining FreeBSD's huge package repository is a daunting task. And even if most of the work is already done by FreeBSD and HardenedBSD, building all those packages is very demanding in resources. The FreeBSD project maintains build clusters of pretty beefy machines building packages all the time. Building an entire set of packages takes those servers a couple of days (when done with that, they begin the next build) for each architecture.

Even if the OPNsense project had the resources to do this, it would largely be wasted compute power. Who would ever run Chromium or LibreOffice on his or her firewall boxes?? Those packages (and their dependencies) alone take hours to build even on modern hardware. Considering to do this is not feasible at all. Therefore OPNsense dedicates its resources on the commonly used packages and is willing to add more if people request it (and the package is not too heavy).

OPNsense does offer easy access to FreeBSD's ports collection however. This "PFC" is a tree of directories which contains various ports. If you don't know what that is: Think "build recipes" for all the software available as packages (and more that e.g. may not be distributed in binary for license reasons). Technically these are Makefile fragments and additional files (patches, init scripts, ...). These ports can be used to build, package and install software directly on OPNsense.

There's a couple of issues with this, though. While it works perfectly well, this traditional way of installing software on FreeBSD is not ... clean. It needs to make available all of the build-time dependencies for your software, too, after all. Sure, you could manually clean up after it, but that's not much fun. Building software like this also means that what's already on the system can influence the result - which is not pretty. It also doesn't scale well: If you are responsible for more than a couple of OPNsense installations, building packages on all the devices (which are often not extremely fast) is time-consuming and wasting resources.

Solution: Building your own additional packages - properly

While you can still build all your software directly from ports on FreeBSD, the project has solved the above issues by providing a better way. There are two programs that cleanly build packages by creating pristine build environments for each package and tearing it down afterwards.

The newer tool, Synth, is more convenient to use but is in maintanence mode and is written in a less comon language (Ada) that needs a special compiler to build. I like it, but then I'm part of the successor project called Ravenports and thus not cannot claim to be neutral on this matter. The other is Poudriere. It's the officially supported build tool for FreeBSD and thus probably the better fit here despite being more complicated to work with.

If you are interested in making additional programs available on your OPNsense box, you must - at least for now! - be somewhat confident in your basic command line skills. What I'm describing here is not possible using the GUI - unless the community at some point decides to offer a proper plugin.

But enough pie in the sky already: Let's get our hands dirty! I'll try to be gentle and explain things along the way, but if you have no shell experience at all (I haven't forgotten you, Windows users. No guarantees though that I'll succeed in explaining everything sufficiently, but I'll try) this is probably not the kind of journey you'll enjoy. Better do at least a little preparation work beforehand.

[kraileth]

Part 2: Theory & Preparation

Jails

Part 1 spoke of clean "build environments". Poudriere leverages FreeBSD's jails to achieve that goal. But what is a jail?

Think of a "container" if you have no idea at all what this means. It's basically an isolated instance of the operating system programs (the so-called "userland"). Sounds like a virtual machine? Yes, it's closely related to that. Jails are in fact a light-weight means of virtualization. In contrast to full virtualization however it does not involve virtualized hardware. This "additional operating system instance" runs on the same machine as the host system and in fact uses the same kernel.

The latter is the limitation responsible for the fact that you cannot run a foreign operating system in a jail (yes, there are "Linux jails" but they make use of FreeBSD's abilities to translate Linux system calls to native ones, so technically it's still all FreeBSD even if it behaves like Linux).

Processes running in the jail are locked in (hence the name). In general they are not able to change anything related to the host operating system or other jails running besides them. There are ways to escape a jail, but that usually involves deliberate risky configuration of the jail or assistance from the host. The main take-away here is: Don't consider jails a perfectly secure feature under all circumstances but as a nice way to improve security in general.

However in our case we're using the jails system just for providing a second operating system instance where Poudriere controls which packages are installed. This way it can guarantee clean builds without harming the machines main tasks by messing with the packages installed on the host system.

Unix shell prompts and privilege

Unix by default follows a very simple rights model: You are either the almighty "root" user (think "Administrator") or one of the many nobodies on the system. There are multiple ways of elevating your privileges or "becoming root". In this tutorial we are going to use the "sudo" command for that (short for "do as superuser").

The shell prompt shows if you are working as the privileged root user. If so, it will end in the # character. For unprivileged users the symbol is normally a % sign (for derivatives of the c-shell) or a $ (in Bourne shell and derivatives).

In this tutorial all command lines that need to be executed begin with either a percent sign or the pound symbol. Do not include these if you type / copy the command lines. They just show you whether you need to execute the command line as root or as a regular user.

A word on editing files

In this tutorial I'm going to use the vi editor. If you are a Linux user you are likely to know it - or at least know what that means. Be warned though: This is FreeBSD. The "vi" command is NOT an alias for vim. It's nvi, a re-implementation of Joy's original vi. This means that quite some additional functionality you may be used to from vim is not available. For example it provides an exactly one level deep undo buffer. If you screw up more than your last action - too bad for you, consider starting over.

Vim users might consider to just install it like this:

Code: [Select]

# pkg install vim-console
People who do not know what vi is will have to either do a little research or use something else. It is an editor, yes, and in fact a very powerful one. But it will not be what you might be expecting. Vi is not for the uninitiated. There are countless jokes on the net about people who don't even know how to exit it again!

Here's the basics: When you open a file in vi, you cannot just start typing away. By default it is in so-called command mode. You need to press a key to change to insert mode first if you want to input text (e.g. pressing "i"). To exit insert mode, press ESC. In command mode, keys have a different function. "15G" for example means "goto line 15". "c3w" is for "replace the following three words with new input". And "dG" translates to "delete all content from the current line to the end of the document". Quitting the editor is ":q" and Enter, quitting if changes were made can be done with ":q!" + Enter and saving changes and quit either with ":wq" and Enter or with "ZZ".

Vi is not a intuitive editor. You have to learn it and actually practice. If you are getting started with it, you'll likely hate it the entire first week. You'll get along with it afterwards. And finally at some point you probably find that you've fallen in love with it. A lot of people swear by vi-derived editors!

Sounds great? It sure does, try it out! Vim is probably available for your operating system, too. It comes with "vimtutor" which will gently introduce you to powerful editing.

Sounds disgusting? Well, you only want to edit darn files and not "learn" an editor, right? Simply use something else. FreeBSD comes with "ee", the "easy editor", pre-installed to. Maybe it is for you? There are no modes and if you hit ESC, it provides a friendly menu window. Or install another editor like nano. The choice is yours.

ZFS

ZFS is a modern enterprise filesystem with great features. FreeBSD supports both the venerable UFS filesystem as well as ZFS. OPNsense has always supported ZFS on separate partitions. For a couple of versions now it also supports booting from ZFS. Unfortunately the installer that OPNsense still uses will just use UFS. So if you haven't deliberately taken action to install on ZFS the bad news for you is that you are using UFS.

Having ZFS is very much beneficial to package building as it allows e.g. for clones of so-called datasets as well as instant deletion of them. On classic filesystems whole directories need to be copied each time a clean jail is needed (i.e. for each package to build!) and cleaning up means deleting it again which also needs time.

While it's definitely good to have, ZFS is not a hard requirement, though.

Remote logins

The modern way for remote logins on machines running a Unix-like OS is via the Secure Shell (ssh). FreeBSD comes with the OpenSSH server daemon pre-installed as part of its base system (and so does OPNsense). It's disabled by default, though.

If your workstation is running any BSD or macOS you have the OpenSSH client available. The same is true for all common Linux distributions. Windows users will have to install an SSH client, though. One popular free choice is PuTTY. You could also use OpenSSH if you use WSL.

Without doing anything fancy, SSH allows for two type of authentication: password-based and key-based. Using passwords is a little easier to do but is generally considered much less secure. Seriously: Invest the little time to work with keys and forget pure password authentication.

In case you don't know much about all this, here's a very much simplified stab at "cryptography in a nutshell". The classical way of encrypting something is by using a "symmetric key", which basically means: The same key is used to encrypt and to decrypt. This has the downside that the sender and receiver of the encrypted message must posess the same "pre-shared secret". Imagine all kinds of scenarios where getting your key safely to the recipient goes sideways! This is not well fit for modern network technology.

The model known as "public key authentication" solves this problem by using asymetric keys. It makes use of higher math to achieve one-way functions; (luckily) the details are not important for us users. Imagine you got a set of very special keys: One can only be used to lock up something (but cannot open the lock again) while the other is able to unlock (but not lock) it. Now you can send out duplicate keys of the public key to everyone you know. They can encrypt something but as long as you keep the private key secret, only you can decrypt it.

This can also be used for authentication: If I have your public key, I can encrypt a secret message and send it to you. Now if you can decrypt it, encrypt it with my public key and send it back, I can decrypt it again and compare it. If it matches, you've proven to me that you hold the private key associated with your public key. If I trust the latter, you have authenticated to me.

If you don't have a keypair, yet, create one. On any reasonably new Unix-like system execute the following command on the console:

Code: [Select]

% ssh-keygen -t ed25519
It makes sense to protect your key with a passphrase. You can accept the other defaults offered you interactively.

The two files will be in ~/.ssh/id_ed25519 and .ssh/id_ed25519.pub respectively. The former is your private key - it should never leave your machine! The latter is the public key that can be freely copied everywhere.

If you are on Windows, consult the documentation for your SSH client. It shouldn't be hard to generate keys.

OPNsense configuration

Login via the WebUI using the root user for privileged access. Now check and possibly change some settings required for SSH access.

First go to System -> Access -> Users
If you don't have a user present that matches your username on your workstation, create it (you can SSH into the machine using a different account, but it's easier to just create the matching user). Do not input any password for the user, check the "Generate scrambled password" box instead.

For the shell pick "/bin/tcsh" if you don't prefer any of the other shells. Make sure that your user is a member of the "admins" group. Paste your SSH public key into the "Authorized keys" box. Then save your user.

Now go to Settings -> Administration
Check the box "Enable Secure Shell"
Unless your setup requires access from the outside, change the SSH Listen Interface to "LAN". Configure Sudo with "No password" and allow groups "wheel,admins". Save.

That's it, you should now be able to login to your OPNsense using SSH.

Root access to the console

Let's try it:

SSH into your OPNsense by executing

Code: [Select]

% ssh 192.168.1.1
If your OPNsense follows the default LAN IP scheme, this should be fine. Otherwise use the correct IP. You should be prompted for your SSH passphrase to unlock the key. If you typed the correct one, you will be greeted like this:

Code: [Select]

Enter passphrase for key '/home/kraileth/.ssh/id_ed25519':
----------------------------------------------
|      Hello, this is OPNsense 21.1          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website:      https://opnsense.org/        |         @@@\\\   ///@@@
| Handbook:     https://docs.opnsense.org/   |       ))))))))   ((((((((
| Forums:       https://forum.opnsense.org/  |         @@@///   \\\@@@
| Code:         https://github.com/opnsense  |        @@@@         @@@@
| Twitter:      https://twitter.com/opnsense |         @@@@@@@@@@@@@@@
----------------------------------------------

%
We have a shell prompt for an unprivileged user. You can now issue Unix commands.

Connecting to another machine via PuTTY works differently. Have a look at the documentation.

Execute the following command to gain root privileges (= administrator rights):

Code: [Select]

% sudo -i
You will then see a line like this:

Code: [Select]

*** OPNsense.localdomain: OPNsense 21.1.1 (amd64/LibreSSL) ***
As well as some information about your specific setup and the following menu:

Code: [Select]

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option:
Choose 8. This will give you a "#" prompt, meaning that you are now the administrative root user. Be careful with it. You now hold the power to harm the system!

[kraileth]

Part 3: Basic package building

Cloning Ports and installing Poudriere

OPNsense does not package Poudriere, so we have to build this one from ports. Luckily it's a self-contained program that doesn't require additional dependencies to be built for it. The first thing we have to do is getting the ports tree onto the machine. OPNsense provides a convenient script to do exactly that for you. Simply run the following command as root:

Code: [Select]

# opnsense-code ports
This will install Git and use it to download ("clone") the ports tree which will then be available in /usr/ports. Ports are organized in categories. Poudriere is in the ports-mgmt category which means that the actual port lives in /usr/ports/ports-mgmt/poudriere. Let's build and install the port now:

Code: [Select]

# make -C /usr/ports/ports-mgmt/poudriere install clean
This will fetch the source code for Poudriere, extract and configure it then build the program. When it's done it will install the package and clean up after itself.

Getting the distfiles in place

Before the package builder can be used, a build jail needs to be created. To set one up, Poudriere needs an archive of all the files that make up the base system. It also needs the source code for the OS (some packages like e.g. lsof require the source to be present as they are closely tied to the exact OS version). Unfortunately Poudriere makes some assumptions which are right only for vanilla FreeBSD. It does not support HardenedBSD's FTP structure for example. We are going to trick it by creating a local mirror for the relevant files. To do so we first create a directory structure that fits Poudriere's expectations:

Code: [Select]

# mkdir -p /usr/opnsense-dist/pub/FreeBSD/releases/amd64/amd64/21.1-RELEASE
Unfortunately OPNsense does not offer the complete distsets which we need. So we need to get the system source code next:

Code: [Select]

# opnsense-code src
This will clone the OPNsense source code repository to /usr/src. We want the code at the time of the 21.1 release so we make git switch to the corresponding tag:

Code: [Select]

# cd /usr/src && git checkout 21.1 && cd -
Then we build the required source tarball:

Code: [Select]

# tar -C / -cvJf /usr/opnsense-dist/pub/FreeBSD/releases/amd64/amd64/21.1-RELEASE/src.txz --exclude-vcs usr/src
Now we can either download the pre-built release tarball for the base system like this:

Code: [Select]

# fetch https://pkg.opnsense.org/FreeBSD:12:amd64/21.1/sets/base-21.1-amd64.txz -o /usr/opnsense-dist/pub/FreeBSD/releases/amd64/amd64/21.1-RELEASE/base.txz
Alternatively (no fun on a weak machine) we can build it from source ourselves. To do so we check out the OPNsense tools repository:

Code: [Select]

# opnsense-code tools
We need to define an environment variable because the build system will otherwise complain that we're trying to build "head". This is to prevent people from accidentally building development versions that will contain lightly tested code. In our case we're in "detached head" mode because we asked git to return to a tagged state (for the actual release version).

The "-j 5" parameter means that the building may use up to 5 threads. What is right for you depends on the number of threads that the CPU you are using offers. A general recommendation is to use "n + 1" for building if you want minimum compile time. My machine has 4 threads and does not do heavy package filtering over night, so I go with 5. You probably don't want to do this on your production firewall that needs to continue fulfilling its main purpose:

Code: [Select]

# env SRCBRANCH=HEAD make -C /usr/tools -j 5 base
When it's done (it *will* take a while!), copy off the resulting tarball into the directory holding our distsets:

Code: [Select]

# cp /usr/obj/usr/src/amd64.amd64/release/base.txz /usr/opnsense-dist/pub/FreeBSD/releases/amd64/amd64/21.1-RELEASE
Ok, we have both distfiles required by Poudriere. But as it will freak out if there's no accompanying MANIFEST file, let's create that, too:

Code: [Select]

# cd /usr/opnsense-dist/pub/FreeBSD/releases/amd64/amd64/21.1-RELEASE && /usr/src/release/scripts/make-manifest.sh base.txz src.txz > MANIFEST
Depending on the CPU of your machine this can take a while. The script will calculate the the SHA256 checksums of the distfiles and count the number of files that they contain. Poudriere uses the manifest to check whether the distfiles are good or if they probably were damaged.

Poudriere configuration

You need to know if your system uses ZFS and if it does, what the name of the ZFS pool is. Execute this command:

Code: [Select]

# zpool list
You will get output like this:

Code: [Select]

NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot   472G  2.65G   469G        -         -     0%     0%  1.00x  ONLINE  -
My system has a pool named "zroot" (FreeBSD's default name). If your output is missing the second line, you're on UFS.

The next step is to configure Poudriere. The program comes with a sample configuration that we can use as a template. To do so we copy the file and then edit it:

Code: [Select]

# cp /usr/local/etc/poudriere.conf.sample /usr/local/etc/poudriere.conf
# vi /usr/local/etc/poudriere.conf
If you are using ZFS, find the line that defines ZPOOL, remove the comment symbol ('#') and change the name if necessary. On a UFS system comment in the line "NO_ZFS=yes" instead.

Then look for the line "FREEBSD_HOST=_PROTO_://_CHANGE_THIS_". Change it to:

Code: [Select]

FREEBSD_HOST=file:///usr/opnsense-dist
Finally comment in the line "BUILD_AS_NON_ROOT=no" and save the file.

OPNsense 21.1 is based on HardenedBSD 12.1. Version 12.1 of FreeBSD is already EOL ("end of life") and no longer supported - and so from the perspective of the ports tree we're running on an obsolete release. We need to set a variable to make the build system stop complain and just do its thing:

Code: [Select]

# echo ALLOW_UNSUPPORTED_SYSTEM=yes > /usr/local/etc/poudriere.d/make.conf
OPNsense offers two different variants of the firmware: Using OpenSSL (default) or LibreSSL. After installation I immediately change to LibreSSL as it is the better one of the two IMO. Check what you are using! On the dashboard look in the "System Information" box under "Versions". If you are using LibreSSL, you need to tell the build system to adapt accordingly. Issue the following command (do NOT do this if you are using the standard OpenSSL!):

Code: [Select]

# echo DEFAULT_VERSIONS+= ssl=libressl >> /usr/local/etc/poudriere.d/make.conf
That's it for the configuration.

Build jail creation and initial ports build

We're good to prepare the jail now. Let's tell Poudriere to create one, to call it "opnsense211" (we cannot use periods in the name, hence 211 and not 21.1) and to use version "21.1-RELEASE" for which we've provided the distfiles:

Code: [Select]

# poudriere jail -c -j opnsense211 -v 21.1-RELEASE
[00:00:00] Creating opnsense211 fs at /usr/local/poudriere/jails/opnsense211... done
[00:00:00] Fetching MANIFEST for FreeBSD 21.1-RELEASE amd64
/usr/local/poudriere/jails/opnsense211/fromftp         222  B  307 kBps    00s
[00:00:00] Fetching base for FreeBSD 21.1-RELEASE amd64
/usr/local/poudriere/jails/opnsense211/fromftp         172 MB   79 MBps    02s
[00:00:09] Extracting base... done
[00:01:32] Fetching src for FreeBSD 21.1-RELEASE amd64
/usr/local/poudriere/jails/opnsense211/fromftp         157 MB   63 MBps    03s
[00:01:40] Extracting src... done
[00:03:45] Cleaning up... done
[00:03:45] Recording filesystem state for clean... done
[00:03:45] Upgrading using ftp
/etc/resolv.conf -> /usr/local/poudriere/jails/opnsense211/etc/resolv.conf
sed: /usr/local/poudriere/jails/opnsense211/usr/sbin/freebsd-update: No such file or directory
12.1--HBSD
[00:03:46] Recording filesystem state for clean... done
[00:03:46] Jail opnsense211 12.1--HBSD amd64 is ready to be used
We also want to hook up the ports tree with Poudriere and name it just "opnports":

Code: [Select]

# poudriere ports -c -m null -M /usr/ports -p opnports
[00:00:00] Imported ports tree "opnports" from /usr/ports
The one missing step is to define a list of ports to be built. Initially we're only building one port that we'll make use of in later builds: Ccache. It is a program that allows the caching of compilation artifacts for later re-use. If you are building packages regularly (and you should for the sake of security!) this will save you some time and resources in future builds.

Code: [Select]

# echo devel/ccache > /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-pkglist
Ok, all done! We are finally ready to begin the actual package building. We'll use the only jail that we have ("opnsense211") with the ports tree that we registered with Poudriere ("opnports") and call the package set "customsense":

Code: [Select]

# poudriere bulk -j opnsense211 -p opnports -z customsense -f /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-pkglist
[00:00:00] Creating the reference jail... done
[00:00:01] Mounting system devices for opnsense211-opnports-customsense
[00:00:01] Mounting ports/packages/distfiles
[...]
[00:00:19] Starting/Cloning builders
[00:00:19] Hit CTRL+t at any time to see build progress and stats
[00:00:20] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.15.10_3
[00:09:55] [01] [00:09:35] Finished ports-mgmt/pkg | pkg-1.15.10_3: Success
[00:09:56] [01] [00:00:00] Building devel/ccache | ccache-3.7.1_1
[00:11:12] [01] [00:01:16] Finished devel/ccache | ccache-3.7.1_1: Success
[00:11:14] Stopping 2 builders
[00:11:14] Creating pkg repository
[...]
We've created a pkglist with only one item, but Poudriere needs the pkg package manager, which is always built first. So when it's finished, we have two newly built packages on our system:

Code: [Select]

# ls /usr/local/poudriere/data/packages/opnsense211-opnports-customsense/All
ccache-3.7.1_1.txz      pkg-1.15.10_3.txz
It's actually a valid repository already, but we'll ignore that for a minute. Let's install the ccache package directly (i.e. without using the repository). This is done with the "add" subcommand of pkg:

Code: [Select]

# pkg add /usr/local/poudriere/data/packages/opnsense211-opnports-customsense/All/ccache-3.7.1_1.txz
Installing ccache-3.7.1_1...
Extracting ccache-3.7.1_1: 100%
[...]
And that's really it: We've just built our first own native package for OPNsense and installed it manually. It's a nice start! The next part will show how to make more out of it.

[kraileth]

Part 4: Making things useful

Build cache

We've just installed ccache. Now we need to create a directory for it to use:

Code: [Select]

# mkdir -p /var/cache/ccache/poudriere
Ccache defaults to not use more than 5 GB of space for its cache. I doubt that it makes sense to increase it as repositories for OPNsense probably shouldn't be so big in the first place. On my FreeBSD systems I do increase it, though. If you are low on disk space, you can of course also lower the limit. I'm going to write out the default here for reference purposes:

Code: [Select]

# echo max_size = 5.0G > /var/cache/ccache/poudriere/ccache.conf
To make Poudriere use ccache, we have to edit its configuration again:

Code: [Select]

# vi /usr/local/etc/poudriere.conf
Find the "CCACHE_DIR=" line, comment it in and change it to:

Code: [Select]

CCACHE_DIR=/var/cache/ccache/poudriere
Now we're going to add Poudriere to the list of packages to build so that we will get updates in the future:

Code: [Select]

# echo ports-mgmt/poudriere >> /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-pkglist
Additional packages

Also add the programs that you actually wanted to build. To do so, simply put them into the package list file one per line and specify the programs as "category/name". My example here is the jail manager / virtualization framework cbsd.

You probably don't know the category if you're not working with ports regularly. You can use the "whereis" command to find out if you know the program's name:

Code: [Select]

# whereis cbsd
cbsd: /usr/ports/sysutils/cbsd

So it would be "sysutils/cbsd" that needs to be added to the package list file:

Code: [Select]

# echo sysutils/cbsd >> /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-pkglist
 But what if you are looking for something but don't even know the exact name of the program? The easiest thing is probably to use this website:

https://www.freshports.org

To the right you will find a "search" box that you can use. Explore the site a little, it is very useful if you are working with FreeBSD ports.
When you are happy with your package list, simply invoke the bulk build command again to make Poudriere build the missing packages:

Code: [Select]

# poudriere bulk -j opnsense211 -p opnports -z customsense -f /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-pkglist
Package repositories

Last time we used "pkg add" to install the ccache package manually. While there's nothing really wrong with that, this method is less helpful if you want to keep your system up to date. This is where repositories come into play. A repository is a set of package files with additional metadata to e.g. make it searchable. Don't worry about the details - Poudriere takes care of all that for you. You only need to make the package manager aware of your additional repository. We'll do that now. Let's take a look at a configuration example:

Code: [Select]

# cat /usr/local/etc/pkg/repos/FreeBSD.conf
FreeBSD: { enabled: no }
Pretty simple, eh? This is an override config that disables the repository called "FreeBSD" which comes with the operating system and is otherwise enabled by default. Have a look at the file OPNsense.conf if you are curious.

Now create a new file:

Code: [Select]

# vi /usr/local/etc/pkg/repos/custom.conf
And paste the following into it:

Code: [Select]

Custom: {
  url      : file:///usr/local/poudriere/data/packages/opnsense211-opnports-customsense,
  priority : 20,
  enabled  : yes
}
This will configure a new repository called "Custom" for pkg to use. I'm setting a lower priority than the OPNsense repository has by default. Why? Because we want the official packages to take precedence to not break anything. If you plan on building your packages often, the cleanest solution would be add all required packages to your list and to disable the OPNsense repository completely.

When using multiple repositories you might run into the situation where pkg will not update to newer packages in your repository even if you give it a higher priority. This is due to the (default) option called "CONSERVATIVE_UPGRADE": pkg remembers which repository a package was originally installed from and will generally prefer that repository over others. Edit /usr/local/etc/pkg.conf and switch that option off if you want to go that route.

Now make the package manager read the repository information again:

Code: [Select]

# pkg update
After the previous step, I can simply install my jail manager:

Code: [Select]

# pkg install cbsd
There we go! It will ask me to confirm the actions that it is about to take and a couple of seconds later the program that is not packaged by OPNsense is installed cleanly on the system.

Build options

FreeBSD's ports framework allows for customizing build-time options for many packages. You can issue the following command to configure all of the ports in your list (AND their dependencies!):

Code: [Select]

# poudriere options -j opnsense211 -p opnports -z customsense -f /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-pkglist
===> Setting user-specified options for cbsd-13.0.0 and dependencies
Doing so is not much fun. HardenedBSD added hardening options to each and every port, even those that don't have any options on vanilla FreeBSD - and OPNsense inherited this. So for all the ports potentially involved this will display a configuration menu for you. Most of the time you will probably just hit Enter but you might want to actually customize a couple of ports. Seriously, this is pretty tedious. Cbsd is not an extreme example at all, but look how much configuration dialogs it caused:

Code: [Select]

# ls /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-options/| wc -l
     275
A better way might be to configure only the ports that you are interested in customizing. While cbsd doesn't in fact currenly have any interesting build-time options to tweak, we're going to do it anyway here for demonstration purposes. Here's how to do it:

Code: [Select]

# make -C /usr/ports/sysutils/cbsd config
This will bring up the configuration menu and if you confirm the selection, the options get saved. By default, the ports tree will save its options in /var/db/ports. There should be a "sysutils_cbsd" directory inside now. You can simply copy those over for use by Poudriere:

Code: [Select]

# mkdir -p /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-options
# cp -r /var/db/ports/* /usr/local/etc/poudriere.d/opnsense211-opnports-customsense-options/
That way you can configure only the ports that you actually care about and not potentially hundreds of others as well.

Updating

Staying safe means regularly upgrading the system. How do you do it? The first step is updating the ports tree. Do this by issuing:

Code: [Select]

# cd /usr/ports && git pull && cd -
Step two is to simply start another bulk build the way you've done at least twice now. When Poudriere has finished you will have a repository with fresh packages in it. You can either update via the command line by issuing:

Code: [Select]

# pkg upgrade
Or you simply use the GUI as you're used to. It will show the updates from all configured repositories, so you don't even have to remember how to use pkg.

Have fun with your new even more useful OPNsense box(es) and enjoy a great Open Source project!

What now?

I still have a couple of things on my mind that I could put into another post - mostly ideas to make this whole procedure easier for users. But with these 4 parts people should be able to roll their own packages. So before putting more time and effort into this, I'll wait to see if this tutorial is actually useful to anybody.

Feel free to ask if you have questions about this. Of course comments or suggestions are welcome as well.

[hushcoden]

Even though I don't have the skills to compile software, just to say thanks to share this, I'm sure someone with the right skills will make use of that, well done !

[fbarco]

Hello, thank you very much for this post.
I tried to build "sysutils/py-salt" port on OPNsense 21.7 but it fails on "ports-mgmt/pkg" build

Code: [Select]

[.....]
install  -m 0644 /wrkdirs/usr/ports/ports-mgmt/pkg/work/pkg-1.16.3/NEWS /wrkdirs/usr/ports/ports-mgmt/pkg/work/stage/usr/local/share/doc/pkg/NEWS
====> Compressing man pages (compress-man)
===>   Installing ldconfig configuration file
===========================================================================
=======================<phase: package        >============================
===>  Building package for pkg-1.16.3
cp: /wrkdirs/usr/ports/ports-mgmt/pkg/work/pkg/pkg-1.16.3.pkg: No such file or directory
*** Error code 1
I built it successfully on FreeBSD 13.
With the "-w" command option, it saves the working directory and "pkg-1.16.3.txz" package exists in "/usr/local/poudriere/data/wrkdirs/opnsense217-opnports-customsense/opnports/pkg-1.16.3.tbz" archive.
Perhaps somebody knows if i forgot to do something

thanks

[fbarco]

I tried on OPNsense 21.1, it's the same

[Balogh]

Tried on 21.7.3_3, building pkg gives an error:

Code: [Select]

[00:00:02] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.16.3
[00:01:38] [01] [00:01:36] Finished ports-mgmt/pkg | pkg-1.16.3: Failed: package

Log shows that failure occurs after pkg is successfully compiled:

Code: [Select]

install  -m 0644 /wrkdirs/usr/ports/ports-mgmt/pkg/work/pkg-1.16.3/NEWS /wrkdirs/usr/ports/ports-mgmt/pkg/work/stage/usr/local/share/doc/pkg/NEWS
====> Compressing man pages (compress-man)
===>   Installing ldconfig configuration file
===========================================================================
=======================<phase: package        >============================
===>  Building package for pkg-1.16.3
cp: /wrkdirs/usr/ports/ports-mgmt/pkg/work/pkg/pkg-1.16.3.pkg: No such file or directory
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/ports-mgmt/pkg
*** Error code 1

Stop.
make: stopped in /usr/ports/ports-mgmt/pkg
=>> Cleaning up wrkdir
===>  Cleaning for pkg-1.16.3
build of ports-mgmt/pkg | pkg-1.16.3 ended at Wed Oct 20 09:23:26 EEST 2021
build time: 00:01:36
!!! build failure encountered !!!


[mimugmail]

poudriere currently doesn't work with pkg 1.16, it only works with the build tools from OPNsens itself right now.

[franco]

Changes in pkg 1.17 are a bit destructive and not much care was taken in the ports tree for the situation. We will be migrating to 1.17 eventually on our way to 22.1, but there is no pressing need for it for the moment.

Situations like these are why we do not use poudriere if anyone was wondering.


Cheers,
Franco