Topics

Part 1: Introduction

Situation

FreeBSD offers over 30,000 ready-to-install binary packages. OPNsense provides only a couple hundred (just over 700 right now). This is because the FreeBSD project maintains a general purpose operating system that needs a wide range of software available to be most useful. OPNsense as a much smaller effort concentrates an staying light and focusing on what's common to do on the special-purpose system that it is.

For people who'd like to do a little more with their OPNsense, it suggests itself to extend it with additional packages. My router running it is an always-on device after all and perfectly fit to host my Git repositories for home use for example. Having a candidate that could provide services always available on my net while the system is running on embedded hardware to achieve low power consumption is very intriguing.

When I decided to do this project next after finishing the previous one (in early January), I did not know of the Community Repo (https://forum.opnsense.org/index.php?topic=20827.msg97031#msg97031). So this was not done deliberately as an action to harm that effort. In fact it is much easier to use pre-built packages than going down the route that I describe here. Still some people might be interested in doing it for themselves or have organizational rules that make trusting third party packages complicated (or impossible).

Why not use foreign or offer native packages?

So why not just install a couple of FreeBSD packages on the system? OPNsense is FreeBSD-based after all, right? Yes and no. Today OPNsense is an indirect FreeBSD-derivative system. The upstream for the OS is HardenedBSD, a security-focused close fork of FreeBSD that regularly pulls the latest changes from its mother.

But OPNsense is not just HardenedBSD with a couple of additional packages on top. There are some pretty invasive changes from system startup to the running OS. It's meant to be administrated entirely (or mostly) via its GUI whereas vanilla FreeBSD and HaredenedBSD are configured in classic Unix style on the command line and via editing text files.

Installing FreeBSD's packages on OPNsense is a particularly bad idea. But even using the HardenedBSD packages is not a good one. It's closer to OPNsense but incompatibilities probably do exist and in general native packages are the most sensible solution anyway.

Then why doesn't the OPNsense project simply ship most of or all the FreeBSD packages? Most of them could probably be built for OPNsense, too, after all. Well, maintaining FreeBSD's huge package repository is a daunting task. And even if most of the work is already done by FreeBSD and HardenedBSD, building all those packages is very demanding in resources. The FreeBSD project maintains build clusters of pretty beefy machines building packages all the time. Building an entire set of packages takes those servers a couple of days (when done with that, they begin the next build) for each architecture.

Even if the OPNsense project had the resources to do this, it would largely be wasted compute power. Who would ever run Chromium or LibreOffice on his or her firewall boxes?? Those packages (and their dependencies) alone take hours to build even on modern hardware. Considering to do this is not feasible at all. Therefore OPNsense dedicates its resources on the commonly used packages and is willing to add more if people request it (and the package is not too heavy).

OPNsense does offer easy access to FreeBSD's ports collection however. This "PFC" is a tree of directories which contains various ports. If you don't know what that is: Think "build recipes" for all the software available as packages (and more that e.g. may not be distributed in binary for license reasons). Technically these are Makefile fragments and additional files (patches, init scripts, ...). These ports can be used to build, package and install software directly on OPNsense.

There's a couple of issues with this, though. While it works perfectly well, this traditional way of installing software on FreeBSD is not ... clean. It needs to make available all of the build-time dependencies for your software, too, after all. Sure, you could manually clean up after it, but that's not much fun. Building software like this also means that what's already on the system can influence the result - which is not pretty. It also doesn't scale well: If you are responsible for more than a couple of OPNsense installations, building packages on all the devices (which are often not extremely fast) is time-consuming and wasting resources.

Solution: Building your own additional packages - properly

While you can still build all your software directly from ports on FreeBSD, the project has solved the above issues by providing a better way. There are two programs that cleanly build packages by creating pristine build environments for each package and tearing it down afterwards.

The newer tool, Synth, is more convenient to use but is in maintanence mode and is written in a less comon language (Ada) that needs a special compiler to build. I like it, but then I'm part of the successor project called Ravenports and thus not cannot claim to be neutral on this matter. The other is Poudriere. It's the officially supported build tool for FreeBSD and thus probably the better fit here despite being more complicated to work with.

If you are interested in making additional programs available on your OPNsense box, you must - at least for now! - be somewhat confident in your basic command line skills. What I'm describing here is not possible using the GUI - unless the community at some point decides to offer a proper plugin.

But enough pie in the sky already: Let's get our hands dirty! I'll try to be gentle and explain things along the way, but if you have no shell experience at all (I haven't forgotten you, Windows users. No guarantees though that I'll succeed in explaining everything sufficiently, but I'll try) this is probably not the kind of journey you'll enjoy. Better do at least a little preparation work beforehand.

[executable]

Hi there,

while preparing to write a little howto on using Jails with OPNsense, I noticed that the distribution only packages a small subset of software that people might want to use. That's perfectly fine for a firewall OS, but it's all FreeBSD under the hood, right? So we can just add missing stuff ourselves. I thus figured I'll write about building packages for OPNsense first.

I was pretty happy to see that the ports tree can be used with OPNsense (it doesn't work with pfSense due to some customizations they are doing!). So I checked out the HardenedBSD ports tree and was able to build Synth, my package builder of choice. But then Synth would fail to build any package.

After running out of ideas, I created an issue with the project on Github. After a little back and forth the cause for failure was found: Synth uses the binary /bin/sh to detect the platform that it's running on. And this is where OPNsense differs from FreeBSD - running file on the binary will return a different string (and one that puzzles me to be honest).

Here's the output on my FreeBSD workstation:

Code Select Expand

% file /bin/sh
/bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.1 (1101501), FreeBSD-style, stripped


And this is what OPNsense shows:

Code Select Expand

/bin/sh: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.0 (1100122), FreeBSD-style, stripped

Pretty close, but mind that the FreeBSD binary is correctly detected to be an executable whereas OPNsense's binary claims to be a shared object!

I found out that this is also the case on HardenedBSD, and that's likely where OPNsense inherited it from. But why is it like that? I suppose that it has something to do with hardening measures that file doesn't know about and thus mistakes HardenedBSD/OPNsense executables as shared objects.

As a quick and dirty "fix" I scp'd the statically linked /rescue executable from my workstation over to my test machine with OPNsense and put that in /bin/sh's place. Synth then works like a charm. But I wonder what exactly is happening here. Can anybody shed some light on this? Would be interesting to know if it needs to be fixed e.g. with file or if string detection with Synth has to consider another possible case.

Hi everyone,

Having followed pfSense on and off for years, I was a little biased towards it when the fork happened. I took a look at both operating systems, though, but soon stopped due to a lack of time. Now I've revisited this case and decided to write a little series about it (I may link the relevant parts in the howto section, too). I've come to really like OPNsense and will definitly write more about it to keep spreading the word and make it more popular.

However I'm a newcomer and I'm not sure that I got everything right (I end up recommending OPNsense over pfSense in the end so it cannot be that bad, eh? ;)). Still I would appreciate some feedback, taking my first steps in the community. If you're interested, please have a look here: pfSense vs. OPNsense.

So far I have had little luck on the forums with the few posts that I made. I have a FreeBSD background and like to tinker with things. And I like to become part of the community of a project that I use and thus would love to find my place with OPNsense, too. The next topics that I intend to write about are jail management and building additional packages on more powerful hardware.

[(failure)]

Hi everyone,

A while ago I decided to build a home router and write about the process on my blog. As a person who uses FreeBSD as a daily driver even on the desktop, I of course wanted to use *BSD on there. I decided to get an APU2c4 and just start with it.

I had e a little bit of experience with pfSense in the past and I've been following OPNsense since it was first released, trying out each release in a VM. There's a lot of drama on the net and I didn't find a somewhat objective comparison pfSense vs. OPNsense and for that reason I'll try and write one up after testing both for a while.

I like OPNsense's interface a lot better and totally love that it's easily possible to switch to LibreSSL. The collaboration with HardenedBSD is also great. However after I found that pfSense works well on the hardware (also the FreeBSD 11.0-based 2.4 beta version), I was surprised to find out that installing OPNsense failed on me! I tried to provide some information on this here, offering to work with the devs if anybody was interested. Obviously I came along at the wrong time when everybody is busy polishing up 17.7 or so. For that reason I continued on my own and found several ways to accomplish an installation. I want to document those here so other people who might be in need find the info (and perhaps to give the devs some more clues on what is happening there).

• The trouble begun when I booted of a memstick that I had dd'd the serial version of 17.1 on: The kernel booted but I was hit by the infamous "error 19" mountroot screen. On the net there are several suggestions on what loader variables to set - none of them (and no combination) helped. I only ever got the mountroot screen and using ? confirmed that no da0 was present. (failure)


• On this forum a user reports that using the SD card instead of trying to install to the mSATA from USB works. I could not try that out because I have no SD card at hand. (not tested)


• My next attempt was to use a device that can mount ISO images and provides a virtual CD drive. I set the loader variables to use the serial console (the cdrom image obviously wasn't meant for serial) and that worked: The system booted all the way up to the login! However the TTY was not configured for serial as it seems and as soon as the login prompt appears, that is no longer forwarded over the serial connection, making it impossible to login and do something... (failure)


• Since everything before the login works, I thought that I might just configure the interfaces and then SSH in. I was able to select which  interfaces should assume what role. But it does not seem possible to configure a LAN address that way! So that also proved to not really help. (failure)


• I had read that the previous version 16.1 used to work, so I decided to give that a try. Unfortunately you guys seem to be extremely tidy when it comes to your mirrors... I've searched for a 16.1 image for hours with no success. I must say that despite that being rather frustrating, I'm kind of impressed. Several search engines, ftp search engines, torrent search engines etc. later, I still had... nothing. Even looking for old cruft from the 90's usually wields a result if you're patient enough, but here I tried every single result - but every link was dead. (failure)


• Then I tried 16.7 again. Didn't work despite the various loader variables (and combinations thereof) that I tried - except for one time when it would suddenly work! I tried to figure out what I had done differently that time. It took me quite a while to realize that I had just forgotten the double quotes when setting the boot delay! And really, it proved to be repeatable: set kern.cam.boot_delay="10000" doesn't work, set kern.cam.boot_delay=10000 does! I was able to install OPNsense and then go through the regular update process with multiple reboots. That's a bit cumbersome and time-consuming, but it works! (success)


• After I had a working OPNsense on my APU I made a backup of the configuration with an IP set for the LAN interface and DHCP enabled for it. Then I formated a USB stick with UFS and put that xml file on there and booted the cdrom ISO again, trying to load that configuration using the configuration importer. It found the drive (da0) but did not find the configuration on there. I tried to rename it to simply config.xml which that didn't help, either. I took a look at the documentation for the importer but that did not mention how to name it. So I looked at the source. Fortunately it's shell code so I could figure out the path and name to use. Booting from the virtual CD and loading the configuration worked. I didn't even need SSH: Now logging in worked over the console as well! (success)


• The previous method was more to my liking but not perfect. I read that such problems could probably be solved by disabling XHCI in the BIOS or use an USB2 port instead of a USB3 one. Disabling it in the BIOS is not possible with the APU2 as far as I can tell and it only has 2 USB3 ports. But I also suspected that it would come down to USB3/XHCI being the cause for error 19. I played with the idea of building a custom kernel without XHCI just to test it, but I'm not familiar with how the OPNsense project builds the ISOs. However that proved not to be necessary: The APU2 board has internal USB2, too! I attached a cable to it that provides a USB port, plugged in a memstick with OPNsense 17.1 serial - and it just worked like there had never been any problem! (success)

So this issue is definitely directly connected to USB3/XHCI. The only thing that's weird is that pfSense 2.4 (which is FreeBSD 11, too) and vanilla FreeBSD 11.0 seem to be able to mount the UFS filesystem using a memstick. Any idea what could be done to fix the images for successfully booting off of USB3?

Again I'd like to offer help in trying out any suggestions if I can. I'm not afraid of building custom stuff or anything. So if anybody has any ideas, please tell me.