OpnSense on FreeBSD 13.1/14-CURRENT in RPI 4 CM isit possible?

[thuttu77]

Is there way to build OpnSense to top of FreeBSD 14-Current since it boot nicely in RPI 4 Compute Module.

I have tried all images I find to Rpi4 CM and no luck I have tred to do several FBSD 13.1 installs and
changed pkg to one from opnsesnse ports but still building packages fails.

Has anyone gotten OpnSense running on top of RPI 4 Compute module?
and even with Dual ethernet mainboard?

EDIT: It seems that RPI 4 CM can boot 13.1 release image but not stable images Interesting.

[thuttu77]

Error what I get with every package is like this

make: stopped in /usr/ports/www/py-requests
[20221205204144] ===> Cleaning for pkg-1.17.5_1
[20221205204144] ===> Cleaning for pkg-1.17.5_1
[20221205204144] ===> Cleaning for py39-requests-2.28.1
[20221205204145] ===> License GPLv2 accepted by the user
===>   squid-5.7 depends on file: /usr/local/sbin/pkg - not found
[20221205204145] ===> Configuring for pkg-1.17.5_1
No installed jimsh or tclsh, building local bootstrap jimsh0
No working C compiler found. Tried cc and gcc.
[20221205204146] ===> Script "configure" failed unexpectedly.
Please report the problem to pkg@FreeBSD.org [maintainer] and attach the
"/usr/obj/usr/ports/ports-mgmt/pkg/work/pkg-1.17.5/config.log" including the
output of the failure of your make command. Also, it might be a good idea to
provide an overview of all packages installed on your system (e.g. a
/usr/obj/usr/ports/ports-mgmt/pkg/work/pkg-1.17.5/src/pkg-static info -g -Ea).
*** Error code 1

there is pkg 1.17 on /usr/local/sbin/pkg

[thuttu77]

It seems that xtools is the problem build seems to work without xtools but takes days
I am buiilding it currently in 13.1

[almodovaris]

Even if it worked, Rpi4 is too slow to do anything meaningful with it.

[dbell37]

I emphatically disagree. I’ve been running a Pi 4 Compute Module (CM4 2GB ram) as a router for well over a year.
Since the CM4 exposes the PCIe bus, a second nic can be added via PCIe instead of USB.
I’ve been running on the DFRobot dual Gb board:
https://www.dfrobot.com/product-2242.html
With Opnsense (thank you XYZ for the image!), I’m able to achieve ~350Mb/s up and down with CPU maxing out at ~50% load. The upload and download speeds should be higher but that’s due to FreeBSD not the CM4 itself. With OpenWRT I can achieve ~960 Mb/s download and ~980 Mb/s upload on this same CM4 and DFRobot board. With Opnsense being far superior, I accept the slower speeds in exchange for all the benefits Opnsense provides over OpenWRT (again, speed issues are due to FreeBSD NOT Opnsense).
It is important that people understand the vast speed improvements of the PI4 over the earlier and much slower PI1/2. Those were indeed hobbyist boards relegated to a few lightweight tasks. The CM4 with accessible PCIe bus opens the door to a lot of options.
I migrated my Opnsense from a 64bit intel system using ~15-30watts to the CM4 router board above using ~3watts a year ago and I am never going back. I hope more in the community start exploring aarch64 chips as ARM has come a long way since the first single core 700MHz Raspberry Pi.
p.s. There is no official ARM build for Opnsense so you’ll either need to rely on someone in the community to provide an image or build your own. I tried to cross-compile my own and failed miserably so I am now trying to compile natively on a Pi4 which takes several days and so far still results in errors. But, I will continue trying and hopefully succeed.
Cheers,
-Dan

[almodovaris]

Yup, OpenWRT does a trick to speed the NAT, I don't think that's available in OPNsense.

[dbell37]

I finished building an image of OpnSense 23.1.9 last Sunday and have been running it on a RPI CM4 dual gigabit routerboard from DFRobot for the past week. It has been stable and performance has been great.
I used a Raspberry Pi 4 running FreeBSD 13.2 Stable to build it:
pkg install git
pkg install groff
pkg install u-boot-rpi4
pkg install rpi-firmware
cd /usr
git clone https://github.com/opnsense/tools
cd tools
make update
make base kernel packages arm-3G DEVICE=RPI

The build scripts expected FreeBSD 13.1 for the build system so I had to edit one of the files in the /usr/tools/config directory. The line in the file was “OS=13.1” and I had to change it to “OS=13.2”.
After that the build ran for 3.5 days. It ended with errors on some packages like Apache and failed to build the SD card image. Since I didn’t need the Apache package, I didn’t try to fix it. To finish building the image I ran:
make arm-3G DEVICE=RPI
Then I wrote the image to the RPI CM4, renamed config_rpi4.txt to config.txt on the fat partition of the image.
Finally I extracted the packages tarball onto a webserver (my webserver IP is 192.168.0.16) on my lan and added the following to /usr/local/etc/pkg/repos/OPNsense.conf:
OPNsense: {
  fingerprints: "/usr/local/etc/pkg/fingerprints/OPNsense",
  url: "http://192.168.0.16/opnsense-repo/${ABI}/23.1/latest",
  signature_type: "NONE",
  mirror_type: "NONE",
  priority: 11,
  enabled: yes
}

Note: My webserver above is not accessible from the internet, so I didn’t bother setting up HTTPS.

Thank you to everyone who has made building OpnSense on ARM possible.

[dbell37]

I’ve been running the OPNsense image referenced above for several months on the Pi CM4 and DFRobot router board. And I have some updates.
I’m using OPNsense as my home firewall with AT&T fiber symmetric gigabit <rant> my neighborhood shockingly has several ISPs to choose from, why isn’t this the norm in the US? </rant>
Within OPNsense I’m using dynamic DNS with a Wireguard road warrior VPN config and HAProxy with SSL in front of my Nextcloud server (running on another server). I’m also running the Telegraf plugin to send OPNsense metrics over MQTT to another server so I can monitor CPU usage, CPU temps, memory, etc

This has all worked quite well and with average CPU utilization being 3%, there is ample left for enabling additional features like Netflow, Ad blocking, etc; all while consuming 2-3 watts.

However, there are a few issues to be aware of. The first being the router board above uses a Realtek NIC and suffers from Watchdog timeouts under heavy load due to FreeBSD having a very old driver. The FreeBSD maintainer’s solution to this is to tell everyone “just don’t use Realtek”, but this isn’t always practical. Realtek is cheaper than Intel or Broadcom and is unfortunately quite prevalent. Every other router board I’ve looked at also use Realtek, so I needed to solve this problem. I compiled the latest driver from Realtek (1.98), but found OPNsense would crash during boot. The crash is due to the Realtek driver expecting a hard-coded MAC address which the Pi CM4 does not have. A few awesome people created a patch for this obscure issue, I patched the 1.98 source with this and installed the driver. OPNsense booted, created a MAC, and has been completely stable no matter what I throw at it.
The second issue I had was when I created the latest OPNsense 23.7.5 image on FreeBSD 13.2, using everything I learned above and found it crashes on boot. This new issue is due to recent FreeBSD 13.2 changes to the /boot/dtb files. I fixed this by replacing the contents of the /boot FAT partition with the contents from an earlier OPNsense build which used FreeBSD 13.1 Stable.

After chronicling all this, I realize no one will probably read this wall of text. If I have time at some point I’ll write up some brief instructions so others can build their own image with the fixes identified above.

[MariuClaudiu]

anyone can post a config.txt for rpi4 please
thank s a lot

[almodovaris]

If you seek a compromise for power consumption, use Minisforum UN100C. Its CPU has 6 W at peak usage. It also has some USB ports which can be turned off from tunables. It has Realtek Ethernet (2 ports, 1 Gigabit), and works okay with vanilla OPNsense (including Gigabit Zenarmor).

[dbell37]

anyone can post a config.txt for rpi4 please
thank s a lot

Here’s the config.txt from my CM4 Opnsense:
arm_64bit=1
dtoverlay=disable-bt
dtoverlay=mmc
device_tree_address=0x4000
kernel=u-boot.bin
armstub=armstub8-gic.bin
hdmi_safe=1

#Enable console on serial port
uart_2ndstage=1
enable_uart=1

#Uncomment below if you want to try overclocking. I haven’t found it necessary though
#over_voltage=6
#arm_freq=2000