24.7 build process, how to cut some packages for faster building time

[none]

Hi,

I am trying to build opnsense 24.7 for a Nanopi R5S and I need another aarch64 box for it (tried on amd64, always had trouble). So the less packages I have to build, the better.

So I did this:

Code: [Select]

root@R4S_imagem:/usr/tools/config/24.7 # cat ports.conf | grep "^#"
#ORIGIN                                         IGNORE
#emulators/virtualbox-ose-additions-nox11       arm,aarch64
#net-mgmt/zabbix5-agent                         arm
#net-mgmt/zabbix5-proxy                         arm
#net-mgmt/zabbix6-agent                         arm
#net-mgmt/zabbix6-proxy                         arm
#net-mgmt/zabbix64-agent                                arm
#net-mgmt/zabbix64-proxy                                arm
#net-mgmt/zabbix7-agent                         arm
#net-mgmt/zabbix7-proxy                         arm
#net/asterisk18                                 arm
#net/bird2                                      arm
#net/frr8                                       arm
#net/haproxy28                                  arm
#net/ntopng                                     arm
#net/siproxd                                    arm
#opnsense/py-haproxy-cli                                arm
#security/clamav                                        arm
#security/suricata                              arm
#security/tor                                   arm
#sysutils/apcupsd                               arm
#sysutils/munin-node                            arm
#sysutils/nut                                   arm,aarch64
#www/sarg                                       arm

the plugins.conf is also edited:

Code: [Select]

root@R4S_imagem:/usr/tools/config/24.7 # cat plugins.conf | grep "^#"
#ORIGIN                                         IGNORE
#net-mgmt/zabbix-agent@zabbix5                  arm
#net-mgmt/zabbix-agent@zabbix6                  arm
#net-mgmt/zabbix-agent@zabbix64                 arm
#net-mgmt/zabbix-agent@zabbix7                  arm
#net-mgmt/zabbix-proxy@zabbix5                  arm
#net-mgmt/zabbix-proxy@zabbix6                  arm
#net-mgmt/zabbix-proxy@zabbix64                 arm
#net-mgmt/zabbix-proxy@zabbix7                  arm
#net/frr                                                arm
#net/haproxy                                    arm
#net/ntopng                                     arm
#net/realtek-re
#net/siproxd                                    arm
#security/clamav                                        arm
#security/tor                                   arm
#sysutils/apcupsd                               arm
#sysutils/apuled                                        arm
#sysutils/munin-node                            arm
#sysutils/nut                                   arm,aarch64
#sysutils/virtualbox                            arm,aarch64
#sysutils/vmware                                        arm
#sysutils/xen                                   arm,aarch64
#www/web-proxy-sso                              arm

but I keep seeing this kind of error:

Code: [Select]

nstalling zip-3.0_2...
Extracting zip-3.0_2: 100%
Installing py311-duckdb-1.0.0...
`-- Installing py311-pandas-2.0.3_2,1...
|   `-- Installing py311-numpy-1.25.0_7,1...
|   `-- Extracting py311-numpy-1.25.0_7,1: 100%
|   `-- Installing py311-numexpr-2.10.1...
|   `-- Extracting py311-numexpr-2.10.1: 100%
|   `-- Installing py311-bottleneck-1.3.8_1...
|   `-- Extracting py311-bottleneck-1.3.8_1: 100%
|   `-- Installing py311-tzdata-2024.1...
|   `-- Extracting py311-tzdata-2024.1: 100%
|   `-- Installing py311-pytz-2024.1,1...
|   `-- Extracting py311-pytz-2024.1,1: 100%
|   `-- Installing py311-python-dateutil-2.9.0...
|   | `-- Installing py311-six-1.16.0_1...
|   | `-- Extracting py311-six-1.16.0_1: 100%
|   `-- Extracting py311-python-dateutil-2.9.0: 100%
`-- Extracting py311-pandas-2.0.3_2,1: 100%
Extracting py311-duckdb-1.0.0: 100%
=====
Message from py311-pandas-2.0.3_2,1:

--
Install math/py-statsmodels to enable parts of pandas.stats.
Install devel/py-xarray to enable the to_xarray() function.
Installing py311-numpy-1.25.0_7,1...
the most recent version of py311-numpy-1.25.0_7,1 is already installed
Installing py311-pandas-2.0.3_2,1...
the most recent version of py311-pandas-2.0.3_2,1 is already installed
Could not find package: suricata
*** Error code 1

Stop.
make: stopped in /usr/tools/

Can I not build all ports there so it can be faster? How can I do it?

Thanks,

none

[franco]

Without tainting the final image, not really. One of the big offenders is Rust which is implied by Suricata, which is required by the core package.

You could strip all plugins and its dependencies but it's a manual effort.


Cheers,
Franco

[Maurice]

If you're okay with not building everything from source, you could prefetch the packages set from my aarch64 repo:

# make prefetch-packages SETTINGS=24.7 VERSION=24.7.r2 MIRRORS=https://opnsense-update.walker.earth

Cheers
Maurice

[none]

If you're okay with not building everything from source, you could prefetch the packages set from my aarch64 repo:

# make prefetch-packages SETTINGS=24.7 VERSION=24.7.r2 MIRRORS=https://opnsense-update.walker.earth

Cheers
Maurice

Hi Maurice,

thanks for the help

I can use it now, thanks

But as my goal is to build image for the Nanopi R5S, I plan on having it all here. Furthermore, can you point me where I can learn on how to set this kind of repository? I have more then one opnsense on arm64 here, and I build things from time to time.

Thanks

none

[Maurice]

I'm not aware of a step-by-step tutorial.

Set up a web server. Copy the base, kernel, packages and aux sets from your build system to /FreeBSD:14:aarch64/24.7/sets/. That's all you need for prefetching sets when building images.

If you also want to use the server for installing updates and plugins on running OPNsense aarch64 systems: Extract the packages set to 24.7/MINT/24.7[.x]/latest/. Create a symlink 24.7/latest/ which links to the latest 24.7/MINT/24.7[.x]/latest/ directory.


Cheers
Maurice

[franco]

If you also want to use the server for installing updates and plugins on running OPNsense aarch64 systems: Extract the packages set to 24.7/MINT/24.7[.x]/latest/. Create a symlink 24.7/latest/ which links to the latest 24.7/MINT/24.7[.x]/latest/ directory.

This is done for staging and control purposes (like using opnsense-revert). Under normal conditions you can get away with using 24.7/latest/ as a directory to unpack the packages set to. That's how we do it for snapshots anyway and what FreeBSD pkg intended.


Cheers,
Franco

[none]

Hello and thanks for you both, Maurice and franco. I kinda created a directory that replicates the files I saw on Maurice's server. That is done, but my opnsense won't use it, complaints about:

pkg: No signature found

and in the end I get the message:

The repository has no fingerprint.

That got me blocked

I am now searching the web about how to create the .sig files for the sets and the fingerprint for the server.

I found this thread here: https://forum.opnsense.org/index.php?topic=37702.0, but so far I am yet to solve this:

Code: [Select]

root@rpi5:/usr/tools # make fingerprint
>>> Executing build step fingerprint on 24.7
root@rpi5:/usr/tools #

The thread suggests it would echo on stdout some code to redirect. Here there is no code

Thanks for all the help.

none

[Maurice]

You need to create a key pair before building the sets:

Code: [Select]

cd /usr/tools/
openssl genrsa -out config/24.7/repo.key 4096
openssl rsa -pubout -in config/24.7/repo.key -out config/24.7/repo.pub
make fingerprint

Save the fingerprint to a file. On the machines which are supposed to use the repository for installing updates and plugins, place this file in /usr/local/etc/pkg/fingerprints/OPNsense/trusted.
You can optionally inject the fingerprint file into the OPNsense package during the build process (see my last comment in the thread you linked to). This is useful if your images are used on many systems because you then don't have to manually copy the fingerprint to each machine.

Cheers
Maurice

[franco]

Hint: you only need fingerprints if you want to feed your builds secure updates. You can also avoid fingerprints and adjust the pkg repo config accordingly.


Cheers,
Franco

[none]

You need to create a key pair before building the sets:

Code: [Select]

cd /usr/tools/
openssl genrsa -out config/24.7/repo.key 4096
openssl rsa -pubout -in config/24.7/repo.key -out config/24.7/repo.pub
make fingerprint

Save the fingerprint to a file. On the machines which are supposed to use the repository for installing updates and plugins, place this file in /usr/local/etc/pkg/fingerprints/OPNsense/trusted.
You can optionally inject the fingerprint file into the OPNsense package during the build process (see my last comment in the thread you linked to). This is useful if your images are used on many systems because you then don't have to manually copy the fingerprint to each machine.

Cheers
Maurice

Thanks Maurice. Do I need to recompile it all so it is signed? If it all needs to be recompiled, I will use for now the way franco said and next build I will add the fingerprint

Thanks you both, again

none

[Maurice]

No need to recompile:

https://github.com/opnsense/tools?tab=readme-ov-file#using-signatures-to-verify-integrity

[none]

Hi guys,

thanks for the help here. it worked just fine, I can build images and update them with no need to a full reinstall

I created some simple and not best written scripts to automate this process as a guide for any new comers.

Its available here: https://github.com/matheusber/opnsense

Thanks again for the help.

Franco, what is the process to add new devices?

Thanks again

none

PS: if changing the thread subject would help in people finding this repo building, its fine by me

[franco]

> Franco, what is the process to add new devices?

Just raise a new PR with the device file and we'll check it. It depends on what trickery the device requires and what we can actually accommodate for.


Cheers,
Franco

[459below]

Hint: you only need fingerprints if you want to feed your builds secure updates. You can also avoid fingerprints and adjust the pkg repo config accordingly.


Cheers,
Franco

I wanted to add how I effectively disabled the fingerprint check, since I for one was not able to follow Franco's suggestion immediately.

1. Add URL to content of e.g. sets-24.7.3-aarch64/packages-24.7.3-aarch64.tar as custom mirror under https://opnsense.example.com/ui/core/firmware#settings
2. Login to the shell of your OPNsense instance as root user
3.

Code: [Select]

vi /usr/local/etc/pkg/repos/OPNsense.conf4. signature_type: "fingerprints"→ signature_type: "none"
5. Mind that the OPNsense.conf file would be overwritten any time you change it under https://opnsense.example.com/ui/core/firmware#settings

I can well imagine this is not the most clean way to disable signature verification.

Using the great info every one collected in this thread to setup fingerprints properly for my builds soon. Thanks for taking the time to share it everyone!

[tax]

Hi guys,

thanks for the help here. it worked just fine, I can build images and update them with no need to a full reinstall

I created some simple and not best written scripts to automate this process as a guide for any new comers.

Its available here: https://github.com/matheusber/opnsense

Thanks again for the help.

Franco, what is the process to add new devices?

Thanks again

none

PS: if changing the thread subject would help in people finding this repo building, its fine by me

Hello friend!

  I have a NanoPi R5S and trying to build OPNsense, I followed the construction files that you made available on github and managed to reach the final step and generate the OPNsense IMG, but when writing to the SDCARD with win32diskimage it does not initialize on the NanoPI R5S, it would have Any steps I missed? Could you provide a working image for the NanoPi R5S that you managed to build?
 I'm starting to study OPNsense Tools on github to understand how the build works.