questions about contributing binary package repo and release engineering toolkit

[egghunter05]

Hello,

I have some questions about contributing package repo for specific packages and engineering toolkit.

1.

If I want to contribute to this binary package repo:
https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/latest/All/

Is there a specific procedure for this?

2.

Which repo address should be used for commiting packages?

3.

I have found this repo:
https://github.com/opnsense/ports

Do you generate all packages on the port tree?  Or how do you decide which packages will be published on the binary package repo?

4.

All minor versions are using the same repo?  For instance: 18.7.5 and 18.7.6 are using the same repo?

https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/latest/All/


5.
I am trying to discover release engineering toolkit.

For generating 18.7.5 dvd .iso image I am running these steps on FreeBSD 11.10 x86_64:

Code: [Select]

cd /usr
# git clone https://github.com/opnsense/tools.git
# cd tools
# make update
# git checkout 18.7.5
# make iso

Is it correct method to do that?

6.


https://github.com/opnsense/tools

On the main documentation I see different commands like:
# make step OPTION="value"
# make base
# make nano
...

If I want to modify base system and create a dvd, can I do that with this tool? If you give an example, I will be glad. For instance: If I want to create a file on the base system like: /etc/egghunter-test what should I do?


Kind regards

[egghunter05]

I discovered "Development manual" page on https://docs.opnsense.org/development/workflow.html, but I couldn't see any information about package repo management policy

Kind regards

[fabian]

What do you plan to do?

Things that need to be built (C, C++, Go, ...) -> build script to ports repository or request it to be built in the tools repository
Plugin that can be installed -> make a pull request to the plugins repository
Core changes (what you can do in the web interface after installation) -> core repository

[franco]

Hi egghunter05,

1.

If the port is already in FreeBSD ports a PR to tools.git is enough. But we need to investigate build time, dependency chains and software usefulness within the OPNsense scope. We can't take all / any package.

If the port is not in FreeBSD ports yet, it should be added there first. It will automatically land in ports.git which is synced daily with FreeBSD/HardenedBSD. We do not take PRs to ports.git unless really really necessary and purpose-built for OPNsense (and we try to avoid it as much as possible).

2.

Should be answered by 1.

3.

Yes, the packages described in ports.conf in tools.git are generated from ports.git.

4.

Yes, same repository and branch, but different tags. https://github.com/opnsense/ports/releases

5.

Almost. "iso" was replaced by "dvd", see https://github.com/opnsense/tools#tldr

6.

If you want to add files to the operating system you need to commit into src.git. But mostly that should be avoided and plugins.git is used to install add-ons.

Hope this helps.


Cheers,
Franco

[egghunter05]

Hello Franco,

I want to ask some questions again.

1.

This pkg repo url is for latest 18.7 version I think:
https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/latest/All/

And these are for specific tags:
https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/MINT/18.7.6/OpenSSL/All/
https://pkg.opnsense.org/FreeBSD:11:amd64/18.7/MINT/18.7.5/OpenSSL/All/

Am I right?

2.

I need 18.7.6, 18.7.5, 18.7.4 iso files.
Where can I find old iso downloads?

3.

I want to build release iso file for a specific version.

I tried to build dvd with these commands on FreeBSD 11.10

Code: [Select]


# cd /usr
# git clone https://github.com/opnsense/tools.git
# cd tools
# git checkout 18.7.5

# cd /usr
# git clone https://github.com/opnsense/ports.git
# cd ports
# git checkout 18.7.5

# cd /usr
# cd tools
# make dvd


and I got this error:

Code: [Select]

>>> /usr/ports does not match expected branch: 18.7.5
>>> To continue anyway set PORTSBRANCH=HEAD

I passed it with:

Code: [Select]

make PORTSBRANCH=HEAD dvd

command. But I want to be sure I am running correct commands for generating native release dvd iso file for a specific tag.

I need i386 and amd64 iso files. If you can share correct commands I will be glad.

Thanks

Kind regards

[egghunter05]

Hello,

Which HardenedBSD version are we using on OpnSense 18.7 and 19.1?

    hardenedbsd-11-stable-20180808-1/   2018-08-08 17:08   -   
    hardenedbsd-11-stable-20180814-1/   2018-08-14 18:53   -   
    hardenedbsd-11-stable-20180826-1/   2018-08-26 21:16   -   
    hardenedbsd-11-stable-20180913-1/   2018-09-12 23:12   -   
    hardenedbsd-11-stable-20180927-1/   2018-09-27 18:51   -   
    hardenedbsd-11-stable-20181018-1/   2018-10-17 22:31   -   
    hardenedbsd-11-stable-20181104-1/   2018-11-04 21:11   -   
    HardenedBSD-11-STABLE-v1100056.8-amd64-disc1.iso (Latest stable)

I think on Opnsense 18.7 we are only using ports tree? or base system as well? What about 19.1? Is it completelty HardenBSD? I am asking it because I want to generate elasticsearch5 port/packages on the correct version for testing on OpnSense.

Kind regards

[AdSchellevis]

Hi egghunter05,

We're using our own repository (https://github.com/opnsense/src), which will start using upstream HardenedBSD as of 19.1 (as Shawn mentions nicely https://twitter.com/lattera/status/1062780843595350016).

You best use our repositories when trying to fix elasticsearch, all are available at https://github.com/opnsense/

When building specific versions, don't forget to checkout the same version on all repositories.

Best regards,

Ad

[franco]

Hi there,

1.

Yes. But we mainly only keep those versions for "opnsense-revert" to be able to revert individual packages to a different release state for debugging / delivering workarounds.

2.

We don't officially provide images for each minor version as vetting images for integrity takes way too much time. It's possible to build them from the (prebuilt) kernel base and packages set, however. And we do so on occasion, mostly also for debugging / delivering workarounds.

3.

The build is tailored for continuous building. So whenever we build a version it happens to be correctly tagged on the used branches. If you check out a tag it says "HEAD" instead of the respective branch and that confuses the build, but it's correct...

There's a tools.git extension called "make rewind VERSION=18.7.5" which lets you know which tags to check out as we sometimes do not have a tag in an associated repository (mostly src.git).

For the build to shut up about "HEAD" you can use:

# cd /usr/core && git checkout stable/18.7 && git pull && git reset --hard 18.7.5

Which will retain the branch information and return core.git into its historic (continuous) state.

I can provide you with a packages set for a 18.7.x release if you want, but I suppose you want to rebuild some of it anyway so maybe it's better to generate them from scratch?


Cheers,
Franco

[egghunter05]

Hello Ad and Franco,

Thank you for answers.

My first goal in this study is building elasticsearch5 package on the correct platform by using the correct port tree for opnsense 19.1. After built it I want to perform installation tests on the latest 19.1 iso version and if any problem happens I will try to fix it.

I have new questions.

1.

When I checked OS version on 19.1b installation I see:

Code: [Select]

11.2-RELEASE-p4-HBSD 4b21e5691ff

Which hardenedbsd installer iso is used for this version "11.2-RELEASE-p4-HBSD 4b21e5691ff"?

I am asking it becase if the version changes, I think we should rebuild and retest elasticsearch5 package on the new version as well.

2.

Do you think that should I use hardenedbsd latest stable 11 version for package build/installatıon tests for an option?  We are planning to use these packages on opnsense 19.1 version.

3.

For building elasticsearch5 package on opnsense 19.1b setup, I need a port tree.

a)
So for installing port tree I can use this repo:
https://github.com/HardenedBSD/hardenedbsd-ports

or

b)
this repo:
https://github.com/opnsense/ports

If you advise to use option b) https://github.com/opnsense/ports. I can't see any 19.1 version tag on the releases section (https://github.com/opnsense/ports/releases). Should I use master branch?

Which port tree should I use building elasticsearch5 on 19.1b version?

4.

I can't see any tags for 19.1 on core,ports repo, how can I build 19.1b version by using "make dvd" command on the build toolskit. Is it posible?

5.

I have discovered this repo for 19.1 version: https://github.com/opnsense/src/releases/tag/19.1.a
Do you advise building latest dvd iso from strach for elasticsearch5 depended package installation tests? If you think, dvd iso is not necessary for package installation tests maybe I should skip this requirement.






Kind regards

[franco]

1.

When you use the build system SETTINGS=18.7 is implied so you are building src.git SRCBRANCH=stable/18.7.

When you explicitly force SETTINGS=19.1 *on every step* you build src.git SRCBRANCH=master which is 19.1.b

You're able to find 4b21e5691ff on the master branch of src.git easily.

https://github.com/opnsense/src/commit/4b21e5691ff

We don't use ISOs.... we build our own HardenedBSD from this exact commit for example.

Not sure what you mean about the version change. If we switch from 18.7 to 19.1 eventually all packages are rebuilt and updated during the major upgrade for ABI safety.

2.

It doesn't matter much if you use 19.1 or 18.7. Since 19.1 (11.2) isn't out yet for us you may run into issues during package builds. Since 18.7 (11.1) is EoL in FreeBSD you may run into issues. Overall both should work fine. Happier if you stick with 19.1 maybe to make your work future-proof.

3.

"make update SETTINGS=19.1" will take care of pulling in the correct ports tree. Check your /usr/ports

It's always *our* repos, that's why we have them and where we tag.

4.

BETA and DEVEL don't have explicit version tags. They construct from the latest tag + commits since, e.g 19.1.b_4

Remember, you are building continuous versions a.k.a. snapshots.

5.

Never mind the tags... 19.1.a is just an annotation that 18.7 development stopped and 19.1 development started. You can build from the latest commit of the branch with no issue.

ISO is not really required as you only need the packages set to be able to install elasticsearch5 in a running system or put the packages into your own mirror and use the GUI firmware settings mirror to point to it for updates.


Cheers,
Franco

[egghunter05]

Hello Franco,

1.

I want to build packages on the correct platform for 19.1 release, before you release it. So I understand that I need to compile opnsense master branch when its state came to 19.1 release candidate.

When Opnsense's src repo (base system and kernel) will be ready for opnsense 19.1 release candidate? Or is it ready now? (base system and kernel)

If it s not ready how should I test /usr/ports package creation tasks for 19.1 release. On which platform?

2.

You have 19.1b iso file. Do you think that I should use it for package creation tests of 19.1 release? If you confirm, I will checkout /usr/ports and I will try to run "make package-recursive" for elasticsearch5 on 19.1b system.

If it is suitable platform for the release (base system and kernel). I won't waste time with compiling whole OS.

3.

I think I should install latest state of master branch on opnsense/src repo for creating 19.1 beta platform.

How can I do that by using build system on the "FreeBSD 11.1"?

I tried to do that by using:

Code: [Select]

make update SETTINGS=19.1
make release VERSION=19.1

commands.

I got some errors with these commands. If my method is correct, and if you want I can share errors with you. If my method is not correct can you share correct commands?

4.

What kind of platform are you using for build task, each rebuild task is taking too much time you know

Thank you

Kind regards

[franco]

1.

It's ready for use. It's not yet RC time, which only means there are 0 - x patches missing. We don't know how many, but the OS is up to date and building and working fine.

2.

Since we go from 11.1 to 11.2 the ISO has 18.7 packages which are fully compatible, but you can build them against 19.1 too. These will not be compatible to 18.7, but in your case it doesn't matter. For the BETA images it does so they can keep updating from the new 18.7.x releases.

It's the best option for you to build 19.1 to be future proof, yes.

3.

ALWAYS set SETTINGS=19.1 .. you can also edit tools.git/Makefile to reflect that. I don't know how to stress this any more.... SETTINGS=18.7 is implied, but if you want to diverge ALWAYS set SETTINGS=19.1 either explicitly or implicitly. "make release VERSION=19.1" is wrong on several levels. What you meant to do was "make release SETTINGS=19.1". Avoid VERSION, it's for numerically matching version tags which do not exist for 19.1....

If you build 19.1 11.1 build system is ok, but you could also use 11.2. In that case it's harder to build 18.7, see 2 for a similar explanation.

4.

Normal builds take 5-7 hours to complete. Yep. https://nightly.opnsense.org/


Cheers,
Franco

[egghunter05]

Hello Franco,

I know I'm bugging you, but looks like I need more help.

I edited tools.git/Makefile and changed SETTINGS variable to 19.1 and when I run

Code: [Select]

make update
make release

command I got "Package file lists do not match" error. As per the on-screen instructions,
I executed "cd /usr/core; make plist-fix" command. This also did not fix the error.


I'm attaching some screenshots. Maybe they can shed more light.

I will try to delete src, tools, plugins, ports and core directories and I will try to build it again from scratch.

Maybe you advise another solution so I wanted to ask.

Your help much appreciated.

Cheers

[franco]

Hi egghunter05,

No worries, almost there.

"release" is just a composite target that runs "dvd nano serial vga" and then executes "release.sh" to add all those images to a file with checksums and such.

As it is intended for building actual versions there is a failsafe in there when the "opnsense" package is not for the correct release version. If you don't want the release set -- just all images -- substitute the targets or for the unversioned release set remove this one "exit 1" to continue anyway:

https://github.com/opnsense/tools/blob/master/build/release.sh#L49

The core packaging issue was fixed via https://github.com/opnsense/core/commit/e494ebd94e

We use "plist" for QA purposes so we make sure that all included files are properly packaged and also visibly registered as packaged files.


Cheers,
Franco

[egghunter05]

Hello Franco

I have generated packages by using "make ports" command.

1.

I see these files on SETSDIR directory.

Code: [Select]

ls /usr/local/opnsense/build/19.1/amdb64/sets/
base-19.1.b_19-amd64.obsolete
base-19.1.b_19-amd64.txz
kernel-19.1.b_19-amd64.txz
packages-18.7.8_138-OpenSSL-amd64.tar

Looks like base and kernel packages are ready, but I am not sure about plugins, core build outputs.

Ports are ready on /usr/ports

Do you think that there is a missing file on sets directory?

2.

Without having an iso installation image file, how can I generate opnsense platform (including os) on a running freebsd 11.x system?

I am not sure but I have two ideas.

a) First one is installing it by using make buildkernel,buildworld,installkernel,installworlds commands.
https://people.freebsd.org/~rodrigc/doc/handbook/makeworld.html

following guide for building world and kernel on /usr/src directory.

b) Installing files in the /usr/local/opnsense/build/19.1/amdb64/sets/ directory.
If so how should install those files?

base-19.1.b_19-amd64.txz
kernel-19.1.b_19-amd64.txz
packages-18.7.8_138-OpenSSL-amd64.tar

I think I can install *.txz files by using pkg tool. What about packages-*.tar?
Is there a missing file?

3.

I built for 19.1 but package filename (packages-18.7.8_138-OpenSSL-amd64.tar
) includes 18.7.8, is that normal?


4.

I see comment char at the begining of the line which includes elasticsearch5 in config/19.1/ports.conf file.

If the futureproof is successful, we should remove comment char only and commit it. That will be enough for adding elasticsearch and its depends to installation right?

Cheers