Building a specific OPNsense version (Git tag)

[Maurice]

Hello everyone,

I've recently built lots of OPNsense images. Took me a while to understand targets, devices, extras etc., but I think I've got it now. With one exception: versions. I'm struggling to reproducibly build a specific OPNsense version (like Git tag 23.7).

According to the opnsense/tools readme:

   VERSION: a version tag

So building a 23.7 DVD should work like this:

Code: [Select]

# cd /usr
# git clone https://github.com/opnsense/tools
# cd tools
# make update dvd VERSION=23.7

The resulting image is named OPNsense-23.7-dvd-amd64.iso, but identifies as OPNsense 23.7_15. And make info says:

Code: [Select]

tools /usr/tools 23.7_1 5616784d9 master
src /usr/src 23.7_6 6cf2e77cb stable/23.7
ports /usr/ports 23.7_81 e9b5a0ed7 master
plugins /usr/plugins 23.7 f183c06d8 stable/23.7
core /usr/core 23.7_15 0ff09cab7 stable/23.7

It seems the heads of the master and stable/23.7 branches are used, not the tag 23.7. Probably a pebkac, maybe someone can shed some light on this.

Cheers
Maurice

[franco]

Hi Maurice,

Well in general the build works only with the checked out (and appropriately tagged) repositories in place. VERSION is an override that offers multiple interpretations depending on which build step is being invoked.

That being said you want to check out each repository to show the appropriate tag or the nearest one below it...

# make update
[...]
# make info
tools /usr/tools 23.7_1 5616784d9 master
src /usr/src 23.7_6 6cf2e77cb stable/23.7
ports /usr/ports 23.7_81 e9b5a0ed7 master
plugins /usr/plugins 23.7 f183c06d8 stable/23.7
core /usr/core 23.7_15 0ff09cab7 stable/23.7

And here comes the important bit which is sadly undocumented because I don't personally use it although I thought it would be helpful...

# make rewind VERSION=23.7
[...]
# make info
tools /usr/tools 23.7 fc7960e36 master
src /usr/src 23.7 f223233ee stable/23.7
ports /usr/ports 23.7 dc1e7fe65 master
plugins /usr/plugins 23.7 f183c06d8 stable/23.7
core /usr/core 23.7 b35678139 stable/23.7

While rewind does tag approximation I'm not sure if it's perfect just yet.

After that you stay away from update and VERSION use until the correct version has been built.


Cheers,
Franco

[Maurice]

Thanks a lot Franco, rewind was the missing step! I would have never figured this out.

VERSION is an override that offers multiple interpretations depending on which build step is being invoked.

So rewind seems to be the only build step where VERSION is actually interpreted as a Git tag?

And here comes the important bit which is sadly undocumented because I don't personally use it

Which makes me curious what your process is. Tag the latest commits in the Git branches, then immediately build the images before any other commits are being made?

After that you stay away from update and VERSION use until the correct version has been built.

I get staying away from update because it doesn't care about Git tags and would pull the branch heads again. But what's wrong with running make dvd VERSION=23.7 after rewinding? This seems to work fine for me:

Code: [Select]

make update rewind dvd VERSION=23.7
Cheers
Maurice

[franco]

So rewind seems to be the only build step where VERSION is actually interpreted as a Git tag?

More or less, yes. VERSION is just a tool for when you don't want to steer the VERSION from an automatically generated date-based version (images) or Git tag (sets).

Which makes me curious what your process is. Tag the latest commits in the Git branches, then immediately build the images before any other commits are being made?

Yes, I do have the benefit that I'm the one tagging so my Git repos are mostly the top commits anyway.

I get staying away from update because it doesn't care about Git tags and would pull the branch heads again. But what's wrong with running make dvd VERSION=23.7 after rewinding? This seems to work fine for me:

Code: [Select]

make update rewind dvd VERSION=23.7

Nothing wrong with that except when you are on a tag mismatch you overwrite the set names with VERSION leading to a state misrepresenting the actual tags that would have been used.

Looking at your command: maybe we should fold "rewind" into "update" when VERSION is explicitly set?


Cheers,
Franco

[franco]

PS: The composite targets "distribution" and "factory" do offer some insight in how VERSION is intended to be used with images to set the correct file name.

[Maurice]

Yes, I do have the benefit that I'm the one tagging so my Git repos are mostly the top commits anyway.

Understandable, though not using your own tools can have its downsides.

Looking at your command: maybe we should fold "rewind" into "update" when VERSION is explicitly set?

Yes, that would be nice and exactly how I originally assumed update would behave. My bad. Never assume anything. Someday I'll learn. Maybe.

The composite targets "distribution" and "factory" do offer some insight in how VERSION is intended to be used with images to set the correct file name.

Nothing says "well documented" like "just look at the code".
Just kidding. I did take a look at factory before, but not distribution. Thanks for the pointer!

Cheers
Maurice

[franco]

I made a reminder ticket https://github.com/opnsense/tools/issues/373

In the case of rewind it's difficult to use it, because it breaks when building the OPNsense release with development packages added as it requires EXTRABRANCHES which screws with checking out a specific tag...


Cheers,
Franco

[Maurice]

Thanks a lot Franco, amazingly quick!

Which brings me to the next question: src doesn't have tags for all OPNsense releases, e.g. there are no 23.1.9, 23.1.10 or 23.7.1 tags. Do these builds use the previous Git tag or do they include some untagged commits, too?

Code: [Select]

root@freebsd:/usr/tools # make update VERSION=23.7.1
[...]
root@freebsd:/usr/tools # make info
tools /usr/tools 23.7.1 d83d6ab91 master
src /usr/src 23.7 f223233ee stable/23.7
ports /usr/ports 23.7.1 e220eb52d master
plugins /usr/plugins 23.7.1 cc4ed826f stable/23.7
core /usr/core 23.7.1 2c6483500 stable/23.7
So it would use src tag 23.7 for a 23.7.1 build. Is that how it's supposed to be?

Cheers
Maurice

[franco]

Well, src.git 23.7.1 wasn't pushed properly it seems so I fixed that. Thanks for noticing!

In general that's what I meant by "tag approximation" in terms of why "rewind" existed and why it's a bit more complicated. It selects the nearest historic tag. In this case 23.7 which was correct at least from the technical perspective at the time the tag 23.7.1 wasn't there.


Cheers,
Franco

[Maurice]

Thanks for fixing!

[Maurice]

Franco, could you please push the 23.7.5 tags to GitHub?

Thanks!
Maurice

[franco]

Sure, forgot it yesterday


Cheers,
Franco

[Maurice]

23.7.6 tags, pleeease. ;-) I'd like to build the aarch64 packages over night.

Generaly speaking, is there an established timeline for binary releases vs. git tags?

Thanks
Maurice

[franco]

Pushed. I normally push them after publishing the release notes but haven't been on a roll in this regard lately. Sorry!


Cheers,
Franco

[Maurice]

No worries! Build underway.

Cheers
Maurice