OPNsense Forum

English Forums => Development and Code Review => Topic started by: Fabricio on April 16, 2018, 08:56:35 pm

Title: Building from Sources - General Questions
Post by: Fabricio on April 16, 2018, 08:56:35 pm
Hello OPN family,
I am just starting here after several years and "frustrations" with the current situation at the "pfsense".
I think at this point I don't need to explain what happened. Everybody knows that it is NOT an opensource anymore.
After version 2.4.3 I can't even compile my own customized kernel anymore.
After all these years, contributing and supporting the product...
That's when a good friend advised me to forget about it and come to OPN family.
I accepted the challenge and here I am.
I have to confess...I am totally amazed with the things I see here. I didn't know the OPN was sooooooooo advanced.  W.O.W.
I was keeping a customized fork of pfs for years with proprietary authentication (winbind) and recently developing a new authentication mechanism using WMI (wmic - wmi client) since I have very good expertise and background over MS WMI queries/protocol. That's when I got SERIOUS problems with it. I got locked/stuck. I can't even compile my custom kernel (Yes, I need one to support my proprietary LCD based on Arduino - uchcom driver with NIS).
So, before I start shooting repetitive questions, here is a doub:
Is there a doc/how-to with steps on how to build OPNSense from the sources? I will probably have to fork the github and customize it according to my needs. Any help will be greatly appreciated.

Thanks Much!
Fabricio.
Title: Re: Building from Sources - General Questions
Post by: fabian on April 16, 2018, 09:28:21 pm
you only need some make commands: https://github.com/opnsense/tools#how-to-run-individual-or-composite-build-steps
Title: Re: Building from Sources - General Questions
Post by: franco on April 16, 2018, 10:54:06 pm
Hi Fabricio,

Thank you for the warm words. You will find all the source code on GitHub, and the README for the tools that Fabian posted will get you started with the build process.

The whole thing should just work, follow these simple steps to build yourself a full build environment:

https://github.com/opnsense/tools#setting-up-a-build-system

There are shortcuts and caching mechanism, but it will become clearer when you start modifying /usr/core or possibly /usr/plugins for your needs. The same goes for the kernel/base under /usr/src.

If you have any questions let us know here or via raising tickets (even if just a support question or problem with a build run with an error message).

Welcome to the family. :)


Cheers,
Franco
Title: Re: Building from Sources - General Questions
Post by: Fabricio on April 16, 2018, 11:11:33 pm
Hi Franco,
It's me who have to thank such warm message.
I will start working on it immediately. Probably will bring doubts later.
Let me know (in private if you prefer) on how to contribute with the project (financial and intellectually)

Thanks Much!
Fabricio.
Title: Re: Building from Sources - General Questions
Post by: franco on April 16, 2018, 11:25:21 pm
I've sent you a mail to discuss expectations. :)


Cheers,
Franco
Title: Re: Building from Sources - General Questions
Post by: Fabricio on April 17, 2018, 06:50:26 pm
Thanks Franco, thanks Fabian.
I have responded to your e-mail Franco. Thanks a lot.

All right. Before I start trying to customize anything, I am following the link sent by Fabian (the way it is FIRST, so I can get familiar with the steps/processes) and here is what I got.
****************************************************
background OS-->  root@opnsource:/usr/tools # uname -a
FreeBSD opnsource 11.1-RELEASE-p9 FreeBSD 11.1-RELEASE-p9 #0: Tue Apr  3 16:59:16 UTC 2018     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

Clang--> root@opnsource:/usr/tools # clang -v
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
Target: x86_64-unknown-freebsd11.1
Thread model: posix
InstalledDir: /usr/bin

******************************************
make update--> >>> Removing /usr/ports... rm: /usr/ports: Device busy
*** Error code 1
Stop.
make: stopped in /usr/tools
******************************************

I found out that I can't use ZFS. After rebuilding the BASE-server with UFS, everything went OK.

Now building the image.

Thanks Much!

Fabricio.
Title: Re: Building from Sources - General Questions
Post by: franco on April 17, 2018, 08:20:13 pm
Haven't used a ZFS in a while. The tools probably just need a small fix, /usr/... has weird permissions / behaviour on ZFS default install.


Cheers,
Franco
Title: Re: Building from Sources - General Questions
Post by: Fabricio on April 18, 2018, 12:18:33 am
Just curious... How long to build an image from scracth?
I am using a Hyper-V VM with 20 Cores and 12 GB mem, under a physical host with Dual Xeon E5 2667 family (24 Cores) with 32 GB Mem. So long, it took ~5 hours and couting...

I see the script is downloading additional packages in real-time, which takes soooo much time to complete. Wouldn't be the case to run a "portsnap fetch extract" before start the final  "make" ?

Regards
Fabricio
Title: Re: Building from Sources - General Questions
Post by: mimugmail on April 18, 2018, 05:38:34 am
It fetches the current src of OPNsense/FreeBSD and GitHub sometimes is really slow. Next run it only fetches deltas
Title: Re: Building from Sources - General Questions
Post by: franco on April 18, 2018, 07:16:34 am
The build is just that. It takes a long time to build all packages, mainly because we use OpenSSL/LibreSSL from ports and there are no precompiled packages for that.

You can, however, speed up the process with two helper targets:

# make distfiles

This will create a distfiles set with all the source code for port package builds so during "make ports" no files need to be fetched. This only works as long as the ports tree is updated, so the right way to rebuild everything using a caching mechanism is:

# make clean-ports distfiles ports

You can also use the following helper target to skip set builds completely:

# make prefetch-packages,base,kernel VERSION=18.1.6

That corresponds to the contents of the sets/ directory on the mirror... https://pkg.opnsense.org/FreeBSD:11:amd64/18.1/sets/

What this will actually do is build you a DVD image in about 30 seconds minus the time it takes to download all the sources:

# make prefetch-packages,base,kernel dvd VERSION=18.1.6

Last but not least, for everyone building from source it may be useful to tweak ports.conf or create your own config/xxx directory with a lightweight build. Based on this you can use the following to build custom sets nightly complete with build logs and such:

# make nightly SETTINGS=xxx

It takes care of distfiles caching, only rebuilding packages (not base or kernel) and is what we use for online updates in e.g. 18.1.x.


Cheers,
Fracno
Title: Re: Building from Sources - General Questions
Post by: franco on April 18, 2018, 07:18:04 am
PS: On a reasonably sized build system "nightly" with most caching in place from previous runs still runs about 4-5 hours for each flavour (LibreSSL or OpenSSL).
Title: Re: Building from Sources - General Questions
Post by: Fabricio on April 18, 2018, 07:58:57 pm
Thanks Franco, Thanks MIMUGMAIL.

Really great information. I did a first try yesterday. It took no more than 5h:30 mins at all. (from scratch)
Today, I customized some items and added the winbind/samba scripts to the main menu, also a few cosmetic changes. I will only clone the tools again and try the  "make update", maybe it will download the "changed" info only, right? (delta)
I am enjoying this phase.  Really great.

Thanks Much
Fabricio.

Title: Re: Building from Sources - General Questions
Post by: Fabricio on April 18, 2018, 09:46:49 pm
Quick question:
once I have created my first dvd -image, is there a way to update the "bsdinstaller" contents only? or Do I have to update the whole thing? I didn't identify the "bsdinstaller" cloning process during the building.
I think it got the contents from the official opnsense repository, not the one I forked. the other branchs went really fine, it got from my fork.

I bet I commited a mistake during the process.   :-)

Thanks Much,
Fabricio.
Title: Re: Building from Sources - General Questions
Post by: Fabricio on April 19, 2018, 11:08:28 pm
Hello Gentlemen,
Hmmm... trying to understand how to update bsdinstaller (only) without having to update other repos.
Any tip for that?
I already generated a DVD ISO, but I have found I missed some items and I had to change it later. Now I just want to update/clone the bsdinstaller only. after that I will try to re-generate the DVD with the updates.

Thanks!!
Fabricio.
Title: Re: Building from Sources - General Questions
Post by: franco on April 20, 2018, 01:43:21 pm
Hi Fabricio,

The build will try to cache what it can, but there are a few tricks to force rebuild of a particular stage:

# make ports-squid,phalcon

This will rebuild the ports and prior to rebuild delete "squid" and "phalcon" package to rebuild it instead of using a previously compiled version.

If you then want an image with these rebuilds, you can force the image rebuild:

# make dvd-again

"again" is just a non-empty marker for the script to force a new image. Building here will also build "make plugins" and "make core" automatically since "make ports" clears these stages. But that's more of a safety caveat than anything else.

bsdinstaller has its own repo and this is where it gets a bit complicated. What we try to do is make these software packages look normal to the ports tree, so it is built from there to look like a native FreeBSD port / package. Here is our own ports category directory:

https://github.com/opnsense/ports/tree/master/opnsense

Knowledge of FreeBSD ports is required when modifying these packages.


Cheers,
Franco
Title: Re: Building from Sources - General Questions
Post by: Fabricio on April 20, 2018, 04:10:52 pm
Hi Franco.
Thanks Much! It's always clarifying to talk to you guys.

So, I am asking about bsdinstaller because the " OPNsense Installer" is still showing the original Welcome message, MOTD and hourglass logo, even after all the customizations. The only location I could see the "Welcome message and logo" was in the bsdinstaller folder.
That's why I have imagined the scripts are still calling/cloning the original opnsense repositories, not the forked one.
The interesting party is that all the other customizations are working really fine. just the "Installer" is still showing the original messages.

Appreciate the help!

Fabricio.
Title: Re: Building from Sources - General Questions
Post by: franco on April 20, 2018, 04:21:08 pm
Hey Fabricio,

It's simple if you follow this file: https://github.com/opnsense/tools/blob/master/config/18.1/ports.conf

Everything in this file is built from the ports tree /usr/ports during the "ports" build step.

You can see bsdinstaller is listed under opnsense/bsdinstaller, so you need to modify the source tree, particularly the makefile for it:

https://github.com/opnsense/ports/blob/master/opnsense/bsdinstaller/Makefile#L18-L20

Here you see it points to our GitHub location:

https://github.com/opnsense/bsdinstaller

There are two approaches in FreeBSD ports possible: fork the bsdinstaller, adjust ports Makefile accordingly in your local environment *or* add patches to the files/ directory for manual overrides. Unfortunately these two are a bit harder to explain so I have to refer to the porters handbook:

https://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/

But what you can also do is see the history of the opnsense/bsdinstaller directory to get the hang of how it is modified. :)


Cheers,
Franco
Title: Re: Building from Sources - General Questions
Post by: Fabricio on April 20, 2018, 04:30:45 pm
Hi Franco. 
Now I got the point!
I am trying this "or" that, during the day.

I am having great times, believe me. :-)
Once again, I really appreciate the help.

Fabricio.