About the testing of OPNsense

[thowe]

Hello everybody,

first of all: many thanks to all of you contributing to OPNsense. I am impressed how clean and logical everything is presented. Really a great UI and functionality!

I am currently evaluating a firewall for my home installation. The most important things for a firewall to me are: 1. security and 2. reliabilty.

I am software engineer and like build and test automation a lot. One thing I asked myself is: how are releases of OPNsense built and tested? I could not find anything about the testing process. Is there some kind of an automated build and testing process? Maybe someone can give some insights.

Thanks & kind regards,
Tom

[franco]

Hi Tom,

Packages are built on a nightly basis, see

https://github.com/opnsense/tools/blob/master/scripts/nightly.sh

Into that script we also weave the test suite, which is a conglomerate of sanity checks, integrity checks, syntax checks and unit tests:

https://github.com/opnsense/tools/blob/master/build/test.sh

You can find the unit test framework here:

https://github.com/opnsense/core/tree/master/src/opnsense/mvc/tests

New features and reworks are staged on the development (master) branch and we create new stable branches (e.g. stable/17.7) for each major release from said master branch (twice a year). During the minor releases we cherry-pick changes after they have been buffered in the master branch for at least a week unless the fixes are more critical or receive direct field testing.

To improve testing of patches, packages and releases we also wrote a few tools that make reverts possible or pull code from github directly, install custom kernels, etc. There is also a crash reporter to submit PHP errors and kernel crashes directly to us, or leave a note and contact data in case a hard to find issue is at hand.

https://github.com/opnsense/update#opnsense-update-utilities

The tests aren't perfect and can certainly be enhanced into many directions. But it will also keep contributions from being merged or new features being developed. It's a trade off we try to manage.

If you want to know more please let me know.


Cheers,
Franco