16.1 Development Milestones

[franco]

Why hello there,

It's almost time. A lot has happened. We are super excited. And we're definitely on time with 16.1.

Most of the additions have already been rolled out, while some have not. The menu and layout rework has been carried out and moved to 15.7 for early access. It's been a tremendous switch from the previous major release in terms of look and feel, moving through the GUI is way more consistent and efficent. One could say the GUI got out of the way to enable users to do what they want. There's also firmware plugin management support now. And probably something important that we forget.

On the other hand, the captive portal implementation switch is imminent and FreeBSD 10.2 underneath will help newer hardware to run more smoothly. Translations diverged and progressed quite a bit in the development version, it was impossible to merge it back into 15.7 without losing half of it, but it'll be worth the wait with many additions for German and French.

And here is a thorough list of key points:

o switched to FreeBSD 10.2 for latest driver support
o seamless firmware transition from OpenSSL to LibreSSL and back
o use the more flexible tmpfs instead of mfs for memory disk mode
o fine-grained firmware packages and plugins management
o parallel opnsense-devel package for development previews
o redesigned the menu presentation for clarity and consistency (almost no more tabs, status and diagnostic settings merged into their respective place in system, vpn, firewall and services)
o layout: reduced the previously excessive padding and removed spurious container wrappings from forms
o log pages usability has been improved by providing a useful tag type search filter to drill down into the log contents
o firmware mirror selection support
o revised LDAP bindings and user import
o improved the crash reporter to be a general tool for direct bug submission
o revised and refreshed all System, Interfaces, VPN and Firewall pages
o only 3 images for all of 15.7 instead of one per minor release  (3 images vs. 26 releases total says a lot about stability and security in 6 months, maybe we can get this down to 1 image in all of 16.1)
o added hotplug support for the menu and page access control
o replaced RRD graph frontend with a modern and flexible D3.js alternative
o greatly improved the usability of the translation
o added a central hub for translation contributions at https://translate.opnsense.org
o improved overall security of the code e.g. by fixing https://www.exploit-db.com/exploits/39038/ a few months earlier than announced
o rewrote the captive portal using new components and better sandboxing + authentication/accounting
o plugins for VMware and Xen for seamless guest integration
o added the simple rc.syshook framework for persistent service start/stop and custom scripting
o introduced a pluggable authentication backend for easier integration of new methods
o steady stream of French and German language updates
o the API gained a machine key authentication mechanism
o new IPS support using FreeBDS's netmap and the latest and greatest Suricata 3.0
o introduced the opnsense-bootstrap utility which can transform a stock FreeBSD securely into OPNsense
o assorted user experience treatments in the firewall section
o introduced opnsense-sign and opnsense-verify to tie arbitrary file signing directly to FreeBSD's pkg readily available key store mechanisms
o rolled out opnsense-update using the new fingerprint verification for kernel and base upgrades
o the nifty quick search feature! (<tab> -- type -- <enter> -- done)
o unbound DNS resolver now supports MX records
o automatic PHP extensions detection for plugins or custom additions
o compressed blacklist support for the proxy server


Feel free to discuss, comment or ask questions. We'd love to hear what you think (and still miss).


Cheers,
Ad, Franco and Jos

[fabian]

I would like to say thanks for your great work.

[interfaSys]

I'm confused by this:

improved overall security of the code e.g. by fixing https://www.exploit-db.com/exploits/39038/ a few months earlier than announced

There is an exploit in the wild and the current release version hasn't been patched, but the dev version has? And the original plan was to wait a few months?

I'm new to the project and trying to understand how I would patch our instance against 0days.

[Aadolf]

Great Work...
Thanks.

[franco]

There is an exploit in the wild and the current release version hasn't been patched, but the dev version has? And the original plan was to wait a few months?

I'm new to the project and trying to understand how I would patch our instance against 0days.

Well, there are two types of releases... one is the development release, the other one is the stable release. When a stable release is out, the development version is updated as well. This gives us the opportunity to try new features or tricky patches without jeopardising the stable version. Usually, after a release or two, the changes from the development version are moved to the stable version as well. The list you're seeing here is partially integrated in the 15.7 series for this reason.

For security and bug fixes we go straight for the release version so simply staying up to date with 15.7.x (or 16.1.x  soon enough) will be all one has to do as a user in order to stay safe.

With that being said since we are a pfSense fork we shared a lot of the same code base at one point, so generally this LFI vulnerability must be checked against in our code. However, the vulnerability in question was released in December and has since been fixed by pfSense, but was fixed in OPNsense in June and September, respectively, as part of general sanity cleanups.

https://github.com/opnsense/core/commit/43ae21efc3cfff404
https://github.com/opnsense/core/commit/f5eb5ea80e27a79

It stands as a good example for the how we've cleaned up the code since we've forked. There have been many more such cases, too many to track or tie to explicit vulnerabilities.

I hope this helps.

[interfaSys]

Thank you very much for the clarification. It was just something which got lost in translation. I read it like you planned on fixing the problem in a few months rather than "it's been fixed for months"

I have a suggestion for your release notes. I know it's painful to identify every change as a bug fix (-) or a new feature (+) and it's best to spend your time coding , but could you at least mention if a fix was security related? Something like

Code: [Select]

* [security] ports: ntp 4.2.8p5[7]

* ports: suricata 2.0.11[2], dhcp6 20080615_5[3], lighttpd 1.4.39[4]
* ports: syslogd 10.2, mpd 5.8[5], ca_root_nss 3.21, dnsmasq 2.75_1[6]
* ports: php 5.6.17[8], python 2.7.11_1[9]
* ports: miniupnpd 1.9.20151212, openvpn 2.3.10[10]
* opnsense-update: add opnsense-verify and opnsense-sign
* opnsense-update: improve verification of signatures of kernel and base upgrades
* menu: bring back dashboard entry due to popular demand
...
I think it would make it easier to quickly assess the risk of running the current version.

[franco]

I used to do it, but it takes a lot of work to extract CVE's, review code and potential issues. The third party software now provides thorough and meaningful links to release notes, which basically means these are the security-related topics. I repeat, if I see a security-related issue, I will add the link.

Assessing our code is harder, but I will try to make the bugfix and feature distinction clearer. Thanks for the suggestion.

[interfaSys]

OK, thanks

[interfaSys]

and FreeBSD 10.2 underneath will help newer hardware to run more smoothly

uname tells me I'm still on 10.1. Is 10.2 coming later?

[franco]

16.1 is still scheduled for January 28 (hence "development series"), see https://opnsense.org/about/road-map/

We're not yet ready to provide reliable test kernels for development releases, but everything is backwards compatible. One of the biggest hurdles is that downgrading the base/kernel is not a supported transition as well as having to build multiple package mirrors for different OS versions. Too much strain for our project at the moment to handle.

PS: If you want to try 10.2, PM me for details.

[interfaSys]

16.1 is still scheduled for January 28 (hence "development series"), see https://opnsense.org/about/road-map/

I know, but it's really soon and people should be testing the new kernel as well, no?

We're not yet ready to provide reliable test kernels for development releases

OK, then, no, we don't want to test it if it crashes too often

as well as having to build multiple package mirrors for different OS versions. Too much strain for our project at the moment to handle.

I understand, so it's quite risky to update to a fresh image then as packages will not have been well tested outside of your team. I guess that's when the beta phase starts.

Thanks!

[franco]

I've been too unclear about this, sorry. We actually wrapped up 10.2 testing back in October, thinking we could skip it entirely back then, but some driver issues appeared in December that make the switch more interesting so we decided to revive the effort for 16.1.

https://forum.opnsense.org/index.php?topic=1302.0

The current test version can be installed like this:

# opnsense-update -bkr 15.7_38 && /usr/local/etc/rc.reboot

This version does not have the latest security patches and upgrading to this version may prevent you from switching back to 10.1 or cause your system to get unstable if you try. It gets more complicated, because the update utility will reinstall the old 10.1 if an update requires it. There's a lot of work needed underneath, FreeBSD moving is in a different direction and then we're stuck here because every effort that we put in making our solution better is one step further away from FreeBSD's future. We don't want that too.

I said "reliable test kernels" but I meant "frequently available test kernels". 10.2 is definitely stable.

[interfaSys]

OK, so I read that thread and, indeed, it seems 11 is the one "we" want if we want newer drivers and stacks...

I was hoping to test a 16.1/10.2 which would have a parallel life to 15.7/10.1, but if it doesn't have the latest patches, then I'll stick to 15.7.99

[franco]

I'll build the new FreeBSD 10.2 based on the latest patches till tomorrow. This is likely the one that will be in 16.1, minus the OPNsense branding, which is still in the pipe (minor boot loader stuff). Deal?

[interfaSys]

I'll build the new FreeBSD 10.2 based on the latest patches till tomorrow. This is likely the one that will be in 16.1, minus the OPNsense branding, which is still in the pipe (minor boot loader stuff). Deal?

Works for me