Rolling Updates / Releases versus Stable 18.1 Release?

[mestafin]

Hi,

I do appreciate the willingness of the OPNsense team to fix issues and to respond to any bug reports in a timely (weekly) manner, but it also create some uneasiness.

When is 18.1 Production Series stable enough to install on a critical production site?

Is there not a need for a more "stable" release with a list of known issues that can be used on production sites provided you can live with the the known issues?

The frequent 18.1 releases is a just too "bleeding edge" for me, or am I just too conservative?

[elektroinside]

Agreed. I was actually thinking to propose this myself.
18.1.4 would be the first version in the 18 series that I think could be cataloged as a strong RC.

If I could have a say in this, for a release, I would confirm as bugs & fix these, in this order:
1. Installer issues: without a doubt, there are some -> first user impressions -> can't install -> can't use
2. Dependencies: verify & fix
3. PPPoE issues: no net -> no use in many cases
4. Routing issues involving VLANs -> can't confirm yet if there are any; personally, I didn't get to test VLANs just yet with OPNsense; many people that use custom firewalls/gateways will probably want this to work flawlessly
5. Other stuff I probably don't know about

Calling these service releases beta versions or release candidates has an added benefit:
1. People will accept the fact there are bugs in these versions
2. OPNsense's public image and trustworthiness will not be so affected if major things break during this development phase (if it's the case)
3. Frequent beta's and RCs gives a very positive feeling to users

But, this recipe might need/involve a few things:
1. Users to test betas and RCs in various environments
2. Do not retire the previous stable version until a new stable version is out and confirmed as stable for some time
3. Beta/RC users to report issues on Github, not the forum (mandatory)
4. The rest of things can be dynamically integrated over the course of implementation

[mimugmail]

Hi,

I do appreciate the willingness of the OPNsense team to fix issues and to respond to any bug reports in a timely (weekly) manner, but it also create some uneasiness.

When is 18.1 Production Series stable enough to install on a critical production site?

Is there not a need for a more "stable" release with a list of known issues that can be used on production sites provided you can live with the the known issues?

The frequent 18.1 releases is a just too "bleeding edge" for me, or am I just too conservative?

You mean something like a community edition and a business edition (with some license fees)?

[franco]

Hi there,

When is 18.1 Production Series stable enough to install on a critical production site?

When is it secure? Or stable? The answer depends on the expectations. Some agree to production status, some disagree to whatever is answered or shipped. If it works for you it's enough. :)

Is there not a need for a more "stable" release with a list of known issues that can be used on production sites provided you can live with the the known issues?

Not within an open source scope. There was an idea for https://forum.opnsense.org/index.php?topic=3408.0 but that hasn't been made a reality yet.

BTW: Updates in 18.1.x are not rolling, that's why we have two major releases per year.


Cheers,
Franco

[elektroinside]

When is it secure? Or stable? The answer depends on the expectations. Some agree to production status, some disagree to whatever is answered or shipped. If it works for you it's enough. :)

This is how we usually do it (1 year plan):
- finish scheduled bugfixes (in a specific order)
- finish scheduled features (feature complete)
- finish adding new automated tests
- tests & bugfixes
- code freeze
- start a new set of automated & manual testing
- if no critical issues found, release a beta with release notes (known issues included)
- some more tests & bugfixes
- include user's feedback (bugs, no features)
- release another beta, probably a RC
- more tests & bugfixes (internal + user's feedback)
- release the major version (first stable)
- bugfixes (in parallel, new features development on another branch)
- code freeze
- more tests
- release a SR (second stable)
- release another beta (from the new branch)
- release another SR, but only if necessary (old branch, third stable)

[franco]

I see your point. Is it realistic to do this twice a year?

Smaller iterations are "safer" as less code is turned over in the new version. Larger iterations are "safer" because less code is turned over in the old version.

Extrapolating from feedback over the years, two releases a year seem rushed or premature. In other software, the same happens in 2-3 year cycles. Sometimes I wonder if anyone likes change at all. And if 80% is ok any way, we must go back to the other 20% and keep iterating.

Maybe this isn't appropriate for commercial software, or maybe that is the point to offer a real "USP" to open source that tries to not lack behind?

The point is: tweaking 18.1.x or 18.7.x is not going to solve the issues that others are seeing. We need to talk about these issues from a broad perspective that includes time and effort on both sides. E.g. another stable version in open source terms is not viable.


Cheers,
Franco

[elektroinside]

It's not the numbers that matters, it's how you do it, IMO.
I certainly have no experience with opensource projects. But I don't think that really matters, minor details here and there, the audience is the same. I could be wrong of course.

I'm trying to help, not to judge. So here it goes:

Frequent release cycles are fine as well, if you could define (the OPNsense way) "stable". You have to define it at some point and stick to it if it works. If not, dig deeper and find a way.

I know you rely heavily on FreeBSD and their releases and commits, which is also fine as long as you stick to your defined "stable". These three months with you guys have been a new experience for me, and coming from the outside, I somehow feel that "stable" is not yet defined. Maybe you don't want too, it's one way to macro manage a product, which sometimes works in such a dynamic and ever changing software. But it's a security product at the end of the day, it's not just a web interface on top of an existing OS, even if it's opensource. The concept of "stable" is most probably arguable, but I don't think it cannot be defined, the OPNsense way.

I know it's difficult to work with people, everybody is an expert in everything and everybody knows better. I also know the reality is totally different and only a handful of people actually deliver something really valuable. However, I learned the hard way that clearly defining some stuff is for the greater good and allowing users to do that for you (or try) is not necessarily that good. The issue here I think is exactly that we are having this conversation (IMO). We shouldn't :-)

Personally, I am not convinced that a clearly defined stable channel is not viable. Call the other frequent releases something else, don't call them alpha, beta, sr, rc etc, invent something else, like feature release or anything else, whatever works is fine and pin it to the forum and/or website.

Ok, I'll stop talking about this as it is a sensitive area and I probably already said too much. Sorry if I made anybody uncomfortable, I meant well. I'll keep trying to help, as always. Take into consideration whatever you think is worthy, or don't, no hard feelings here :-)

[mimugmail]

Believe me, the guys feeling responsible can go fine with critism :) Just go ahead, if 1% of it will respected you influenced the process by x percent  :P

If franco and the other core devs would only go stable this firewall would be without any features.
I'm way no programmer and have no idea of it. I can do some basic if/then/else and perfectly copy+paste, and with this I've done around 12 plugins. The bad side is I'm a network engineer, I'm not that good when it comes to Q&A and that's why there will be probably some bugs with updates.

[NOYB]

I like the two major releases per year model.  But reserve the yy.1/7.x (dot x releases) for security and critical functionality only.  That would be my definition of "stable".

As the product becomes more mature and development beat rates slows then maybe yearly major releases.

[elektroinside]

Stable is good, features are good. Why not combine these somehow so that everybody is free to choose whichever? For production, one could choose the stable release, for features, one could choose the less stable but more feature rich release. When some features get stable enough, add them to the stable release and so forth. Damn it, i'm talking again :p

[mimugmail]

Good idea! But this is extra work and I don't know who should do this since everyone does the development on their spare time. Only Deciso works for it, but since OPNsense is responsible for their income I'm not really sure they will give their extra time for a stable distribution for free.

[Davesworld]

Hi,

I do appreciate the willingness of the OPNsense team to fix issues and to respond to any bug reports in a timely (weekly) manner, but it also create some uneasiness.

When is 18.1 Production Series stable enough to install on a critical production site?

Is there not a need for a more "stable" release with a list of known issues that can be used on production sites provided you can live with the the known issues?

The frequent 18.1 releases is a just too "bleeding edge" for me, or am I just too conservative?

You mean something like a community edition and a business edition (with some license fees)?

Oh hell no! I can't stand using crippled or beta distros because I cannot pay for a fully functional release copy, Endian which I had used for months is exactly this way. I wound up sending patches and changes but did I get access to the stable version with all the features? Nope. I was just testing beta quality yet crippled software.

Anyone can switch their updates to go from stable to bleeding edge in OPNsense, it's a feature recently added. These ARE stable releases. The community edition model would be akin to being forced to use the development version unless you are paying thousands and no other choice. Anyone looking for something 100% stable and bug free, get back to me in about 5 billion years just before the Sun is at the end of it's Star life and tell me how it went.

I'm fine with it just as it is. Two majors with minor security and fixes updates between. We just got 18.1, we have some new features. We likely will not see any changes in features if at all until this Summer. Even software costing tens of thousands a year for licenses has unknown bugs that may show up at some point for someone. The openness and powerful feature set where we all get the same stable version whether we are a Corp with a million dollar maintenance contract with Deciso, is exactly why I looked into OPNsense after years of Linux firewalls and a stint with PFsense. I'm getting up to speed with the differences between BSD and Linux but many programs by CLI are the same. 

[Davesworld]

The frequent 18.1 releases is a just too "bleeding edge" for me, or am I just too conservative?

Nope, you just misread the entire situation and badly. These ARE production ready releases, unless you changed your settings in updates to use development snapshots which are bleeding edge testing versions. What version do you think Deciso is using on client's networks? Some of these clients being major players with scaled up networks most of us couldn't even imagine having. At least seven that I know of.

The premise that something is only stable if there are infrequent security and bugfix updates is very misguided. If this were true, IPCop which is in stasis and has not had a cvs commit in nearly two years would be the most stable distribution ever conceived. Actually the IPCop.org domain only has several months left before it will be discontinued. This distro is not small time nor is 18.1 a testbed for something else. You are not a guinea pig unless you switch your update settings to devel snaphots. This can easily be reversed.

I really hope silly ideas that will ruin everything that brought so many of us here are not considered.

[elektroinside]

Good idea! But this is extra work and I don't know who should do this since everyone does the development on their spare time. Only Deciso works for it, but since OPNsense is responsible for their income I'm not really sure they will give their extra time for a stable distribution for free.

Yep, I'm aware.. but because I don't know anything about their resources, I popped the idea :-) I would help, but I'm not a BSD/Linux guy. Very limited knowledge.

[Davesworld]

When is it secure? Or stable? The answer depends on the expectations. Some agree to production status, some disagree to whatever is answered or shipped. If it works for you it's enough. :)

This is how we usually do it (1 year plan):
- finish scheduled bugfixes (in a specific order)
- finish scheduled features (feature complete)
- finish adding new automated tests
- tests & bugfixes
- code freeze
- start a new set of automated & manual testing
- if no critical issues found, release a beta with release notes (known issues included)
- some more tests & bugfixes
- include user's feedback (bugs, no features)
- release another beta, probably a RC
- more tests & bugfixes (internal + user's feedback)
- release the major version (first stable)
- bugfixes (in parallel, new features development on another branch)
- code freeze
- more tests
- release a SR (second stable)
- release another beta (from the new branch)
- release another SR, but only if necessary (old branch, third stable)

Tests tests and more tests, the Sun will burn out before anything bug free is ever released by anyone. If Architecture was as complex as millions of lines of code with each fix possibly creating another problem no one will run into for maybe months or years, a single woodpecker could destroy civilization. A Stable version of Windows often has 60,000 known bugs. OS X had nearly that many when it came out. Even so it was considered stable and most people were not affected by those bugs much of the time.

I like the OPNsense model exactly as it is. Other offerings from Companies that would have you think they are ironed out and bulletproof and cost a lot of money to use for even 50 users per year, are simply not telling you everything. These guys are very transparent and forthcoming and weird obscure problems like I had recently are taken seriously. I've not seen any dstro polish up and improve in three years like this one has.