Messages

Oh, thanks!  That makes the list a lot longer :)

Hi,

what happened to all the plugins?  Their list has become rather short.

In particular, what happened to the os-iperf plugin?  Is there a replacement for it?

Most migrations happen automatically under the hood without user interaction.

For firewall rules an automatic forced migration would have been far worse than an optional one.

After an upgrade nothing is broken, everything is 100% transparent. You can even compare old and new ruleset before deleting the old rules.

We did the right approach here, your reaction shows nothing would have been appreciated except 100% perfection. But thats not feasible because whatever we would have done, some concepts would have been lossy one way or another.

Having an optional instead of a forced migration is great for the user.

You are putting words into my mouth :(

I've said nowhere that the switch should have been done automatically and that I would expect 100% perfection.  I have also not said anything about wether doing it automatically or manually would be a wrong or a right approach.

I have said, and I keep saying so, that it has been handled badly because there is a lack of information, warning and documentation about this switch.

Besides, if something was handled badly or not doesn't have necessarily have to do anything with picking the right approach or not.  You can pick the right or the wrong approach and still handle it badly or well regardless.

Why don't you just put better information and a warning on the page of the migration assistent so anyone who considers using it doesn't think it'll be an easy transition instead of falsely blaming me?  It's really easy to do.  It's not necessary to make a big deal of it.  Just do it.

Most migrations happen automatically under the hood without user interaction.

For firewall rules an automatic forced migration would have been far worse than an optional one.

After an upgrade nothing is broken, everything is 100% transparent. You can even compare old and new ruleset before deleting the old rules.

We did the right approach here, your reaction shows nothing would have been appreciated except 100% perfection. But thats not feasible because whatever we would have done, some concepts would have been lossy one way or another.

Having an optional instead of a forced migration is great for the user.

You are putting words into my mouth :(

I've said nowhere that the switch should have been done automatically and that I would expect 100% perfection.  I have also not said anything about wether doing it automatically or manually would be a wrong or a right approach.

I have said, and I keep saying so, that it has been handled badly because there is a lack of information, warning and documentation about this switch.

Why don't you just change that instead of falsely blaming me?  It's really easy to do.

I've seen the new rules being there since quite a while, and I didn't want to update.

Then why did you ??

Because these assumptions :

But it looks as if it won't be long before the so-called 'migration assistant' will be gone and you'll suddenly find yourself without a way to switch after an update.

Yes, what else are users supposed to assume?  Assuming something else
would be unreasonable and contradict all experience.  Software changes
fast and all the time.

I already had waited an uncomfortably long time before switching to
the new rules.  I didn't expect a total disaster.  For at least four
years now I've seen opnsense working great.  Despite all the many
updates of several machines, nothing went wrong.  That's quite an
accomplishment.  The developers have done an outstanding job.

So what was I supposed to expect?

Is there a way to switch from DHCP to kea yet?  When will DHCP be gone?  It would really suck if we had to convert everything manually.

Are simply NOT TRUE !!

That's not an assumption but a question.  I'd gladly switch if I
wouldn't have to transfer all the information painstakingly and error
prone by hand from ISC DHCP to kea.  Maybe there will be a 'migration
assistant'.

Moving from ISC to either KEA or DNSmasqd has been discussed in
multiple topics by now which you can find in the 25.7 and 26.1
sections or just use the Forum Search ;)

Again, I don't come to the forum unless I can't avoid it since I was
told to stay away.  The search isn't really helpful.

I just checked the documentation.  It 'strongly suggests' to switch
--- so I'm not wrong this either.

But why doesn't the documentation say when DHCP will go away at the
same place where it tells you to switch?

IMHO it's not that complicated, but you need to know what you are doing, so please also read the information @ https://docs.opnsense.org/

Ok where does it say when DHCP will be gone?  Does 'strongly suggest'
mean it'll be gone soon?  When is 'soon'?

I admit that I don't want to read all the documentation.  It's a lot
to read and I'm assuming that the information is probably not in it
anyway.

Why would developers want to let us define firewall rules on a concept that doesn't exist?

They don't and making such statements isn't going to get you much help from them either...

You misunderstood what I was saying.

I don't understand why group rules would have to be in a specific order, i. e. between floating rules and interface rules.
I can understand that they can be useful in the way I'm using them, but that isn't a good way at all and only due to a lack of better alternatives.

AFAIK it's explaned @ https://docs.opnsense.org/ and you can see there that they all have what are called 'Sequence Numbers' if I am not mistaken :
10000
20000
30000
40000
etc.

Right, and I understand that the rules are numbered so the rules can
easily be sorted by the numbers.  I also understand why the rules
need to be in a desirable order.

That doesn't explain why the rules would have to be in a specific
order.  When a user moves a rule, the rules need to be renumbered
anyway to bring them into the order the user desires them to be in.

So why do group rules have to be between floating rules and interface
rules?  Just renumber the rules according to where the user moved a
rule to so they can be brought into the desired order.

Use just these numbers or create some kind of hybrid between the current system and them...

This is the same question: Why do the rules need to be in a specific
order which then can be different from the order the user may want
them to be in?

'Floating' seems like a kinda random term for the rules that get applied first.
You could as well, or better, call floating rules, group rules and interface rules first rules, second rules and third rules, respectively.

Actually the current system makes a lot of sense to me, but as with everything : You need to read the documentation and get used to new things!

Do you mean the old rules or the new?  There are currently two systems.

Now I do have to admit that I did not do that for a lot of OPNsense things because I knew what I was doing from the start, but for some things you simply have no other choice and IMHO it's not something to complain about either...

I'm not complaining, I'm merely questioning why there is no other and
possibly better choice.  IMO having to create overkill rules isn't a
good design and there's currently no other choice with the new rules.
Or is there?

Perhaps with the new rules, I could omit the floating rules and the
group rules and instead put a whole bunch of interface rules to block
all traffic between all interfaces after allowing what needs to be
allowed.  That would be much worse in that it would be a lot of rules
and very error prone.  Interface groups are very useful.

It's only the inability to put the rules into the desired order that
forces users to make overkill rules.  Change that and there wouldn't
need to be overkill rules.

I always read the notes that are being shown when I'm about to update.  I don't recall any mentioning of the new rules at all, though my memory isn't what it used to be.

Yet if there had been a suitable warning about the new rules like I suggested in my previous post, I would have remembered it.

The message that appears after checking for updates clearly displays it : Your choice to read it or not.

No it doesn't.  The message didn't say not to use the so-called
migration assistant and didn't say that the firewall rules will be
screwed up when using it.

I usually don't look into the forum.  One of the moderators drove me away by telling me to stop posting when I was trying to help, so now I only come here when I can't avoid it.

Sorry you had to go through that...

I know the feeling : I left a forum last year that I was a member of since 2002 because of the same reason!

It's ok; I told him that I'm happy to help, especially when I'm
getting help myself, and that I have no need to spend my time on doing
that.  I have plenty other things to do, and when it's not wanted I'll
just stop.

Generally, I'm done helping, done making bug reports, done
contributing, done asking questions.  It's not wanted and usually
results in being insulted and/or censored or blocked, and almost never
gets any useful answers.  There are few exceptions.  The internet
keeps getting more fiendish and unfriedly all the time and is way past
any reasonable limits in regards to respect and simple decency.

I remember different times.  Just sometimes it can't be avoided to ask
questions.

Besides, people are taking way too long to figure out what I'm saying.

I clearly remember the advice and warnings in the documentation on 26.1, before I tested then fully migrated without issue.

Yep! :)

Which documentation was that?  Like I said, I can only go by what is
being presented when I'm updating, not by posts hidden in some forum
I'm not even supposed to look at.

When you have warnings to give, present them to the users where they
see them.

In general terms, my advice is first allow what you must, then block the rest.

Actually a Firewall should always be : Block First then Allow what you need to allow.

Hm.  Good point.  Ok.

How do you do that?

To give you a simplyfied example:


Suppose you have a setup with a WAN interface, a LAN interface and a
GUEST interface.  Both LAN and GUEST networks are private networks
meant to allow internet access for clients through the WAN interface.
They both have IPv4 and IPv6 enabled, but unfortunately you don't have
a static IPv6 prefix.

You want to block all traffic between the LAN and GUEST network.

You also have a SERVICE interface that also has IPv6 on its network,
and IPv4 in another private network. You have an XMPP server on the
SERVICE network.  Clients on LAN and GUEST networks need to
communicate with the XMPP server on the SERVICE network.


When you design your firewall to first block all traffic, then how do
you allow traffic between your server and the clients on the LAN and
GUEST networks?  Keep in mind that you can't tell what IPv6 address
your server will have, and that its IPv6 address keeps changing.

OPNsense does this too : When you create a new network there is no Internet access at all : It's blocked.
_{(This is simply put ofcourse. We all know it's not exactly like that, but most people experience it like that!)}

Right, that's not how it works.  It's not unreasonable to assume that
a lot of people have a guest network and don't want all their guests
to be able to access their workstations on the LAN because there is no
way to block the IPv6 traffic when you don't have a static IPv6
prefix because you don't know what you need to block.

Maybe not so many have an XMPP server but some may have some
installation of homeassistant (though XMPP is great for automation)
...

If everything goes soo wrong that I need to reinstall : So be it! :)

You wouldn't have much choice.  However, that can get pretty
expensive.

Maybe it's you that are handling this badly?

Hate to say it, but that's probably the case here...

No, it's not.

Instead it's being suggested that there's the so-called migration assistant which users are supposed to use soon before it goes away

Where exactly has this been suggested? Particularly the "soon" and "goes away" parts?

It's being suggested by the so-called migration assistant showing up in the web interface.  It makes it obvious to users that things are changing and that they better keep up with the changes and andjust and adapt before its too late.

Why doesn't that page simply tell the users something like "Don't use this feature, it'll be years before this becomes relevant, and if you use it, your firewall rules will probably be screwed up."?  At least that way users would know.

I always read the notes that are being shown when I'm about to update.  I don't recall any mentioning of the new rules at all, though my memory isn't what it used to be.

Yet if there had been a suitable warning about the new rules like I suggested in my previous post, I would have remembered it.

It looks like you have not even read my post. There has been a whole lot of mentions to the new rules area, starting with version 26.1. To be clear, I will repeat it again:

I have not read your post in detail.  I saw you put a bunch of links to forum posts and lists of changes.  At that point my firewall rules were already screwed up.

Like I said, I usually don't look into this forum because I was basically told not to do that, and there is insufficient information and warning about these new rules outside of that.  The so-called 'migration assistant' suggest that user will experience a smooth and trouble free transistion.  The documentation about the new rules is also insufficient.

o The firewall migration page is not something you need to jump into right away.

I didn't jump into it but dreaded the switch and waited a while, being afraid that after one of the next updates it would be too late to do it.  It doesn't say anywhere except maybe somewhere in the forum that it would take years before it might be too late.

Just use the 'migration assistant' and it'll be fine obviously ...  If you don't trust the developers you can't use what they're making.  Opnsense updates have always been working without issues over the years, and it's not feasible for me to set up an up to date system to test it in production before updating every time.  There are other precautions.  So after making good experiences with it, why not use the so-called 'migration assistant'.  It'll be fine.

Why is it so difficult for you to admit that better information and warning about the new rules should have been given to the users during updates and on the page of the so-called migration assistant?  Why are you going to all these lengths to defend this oversight?

This just has been handled badly.

Did you absolutely need to migrate to a brand new rules GUI?

No, but I had to do it before the option to switch would have been removed.  Or what would the alternative have been?

Did you check the exported ruleset on a spreadsheet?

Yes, and there isn't anything wrong with the old rules and apparently nothing wrong in the spreadsheet.

Did you have a configuration backup?
Did you have an snapshot?

both

Did you try it on a test system beforehand?
Did you have a replicate machine to test the changes beforehand?

No, I don't have test system for that.  Once you understand how freaking expensive electricity is in this country (on top of everything getting more expensive and worse every day), you might understand why.  Also I have only limited room.  Do you expect everyone to buy additional machines and rent some office space to set up a bunch of testing environments for every update and every change?

And why would I have done that?  Just use the migration assistant and it'll be fine.

Do you have a written design of your ruleset?

No, I don't need one.

Maybe it's you that are handling this badly?

No, definitely not.

so I don't understand your anger with the devs about something absolutely new, and not forced until (at least) two more years (wow) of development.

I'm not angry about the new rules.  I like them better than the old ones, and they seem like a really nice improvement has been made.  And of course I appreciate that development goes on and that so much work is being put into it.

I'm merely saying the transition needs to be handled better by giving better information and warning at the very place users actually look at.

How do you suggest that users know about all the forum posts you provided links to before you put them into your post --- especially when they were told to stay away from the forum --- and when the documentation doesn't have sufficient warning and information either?

In general terms, my advice is first allow what you must, then block the rest. Design and review is a task prior to construction, like any systems design, so in my view your problem existed before your migration exposed it.

That's what I have been doing and am still doing.  Once something has been blocked there is nothing a hole could be punched into.

no more warning than to back up your configuration and/or to take a snapshot

If you are not taking a snapshot before any similar system modification then you are making a mistake with which I cannot otherwise help you.
[/quote]

I did take a snapshot and took a backup of the configuration.

That doesn't mean that better information and warning about switching to the new rules should have been and should be given.  It's not like that would be difficult to do, and it can still be done.  Instead it's being suggested that there's the so-called migration assistant which users are supposed to use soon before it goes away, and which would make the transition smooth.  After all it's a 'migration assistant', so obviously, nothing could go wrong.

You're a little bit all over the place with some of your interpretations and you're putting words in my mouth while trying to draw me in to your attacks against OPNsense.  For example I never said groups don't exist or that they cause chaos.

It's not my intention to put words into anyones mouth.  Where do you think I would have done that?

I haven't attacked opnsense either.  Are you putting words into my mouth?

Please, read the release notes before upgrading. They are posted on a dedicated section of this forum and they are showed to the admin before upgrading, either by GUI or console.

I always read the notes that are being shown when I'm about to update.  I don't recall any mentioning of the new rules at all, though my memory isn't what it used to be.

Yet if there had been a suitable warning about the new rules like I suggested in my previous post, I would have remembered it.

I usually don't look into the forum.  One of the moderators drove me away by telling me to stop posting when I was trying to help, so now I only come here when I can't avoid it.

This just has been handled badly.

But where do they come from then and why were they invented the way they are ?

I didn't know this coming in to OPNsense but interface groups are a BSD networking construct, not at all related to the firewall: https://man.freebsd.org/cgi/man.cgi?ifconfig. 

The manpage isn't very illuminating about this concept.  What's the idea with these groups?

It makes sense now why groups are printed in 'ifconfig' output and I can appreciate why developers would want to let us define rules at that level.

Why would developers want to let us define firewall rules on a concept that doesn't exist?  Is there any kind of relation between interface groups and firewalls?  You're saying there isn't.

I don't understand why group rules would have to be in a specific order, i. e. between floating rules and interface rules.  I can understand that they can be useful in the way I'm using them, but that isn't a good way at all and only due to a lack of better alternatives.  I wish there were a better way.

The way it is now makes it so that having to have the one group rule I actually need leads to a requirement for a bunch of other (overkill) group rules which shouldn't be needed.  That's only because there is no other way to put the rules in the desired order.  That makes me feel like that this is a pretty bad design.

It might be messier in the back end ruleset and lead to higher rule counts, but as an administrative abstraction it's quite useful.  Especially paired with non-'quick' rules that can be overridden at interface level (though these can lead to subtle ruleset bugs which are not obvious if they're used sloppily; you have to mind the ordering of pass/block rules and the successive scope of the matches).

That sounds like a recipie for chaos.  I like it that the new rules are now all in the same place and thus in the 'right' (obvious) order instead of kinda hidden over a bunch of interfaces where you can't really see them unless you flip around all these interfaces all the time.  That alone would make a distinction between different types of rules obsolete, but then there still needs to be a way to block traffic between interfaces wich comes about through interface groups.  If there was another way to do that, what would we need interface groups for?

The 'floating' concept confused me as well because the pf manual makes reference to 'floating' also, so as a newcomer I conflated them.  Totally separate things: pf talks only about floating states, not rules.  @Monviech's explanation above is much needed context for new users, IMO.

'Floating' seems like a kinda random term for the rules that get applied first.  You could as well, or better, call floating rules, group rules and interface rules first rules, second rules and third rules, respectively.  (That's ignoring that such a distinction isn't great for the new rules ...  They are already in some (i. e. one) order and we can actually see that now; only we can't change the order the way we would like to (yet?) because the distinction into types of rules is in the way.  Maybe that makes things difficult to explain.  But if you explain it like the other way round like this (I mean kinda backwards), it's probably easy to understand once a user thought about it for a while.)

Why didn't the developers just do that?

Maybe also tell them that the new rules are a work in progress which is in its early stages; that it will be improved upon and that it might be easier later to create a new firewall with the new rules.

I totally agree with you on this one considering the amount of people on this forum that did not know about the fact that the migration is not mandatory at all and can be ignored for a long time into the future :)

So I'm not the only one ... and it's still time to give better information to the users.

I've seen the new rules being there since quite a while, and I didn't want to update.  But it looks as if it won't be long before the so-called 'migration assistant' will be gone and you'll suddenly find yourself without a way to switch after an update.

Things like this need to be handled way better.  I guess next time everyone needs to ask on the forum when something new shows up.

Is there a way to switch from DHCP to kea yet?  When will DHCP be gone?  It would really suck if we had to convert everything manually.

Please keep your annoyance in check, it's quite hard to go from a historically grown legacy implementation to a new one without having some edge cases. That it seems to overly affect you personally is nothing that could have been anticipated.

https://github.com/opnsense/core/issues/9652

Don't be surprised when users are very annoyed.  Having the so-called migration assistant and no more warning than to back up your configuration and/or to take a snapshot makes users think that it's an easy and simple migration that will take maybe 5 minutes, if that, and that in the unlikely case in which one might find an issue, it'll be easily resolved by skimming over the exported rules in ones favourite spreadsheet.

You need to tell users that this is NOT a migration but merely an export and import of existing rules.  You need to tell them that their old rules will inevitably mean something else in the new rule section and that the old rules will only give you some clue as to what was intended with the old rules.  You need to tell them that they will have to set up their whole firewall again as if from scratch.

You also need to tell them that they have at least 2 years before making this change and that they better don't do it but wait until they actually want to rebuild their firewall from scratch and do that with the new rules.

Maybe also tell them that the new rules are a work in progress which is in its early stages; that it will be improved upon and that it might be easier later to create a new firewall with the new rules.

Only if you do that, the users may not be too annoyed but might even be glad that there was an improvement made (or prepared) by switching to the new rules.

The easiest way to create a floating rule right now if you really need to for some reason is to create a random loopback interface(Interfaces -> Devices - Loopback), assign it, name it "Floating" for example, and add it as second interface to any rule that should be floating.

But please keep in mind that this is not the intended way to use the ruleset, only a workaround if you really need it for some reason.

I made floating rules because I needed them, without creating extra interfaces :)  I still need rules that do what the floating ones did, only now floating rules don't work anymore and group rules are required instead.

-----

Also group and floating are not concepts of PF (the paket engine thats configured) Check out /tmp/rules.debug, there is no floating or group in there. PF only goes by a global order.

Mmh, that's what I thought.

If you unhide the advanced options in a new rule you can see the priority group and the seqence. Ideally, there should only be a global sequence and no priority group (group and floating), but getting to there and resolving the sins of the past is still up in the air.

If that's so easy, then why did the GUI prevent me from moving my LAN rule above the blocking rule?  Is that a bug in the GUI?

Sure here is the relevant section. Please note group rules are not a new concept, they also exist in the old firewall rules since ages.

Well, I never had to mess with group rules before.  I'm finding interface groups for things like VPNs very strange, if not to say annoying.  What's the point of forcing entirely different, unrelated interfaces into a group?  I have to have specific rules for each connection anyway, and only if I should ever need to have the same set of rules for a bunch of connections (interfaces), only then I should be able to optionally create a group interface.

Floating rules seem fine.  I made some floating rules according to https://forum.opnsense.org/index.php?topic=28447.msg138309#msg138309 because, unless that changed in the meantime, is the only way to deal with IPv6 when you don't have a static prefix.  It required to make an interface group but no group rules.

After this migration it seems that the floating rules need to be converted into group rules because they aren't working anymore.

Floating is an annoying concept that still has to be decided on most likely at some point. For now it is what it is.

Floating is very nice indeed.  Mankind has been dreaming of it for thousands of years before we finally managed floating :)

I think the problem is more with the arbitrary numbering of rules.  What matters is the order of the rules, not their numbers, but with the new rules, things got worse because the arbitrary numbering didn't go away but the distinction between different types of rules did.

I'm not sure if would make sense, but I might suggest to explicitly introduce types of rules, and for each type its own number range.  Each of the number ranges could go from 0 to N to order the rules of each type in a desirable way.  And the overall order of rules would be determined by their types.  It would be more user friendly to have types of rules.

It kinda shows in the GUI with the new rules in that the new rules are in a particular order.  That even begs the question why there are different types of rules.  What's the point of having these types?  It seems the order of the rules matters, not their types ...

All it currently does is preventing me from moving my LAN rule before the blocking rule.  If I could do just that, things would be much easier, and there wouldn't be a need to create overkill rules (unless I'd actually need one).

Also you are not forced to migrate yet. The old rules stick around for 2 more years at least. No rush.

Why didn't it say that?

But I'll have to do it anyway, eventually ...

-----

Since Firewall ‣ Rules [new] and Firewall ‣ Rules implementations exist side by side, there are some additional considerations regarding the processing order of rules.

If a Firewall ‣ Rules [new] filter rule has:

a single interface defined, it is an Interface Rule
a group interface defined, it is a Group Rule
any number of interfaces or one inverted interface defined, it is a Floating Rule

Ah yes, I saw that but couldn't find it when I tried to find it again later.

This makes no sense.  The order of the rules should matter, not what their arbitrary types are.  The GUI to create rules is already the same for all of them.  It only needs to allow us to move our rules to where we want them.