Set gateway group as default gateway

[drivera]

Hi!

I found another thread asking this same question, and this appears to be an unresolved topic. However, it's worth asking again.

I have some special routing needs which necessitate the definition of gateways to handle the traffic. My problem is that the firewall won't select the correct gateway to the internet, and so it'll occasionally become isolated and won't be able to check for updates (for instance).

The rest of the traffic works and flows just fine, though.

There's only a checkbox to select a default gateway, but my problem is I have a multi-wan setup.

Is there a way to configure OPNSense and tell it, specifically, which gateways lead to the internet so that it can rotate through them in the event of circuit failure (i.e. failover)? Alternatively, is there a means to set a gateway group as the default gateway?

Thanks!

[franco]

This is a very specific question. AFAIK, this exists in pfSense only since 2.4.4, which is effectively 1 month old.

Default gateway switching has always done roughly the same thing as you cannot have more than one gateway active at all times in the routing table anyway.


Cheers,
Franco

[drivera]

Yes, but how does OPNSense decide which gateway should be used instead of the one that just went down? In the end, I have no means to tell OPNSense which gateways lead (eventually?) to the internet, and which ones just lead deeper into my network.

So... thoughts? Is there an already-planned feature that would cover this use case?

[mimugmail]

ATM this decision is random, probably the order the gateways were set. Some months ago it was also possible when you have a internal gateway for a development network or whatever, this could be your next default

Ad promised me that he'll add a similar thing like default gateway groups .. I just have to catch him in a good moment

As you are new to the project, it might be a good idea to flow over all open issues in core .. it gives you a better overview on what is worked on

https://github.com/opnsense/core/issues/2279

[drivera]

Being new, I don't mean to be bossy.  However, this seems like a truly BIG defect that shouldn't have gone neglected this long.

After looking at most of the issues, they're valid and definitely need fixing. And, of course, one can't neglect the fact that there are paying customers out there requiring these fixes.

However, this is the core routing not functioning as one would expect it to.  In short: multi-wan can't be considered robust unless the FW only has one non-public network behind it (ie. all configured gateways are WAN gateways).  If I have at least one non-WAN gateway that needs configuring for proper routing, the FW's behavior on a failover (and, worse-still, fail-back) scenario is inconfigurable, which makes it non-deterministic.

Needless to say, this is a VERY BAD THING ^(TM) for a firewall to (not?) do

So, I'll be happy to pitch in and lend a hand here if that's the main challenge. Not very well-versed on *BSD though. Not sure if OPNSense already does this, but perhaps manual management of the routing table (to apply configured priorities, allow gateways to be marked as non-WAN for ignoring, etc) is an option?

Again - don't mean to preach or be bossy. Just a little surprised something this key and this big doesn't even have a plan-of-attack for a solution yet.

I'll start poking around the code on my spare time in hopes of understanding how a solution might be implemented.

Cheers... and thanks for the rapid responses!!  All of this bitching aside, congratulations on a great product! I'm very impressed with all of it (except, obviously, the above) so far!

[mimugmail]

Routing is one thing you can't understand when you come from Linux. As also pfSense added this feature (or fixed the bug as you say) in September, this is not an often requested feature, also no paying customers of course

If you think this is a big issue, you can always go along to deciso and try to pitch this for money, no idea.

In sum, it's not a bug anymore, internal gateways can be marked as down for dpinger/apinger process, so they wont be used. Also, when you have 2 WANs it works perfectly, also when you have 3 or 4 WANs, there's only the downside that you can't decide which one's selected first. This is a missing feature from FreeBSD itself (no metrics) .. as I said, when you come from Linux there are 2-3 cases where you thought this is standard, but not in FreeBSD.

I also thought about switching back to Linux .. but the chances you have here with changing the code yourself and adding features it too sweet.

And as always .. try to understand the code and fix the bugs/issues with all of the guys here. This is a community product .. it's not FOR the community, it's FROM the community Deciso is just a holder of the trademark (and also nicely adding much of the code)

Trust me, I know what businesses need .. that's why most of the bugs you complain where already issued by me in github, I also have many customers running opnsense in production, and I know many companys with 500+ emps running it 

[drivera]

Interesting tidbit: if a gateway is marked as down, but monitoring is also turned off, it's still considered "up".

Shouldn't the "mark as down" option supersede the effects of "disable monitoring"? Perhaps I want to mark it as down and disable monitoring as well to save CPU cycles, but still always consider it down?

Perhaps a better UI choice is to have a 3-state selector condensing those two options where one can select whether a Gateway is "Always Up", "Always Down", or "Autodetect" (i.e. "monitor via ping")?

I think that'd be much more consistent and self-explanatory...

[mimugmail]

Mark as down only means dpinger does not respect it in the decision process.

[drivera]

Final question: can you point me in the code to where the routing rotation is carried out? I'd like to see how much effort it'd be to add the "ignore this gateway when switching them" tidbit...

Is there any doc where the architecture/design of that area is described?

[franco]

https://github.com/opnsense/core/blob/622ef1da5a9094792efba6507ccbe90d401a61a6/src/etc/inc/gwlb.inc#L878

There is no documentation in the form that you seek. I don't believe the original authors intended to provide a high level concept of their work or were consciously aware of one.

The code was therefore scheduled for removal at some point in the past, only prevented by Michael's testing, prodding and ultimately his help in making it work more reliably, which was very much appreciated.


Cheers,
Franco

[drivera]

I see it, and it should be about as simple as the

Code: [Select]

if (isset($gwsttng['disabled'])) check, except the setting name might be something like "non_wan_gateway".

Now I have to hunt down the UI page where the controls are, and add the new checkbox

I might submit something by this Sunday.

Cheers!

[franco]

Virtually that is true. Practically it's a workaround on top of more workarounds. We can already disable individual gateways and the gateway group approach is a good one. What needs to be implemented is:

1. easily manageable gateway group edits (bind to group, not edit multiple single gateways)
2. priority / weight for all gateways to make ordering possible (bind to group, not edit multiple single gateways)
3. bind group to gateway switching feature, but retain backwards-compat with current installs
4. the switching code would need to be refactored / reordered to make 1-3 possible

https://github.com/opnsense/core/issues/2279


Cheers,
Franco

[drivera]

How about this approach:

• Gateway & Group management remains as-is
• Add a new page called "Gateway Priority" where one can add single gateways as well as gateway groups, and choose the order in which they're selected for use (i.e. the priority) in the same manner as one chooses the order of firewall rules (for UI consistency)
• The gateway selection algorithm walks over this new list (Gateway Priority) to select which gateway to use next upon a gateway failure/recovery. Groups already have priority internally, so when a group is found we either apply a "mini" version of the algorithm within the group's members (might me more elaborate), or simply "expand" the groups on the list in the proper order so it's just one big list of gateways (i.e. pre-process the list prior to running the selection algorithm, which might make it simpler)

Thoughts?