Stop automatic default route generation

[JimIFN]

Hello:

I run a small ISP, and I'm using OPNsense in a failover pair as my core ISP router.  I have some public IP address ranges assigned, and I am announcing them via BGP to my upstreams.  I also receive via BGP default (and some additional) routes.

Currently, in my route table, the kernel route is inserted by the gateways.  This overrides the default route from BGP.  Unfortunately, this means if BGP session drops to a peer, the default route remains in place..but I do NOT want that.  I want my default route managed by BGP.

I tried deleting the gateways, but "bad things" happened.  Is there some way to disable the gateway system, or at least suppress insertion of the default route into the kernel routing table?

[franco]

Removing drops you back to enabled defaults. Try disable or unchecking "Upstream Gateway".


Cheers,
Franco

[mooh]

Have you looked into the Firewall:Settings:Advanced:"Disable force gateway" setting? By default OPNsense creates a default policy route for traffic originating from the FW itself.

[Maurice]

If the gateway is needed for static routes or policy routing, keeping it enabled but marking it as "down" is what worked for me. Not sure whether this (still) is best practice though.

@franco Doesn't unchecking "Upstream Gateway" only lower the priority? From my experience, this doesn't prevent a gateway from becoming the default. Has this changed?

@mooh These firewall rules have nothing to do with the default route in the routing table.

Cheers
Maurice

[franco]

The default code treats it like this:

https://github.com/opnsense/core/blob/092433dae62be475918f498b101296cb41810d90/src/opnsense/mvc/app/models/OPNsense/Routing/Gateways.php#L497-L500

So disabled should work.

I'm a bit unsure about 'defaultgw' property. It has been weird historically in that it orders gateways, but it's more of a hint of what should be a default gateway so that may not work indeed.

But if you want a default route that is configured as a gateway to stick only checking that as upstream (defaultgw) will do the job so nothing else will create a different gateway.


Cheers,
Franco

[mooh]

@mooh These firewall rules have nothing to do with the default route in the routing table.

I agree, it doesn't change the kernel routing. Thanks to your response I now understand the question better, so please ignore my comment.

[Maurice]

But if you want a default route that is configured as a gateway to stick only checking that as upstream (defaultgw) will do the job so nothing else will create a different gateway.

That's a little off topic here since the OP doesn't want the default route to be set by a gateway at all, only by BGP. But while we're at it: Marking a gateway as upstream doesn't reliably prevent non-upstream gateways from becoming the default gateway.

I've had a situation with three gateways, 1 and 2 marked as upstream and with gateway monitoring enabled, 3 not marked as upstream. The intention was default gateway switching between gateway 1 and 2 (failover), while using gateway 3 only for some specific static routes.

This worked as long as gateway 1 and / or 2 were up. But when both went down, gateway 3 became default (which should never happen). The only way to prevent this was marking gateway 3 as "down".

If anything has changed in this regard, I'd be happy to learn about it.

[Monviech (Cedrik)]

From what I know if you have 3 gateways that all have priority 254, setting one with the upstream gateway flag will prefer it as candidate before the other active gateways with the same priority.

I always disable gateways I only need for installing static routes. And my (test) BGP setup does the same where I install a default route via BGP. All gateways disabled, no interface on DHCP.

[Maurice]

@Monviech Exactly, "Upstream Gateway" is a preference setting, not a "this gateway will always / never be upstream".
Having two preference settings (numeric "Priority" and binary "Upstream Gateway") has always been a bit confusing.

[franco]

That's what I meant by "weird" earlier: historically, a default route was always enforced even if the user didn't want that because in the average case the user wants a default route and if he doesn't have one he's going to complain if he can actually manage to get on the interwebs ;)

As mentioned by Maurice force down works, same as disable. I don't think we should change the "upstream" behaviour.

Weaving gateway monitoring into the construct is a different level of complexity, too.

Also, can't BGP do the split default route thing that WireGuard does? Adding 0.0.0.0/1, 128.0.0.0/1 and/or ::/1, 8000::/1?


Cheers,
Franco

[JimIFN]

So we finally got the opportunity to implement this.  We've deleted all gateways except our primary upstream (BGP peers) and marked both of those as default.  Yet, the kernel default route to one of the gateways remains, and is the selected route (ignoring the BGP-provided default route).

How do we fix this going forward?

[JimIFN]

Actually, it appears a reboot was required to clear out the installed default routes.  Its now working as desired/expected.

[Seimus]

You can as well delete the routes in the table directly. If there are no statics/active GW for those statics when you delete them from RIB they will not be populated back.

Regards,
S.

[Seimus]

I tested the GW and behavior cause its an interesting Topic and the Use case the OP has is often used for BGP.

GW = disabled
GW itself or any static routes with this GW will not be in the route table | GW is still selectable in FW for PBR = no traffic will be routed to it

GW = force disabled
GW itselfs appears in the routing table so does any static route with this GW | GW selectable in Rules PBR = traffic is routed

Honestly I have no clue what actually should this Force Disable do in fbsd, but per the behavior I have seen it smells like its to remove it from the default route selection.

Regards,
S.

[franco]

"force down" isn't about fbsd at all, it's a sense thing that came to be with the gateway monitoring and is effectively labelled incorrectly. It's more of a "do not use for automation" flag with the twist that it blanks the status for the gateway. Won't be easy to clean this up.  ;)


Cheers,
Franco