Gateway priority and status not respected in routing table

[zubrick]

Hello,

I am migrating to an OPNsense firewall and have an issue with the routing table.
I've set up 3 IPSec VPNs with a vti interface on each and a corresponding gateway and different priorities on each.
IP monitoring is working correctly on all three gateways and interfaces of disconnected tunnels are quickly marked as down.

The problem is that the static route I add on those gateways are inserted in the routing table regardless of the gateway status or priority, leaving routes on inactive tunnels.
I've tried all the parameters in the gateways.

I saw there are gateway groups, but I cannot put routes on them, only use them in firewall policies which solves the issue if the connections is initiated from the OPNsense side, but still creates asymmetric routes if the connections is initiated from the other side.

Am I missing something ? Is there an option for it to work?
It seems to work correctly with the two default gateways.

UPDATE:
After some tests it seems OPNsense doesn't like to have two routes to the same subnet on two different gatways and acts completely random in that case.
This is the first firewall on which I am not able to do that.
This is really a basic feature for a route based VPN.

The current workaround will be to take advantage of the routing engine more specific route priority and divide the subnets in two on preferred VPN gateway.
So for exemple 10.20.30.0/25 and 10.20.30.128/25 on the primary vpn and 10.20.30.0/24 on the secondary one
This is a lot of work and is really error prone, but I don't see anything else.
And then it still doesn't solve my issue because I need to go disable the routes manually, because OPNsense doesn't remove route of a disabled gateway?????

[kfm]

I can confirm this issue.  Is there any progress or better workarounds?


ISSUE
--------------------------

In my situation, I define multiple non-upstream gateways with different priorities in Gateways -> Configuration:

GW100 192.168.1.100 upstream=no priority=100 monitor=10.0.0.100
GW101 192.168.1.101 upstream=no priority=101 monitor=10.0.0.101

...and then I add the same route for each gateway in Routes -> Configuration:

10.0.0.0/8 via 192.168.2.100
10.0.0.0/8 via 192.168.2.101

The expectation is that the system routing table will populate with a route for whichever gateway is up with the highest priority (in this case, it would be GW100).  Instead, opensense seems to randomly select which gateway will get the route in the system table, regardless even of whether the gateway is up or down. 

This issue seems to only apply to non-default gateways.  Default gateways seem to handle this correctly.




WORKAROUNDS
--------------------------

1. Like zubrick said, you can manually create more precise routes for the preferred gateway to force opnsense to put both routes in the system table.  However, if the preferred gateway goes down, opnsense doesn't update the routing table is not updated to remove the applicable route(s).  Rather, you have to manually disable the preferred gateway to remove the route, which is not optimal.

2. Can't use gateway groups in NAT rules because the traffic is not being NATted.

3. Can't use gateway groups in Routes -> Configuration, because opensense does not allow groups to be the target of route rules.

4. Can create a gateway group and then create firewall rules for all interfaces that force the gateway for traffic going to 10.0.0.0/8, but this would mean duplicating every firewall accept rule (one for local traffic, one for remote traffic).  I'm not sure if maybe there is a smarter way to do this with non-quick rules or marking?

5. Maybe some kind of monit scripting that can watch the gateway status and add/remove routes as needed?  I couldn't find much documentation on it.

None of these options are great.




PREFERRED RESOLUTION
--------------------------

The two easiest ways to solve this from a user's perspective would be

(1) allow gateway groups as targets for static route configuration or

(2) have an option to treat down gateways as disabled for purposes of system routing table generation.