System >> Routes failing after reboot

[furfix]

Hi All,
I've set a static route under System >> Routes >> Configuration, but I notice that after rebooting OPNSense, the route is no longer active, and I need to disable it and enable it again to make it work. The route uses a wireguard gateway and my ISP uses PPPoE. I didn't notice this issue with previous versions to be honest, and since I'm suffering the Kernel Panic reboots, this issue became more visible for me because the box is rebooting more often.
 
I assume (most probably wrongly just because I know nothing) that maybe the tunnel is not up immediately after the reboot, while opnsense is setting up the static route and that's why it's not being set, but is there no time out or something to wait a little for the tunnel/gateway to be up again after reboot or maybe a retry?

Is this a known issue? Sorry if it was already reported. I couldn't find it on the forum,

OPNsense 24.7.2-amd64 (baremetal)
FreeBSD 14.1-RELEASE-p3
OpenSSL 3.0.14

Thanks,
f

[dseven]

I'm not sure if a static route like that is supposed to work after reboot, but I would suggest that you try policy-based routing instead of the static route - create a firewall rule that matches your destination and specifies the VPN gateway....

[furfix]

I remember it used to persist after a reboot and I don't find any reason why it would be disabled after rebooting on purpose. The only logical reason I found, is what I wrote. Maybe the tunnel/gateway is taking longer to be up, hence opnsense is ignoring it, but idk.

Regarding why i didn't use firewall rules, it's because nobody on the network needs to reach this endpoint through this gateway more than OPNSense host itself. In the other side of the tunnel, there is a database not exposed to internet.

If there is another way to do it, or a better way, I just don't know but I'm open to suggestions always So far, it's working. I just need to remember to disable/enable it after a reboot I post it, in case it's not an expected behaviour. Just tying to help

[doktornotor]

If the WG interface assigned or not?

[furfix]

Sorry @doktornotor, I don't understand your question.

[doktornotor]

Interfaces - Assignment

[furfix]

Interfaces - Assignment

Assuming it was a typo and instead of IF, you wanted to asked "IS the WG interface assigned?", then yes, it's assigned and enabled. As mentioned, there is an enabled Gateway using it.

If that's not what you wanted to ask, sorry. I can't really help you. Try to ask in english again and I will do my best to help you.

[Seimus]

So you have a static route that point thru the WG tunnel?

Most likely that's the problem, if WG is not yet up the route tries to point towards a GW into something that is not yet existing or up, than you may hit this problem.

A static route can not be put into a route table if the egress interface is not existing (or down). How is your GW configured onto which you point the static route? Does it use tracking (dping)?

Regards,
S.

[furfix]

So you have a static route that point thru the WG tunnel?

Most likely that's the problem, if WG is not yet up the route tries to point towards a GW into something that is not yet existing or up, than you may hit this problem.

A static route can not be put into a route table if the egress interface is not existing (or down). How is your GW configured onto which you point the static route? Does it use tracking (dping)?

Regards,
S.

That's what I was suspecting. Thanks S! Do you know any other way opnsense host can reach an specific local ip via that specific gateway without using a static route?

I usually create outbound rules and specific firewall rules to route traffic via specific gateways, but in this case...since it's the host itself, I don't know how to do it.

[furfix]

Just for testing purposes, I set down the gateway and turn it up again without rebooting and the static route it's was deleted from the table and it never came back again.  Maybe the expected behavior, right? But it doesn't make sense to me. If the WAN gateway goes down and up, you get the routes and you are back online, without doiing anything, right? or maybe not?

[Seimus]

I believe the route should be back.

How is your GW configured, is the monitoring turned on that GW? Do you have the proper Egress interface set?

I had few years ago similar issue with static route. Where in my case I did migrate off to VLANs but forgot to change the EGRESS interface on the GW to the VLAN and instead let it on the LAGG which wasn't L3. Each reboot or Interface flap I lost the route and had to manually apply it. After time a realized I actually forgot to change it (STUPID ME).

Regards,
S.

[furfix]

GW is set as Far Gateway and monitoring turned on, pinging fine. Priority 255. On the static route, I've set this gateway as egress for that specific host, and it's working perfectly!

The problem is when the gateway goes down and up, or indeed I reboot the box. The static route is not longer in the table, even if OPNsense is showing it as Enabled in System:Routes:Configuration.



ps. just for the records, I'm not complaining. I'm just another home user. I can disable and enable the static route every time I boot, I don't mind. I'm sharing it in here, in case it's something not expected to happen and Franco and the team wants to take a look, for benefit of the real users and I know they are busy after the release. definitely not in top of the list this thing.

[Seimus]

When we stick to the base concept of routing and routes itself.

A route will be put into uRIB (unicast routing table) only in case the destination where its pointing to is reachable. E.g we have that specific destination in the route table and we are able to do recursive lookup.

Example:

Source A wants to go to destination M

1st lookup:
Destination M is reachable using static route over Gateway B

2nd lookup:
Destination B is reachable over C

2nd lookup:
Destination C is reachable over XYZ

etc.

This is happening until the router in this case OPNsense will not hit such a recursive lookup that will point it to an Egress Interface.

So my question now is, when you dont have the route present in the route table. Does OPNsense know where is the GW of that static route? Do you have it in ARP & Router table?

Regards,
S.

[chemlud]

There is a Cron job configurable in the GUI called "Renew DNS on Wireguard for stale connections". Try to run every min. and see if route is added. Just an idea for a crude hack...

[furfix]

When we stick to the base concept of routing and routes itself.

A route will be put into uRIB (unicast routing table) only in case the destination where its pointing to is reachable. E.g we have that specific destination in the route table and we are able to do recursive lookup.

Example:

Source A wants to go to destination M

1st lookup:
Destination M is reachable using static route over Gateway B

2nd lookup:
Destination B is reachable over C

2nd lookup:
Destination C is reachable over XYZ

etc.

This is happening until the router in this case OPNsense will not hit such a recursive lookup that will point it to an Egress Interface.

So my question now is, when you dont have the route present in the route table. Does OPNsense know where is the GW of that static route? Do you have it in ARP & Router table?

Regards,
S.

Bare in mind, my knowledge is limited so I will answer in that way. Limited.

The host destination is on the same subnet than my opnsense, but it's phisically in a different location, that's why I'm using a tunnel to hit this host.

If the static route is not in the routing table, opnsense tries to hit the IP on its own network, but the host is not there of course.

So I've created this static route to let OPNSense know that when the host try to reach ie 192.168.100.100/32, it should use the wg0 gateway, and it's doing it fine. The host, in the other side of the tunnel reply without any issue and they talk each other fine.

...and I'm doing it in this way, just because the APP trying to reach this remote/local host, is running in OPNSense itself. If it would be a client connected to any OPNSense interface, I would created a firewall rule...to route the traffic via that Gateway.

Not sure if it makes sense, what I'm explaining. Sorry. Basically, everything works as expected, but as soon as the gateway goes down and up, the route (UGHS) is no longer in the table.


Trying to make it more clear:

With static route enabled:

root@opnsense:~ # ping 192.168.100.100
PING 192.168.100.100 (192.168.100.100): 56 data bytes
64 bytes from 192.168.100.100: icmp_seq=0 ttl=64 time=2.148 ms
64 bytes from 192.168.100.100: icmp_seq=1 ttl=64 time=2.039 ms
64 bytes from 192.168.100.100: icmp_seq=2 ttl=64 time=2.267 ms
64 bytes from 192.168.100.100: icmp_seq=3 ttl=64 time=2.202 ms
64 bytes from 192.168.100.100: icmp_seq=4 ttl=64 time=2.184 ms
64 bytes from 192.168.100.100: icmp_seq=5 ttl=64 time=2.292 ms


With static route disabled after GW flipped:

root@opnsense:~ # ping 192.168.100.100
PING 192.168.100.100 (192.168.100.100): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down