[Solved] Static routes on wrong iface after 23.1.7/8 upgrade

[idanoo]

I recently upgraded from 23.1.6 -> 23.1.8, so the issue may be related to the 23.1.7 update where I saw some gateway/routing changes. For simplicity I have disabled HA and CARP config.

<Upstream GW|185.121.168.1>
WAN
<OpnSense WAN vtnet0|185.121.168.232 - LAN vtnet1|10.0.0.1/24>
LAN
<185.121.168.233+>

I have several gateways set up on the LAN interface which serve public IPv4 addresses. The gateways have LAN interface selected. I have then set up static routes for those public IP's to route via those gateways.
This is happening to all Gateways/Routes I have setup but will be using 185.121.168.233 as an example here.

Images:
- Gateway config
- Route config
- Route status on 23.1.6
- Route status on 23.1.8

On 23.1.6 the routes were on LAN interface (correct), however after upgrading they're being created on WAN interface, even though the gateway has LAN set as the interface.

Before:

Code: [Select]

# route get 185.121.168.233
   route to: mastodon.nz
destination: mastodon.nz
        fib: 0
  interface: vtnet1
      flags: <UP,HOST,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

After:

Code: [Select]

# route get 185.121.168.233
   route to: mastodon.nz
destination: mastodon.nz
    gateway: mastodon.nz
        fib: 0
  interface: vtnet0
      flags: <UP,GATEWAY,HOST,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Does anyone have any idea here? I had to roll back to get connectivity but I have a second install setup via HA so I can check/confirm any suggestions. It does look similar to some of the issues in the following thread, and there was a duplicate WAN_GW in the config which I have since removed and the issue persists
https://forum.opnsense.org/index.php?topic=33864.0

[franco]

Since it's a host route I think your upstream DHCP supplied name servers are messing with this...

# configctl system list nameservers debug

and

# pluginctl -r host_routes


Cheers,
Franco

[idanoo]

Hmm I thought it may have been too, but doesn't look like it

Code: [Select]

# configctl system list nameservers debug
[
    {
        "host": "8.8.8.8",
        "gateway": null,
        "interface": null,
        "source": "config"
    },
    {
        "host": "1.1.1.1",
        "gateway": null,
        "interface": null,
        "source": "config"
    }
]
# pluginctl -r host_routes
{
    "core": {
        "8.8.8.8": null,
        "1.1.1.1": null
    }
}



[idanoo]

Just for testing purposes, I've created a Far Gateway for 123.123.123.123 on the LAN interface, and then added a static route to 123.123.123.123/32 via the above gateway - It's still routing via WAN interface.

It looks like any static routes that fall outside the LAN subnet, regardless of "Far Gateway" or Interface setting are being routed via default routes.


Edit; I've found a bug. Getting a PR up now

[idanoo]

https://github.com/opnsense/core/pull/6580

This resolves the issue for me.

[franco]

Thanks, my fault then entirely. Though I wonder why traditionally "add" operations for static routes are muted meaning no error shows up in the log. Default route doesn't do that for obvious reasons and would have caught this very quickly... possibly during development phase too.


Cheers,
Franco

[franco]

PS: https://github.com/opnsense/core/commit/07f07fff3b5 seems to make sense in this context