openvpn internal network route ignored - after multiwan added to an endpoint

[tedly]

Hi. I've set up countless (open)vpn site2site setups over the last decade with pfsense. Now I'm all-in on opnsense. I had it working fine 12 hours ago before I added in multi-wan. Now that multi-wan is going, openvpn no longer routes properly. Rather than use the VPN tunnel IP to route traffic, it uses the upstream hop. See below:

1?: [LOCALHOST]                      pmtu 1500
1:  192.168.1.1                                           0.965ms asymm  2
2:  100.64.0.1                                           39.816ms asymm  4
3:  172.16.251.70                                        38.100ms asymm  4
4:  undefined.hostname.localhost                         51.994ms (This broken router returned corrupted payload) asymm  8
5:  undefined.hostname.localhost                         42.959ms asymm  6
6:  den-b3-link.ip.twelve99.net                          43.309ms !N
     Resume: pmtu 1500

Note that 192.168.1.1 is my upstream hop because I have cgnat behind starlink.

My source network is 192.168.150.0/23 and my destination is 192.168.148.0/23. Each end of the site-to-site connects to a hub opnsense host and that hub communicates traffic between the two networks. Again, something I've done many times.

The remote end (192.168.148.0/23) can ping and communicate with the local side (192.168.150.0/23). When the remote side does a traceroute, it correctly talks to the VPN's tunnel subnet (172.30.1.16/28).

But when the local side tries to connect to the remote network, it skips routing through the tunnel's subnet gateway (172.30.1.17). And goes out the the public (192.168.1.1) gateway. And as you can see in the example above, it doesn't reach the real end point.

I have verified that the local opnsense has a route setup for 192.168.148.0/23 to go to the tunnel subnet gw (172.30.1.17). But it is being ignored anytime I send traffic.

As mentioned at the start of the post. This was working until I added multiwan on the local (192.168.150.0/23) opnsense.

I've rebooted. I've deleted and recreated the openvpn client configs. I've scoured the configs for 3-4 hours now. The VPN connects but the route is just broken.

Any ideas?

[tedly]

I just observed a new symptom. I can't even ping my own LAN gateway. And if I traceroute to it, it give me that same goofy route outside of my LAN to starlink's CGNAT gateway.  :'(

Route table and Tracepath results in attached screenshot.

How in the world could a gateway / router send traffic heading to itself to the public internet?

FYI - i am on the network typing this and can use the same opnsense gw just fine to browse/anything currently.

[tedly]

I found that if I use an external search (google), i get much better results than the forum search. Here's several people talking about the same thing as I:

https://www.google.com/search?q=opnsense+vpn+client+use+gw+group+multi-wan+site%3Aforum.opnsense.org

None of the threads answer the question, dating back to 2016. Looks like it may just be a short coming of opnsense that isn't getting much attention.

VPN routing works fine if one disables the multi-wan setup on the device.