Problem - Asymmetric routing with two gateways

[HatalaTitla48]

Hi, I have old box with multiple vlans/subnets connected to switch. On same network/vlan I have connected new opnbox to the same switch and created in opnsense direct route(192.168.12.0/24 192.168.12.1 - far gateway) to the subnet 192.168.12.0/24 on old box(it has own gw on 192.168.12.1).
Now comunication and routing works ok between boxes and subnets until I try access from openvpn client subnet(192.168.202.0/27) to the 192.168.12.0/24. On the old box i did put direct route to the networks on opnbox "route add -net 10.88.1.0 netmask 255.255.255.0 gw 10.88.1.254 eth5" and "route add -net 192.168.202.0 netmask 255.255.255.224 gw 10.88.1.254 eth5". Both work ok, I get even response from ip delegated to vpn user on its notebook. But ping from notebook to 192.168.12.1 is without response.
Ive been looking for a cause and found out about asymetric routing, both boxes have different gw. Ive tried on opnbox outbound(snat) NAT, to mask opnbox gw 10.88.1.254 for 192.168.12.5(ip of opnbox interface connecting to oldbox) but so far without success. Probably easy fix...where I should look?

Thx.

[viragomann]

created in opnsense direct route(192.168.12.0/24 192.168.12.1 - far gateway)
"route add -net 10.88.1.0 netmask 255.255.255.0 gw 10.88.1.254 eth5"

These routes don't make any sense to me as the gateway is inside to subnet you try to route.
If the gateway IP is inside a subnet, a route for this one is useless at all.
So your network is pretty unclear to me.

But ping from notebook to 192.168.12.1 is without response.
Ive been looking for a cause and found out about asymetric routing

Pings or even the whole ICMP protocol doesn't care about asymmetric routing. This rather affects TCP.
There might be another issue causing the ping faults. Use packet capture to investigate.

[nero355]

Probably easy fix...where I should look?

I am guessing Policy Based Routing on the Server with multiple NICs/Interfaces itself instead of OPNsense :)

Imagine something like this :
eth0 - 192.168.1.0/24
Only this Interface has a Gateway configured : 192.168.1.1/24
eth0.10 - 10.0.0.0/24
eth0.172 - 172.16.0/24
These two are just P2P Interfaces basically.

And now you grab a Client that's connected to either the eth0.10 or the eth0.172 network and SSH into this Server to IP Address 192.168.1.1/24 :
- The session times-out in about 30 to 60 seconds.
- The cause is because the Server uses both Interface eth0 and one of the eth0.10 or eth0.172 Interfaces depending on where you are connecting from to communicate with the SSH Client.

And to fix this you simply add a Policy Based Routing Rule to the Server telling it that it should keep on communicating via it's Default Gateway with that Client instead of the directly connected Interfaces :)


Fun fact about this is that I only found about it after I had started using OPNsense and the Default Drop/Deny Rule got triggered again and again, while my old Router never complained about it for whatever reason...