dual gateawy, bgp /32 and policy based routing question

[cloudsense]

Hi,

I searched for similar posts, but could not find an example use case, so posting here.

I have opnsense with the following

- vtnet0 - ISP1 
- vtnet1 - LAN 
- vtnet2 - ISP2 

ISP1 is the default gateway, and all machines in the LAN  use send/receive using ISP1.

suppose the IPs are like this.

ISP1 is 1.1.1.1
ISP2 is 2.2.2.2
LAN  is 192.168.0.1
test-machine is 3.3.3.3 ( outside ip)

I am using cilium bgp in k8s and announcing the load balancer ip to opnsense.
I can see the 10.101.101.0/32 - announced OK.


since 10.101.101.0/32 is in route in opnsense, if I do http/curl 10.101.101.0, i get nginx test page from opnsense and all machines in the lan.

i also have following forwarding done in opnsense.

1.1.1.1 80/443 ->  10.101.101.0 80/443
2.2.2.2 80/443 ->  10.101.101.0 80/443

from 3.3.3.3 if I do curl 1.1.1.1, i get the nginx OK.
from 3.3.3.3 if I do curl 2.2.2.2, i do not get the page.


what i see when I try to do curl 2.2.2.2 is

vtnet2(ISP2) -> 3.3.3.3.xxxx -> 2.2.2.2.80
vtnet1(LAN) -> 10.101.101.0.80 ->  3.3.3.3.xxxx
vtnet0(ISP1) -> 2.2.2.2.80  ->  3.3.3.3.xxxx

here 2.2.2.2 ( ip from isp2) is seen as trying to go out via isp1

how do I add route/policy such that when  request is made to 2.2.2.2(ISP2), it is returned via ISP2 and not via ISP1.



Thanks

[seed]

https://docs.opnsense.org/manual/firewall.html#policy-based-routing

reply-to
   

By default traffic is always send to the connected gateway on the interface. If for some reason you don't want to force traffic to that gateway, you can disable this behaviour or enforce an alternative target here.

This might be it.