Policy based routing not working

[karan293474]

Hi

I have a host for which I want to route via a different gateway so I used policy based rule. The traffic seems to go out to internet and seen back on the interface connecting to secondary gateway however web access does not work. To debug I did captures on both LAN and new WAN interface and to my surprise I see both incoming/outgoing packets on new WAN interface but I only see outgoing packets on LAN interface. Any idea what may be wrong because I would expect firewall to keep state of the session and should implicitly allow response packets from internet but I do not see this working so not sure what is wrong. Below is very basic topology:

client--->LAN1----->WAN1------Internet

Packet capture logs:
1. LAN1 sees only outgoing packets
2. WAN1 sees both incoming/outgoing packets
3. Traffic is permitted from LAN1 to WAN1 and no firewall deny logs so I would not see this to be firewall permissions issue.

Thanks
Karan

[patient0]

Hi (again :),

Does ping to a public IP work and does a DNS query from the client work?

[karan293474]

Hi

Ping does not work. If I look at TCP handshake I see no ACK back from client on firewall in packet captures. So I am not sure what is wrong. I doubt if SYN/ACK from server made to client but I cannot confirm that because on LAN1 I see no return packets from server in capture files.

The return packets on LAN1 are not seen even in working scenario(with default gateway and no PBF) that is if I capture same client traffic to google.com LAN1 captures show packet from client to server but no packets from server to client but site is accessible.

I doubt if firewall is handling PBF return packets correctly but I have no way to diagnose that.

Thanks
Karan

[patient0]

Sorry for the very, very late answer. To recap:

Normal traffic on LAN1 using the default GW does work correctly. But you don't see any returning packages from server to client on LAN1 if you do a package capture.

How to you capture the packages, what command and what options to you use (on what interface, ...)?


According to your package capture, PBR (Policy Based Routing) traffic goes out and on WAN you see them in both directions but - as with the normal traffic - you don't see any returning packages on LAN1. And as a result you can't debug the issue.

How is the policy routing setup? Can you post a screenshot of the all the rules on you LAN1 interface (redact public IPs before posting)?

[karan293474]

Thanks for following up. I dug further and below are my observations:

Scenario 1: using normal default gateway and it works

traffic from client on VLAN 2 to server goes via a L3 switch which is gateway for client and then towards firewall on VLAN1.

VLAN 2---->(VLAN2)switch(VLAN1)---->VLAN1(Firewall)------Internet

Return traffic:

Internet----firewall(VLAN2)---->Client on VLAN2: switch is bypassed as firewall and client are on same network

Note: Firewall has interface on same network as client so when traffic is returned firewall sends it directly to client and this time I could see packets in capture but on interface where firewall is directly connected to client on VLAN 2. I reviewed firewall configuration and figure out how firewall is sending return packets so I was able to run packet captures on firewall's VLAN 2 interface and see them come in from internet. This is asymmetric but is working on OPNsense.

Scenario 2: Using PBF to different gateway:Not working

Client(VLAN2)-----(VLAN2)switch(VLAN1)-----(VLAN1)firewall(VLANPBF)-----internet

Internet-----(VLAN PBF)firewall-----No logs or traffic seen on packet captures on VLAN 2 interface of the firewall

Here traffic is received by firewall on VLAN 1 interface and using PBF is sent to internet on VLAN PBF interface. Return packets are seen on VLAN PBF interface of the firewall and since client is on VLAN 2 as is firewall I would expect to see packets on VLAN 2 interface of the firewall but there are no return packets. So I am not sure what is firewall doing with those return packets received on VLAN PBF and why it is not forwarding out the directly connected interface to the client.

IPs as below to help understand:
client(VLAN2): 10.10.10.45/24
server/google: 8.8.8.8
client gateway(switch VLAN 2): 10.10.10.1/24
switch(VLAN1): 20.20.20.45
switch gateway(firewall VLAN1): 20.20.20.1
firewall(VLAN 2): 10.10.10.2/24

If you see forward traffic from client goes via switch to firewall and then to internet but when packet is returned it goes via firewall directly to client and is apparent from packet captures I took on VLAN 2 interface of the firewall(10.10.10.2). This is asymmetric routing but is working on OPNsense whereas it does not appear to work when traffic is routed out via another gateway using PBF rule.

[patient0]

I see, I won't be of much help here, someone with more knowledge would have to step in.

You haven not shown any details of rules in you setup for neither the normal nor the PBR (what is PBF standing for?). Is the async routing what you want and you set it up that way? Not sure anyone can help you with that little information that is here.

[karan293474]

Attached document for better details. I don't see any opnsense experts here so not sure how to get attention.

[patient0]

Attached document for better details. I don't see any opnsense experts here so not sure how to get attention.

In the doc you repeat what you wrote before, sure.

Let's see, in the other forum you mentioned that there's another router in front (RTR2, no idea what that is).

"client--->LAN1--(PBF)-->LAN2----RTR2----Internet"

You start here with a simple drawing, not complicated at all.

"client--->LAN1----->WAN1------Internet"

Then you write that there's a L3 switch involved and asymmetric routing for the regular traffic. I asked if you set the symmetric routing up, you didn't answer. Now you show one rule - not including the interface it's on, I assume that is on some LAN and the same package capture again. Sure I get it, the packages don't go the way you expect them.

So I'm not sure if you don't know more or if you want to waste time or hoping for someone with telepathic abilities. It's not gonna be me.

If you browse the recent topic you see a few people who are answering a lot and are very knowledgable.

[karan293474]

OK leave this. I don't want to waste any more time of yours. Thanks for your help and attention so far.