IPsec Traffic - Firewall Rule not Evaluating(Asymmetric Routing)[SOLVED]

[markh0ppus]

Hi Everyone, 

I am not sure if the issue that I am facing is a bug or a technicality that I am missing.

1) Network Diagram opnsense-ipsec-high-level.png

The diagram attached describes my setup in a high level.

2) ICMP Traverses the Tunnel

Ping flows over the tunnel, both hosts can ping each other 192.168.20.11 <-> 192.168.1.100.

3) Traffic Rules fireall-rules-vlan02.png

However, HTTPS traffic is being blocked even though there is a rule to allow it.

It seems that for any reason the rule is not evaluating( it is a broad rule at the moment for testing purposes ).

4) Blocked Traffic blocked-rules.png

As seen on the screenshot, the traffic is captured by the block all rule. However, if I disable it the webUI loads on the other side of the tunnel.

I have tried quite a few changes but none worked.

Some help would be much appreciated if someone with more experience are there reading my post.

Let me know if you need more detail about the setup.

Thanks. 

[markh0ppus]

Hi everyone,

I am attaching the IPSec ruleset in case it helps.

[Monviech (Cedrik)]

I had a problem like this too sometimes where traffic for Ipsec rules doesn't match and I solved it by using the following in IPsec rules:

Instead of
Protocol: TCP
try
Protocol: any

[markh0ppus]

Hi @Monviech,

Thanks for the update.

I have already tried to change the protocol to Any. However, the returning traffic is still being blocked.

Another behavior that it is not logical to me, is the fact that when the traffic matches a rules in the `IPsec` Ruleset it is allowed without hitting the Ruleset for the interface it is forwarded.

i.e) Traffic from home - 192.168.20.11 > 192.168.1.100 is allowed in the `IPsec` Ruleset. However it passes even if explicitly blocked in the interface VLAN02 ( 192.168.1.0/24 Subnet ).

Kind regards.

[Monviech (Cedrik)]

The first matching rule wins.

If there is a rule in "Firewall: Rules: IPsec" with "Direction: in" and it matches a packet, it is allowed to go to the "destination" thats specified. It won't go inbound to the next interface on the same firewall (EG: Firewall: Rules: LAN) and then be evaluated again, it goes straight to the destination.

If you have a mix of "Direction: in" and "Direction: out" rules, you should first try to disable the "out" rules and only work with "in" rules until things are working as expected.

You can try to use "Firewall: Rules: Floating" where you can create "Direction: in" rules that match before all other rules, in order to troubleshoot with log enabled. You can select multiple matching interfaces there (For example IPsec and VLAN02), or select ANY interface by leaving it unselected.

[markh0ppus]

Hi @Monviech,

Thanks for the reply. Great explanation

The rule logic makes more sense now.

Also, great tip in regards using the "Floating rules" to troubleshoot. It's something I haven't thought of.

It's quite late already and I am on call tonight but I will play around tomorrow.

Many thanks.

[markh0ppus]

Hi @Monviech

I have tried the "Floating Rules" and to my sadness it had not worked.

The HTTPS traffic returning "HOME" is still being blocked inbound by the "Block All" rule on VLAN02 Ruleset.

It is funny that ICMP hits the "Floating Rule" to allow Any Protocol from VLAN02 to HOME(192.168.20.11) but HTTPS simply ignores it and it is blocked.

At this point I believe that this could be a bug or most probable some sort of misconfiguration. I will keep poking around, but to be honest I am already planning to bring my pfSense back.

I am migrating to OPNsense because it offer VxLAN whilst pfSense only the paid version this feature is enabled

I will update this thread if I find a solution to this issue.

Many thanks.

[Monviech (Cedrik)]

If ICMP matches but HTTPS doesnt, there might be asynchronous traffic involved.

You could try to go to "Firewall: Rules: Floating" and open the firewall rule that should match the HTTPS traffic. Then you open the "Advanced features" in it and scroll all the way down to "State Type" and set it on "sloppy state". That will circumvent async traffic.

You can also check

"Firewall: Settings: Advanced" where you can "Disable Firewall" and see if routing only without firewall works. But be sure to read the help of that feature.

[markh0ppus]

Hi @Monviech,

Your help is much appreciated.

The problem was solved with your tips, and you were spot-on, pointing to Asymmetric Routing as the culprit. Also, pointed out as a probable cause of the issue by a nice fella(JN) at work.

I started to play around with the floating rules as suggested but could not get it to work. When turning off the "Block All" rule, the traffic flowed, indicating that routing was not a problem.

After some research, an article from pfSense explained possible asymmetric routing scenarios and how to solve them. Their example seemed to fit my issues.

Also, I saw some people online suggesting enabling "Static route filtering : Bypass firewall rules for traffic on the same interface" option under "Firewall>Settings>Advanced." It did nothing in my case because it auto-generates rules; however, I have modified OPNsense's source code not to generate automatic rules.

SOLUTION

pfSense's documentation suggested a manual fix adding specific rules to the firewall.

1) A rule was created under the VLAN02 ruleset.

Code: [Select]

Action: pass
Direction: in
Interface: VLAN02
Protocol: TCP
Source: VLAN02 Net (192.168.1.0/24 )
Destination: HOME Subnet (192.168.20.0/24)
Under Advanced
TCP Flags: Any Flags
State Type: sloppy state
2) And finally, a Floating rule to solve the issue.

Code: [Select]

Action: pass
Direction: out
Interface: VLAN02
Protocol: TCP
Source: VLAN02 Net (192.168.1.0/24 )
Destination: HOME Subnet (192.168.20.0/24)
Under Advanced
TCP Flags: Any Flags
State Type: sloppy state
Traffic now flows between the remote subnets

Thanks for all the help and tips provided.