Strange behaviour of firewall OUT rules

[it_hardway]

Hi there,

I've started to use the OPNsense for couple of weeks and really like it. It is a great software, many thanks to all contributors!

What I can't understand right now, is how rules with the direction "out" are working. For testing I've created a VM with a single network interface, configured by DHCP. After a standard OPNsense installation and applying updates, all works as expected. Traffic goes from the OPNsense host to the Internet (and back) without any problems. But when I'm creating a rule with the direction OUT and the action PASS (all other fields are default), it just stops working. Packets are leaving the firewall, but not coming back. For example, ping 8.8.8.8 doesn't receive any single packet.

After some trial and error I've found out, that the rule works as it should work, when the gateway is set explicitly in the rule. The host has only one network interface and the system routing table is correct and pretty simple. What is the reason for a such behaviour?

[nzkiwi68]

When writing firewall rules, you need to think of packets from the firewalls point of view.

That's very important.

Firewall rules, in general, are set again an interface, and the direction is normally IN. That's because, from the firewall's point of view, a packet going out to internet from a LAN connected device, is received (hence IN) on the firewall's LAN interface. From there the packet does through the firewall rules, NAT and then leaves (out) on the WAN interface. With stateful firewalls (and OPNsense is most certainly a stateful firewall) you only write a firewall rule for the FIRST packet.

See the OPNsense documentation:
https://docs.opnsense.org/manual/firewall.html

[it_hardway]

Hi nzkiwi68, thank you for the firewall basics introduction. I've read the OPNsense documentation and also checked a couple of PF mans. Anyway, the logic of OUT rules is still unclear.

My setup is quite simple - just a single WAN interface, without LAN and without NAT. The interface is also the default gateway. As of my understanding, the only way to limit the outgoing traffic in this case - is to create an OUT rule. But even an empty OUT rule with the action PASS just blocks the traffic (without any log messages).

[nzkiwi68]

That's an unusual setup.
I wouldn't recommend this setup.

1.
If the VM has the ISP default gateway, then, the traffic won't be touching the firewall at all.
Consider this example:
FW WAN ip is 202.202.202.1
VM IP: 202.202.202.100
ISP gateway: 202.202.202.254

Then, the client traffic will go directly to the ISP gateway and the firewall will never see the traffic, hence rules not working.

2.
If the VM does have the FW's IP address as the gateway, then:
Make sure under:
Firewall > Settings > Advanced
"Static route filtering" is not ticked to make the firewall check WAN traffic that arrives (in) and then leaves again.

If it is ticked, then since the traffic enters the WAN and then leaves the WAN, then "Bypass firewall rules for traffic on the same interface" operates.

[it_hardway]

Yes, this is not a productive setup, but it allows to learn and test some basic concepts. For example, I've reproduced your case 1. and can confirm, that it is possible to see and filter the traffic on the firewall. For the verification I've created a single floating rule with follow settings:

Code Select Expand


Action: Pass
Quick: Checked
Interface: WAN    <<< Important!
Direction: Any
TCP/IP Version: IP4
Protocol: ICMP
Log: Checked
Description: Test ICMP
Gateway: default   <<< Important!


Test 1.1 - ping with states

For testing I've just issued a single ping:

Code Select Expand

ping -c1 8.8.8.8

The ICMP packet was captured by the firewall as an outgoing traffic (direction OUT):



The ping answer from 8.8.8.8 was not captured by the firewall, as a session was created by the outgoing ping request:



Test 1.2 - ping without states

Now let us test the same setup simulating a stateless firewall. For this purposes the rule should be extended with the follow Advanced Option:

Code Select Expand


State Type: None


The same ping command leads now to follow entries in the log:



The firewall captured 2 packets - one for outgoing traffic (direction OUT) and one for incoming traffic (direction IN). The state was not created, so the ping answer was captured by the firewall.

From my point of view, both test cases are working as desired.

Test 2.1 - ping with states with an OUT rule

In the last couple of tests we could see, that a rule with combined direction (ANY) works good. But if the direction is set explicitly, the incoming traffic is just blocked. For reproducing I've cloned the rule and set "Direction" to "In" in the first one and to "Out" in the second one. The ping receives no answer now:



In the firewall log we can see 2 outgoing packets, but no incomming:



As for me, the result looks pretty strange. First of all, why the upper packet with the source 8.8.8.8 was captured by the OUT rule? It is obviously an incoming traffic. Furthermore, the rule option "State Type" is set to the default value "Keep state" and 2 sessions was actually created:



Because of the created sessions the incoming packet should bypass the firewall rules. But as we can see, it is not so. BTW, why 2 sessions? The source and destination are the same. Should it not be 2 states, but a single session?

Test 2.2 - ping with states with an OUT rule with a gateway

Now let us set the parameter "Gateway" in the OUT rule. As we have only one interface, it can be done pretty easily:

Code Select Expand


Gateway: WAN_DHCP


The ping works again now!



There is only one outgoing packet in the log:



This is correct, because a session was created. And this time only a single one:



Conclusion

The test 1.2 shows, that an OUT rule without gateway leads to problems with states and blocks incoming traffic. Is it an error, or am I missing something?

[meyergru]

For starters, you should look at how the pipeline of rules for floating and interfaces work and realise that in order to have a working rule for direction "any", packets not only have to pass the firewall pipeline once, but twice (and in reverse directions).

Also, I guess you also use NAT with implicit firewall rules and implicit back-translation of incoming traffic that might get in the way of your rules.

I never tried it, but I guess that when direction "any" rules seldomly make sense, they make even less sense with NAT involved. There are simply too many variables involved, you should try to isolate a single "problem" before jumping to conclusions (and I doubt you'll find any).

P.S.: What do you think a "session" is with ICMP packets?

[it_hardway]

Hi meyergru, thank you for the reply.

Let me start from the last question. There is a section in the man pfctl, describing the ping states:

As we can see here, only one packet - the initial ping request - matched the table, but all packets passing as the result of the state are correctly accounted for.

There is also a description of UDP states in the OpenBSD PF FAQ:

While it is true that a UDP communication session does not have any concept of state (an explicit start and stop of communications), this does not have any impact on PF's ability to create state for a UDP session. In the case of protocols without "start" and "end" packets, PF simply keeps track of how long it has been since a matching packet has gone through. If the timeout is reached, the state is cleared. The timeout values can be set in the options section of the pf.conf file.

So I'm pretty sure, that the OPNsense treats the ICMP packets in the same way as UDP. Couple of screenshots of such sessions you can find in my last post.

...in order to have a working rule for direction "any", packets not only have to pass the firewall pipeline once, but twice (and in reverse directions).

No, not really. A rule with the direction "any" matches both outgoing and incoming packets. Even if you issue a single outgoing packet without any response, the rule will capture it. Proof:
1) create a rule with the direction "any", action "pass", and active logging
2) start tcpdump with the port 53
3) use Interfaces -> Diagnostics -> DNS Lookup for issuing of UDP packets that will surely get no replies:


Result

The outgoing packet was captured by the firewall even though there was no incoming packets as per tcpdump:

Also, I guess you also use NAT with implicit firewall rules and implicit back-translation of incoming traffic that might get in the way of your rules.

No, I haven't:



I'm really trying to follow your proposal to "isolate a single "problem". That is why I've prepared such a simple setup with a single network interface.

[Fright]

or am I missing something?

maybe.
try to look at actual rule syntax at Firewall: Diagnostics: Statistics -> rules
is "reply-to" is there for your "out" rule with "Test 2.1" setup?
and  "strange" outgoing record with 8.8.8.8 source will stop be a "pretty strange"  ;)
(its a new (state creating) packet, that OPN sends from WAN to the "WAN_DHCP" gateway)

[it_hardway]

Hello Fright, thank you for the tips. I've checked the statistic as you suggested, but not sure what exactly can be wrong here. These are the stats for my IN and OUT rules:



The reply-to interface seems to be correct:

and  "strange" outgoing record with 8.8.8.8 source is not a "pretty strange"  ;)
(its a new (state creating) packet, that OPN sends from WAN to the "WAN_DHCP" gateway)

But if OPNsense sends the packet from the WAN to WAN_DHCP, why the source is 8.8.8.8?

[Fright]

But if OPNsense sends the packet from the WAN to WAN_DHCP, why the source is 8.8.8.8?

because it received this with 8.8.8.8 source (and your WAN as a dest). its a ICMP echo reply packet
(i think you can try to tcpdump this to make sure)

[Fright]

not sure what exactly can be wrong here

not "wrong" per se but..
@69 rule says "create state for outgoing icmp on em0. AND (reply-to) when reply comes - send it to *.*.55.1"

just check with "reply-to" disabled on "out" rules.

upd: forgot to say, when you set gateway (test 2.2 setup) this removes "reply-to" from rule and icmp replies goes  as expected

[it_hardway]

just check with "reply-to" disabled on "out" rules

That is a really good point! If a Gateway is set in the rule, then there is no reply-to in the stats:



It is exactly inverted, I've double checked it! Also, without a gateway in the rule, there is a reply-to option in the stats. And if a gateway is set, then there is no reply-to in the stats. Little bit confusing, but perhaps there are reasons for a such behaviour. Btw, a rule with the direction "any" has no reply-to by default:

because it received this with 8.8.8.8 source (and your WAN as a dest). its a ICMP echo reply packet
(i think you can try to tcpdump this to make sure)

The tcpdump shows follow picture:

Test 1.1 - ping with states (same as before, without reply-to)



The output looks as expected: the first packet (icmp request) with source ip WAN and destination IP 8.8.8.8 leaves the firewall and the second packet (icmp reply) with the source ip 8.8.8.8 and the destination IP WAN comes from the Google DNS to the firewall. Now let us check the test with an OUT rule:

Test 2.1 - ping with states with an OUT rule (same as before, without gateway hence with reply-to)



There are 3 packets after a single ping! One request and 2 replies.

Now hopefully I can understand, what happens:

• The firewall sends an outgoing icmp request packet from the WAN interface to the destination 8.8.8.8
• The firewall receives a reply from the 8.8.8.8 with destination WAN
• Because of the reply-to option, the reply packet is just forwarded to the reply-to IP (also, DHCP gateway). This is what we see in the tcpdump in the red square. This explains also, why we see an "out" packet on the WAN interface in the firewall log with the source 8.8.8.8. Just because it is an outgoing packet, cloned from the incoming one!

Fright, many thanks for the showing the right direction! The last point is my assumption, but I will prepare a setup with an intermediate gateway between the provider gateway and the firewall and check if the forwarded reply packet comes to it. After test I will update this post.

[Fright]

It is exactly inverted, I've double checked it! Also, without a gateway in the rule, there is a reply-to option in the stats. And if a gateway is set, then there is no reply-to in the stats.

imho this is exactly what I was talking about: there is no route-to and reply-to in the same rule

I did not notice this in man, but at one time looking at the source code it seemed to me that the routing options (route-to, reply-to, dup-to) are mutually exclusive
https://github.com/opnsense/src/blob/47fa565344c38a0d104e3913920546f07d7807e0/sbin/pfctl/parse.y#L4580
(and thats why im still not sure that enabling a reply-to on "out"-rules by default is the right choice)

There are 3 packets after a single ping

it ("reply-to effect") should be clearly visible if you look at the MAC addresses  imho

[Demusman]

When writing firewall rules, you need to think of packets from the firewalls point of view.

That's very important.

Firewall rules, in general, are set again an interface, and the direction is normally IN. That's because, from the firewall's point of view, a packet going out to internet from a LAN connected device, is received (hence IN) on the firewall's LAN interface. From there the packet does through the firewall rules, NAT and then leaves (out) on the WAN interface. With stateful firewalls (and OPNsense is most certainly a stateful firewall) you only write a firewall rule for the FIRST packet.

See the OPNsense documentation:
https://docs.opnsense.org/manual/firewall.html

Short answer, don't use OUT as a direction. It will only ever apply to floating rules and floating rules should not be used unless absolutely neccessary.

Long, in PF rules are applied on the interface as traffic enters that interface. First rule that matches is applied and no other rules are evaluated, so order matters.
The best analogy I found is using a house.
Your house has many doors in it, these are the interfaces. If you want to stop a packet from exiting your back door, you don't let it in the front door and then tell it the backdoor is off limits. You block it from entering the front door. Thereby blocking all other doors.
So if a packet is allowed in the front door, it then has access to any of the other doors you allowed it from the front door.
It makes sense when you wrap your head around it.

You also need to understand that IN and OUT refer to the same interface.
Take the LAN interface. IN is a packet entering the interface FROM the directly connected network. (LAN)
OUT is a packet leaving the interface TO the directly connected network. (LAN)
A lot of people think LAN OUT is to the WAN or other interfaces, it's not.

So never use OUT as a direction on an interface rule. It really shouldn't even be an option except on Floating rules.
Use direction IN and set the rules to what that packet can access.

[it_hardway]

it ("reply-to effect") should be clearly visible if you look at the MAC addresses  imho

Bingo! The formatting is little bit ugly, so I've marked the WAN mac address with a green box and the gateway mac - with a red one:

imho this is exactly what I was talking about: there is no route-to and reply-to in the same rule

Sorry, I just didn't get your point for the first time. Btw, it seems to be a trick with the default rule "let out anything from firewall host itself". In the UI the rule has no gateway:



But in the stats it has NO reply-to option:



That is why the firewall works by default :) I've checked now the advanced option "Disable reply-to". It removes indeed the "inversion", but as you mentioned, it is off by default.