Hi there,
I've started to use the OPNsense for couple of weeks and really like it. It is a great software, many thanks to all contributors!
What I can't understand right now, is how rules with the direction "out" are working. For testing I've created a VM with a single network interface, configured by DHCP. After a standard OPNsense installation and applying updates, all works as expected. Traffic goes from the OPNsense host to the Internet (and back) without any problems. But when I'm creating a rule with the direction OUT and the action PASS (all other fields are default), it just stops working. Packets are leaving the firewall, but not coming back. For example, ping 8.8.8.8 doesn't receive any single packet.
After some trial and error I've found out, that the rule works as it should work, when the gateway is set explicitly in the rule. The host has only one network interface and the system routing table is correct and pretty simple. What is the reason for a such behaviour?
When writing firewall rules, you need to think of packets from the firewalls point of view.
That's very important.
Firewall rules, in general, are set again an interface, and the direction is normally IN. That's because, from the firewall's point of view, a packet going out to internet from a LAN connected device, is received (hence IN) on the firewall's LAN interface. From there the packet does through the firewall rules, NAT and then leaves (out) on the WAN interface. With stateful firewalls (and OPNsense is most certainly a stateful firewall) you only write a firewall rule for the FIRST packet.
See the OPNsense documentation:
https://docs.opnsense.org/manual/firewall.html (https://docs.opnsense.org/manual/firewall.html)
Hi nzkiwi68, thank you for the firewall basics introduction. I've read the OPNsense documentation and also checked a couple of PF mans. Anyway, the logic of OUT rules is still unclear.
My setup is quite simple - just a single WAN interface, without LAN and without NAT. The interface is also the default gateway. As of my understanding, the only way to limit the outgoing traffic in this case - is to create an OUT rule. But even an empty OUT rule with the action PASS just blocks the traffic (without any log messages).
That's an unusual setup.
I wouldn't recommend this setup.
1.
If the VM has the ISP default gateway, then, the traffic won't be touching the firewall at all.
Consider this example:
FW WAN ip is 202.202.202.1
VM IP: 202.202.202.100
ISP gateway: 202.202.202.254
Then, the client traffic will go directly to the ISP gateway and the firewall will never see the traffic, hence rules not working.
2.
If the VM does have the FW's IP address as the gateway, then:
Make sure under:
Firewall > Settings > Advanced
"Static route filtering" is not ticked to make the firewall check WAN traffic that arrives (in) and then leaves again.
If it is ticked, then since the traffic enters the WAN and then leaves the WAN, then "Bypass firewall rules for traffic on the same interface" operates.
Yes, this is not a productive setup, but it allows to learn and test some basic concepts. For example, I've reproduced your case 1. and can confirm, that it is possible to see and filter the traffic on the firewall. For the verification I've created a single floating rule with follow settings:
Action: Pass
Quick: Checked
Interface: WAN <<< Important!
Direction: Any
TCP/IP Version: IP4
Protocol: ICMP
Log: Checked
Description: Test ICMP
Gateway: default <<< Important!
Test 1.1 - ping with states
For testing I've just issued a single ping:
ping -c1 8.8.8.8
The ICMP packet was captured by the firewall as an outgoing traffic (direction OUT):
(https://www.dropbox.com/s/xjjxblr58llb86r/test_1_1.png?dl=1)
The ping answer from 8.8.8.8 was not captured by the firewall, as a session was created by the outgoing ping request:
(https://www.dropbox.com/s/ippeqjj7z7p4u7g/test_1_2.png?dl=1)
Test 1.2 - ping without states
Now let us test the same setup simulating a stateless firewall. For this purposes the rule should be extended with the follow Advanced Option:
State Type: None
The same ping command leads now to follow entries in the log:
(https://www.dropbox.com/s/hebq4yj9rb7r7jy/test_1_3.png?dl=1)
The firewall captured 2 packets - one for outgoing traffic (direction OUT) and one for incoming traffic (direction IN). The state was not created, so the ping answer was captured by the firewall.
From my point of view, both test cases are working as desired.
Test 2.1 - ping with states with an OUT rule
In the last couple of tests we could see, that a rule with combined direction (ANY) works good. But if the direction is set explicitly, the incoming traffic is just blocked. For reproducing I've cloned the rule and set "Direction" to "In" in the first one and to "Out" in the second one. The ping receives no answer now:
(https://www.dropbox.com/s/1yi4pmucd416q09/test_2_1.png?dl=1)
In the firewall log we can see 2 outgoing packets, but no incomming:
(https://www.dropbox.com/s/chc7bl45ziq21c2/test_2_2.png?dl=1)
As for me, the result looks pretty strange. First of all, why the upper packet with the source 8.8.8.8 was captured by the OUT rule? It is obviously an incoming traffic. Furthermore, the rule option "State Type" is set to the default value "Keep state" and 2 sessions was actually created:
(https://www.dropbox.com/s/ikb0d1bzzg8wqzu/test_2_3.png?dl=1)
Because of the created sessions the incoming packet should bypass the firewall rules. But as we can see, it is not so. BTW, why 2 sessions? The source and destination are the same. Should it not be 2 states, but a single session?
Test 2.2 - ping with states with an OUT rule with a gateway
Now let us set the parameter "Gateway" in the OUT rule. As we have only one interface, it can be done pretty easily:
Gateway: WAN_DHCP
The ping works again now!
(https://www.dropbox.com/s/stpntheskuqptnc/test_2_4.png?dl=1)
There is only one outgoing packet in the log:
(https://www.dropbox.com/s/6x5pcw2j3fzjny4/test_2_5.png?dl=1)
This is correct, because a session was created. And this time only a single one:
(https://www.dropbox.com/s/1fjbsrfjigqr0l6/test_2_6.png?dl=1)
Conclusion
The test 1.2 shows, that an OUT rule without gateway leads to problems with states and blocks incoming traffic. Is it an error, or am I missing something?
For starters, you should look at how the pipeline of rules for floating and interfaces work and realise that in order to have a working rule for direction "any", packets not only have to pass the firewall pipeline once, but twice (and in reverse directions).
Also, I guess you also use NAT with implicit firewall rules and implicit back-translation of incoming traffic that might get in the way of your rules.
I never tried it, but I guess that when direction "any" rules seldomly make sense, they make even less sense with NAT involved. There are simply too many variables involved, you should try to isolate a single "problem" before jumping to conclusions (and I doubt you'll find any).
P.S.: What do you think a "session" is with ICMP packets?
Hi meyergru, thank you for the reply.
Let me start from the last question. There is a section in the man pfctl (https://man.freebsd.org/cgi/man.cgi?query=pfctl(8)), describing the ping states:
QuoteAs we can see here, only one packet - the initial ping request - matched the table, but all packets passing as the result of the state are correctly accounted for.
There is also a description of UDP states in the OpenBSD PF FAQ (https://www.openbsd.org/faq/pf/filter.html#udpstate):
QuoteWhile it is true that a UDP communication session does not have any concept of state (an explicit start and stop of communications), this does not have any impact on PF's ability to create state for a UDP session. In the case of protocols without "start" and "end" packets, PF simply keeps track of how long it has been since a matching packet has gone through. If the timeout is reached, the state is cleared. The timeout values can be set in the options section of the pf.conf file.
So I'm pretty sure, that the OPNsense treats the ICMP packets in the same way as UDP. Couple of screenshots of such sessions you can find in my last post.
Quote from: meyergru on April 10, 2023, 03:56:32 PM
...in order to have a working rule for direction "any", packets not only have to pass the firewall pipeline once, but twice (and in reverse directions).
No, not really. A rule with the direction "any" matches both outgoing and incoming packets. Even if you issue a single outgoing packet without any response, the rule will capture it. Proof:
1) create a rule with the direction "any", action "pass", and active logging
2) start tcpdump with the port 53
3) use Interfaces -> Diagnostics -> DNS Lookup for issuing of UDP packets that will surely get no replies:
(https://www.dropbox.com/s/zpagybc158rqelq/test_3_1.png?dl=1)
ResultThe outgoing packet was captured by the firewall even though there was no incoming packets as per tcpdump:
(https://www.dropbox.com/s/80q8ylnrzwapfxb/test_3_2.png?dl=1)
(https://www.dropbox.com/s/pi6w6eu97t9sqcy/test_3_3.png?dl=1)
Quote from: meyergru on April 10, 2023, 03:56:32 PM
Also, I guess you also use NAT with implicit firewall rules and implicit back-translation of incoming traffic that might get in the way of your rules.
No, I haven't:
(https://www.dropbox.com/s/rtvzbo4pkxuisjz/test_3_4.png?dl=1)
I'm really trying to follow your proposal to "isolate a single "problem". That is why I've prepared such a simple setup with a single network interface.
Quoteor am I missing something?
maybe.
try to look at actual rule syntax at Firewall: Diagnostics: Statistics -> rules
is "reply-to" is there for your "out" rule with "Test 2.1" setup?
and "strange" outgoing record with 8.8.8.8 source will stop be a "pretty strange" ;)
(its a new (state creating) packet, that OPN sends from WAN to the "WAN_DHCP" gateway)
Hello Fright, thank you for the tips. I've checked the statistic as you suggested, but not sure what exactly can be wrong here. These are the stats for my IN and OUT rules:
(https://www.dropbox.com/s/ygpq9iolf2ijs30/test_3_5.png?dl=1)
The reply-to interface seems to be correct:
(https://www.dropbox.com/s/wjux0crzbofzx6k/test_3_6.png?dl=1)
Quote from: Fright on April 10, 2023, 07:30:01 PM
and "strange" outgoing record with 8.8.8.8 source is not a "pretty strange" ;)
(its a new (state creating) packet, that OPN sends from WAN to the "WAN_DHCP" gateway)
But if OPNsense sends the packet from the WAN to WAN_DHCP, why the
source is 8.8.8.8?
QuoteBut if OPNsense sends the packet from the WAN to WAN_DHCP, why the source is 8.8.8.8?
because it received this with 8.8.8.8 source (and your WAN as a dest). its a ICMP echo reply packet
(i think you can try to tcpdump this to make sure)
Quotenot sure what exactly can be wrong here
not "wrong" per se but..
@69 rule says "create state for outgoing icmp on em0.
AND (reply-to) when reply comes - send it to *.*.55.1"
just check with "reply-to" disabled on "out" rules.
upd: forgot to say, when you set gateway (test 2.2 setup) this removes "reply-to" from rule and icmp replies goes as expected
Quote from: Fright on April 10, 2023, 08:48:02 PM
just check with "reply-to" disabled on "out" rules
That is a really good point! If a Gateway is set in the rule, then there is no reply-to in the stats:
(https://www.dropbox.com/s/t4kofhgglm7cnzm/test_4_1.png?dl=1)
It is exactly inverted, I've double checked it! Also,
without a gateway in the rule,
there is a reply-to option in the stats. And if
a gateway is set, then there is
no reply-to in the stats. Little bit confusing, but perhaps there are reasons for a such behaviour. Btw, a rule with the direction "any" has no reply-to by default:
(https://www.dropbox.com/s/ognazucqo689zz6/test_4_2.png?dl=1)
Quote from: Fright on April 10, 2023, 08:33:36 PM
because it received this with 8.8.8.8 source (and your WAN as a dest). its a ICMP echo reply packet
(i think you can try to tcpdump this to make sure)
The tcpdump shows follow picture:
Test 1.1 - ping with states (same as before, without reply-to)
(https://www.dropbox.com/s/y87piaafenrqxwf/test_4_3.png?dl=1)
The output looks as expected: the first packet (icmp request) with source ip WAN and destination IP 8.8.8.8 leaves the firewall and the second packet (icmp reply) with the source ip 8.8.8.8 and the destination IP WAN comes from the Google DNS to the firewall. Now let us check the test with an OUT rule:
Test 2.1 - ping with states with an OUT rule (same as before, without gateway hence
with reply-to)
(https://www.dropbox.com/s/rz8yq8ywb65ibe8/test_4_5.png?dl=1)
There are 3 packets after a single ping! One request and 2 replies.
Now hopefully I can understand, what happens:
- The firewall sends an outgoing icmp request packet from the WAN interface to the destination 8.8.8.8
- The firewall receives a reply from the 8.8.8.8 with destination WAN
- Because of the reply-to option, the reply packet is just forwarded to the reply-to IP (also, DHCP gateway). This is what we see in the tcpdump in the red square. This explains also, why we see an "out" packet on the WAN interface in the firewall log with the source 8.8.8.8. Just because it is an outgoing packet, cloned from the incoming one!
Fright, many thanks for the showing the right direction! The last point is my assumption, but I will prepare a setup with an intermediate gateway between the provider gateway and the firewall and check if the forwarded reply packet comes to it. After test I will update this post.
QuoteIt is exactly inverted, I've double checked it! Also, without a gateway in the rule, there is a reply-to option in the stats. And if a gateway is set, then there is no reply-to in the stats.
imho this is exactly what I was talking about: there is no route-to and reply-to in the same rule
I did not notice this in man, but at one time looking at the source code it seemed to me that the routing options (route-to, reply-to, dup-to) are mutually exclusive
https://github.com/opnsense/src/blob/47fa565344c38a0d104e3913920546f07d7807e0/sbin/pfctl/parse.y#L4580
(and thats why im still not sure that enabling a reply-to on "out"-rules by default is the right choice)
QuoteThere are 3 packets after a single ping
it ("reply-to effect") should be clearly visible if you look at the MAC addresses imho
Quote from: nzkiwi68 on April 10, 2023, 04:53:11 AM
When writing firewall rules, you need to think of packets from the firewalls point of view.
That's very important.
Firewall rules, in general, are set again an interface, and the direction is normally IN. That's because, from the firewall's point of view, a packet going out to internet from a LAN connected device, is received (hence IN) on the firewall's LAN interface. From there the packet does through the firewall rules, NAT and then leaves (out) on the WAN interface. With stateful firewalls (and OPNsense is most certainly a stateful firewall) you only write a firewall rule for the FIRST packet.
See the OPNsense documentation:
https://docs.opnsense.org/manual/firewall.html (https://docs.opnsense.org/manual/firewall.html)
Short answer, don't use OUT as a direction. It will only ever apply to floating rules and floating rules should not be used unless absolutely neccessary.
Long, in PF rules are applied on the interface as traffic enters that interface. First rule that matches is applied and no other rules are evaluated, so order matters.
The best analogy I found is using a house.
Your house has many doors in it, these are the interfaces. If you want to stop a packet from exiting your back door, you don't let it in the front door and then tell it the backdoor is off limits. You block it from entering the front door. Thereby blocking all other doors.
So if a packet is allowed in the front door, it then has access to any of the other doors you allowed it from the front door.
It makes sense when you wrap your head around it.
You also need to understand that IN and OUT refer to the same interface.
Take the LAN interface. IN is a packet entering the interface FROM the directly connected network. (LAN)
OUT is a packet leaving the interface TO the directly connected network. (LAN)
A lot of people think LAN OUT is to the WAN or other interfaces, it's not.
So never use OUT as a direction on an interface rule. It really shouldn't even be an option except on Floating rules.
Use direction IN and set the rules to what that packet can access.
Quote from: Fright on April 10, 2023, 10:15:58 PM
it ("reply-to effect") should be clearly visible if you look at the MAC addresses imho
Bingo! The formatting is little bit ugly, so I've marked the WAN mac address with a green box and the gateway mac - with a red one:
(https://www.dropbox.com/s/v8b845urb7ed0zx/test_4_6.png?dl=1)
Quote from: Fright on April 10, 2023, 10:15:58 PM
imho this is exactly what I was talking about: there is no route-to and reply-to in the same rule
Sorry, I just didn't get your point for the first time. Btw, it seems to be a trick with the default rule "let out anything from firewall host itself". In the UI the rule has no gateway:
(https://www.dropbox.com/s/parodshlyrgkdqa/test_4_8.png?dl=1)
But in the stats it has NO reply-to option:
(https://www.dropbox.com/s/vgstuiqkqcjxnby/test_4_7.png?dl=1)
That is why the firewall works by default :) I've checked now the advanced option "Disable reply-to". It removes indeed the "inversion", but as you mentioned, it is off by default.
Hi Demusman, thank you for the reply, your analogy is great!
Quote from: Demusman on April 10, 2023, 10:43:21 PM
Short answer, don't use OUT as a direction. It will only ever apply to floating rules and floating rules should not be used unless absolutely neccessary.
...
So never use OUT as a direction on an interface rule. It really shouldn't even be an option except on Floating rules.
Could you explain please, why the OUT direction can be evaluated only on floating rules? My original goal was to block some ip addresses in internet in some cases. Basically you are right, it can be done by blocking an inbound traffic on the LAN interface using an IN rule. But it will not block packets, issued by the firewall host itself. In this case there will be no inbound traffic at all, the packets will be created by the firewall host directly on the WAN interface. As of my understanding these packets can be blocked only using an OUT rule. I've created and tested such OUT rule on the WAN interface directly (not a floating one) - and it seems to work.
You can't block anything generated by the firewall itself if that's what you're trying to do.
All rules are evaluated on the IN of the interfaces, the firewall would never generate traffic INTO an interface.
If you want to block traffic from the internet, you put the rules on the WAN.
Again, IN to an interface is traffic generated from the directly connected network of that interface. So the WAN would be directly connected to the internet by way of your WAN Network alias.
Can I ask, why would you want to block traffic from the firewall itself?? What exactly is being generated there?
If you did not try and put all your logic into one rule with direction "any", but separate them out into two rules for in and out, you could then easily switch them on and off at will to further inspect and see which way packets are blocked and go further from there.
That is what I meant by 'isolate a single "problem"'.
Your setup only seemed minimal, since by using only one interface, you had to use the "unusual" out direction (in fact, you even contracted two rules into one by using "any"). Usually, as Demusman said, you filter in the "in" direction at whatever is coming at the firewall from either LAN or WAN. Unless the firewall is infected, you do not have to control what leaves it.
While it may be instructive to bang your head at such points, to get something done, it may be easier to "got with the flow" and do things as they are expected to be done.
Quote from: Demusman on April 11, 2023, 02:51:11 AM
You can't block anything generated by the firewall itself if that's what you're trying to do.
Yes, my goal is to block all kinds of outgoing traffic under certain circumstances, even if it was originated by the firewall itself. It seems to be possible to implement such behaviour. For example, by executing of a ping command directly at the firewall, it creates ICMP packets on the WAN interface. These packets can be captured and blocked by the follow OUT rule on the WAN interface:
Action: Block
Quick: Yes
Interface: WAN
Direction: Out
Protocol: ICMP
Destination: 8.8.8.8
Log: Yes
Description: Block ICMP to Google DNS
Firewall log:
(https://www.dropbox.com/s/pi9fiz9rhcuqw2i/test_5_2.png?dl=1)
The packet was blocked. There are no packets in the tcpdump also:
(https://www.dropbox.com/s/wkyhuy8xclpcw8c/test_5_1.png?dl=1)
Ping to other hosts works properly. Follow your analogy, if a baby is born in this house, the only way to close exits to a dangerous street is to catch it when it will try to go thru a door :)
Quote from: Demusman on April 11, 2023, 02:51:11 AM
Can I ask, why would you want to block traffic from the firewall itself?? What exactly is being generated there?
Well, there are couple of reasons:
- If someone will get a non-admin access to the firewall, he will be able to send information to blocked servers in internet (but not to change rules!). In my situation some colleagues must have such access rights.
- Theoretically a kind of malware can be executed on the firewall, which will forward some traffic to the blocked internet servers. The IN rules will not help in this case.
- If I make an error by maintenance or reconfiguring the firewall, such blocking rule on WAN interface would be an additional protection, that guaranties, that not a single packet will be sent to the blocked servers.
Quote from: meyergru on April 11, 2023, 08:55:12 AM
If you did not try and put all your logic into one rule with direction "any", but separate them out into two rules for in and out, you could then easily switch them on and off at will to further inspect and see which way packets are blocked and go further from there.
The floating rules like "any direction on any interface <some_condition>" are useful for logging packets matching the <some_condition>. So you can discover for example, in which direction and on which interfaces the packets are flowing. Also I've found out, that Any-Direction-Rules have no gateway / reply-to "problem".
Quote from: meyergru on April 11, 2023, 08:55:12 AM
While it may be instructive to bang your head at such points, to get something done, it may be easier to "got with the flow" and do things as they are expected to be done.
I'm feeling a little bit uncomfortable when doing things that I don't understand. And the information security is such a specific topic... You can think you are safe and work a long time without any issues. But if some day the bad-case scenario happens and you discovers, that your rules have not prevented it just because you didn't understand how they work, it will be really disappointing. I would better invest some time now in the learning.