Messages

I've faces a strange system behaviour after disabling of an existing static default route and adding a similar one for testing. The new route was not added after system reboot. But when I've deleted the disable route, the new one started to work (even after rebooting).

How to reproduce:
1. In System -> Routes -> Configuration:
1.1. Add a default route like "0.0.0.0/0 -> GW1" with the comment "New route", do not disable it
1.2. Copy the created route with a comment "Test disabled"
1.3. Disable the copied route
1.4. Save changes and restart the system
2. In System -> Routes -> Status: check, that there is no default route
3. Delete the copied disabled route and restart the system
4. In System -> Routes -> Status: check, that the default route "New route" is available

I've spent quite a lot of time, before figured out, that a disabled item has such side-effects :)

System information:
Type: opnsense   
Version: 23.7.11   
Architecture: amd64   
Commit: 0f520fffd

If you did not try and put all your logic into one rule with direction "any", but separate them out into two rules for in and out, you could then easily switch them on and off at will to further inspect and see which way packets are blocked and go further from there.

The floating rules like "any direction on any interface <some_condition>" are useful for logging packets matching the <some_condition>. So you can discover for example, in which direction and on which interfaces the packets are flowing. Also I've found out, that Any-Direction-Rules have no gateway / reply-to "problem".

While it may be instructive to bang your head at such points, to get something done, it may be easier to "got with the flow" and do things as they are expected to be done.

I'm feeling a little bit uncomfortable when doing things that I don't understand. And the information security is such a specific topic... You can think you are safe and work a long time without any issues. But if some day the bad-case scenario happens and you discovers, that your rules have not prevented it just because you didn't understand how they work, it will be really disappointing. I would better invest some time now in the learning.

You can't block anything generated by the firewall itself if that's what you're trying to do.

Yes, my goal is to block all kinds of outgoing traffic under certain circumstances, even if it was originated by the firewall itself. It seems to be possible to implement such behaviour. For example, by executing of a ping command directly at the firewall, it creates ICMP packets on the WAN interface. These packets can be captured and blocked by the follow OUT rule on the WAN interface:

Code Select Expand


Action: Block
Quick: Yes
Interface: WAN
Direction: Out
Protocol: ICMP
Destination: 8.8.8.8
Log: Yes
Description: Block ICMP to Google DNS


Firewall log:



The packet was blocked. There are no packets in the tcpdump also:



Ping to other hosts works properly. Follow your analogy, if a baby is born in this house, the only way to close exits to a dangerous street is to catch it when it will try to go thru a door :)

Can I ask, why would you want to block traffic from the firewall itself?? What exactly is being generated there?

Well, there are couple of reasons:

• If someone will get a non-admin access to the firewall, he will be able to send information to blocked servers in internet (but not to change rules!). In my situation some colleagues must have such access rights.
• Theoretically a kind of malware can be executed on the firewall, which will forward some traffic to the blocked internet servers. The IN rules will not help in this case.
• If I make an error by maintenance or reconfiguring the firewall, such blocking rule on WAN interface would be an additional protection, that guaranties, that not a single packet will be sent to the blocked servers.

Hi Demusman, thank you for the reply, your analogy is great!

Short answer, don't use OUT as a direction. It will only ever apply to floating rules and floating rules should not be used unless absolutely neccessary.
...
So never use OUT as a direction on an interface rule. It really shouldn't even be an option except on Floating rules.

Could you explain please, why the OUT direction can be evaluated only on floating rules? My original goal was to block some ip addresses in internet in some cases. Basically you are right, it can be done by blocking an inbound traffic on the LAN interface using an IN rule. But it will not block packets, issued by the firewall host itself. In this case there will be no inbound traffic at all, the packets will be created by the firewall host directly on the WAN interface. As of my understanding these packets can be blocked only using an OUT rule. I've created and tested such OUT rule on the WAN interface directly (not a floating one) - and it seems to work.

it ("reply-to effect") should be clearly visible if you look at the MAC addresses  imho

Bingo! The formatting is little bit ugly, so I've marked the WAN mac address with a green box and the gateway mac - with a red one:

imho this is exactly what I was talking about: there is no route-to and reply-to in the same rule

Sorry, I just didn't get your point for the first time. Btw, it seems to be a trick with the default rule "let out anything from firewall host itself". In the UI the rule has no gateway:



But in the stats it has NO reply-to option:



That is why the firewall works by default :) I've checked now the advanced option "Disable reply-to". It removes indeed the "inversion", but as you mentioned, it is off by default.

[without]

just check with "reply-to" disabled on "out" rules

That is a really good point! If a Gateway is set in the rule, then there is no reply-to in the stats:



It is exactly inverted, I've double checked it! Also, without a gateway in the rule, there is a reply-to option in the stats. And if a gateway is set, then there is no reply-to in the stats. Little bit confusing, but perhaps there are reasons for a such behaviour. Btw, a rule with the direction "any" has no reply-to by default:

because it received this with 8.8.8.8 source (and your WAN as a dest). its a ICMP echo reply packet
(i think you can try to tcpdump this to make sure)

The tcpdump shows follow picture:

Test 1.1 - ping with states (same as before, without reply-to)



The output looks as expected: the first packet (icmp request) with source ip WAN and destination IP 8.8.8.8 leaves the firewall and the second packet (icmp reply) with the source ip 8.8.8.8 and the destination IP WAN comes from the Google DNS to the firewall. Now let us check the test with an OUT rule:

Test 2.1 - ping with states with an OUT rule (same as before, without gateway hence with reply-to)



There are 3 packets after a single ping! One request and 2 replies.

Now hopefully I can understand, what happens:

• The firewall sends an outgoing icmp request packet from the WAN interface to the destination 8.8.8.8
• The firewall receives a reply from the 8.8.8.8 with destination WAN
• Because of the reply-to option, the reply packet is just forwarded to the reply-to IP (also, DHCP gateway). This is what we see in the tcpdump in the red square. This explains also, why we see an "out" packet on the WAN interface in the firewall log with the source 8.8.8.8. Just because it is an outgoing packet, cloned from the incoming one!

Fright, many thanks for the showing the right direction! The last point is my assumption, but I will prepare a setup with an intermediate gateway between the provider gateway and the firewall and check if the forwarded reply packet comes to it. After test I will update this post.

[source]

Hello Fright, thank you for the tips. I've checked the statistic as you suggested, but not sure what exactly can be wrong here. These are the stats for my IN and OUT rules:



The reply-to interface seems to be correct:

and  "strange" outgoing record with 8.8.8.8 source is not a "pretty strange"  ;)
(its a new (state creating) packet, that OPN sends from WAN to the "WAN_DHCP" gateway)

But if OPNsense sends the packet from the WAN to WAN_DHCP, why the source is 8.8.8.8?

[Result]

Hi meyergru, thank you for the reply.

Let me start from the last question. There is a section in the man pfctl, describing the ping states:

As we can see here, only one packet - the initial ping request - matched the table, but all packets passing as the result of the state are correctly accounted for.

There is also a description of UDP states in the OpenBSD PF FAQ:

While it is true that a UDP communication session does not have any concept of state (an explicit start and stop of communications), this does not have any impact on PF's ability to create state for a UDP session. In the case of protocols without "start" and "end" packets, PF simply keeps track of how long it has been since a matching packet has gone through. If the timeout is reached, the state is cleared. The timeout values can be set in the options section of the pf.conf file.

So I'm pretty sure, that the OPNsense treats the ICMP packets in the same way as UDP. Couple of screenshots of such sessions you can find in my last post.

...in order to have a working rule for direction "any", packets not only have to pass the firewall pipeline once, but twice (and in reverse directions).

No, not really. A rule with the direction "any" matches both outgoing and incoming packets. Even if you issue a single outgoing packet without any response, the rule will capture it. Proof:
1) create a rule with the direction "any", action "pass", and active logging
2) start tcpdump with the port 53
3) use Interfaces -> Diagnostics -> DNS Lookup for issuing of UDP packets that will surely get no replies:


Result

The outgoing packet was captured by the firewall even though there was no incoming packets as per tcpdump:

Also, I guess you also use NAT with implicit firewall rules and implicit back-translation of incoming traffic that might get in the way of your rules.

No, I haven't:



I'm really trying to follow your proposal to "isolate a single "problem". That is why I've prepared such a simple setup with a single network interface.

[is possible]

Yes, this is not a productive setup, but it allows to learn and test some basic concepts. For example, I've reproduced your case 1. and can confirm, that it is possible to see and filter the traffic on the firewall. For the verification I've created a single floating rule with follow settings:

Code Select Expand


Action: Pass
Quick: Checked
Interface: WAN    <<< Important!
Direction: Any
TCP/IP Version: IP4
Protocol: ICMP
Log: Checked
Description: Test ICMP
Gateway: default   <<< Important!


Test 1.1 - ping with states

For testing I've just issued a single ping:

Code Select Expand

ping -c1 8.8.8.8

The ICMP packet was captured by the firewall as an outgoing traffic (direction OUT):



The ping answer from 8.8.8.8 was not captured by the firewall, as a session was created by the outgoing ping request:



Test 1.2 - ping without states

Now let us test the same setup simulating a stateless firewall. For this purposes the rule should be extended with the follow Advanced Option:

Code Select Expand


State Type: None


The same ping command leads now to follow entries in the log:



The firewall captured 2 packets - one for outgoing traffic (direction OUT) and one for incoming traffic (direction IN). The state was not created, so the ping answer was captured by the firewall.

From my point of view, both test cases are working as desired.

Test 2.1 - ping with states with an OUT rule

In the last couple of tests we could see, that a rule with combined direction (ANY) works good. But if the direction is set explicitly, the incoming traffic is just blocked. For reproducing I've cloned the rule and set "Direction" to "In" in the first one and to "Out" in the second one. The ping receives no answer now:



In the firewall log we can see 2 outgoing packets, but no incomming:



As for me, the result looks pretty strange. First of all, why the upper packet with the source 8.8.8.8 was captured by the OUT rule? It is obviously an incoming traffic. Furthermore, the rule option "State Type" is set to the default value "Keep state" and 2 sessions was actually created:



Because of the created sessions the incoming packet should bypass the firewall rules. But as we can see, it is not so. BTW, why 2 sessions? The source and destination are the same. Should it not be 2 states, but a single session?

Test 2.2 - ping with states with an OUT rule with a gateway

Now let us set the parameter "Gateway" in the OUT rule. As we have only one interface, it can be done pretty easily:

Code Select Expand


Gateway: WAN_DHCP


The ping works again now!



There is only one outgoing packet in the log:



This is correct, because a session was created. And this time only a single one:



Conclusion

The test 1.2 shows, that an OUT rule without gateway leads to problems with states and blocks incoming traffic. Is it an error, or am I missing something?

Hi nzkiwi68, thank you for the firewall basics introduction. I've read the OPNsense documentation and also checked a couple of PF mans. Anyway, the logic of OUT rules is still unclear.

My setup is quite simple - just a single WAN interface, without LAN and without NAT. The interface is also the default gateway. As of my understanding, the only way to limit the outgoing traffic in this case - is to create an OUT rule. But even an empty OUT rule with the action PASS just blocks the traffic (without any log messages).

Hi there,

I've started to use the OPNsense for couple of weeks and really like it. It is a great software, many thanks to all contributors!

What I can't understand right now, is how rules with the direction "out" are working. For testing I've created a VM with a single network interface, configured by DHCP. After a standard OPNsense installation and applying updates, all works as expected. Traffic goes from the OPNsense host to the Internet (and back) without any problems. But when I'm creating a rule with the direction OUT and the action PASS (all other fields are default), it just stops working. Packets are leaving the firewall, but not coming back. For example, ping 8.8.8.8 doesn't receive any single packet.

After some trial and error I've found out, that the rule works as it should work, when the gateway is set explicitly in the rule. The host has only one network interface and the system routing table is correct and pretty simple. What is the reason for a such behaviour?