OPNsense Forum
Archive => 20.1 Legacy Series => Topic started by: pv2b on February 17, 2020, 04:58:44 pm
-
I have some feedback regarding the default behavior for inbound firewall rules on the WAN interface of OPNsense 20.1, and I believe this problem is also present in earlier versions.
Consider a network that looks like this:
__ _
_( )_( )_
(_ _ _) Internet
(_) (__)
|
.---'--------. .-----------.
| WAN Router | | Computer |
| 192.0.2.1 | | 192.0.2.2 |
'---.--------' '-----.-----'
| |
| |
.---'-------------------'---.
| 192.0.2.0/24 WAN Network |
'------.--------------------'
|
| WAN (igb1)
.------'------------.
| OPNsense Firewall |
| 192.0.2.3 |
| 192.168.0.1 |
'------.------------'
| LAN (igb0)
|
.------'---------------------.
| 192.168.0.0/24 LAN Network |
'----------------------------'
Consider that there is a firewall rule the OPNsense box, on the WAN interface which permits traffic from 192.0.2.2 to 192.0.2.3 TCP PORT 443. All other options default.
By default, OPNsense seems to create this firewall rule with a "reply-to" clause, which means that any reponse to the SYN sent by the computer is not sent back to the computer, but instead back to the default gateway on the WAN network.
This will work most of the time (except wasting sending traffic to and from the gateway), but if the gateway is itself another stateful firewall (like another OPNsense box), this causes the connection to never be established.
The solution is to check "disable reply-to" on the inbound firewall rule on the WAN.
Not sure when this became the default, it seems to have been the case since at least 2018. https://forum.opnsense.org/index.php?topic=8841.0 Although I'm not finding this global checkbox that @franco was referring to.
Now, I understand the reason for this default "reply-to" behavior, which is to ensure that in a multi-WAN scenario, you don't get triangular routing, but instead the way it's implemented, you end up with triangular traffic flow in the basic case.
A sane default would be to not enable reply-to unless the user selects a gateway on the rule. (In fact, when you just select "default" as the gateway, the help says the system routing table is used, which is NOT true.)
Another possible fix for a solution that "just works" that both works for preventing triangular routing in multi-WAN scenario and doesn't break local WAN commmunication is to install two pf rules in the back end, which disables reply-to for traffic coming from a host directly connected on the WAN network. Something like:
pass in quick on igb1 inet proto tcp from {{igb1:network}} to {{self}} port {443} keep state
pass in quick on igb1 reply-to { igb1 192.0.2.1 } inet proto tcp from {any} to {{self}} port {443} keep state
Of course all of this can be configured by an admin directly in the GUI, if you don't mind some doubled firewall rules.
To summarize, I don't expect Policy Based Routing to be implemented by default unless i specifically turn it off in advanced settings.
-
Hi,
You can find the general checkbox "Disable reply-to" in system_advanced_firewall.php, which disables the default as Franco mentioned in the other post.
This rule has been the default since probably pfSense 1.2.x, since many installations depend on it, it's unlikely that we're going to change it.
The toggle to disable it was introduced in https://github.com/pfsense/pfsense/commit/efefb2a1e860d082a6024b7c6b67c646b1e8aa6e more than a decade ago.
When designing such functionality now, we might make other choices, but some habits are better left alone.
Best regards,
Ad
-
Thanks for showing me the checkbox.
I wasn't aware this was ever since pfSense 1.2. I'm surprised this hasn't bitten me yet, but then again I rarely deploy an OPNsense box behind a stateful firewall on the WAN interface. But I can definitely say that a lot of traffic has been needlessly flowing through the default gateway on some of my installed sites without me ever realizing, leaving performance on the table due to this misfeature. It just never broke anything because typically the gateway on a WAN is not a stateful firewall.
I would though also point out that the documentation text is misleading. To quote:
With Multi-WAN you generally want to ensure traffic leaves the same interface it arrives on, hence reply-to is added automatically by default. When using bridging, you must disable this behavior if the WAN gateway IP is different from the gateway IP of the hosts behind the bridged interface.
This does not mention that it also forces reply traffic bound for directly for another host on the WAN subnet is then also forced through the default gateway, which is especially surprising given the documentation on the "gateway" setting on creating a new firewall rule:
Leave as 'default' to use the system routing table. Or choose a gateway to utilize policy based routing.
In my opinion, either the help text needs to be updated to reflect the case that all traffic is forced back to the upstream gateway on the interface the packet came in (as opposed to back to the same interface, which is not exactly the same thing) or the rule generation code needs to be updated to not apply reply-to when the source IP address is on the same interface (for example using the technique I suggested in the OP). This would also avoid sending reply traffic bound for another host on the WAN network through the default gateway, and would not be likely to break many setups.
It's also not clear from the docs what's considered a "WAN" interface. Because if I make another interface for my second WAN it's going to be an OPT interface, and I'm not sure how/if reply-to is going to be applied to those rules. Based on a quick reading on the source code it seems to depend on whether you've set an upstream gateway on the interface or not, which makes sense, although it could be more explicit.
https://github.com/opnsense/core/blob/master/src/opnsense/mvc/app/library/OPNsense/Firewall/FilterRule.php#L144-L147
-
Feel free to open a PR with improvements to the help text. Wan in the context of most of these things is defined by an interface with an attached gateway (either via dhcp or statically defined).
Relevant parts of the code can be found here https://github.com/opnsense/core/blob/e690ff6fec3de90949cae42c7620ae6ec6b8beb0/src/etc/inc/filter.lib.inc#L60-L82
-
This "reply-to" just bit me HARD. I have been testing OPNsense in a lab. My WAN gets a DHCP address from my main subnet and the OPNsense LAN is completely separate with its own test computer behind it. I set a rule to allow SSH from the WAN subnet and I beat my head against the wall for days trying to figure it out. Checking "disable reply-to" fixed my issue. I am a network engineer and have been so for well over a decade. Sending traffic to a gateway for an IP that is on the same subnet is completely stupid. You don't need routing on the same subnet. It is only layer 2.
And FYI, on pfSense with "disable reply-to" unchecked for my allow SSH from WAN subnet rule, my connection works. It apparently doesn't send local subnet traffic to the gateway.
-
I have been thinking about this for the past day or so and I have come to the conclusion that OPNsense sending return traffic to the gateway for an IP that is on the local WAN subnet is a bug. This behavior goes against everything I have learned as a network engineer. I think if you asked other network engineers (or hell, people with their CCIE), you would get the same response every time: this behavior is wrong no matter if reply-to is enabled or disabled. Again, this should just be layer-2, not routing.
-
Just set in Interface config in your WAN the upstream gateway to auto-detect and it's gone.
With this settings auto nat for outbound has to be configured manually
-
Oh, doesnt work with DHCP. Maybe there is a chance to negate WAN net in autogenerated reply-to.
Would be better to open an issue in GitHub
-
a direct IP connection is still "layer 3".The return address is then the source IP. To describe it as layer 2 traffic is a bit strange.
If I go to another town with my car it doesn't make my car the road it's traveling on, nor does it make me a car.
-
What he meant is that a reply packet on the same layer 2 network has to go directly and not via default gateway, which indeed is usual behavior.
-
If we want to classify this as a bug, the bug exists in FreeBSD since forever. pfSense merely added a patch and we are no fans of non-standard OS modification unless they serve a higher purpose.
Any sane gateway will bounce the packet back to the destination, otherwise a checkbox to fix it is really not too much to ask from a user perspective especially since talking about the behaviour is proof that the solution has already been found.
Cheers,
Franco
-
Unless the "sane gateway" is a firewall. Because it is a stateful device it drops the packets. Only stateless devices, like routers, will forward it on.
Also, if "a checkbox to fix it is really not too much to ask from a user perspective" then it should not be too much for this behavior to be documented on the setting since this is incorrect network behavior.
-
Luckily it has been documented for quite some time https://docs.opnsense.org/manual/firewall.html#policy-based-routing. Feel free to open PR’s for useful additions on one of our Github repositories, it’s time better spend than complaining how things are different than you’re used to on product X,Y or Z
Best regards,
Ad
-
@mimugmail
i wasn't commenting on what he meant.
I knew what he meant.
I was commenting on what he wrote.
-
@Ad, You're right, my time might be better spent doing a PR. However, you are assuming I am a developer or programmer, which I am not. However, I am most certainly not complaining how things are different vs product x, y, or z. I am complaining that Opnsense's implementation of reply-to doesn't seem follow 802.3 standards. Standards documentation is extremely dry and I don't have the time to dig through it all to back up my claim with evidence. Franco's assertion is that this is an upstream bug with FreeBSD/pf. If he is correct, then proving that should be relatively simple. But first let's establish normal networking behavior versus Opnsense's behavior with reply-to enabled on the rule given the IPs and MACs below.
10.1.1.1/24 - aa:aa:aa:00:00:01 Default gateway
10.1.1.2/24 - aa:aa:aa:00:00:02 Computer A (anything with an SSH client)
10.1.1.3/24 - aa:aa:aa:00:00:03 Computer B (Computer with SSH server) OR OpnSense WAN interface
Normal Behavior (assuming arp tables are empty):
"A" wants to SSH to "B"
1. "A" arps who has 10.1.1.3
2. "B" replies with 10.1.1.3 is at aa:aa:aa:00:00:03
3. "A" sends SYN to "B"
4. "B" arps who has 10.1.1.2
5. "A" replies with 10.1.1.2 is at aa:aa:aa:00:00:02
6. "B" replies with SYN ACK to "A"
7. "A" replies with ACK to "B"
8. SSH connection is now established
OPNsense Behavior with reply-to enabled and a firewall as a default gateway (assuming arp tables are empty):
In this case "B" is OpnSense WAN interface. SSH is and enabled. Private IPs and bogons allowed on the WAN interface.
"A" wants to SSH to "B"
1. "A" arps who has 10.1.1.3
2. "B" replies with 10.1.1.3 is at aa:aa:aa:00:00:03
3. "A" sends SYN to "B"
4. "B" arps who has 10.1.1.1
5. Gateway replies with 10.1.1.1 is at aa:aa:aa:00:00:01
6. "B" replies with SYN ACK to "A" BUT uses the MAC address of the Gateway
7. The gateway drops the SYN ACK
8. Steps 3, 6, 7 repeats a few times.
9. "A" gives connection time-out.
To test with FreeBSD with a firewall as a gateway (I am guessing here and it may be a little simplistic):
1. Setup a freebsd VM with pf and SSH installed.
2. SSH to "B" from "A".
3. Configure pf to have a rule: allow SSH from 10.1.1.0/24 with reply-to enabled using the gateway.
4. Reboot
5a. After reboot, SSH back to "B". If connection establishes then Opnsense's behavior is not an upstream bug and a bug report should be filed on Opnsense's GitHub.
5b. If SSH session does not establish, then the behavior is indeed an upstream bug and a bug report should be filed with FreeBSD.
To be thorough while doing this testing, tcpdumps should be running to verify the behavior. I have never manually configured pf before and my only experience with any flavor of BSD is Opnsense and pfSense so I don't think I can quickly verify FreeBSD's behavior. If anyone who is familiar with FreeBSD and pf want's to perform the test, that would be great.
-
You're free to find the correct paragraph in the rfc's which describes that you're not allowed to forward same net traffic to a specific host, which is what reply-to allows you todo (you likely are not going to find it in 802.3, which describes the ethernet layer, not the IP part).
By my knowledge reply-to on pf is implemented equally on both OpenBSD as FreeBSD, which likely isn't violating the rfc's, but then again, you're always free to prove otherwise (and ask upstream to fix it).
As stated multiple times, you can disable this behaviour, overwrite it for specific nets or protocols using your own rules, which you can all influence from the web interface. When designing such functionality from scratch, we might have chosen other defaults (which is the only thing we're talking about here).
Since changing this behaviour will influence people depending on the current one or adds a lot of additional complexity to guess which nets one wouldn't like to forward, these things are most likely not going to happen.
To write documentation, one doesn't have to be a developer by the way.
As always, feel free to propose useful additions, write documentation or when able, contribute code which helps all use-cases (not only your own).
Best regards,
Ad
-
Maybe it's worth to add a checkbox to interface config, with disabling reply-to for traffic from the same network.
This is the usual rules when WAN has 192.168.99.5 and WANGW is 192.168.99.1:
pass in quick on igb1 reply-to ( igb1 192.168.99.1 ) inet from {(igb1:network)} to {any} keep state label "d13e3b46e38dad18644ef0d12f2cfff7"
And when enabled checkbox change to:
pass in quick on igb1 reply-to ( igb1 192.168.99.1 ) inet from {(igb1:network)} to !{(igb1:network)} keep state label "d13e3b46e38dad18644ef0d12f2cfff7"
No idea if pf allows this syntax
-
Ad, you are right it is not 802.3. It is RFC1122 (Requirements for Internet Hosts -- Communication Layers
). Specifically in section 3.3.1.1.
https://tools.ietf.org/html/rfc1122#page-47
My interpretation is that OPNsense's implementation of the "reply-to" feature appears to be overriding the local/remote decision as described in rfc1122 section 3.3.1.1.
"""
(b) If the IP destination address bits extracted by the
address mask match the IP source address bits extracted
by the same mask, then the destination is on the
corresponding connected network, and the datagram is to
be transmitted directly to the destination host.
(c) If not, then the destination is accessible only through
a gateway. Selection of a gateway is described below
(3.3.1.2).
"""
In section 3.5 INTERNET LAYER REQUIREMENTS SUMMARY, "Use address mask in local/remote decision" is marked as "MUST".
I will go ahead and open up a bug when I have the time.
-
Bug #3952 has been submitted.
-
@ctminime As mentioned earlier, if you believe there would be a bug to report on pf, better do so at FreeBSD, which is where our kernel is derived from.
-
If we want to classify this as a bug, the bug exists in FreeBSD since forever. pfSense merely added a patch and we are no fans of non-standard OS modification unless they serve a higher purpose.
Any sane gateway will bounce the packet back to the destination, otherwise a checkbox to fix it is really not too much to ask from a user perspective especially since talking about the behaviour is proof that the solution has already been found.
Cheers,
Franco
In that case OPNsense itself is not a sane gateway by your own definition. :-)
Consider the scenario where another OPNsense box is your gateway on your WAN. It receives a "reply" to a packet there there's on state. The stateful firewall will drop that packet on the floor rather than route it anywhere.
Also This is not an OS/FreeBSD bug. The OS is doing exactly what the firewall rule says. Any replies are sent back to the gateway on the WAN subnet. OPNsense is the one generating this firewall rule, and FreeBSD is simply doing exactly what the pf rule says to do (even if it doesn't make sense), so if the bug is anywhere, it's in OPNsense, not in FreeBSD.
-
> In that case OPNsense itself is not a sane gateway by your own definition. :-)
Fair enough. I don't mind that much because we only try to curate a decision that was made by nobody who was ever involved in OPNsense.
The way this presents itself over the years is that "please don't break this" and "please change this" are mutually exclusive requirements by the same people loosely related in space and time. :)
Cheers,
Franco
-
I've been beating my head on the wall for 2 hours because a host in my "wan" vlan can't ping the actual pfsense firewall even though FW rules allowed it. I literally just created an account to say "Thank you!" for posting about this!
Is this "reply-to" only on the "wan" interface & therefore avoidable by using/creating additional "opt" interfaces & ignoring the wan interface?
-
You can just go to Interfaces : WAN and don't set an upstream gateway.
Easy ...
-
I've been beating my head on the wall for 2 hours
Welcome to the club.
Is this "reply-to" only on the "wan" interface & therefore avoidable by using/creating additional "opt" interfaces & ignoring the wan interface?
No, because if a gateway exists on an interface, this interface is considered a "WAN-type" interface and "reply-to" will be enabled.
You can just go to Interfaces : WAN and don't set an upstream gateway.
Eh? :o
-
Maurice, sorry, you are wrong :)
You can have 100 Gateways on LAN and set 200 Routes pointing to 100 different gateways.
The reply-to thing happens due to a auto pf rule set where an incoming packet will be replied via gateway in upstream gateway (DONT MIX THIS WITH UPSTREAM CHECKBOX UNDER SYSTEM:GATEWAYS:SINGLE :) )
-
You're right. For static interfaces, reply-to is not enforced if the upstream gateway is set to auto-detect in the interface settings. Even if a gateway exists on that interface. I learned something today. Thanks! :)
But for DHCP interfaces, there is no such setting. Reply-to is always enforced (as long as a dynamic gateway exists on the interface).
And it is unrelated to which interface the active default gateway is on. If you have multiple DHCP interfaces with dynamic gateways, reply-to is always enforced on ALL of them. While on a static interface, reply-to is not enforced even if the active default gateway is on that interface. This is... confusing.
(And it doesn't matter whether it's the default "wan" or "opt1" or whatever. There is nothing special about the default "wan" (correct?). I think that was 5SpeedFun's original question. ;))
-
Maybe we can find a solution for 20.7 .. no promise
-
I am in the club too. A lot of time has been wasted due to default reply-to setting.
-
https://docs.opnsense.org/manual/firewall_settings.html#disable-reply-to
-
Ad, you are right it is not 802.3. It is RFC1122 (Requirements for Internet Hosts -- Communication Layers
). Specifically in section 3.3.1.1.
https://tools.ietf.org/html/rfc1122#page-47
My interpretation is that OPNsense's implementation of the "reply-to" feature appears to be overriding the local/remote decision as described in rfc1122 section 3.3.1.1.
I've been trying to solve something that I believe is the same or similar issue that ctminime describes.
What I am experiencing is that originating traffic (90.90.90.90) from the Internet is passed through a NAT port foward rule that translates the destination IP from the public IP of OPNsense # 1 (100.100.100.100) to a private IP of an internal server (172.20.20.20).
This traffic is then passed through an OpenVPN tunnel to another OPNsense firewall also connected to the Internet (200.200.200.200). The traffic is then routed to the server LAN on the second OPNsense instance and arrives at the server (172.20.20.20). The server responds with the destination of the originating public IP (90.90.90.90), but the second OPNsense instance insists on replying out it's default gateway (200.200.200.1) instead of sending it back through the OpenVPN tunnel to the first OPNsense instance. The packet is dropped at the WAN interface since that traffic is not allowed by default.
If I add a firewall rule at OPNsense # 2 (200.200.200.200) for the server interface to send all traffic to a different gateway, the OpenVPN tunnel, all outgoing traffic from the internal server is sent correctly out via OpenVPN and out at OPNsense # 1's WAN interface as it should. Outgoing traffic from the internal server works fine.
However incoming traffic via OPNsense # 1 (100.100.100.100) going through the OpenVPN tunnel still wants to take it's return path via OPNsense # 2's (200.200.200.200) default gateway and interface.
Adding a firewall rule on OPNsense # 2 to allow the traffic results in that the traffic is then passed back through OpenVPN, but destination translated with OPNsense # 2's IP-address. The result is of course that the traffic is discarded at the Internet client side as the sender address doesn't match any outgoing states.
No matter what I try I cannot make OPNsense not insist on sending the traffic out OPNsense # 2's WAN, perform an outgoing NAT and pass the packet back through the tunnel. I believe it should be passed back through the tunnel unaltered and not via the WAN interface before sending it back via the tunnel.
Client (90.90.90.90) <---> (100.100.100.100) OPNsense # 1 (10.20.30.1 tunnel IP) <----> (10.20.30.2 tunnel IP) OPNsense # 2 (172.20.20.1) <---> (172.20.20.20) Server
ISP (200.200.200.1) <---> (200.200.200.200) OPNsense # 2 (172.20.20.1) <---> (172.20.20.20) Server
OPNsense # 1
WAN : 100.100.100.100 /24 gateway 100.100.100.1
LAN : 192.168.100.1 /24
OpenVPN-server listening on WAN (OpenVPN tunnel IP 10.20.30.1)
Allow anything in/out on OpenVPN interface
Allow udp/53 in on WAN
NAT udp/53 in on WAN to 172.20.20.20
OPNsense # 2
WAN : 200.200.200.200 /24 gateway 200.200.200.1
LAN : 192.168.200.1 /24
SERVERS: 172.20.20.1 /24
OpenVPN-client with 100.100.100.100 (OpenVPN tunnel IP 10.20.30.2)
Allow anything in/out on OpenVPN interface
Allow anything in/out on SERVERS interface
Logs show in order from oldest to newest entries:
OPNsense # 1 firewall log
pass WAN < 90.90.90.90:60872 172.20.20.20:53 udp Allow DNS
pass OPENVPN < 90.90.90.90:60872 172.20.20.20:53 udp let out anything from firewall host itself
OPNsense # 2 firewall log
pass OPENVPN > 90.90.90.90:11595 172.20.20.20:53 udp Allow DNS
pass SERVERS < 90.90.90.90:11595 172.20.20.20:53 udp let out anything from firewall host itself
block INTERNET > 172.20.20.20:53 192.168.1.100:11595 udp Default deny rule
block INTERNET > 172.20.20.20:53 192.168.1.100:11595 udp Default deny rule
Adding the following rule at OPNsense # 2 allows return traffic, but with wrong source IP:
Interface: INTERNET
pass IPv4 172.20.20.0/24 any port to any destination and any port GATEWAY 10.20.30.1 any time
OPNsens # 1 new firewall log
pass INTERNET > 90.90.90.90:25537 172.20.20.20:53 udp Allow DNS
pass OPENVPN < 90.90.90.90:25537 172.20.20.20:53 udp let out anything from firewall host itself
pass OPENVPN > 172.20.20.20:53 200.200.200.200:25537 udp
Note that the last entry has a destination of 200.200.200.200 (firewall 2 WAN IP) instead of 90.90.90.90 (client public IP).
I've tried adding a NO NAT rule in many shapes and forms on OPNsense # 2's INTERNET interface with no change in the results.
Coming across this "route to" thread I tried to disable and enable that on OPNsense # 2 as well. No change.
I don't have multi-WAN, and my Multi-WAN settings are:
- Disabled "Sticky connections"
- Disabled "Shared forwarding"
- Enabled "Disable force gateway" (tried disabling it too)
- Enabled "Disable reply-to" (tried disabling it too)
OPNsense # 2's WAN is usually DHCP assigned, but I tried to set a static IP on it instead and a static default route instead of the DHCP-supplied default route. No changes.
Is it impossible to override that the traffic returning from the SERVERS network has to exit the INTERNET WAN interface, before returning via the OpenVPN tunnel?
Enabling NAT on OPNsense # 1s traffic using 10.20.30.2 as the source translated IP, before passing it through the OpenVPN-tunnel, makes everything work from 90.90.90.90.
-
I have a similar issue as @oddjarle.
Is there any solution to this problem?
-
Just dont use an upstream gateway in Interface : WAN
-
I believe oddjarle's problem is different than the OP. The reply simply travels via default route. removing the gateway from wan won't change that, will it? Maybe continue here:
https://forum.opnsense.org/index.php?topic=20843.0 (https://forum.opnsense.org/index.php?topic=20843.0)