Reply-to on WAN by default is bogus

[pv2b]

I have some feedback regarding the default behavior for inbound firewall rules on the WAN interface of OPNsense 20.1, and I believe this problem is also present in earlier versions.

Consider a network that looks like this:

Code Select Expand


   __   _
_(  )_( )_
(_   _    _) Internet
  (_) (__)
    |
.---'--------.    .-----------.
| WAN Router |    | Computer  |
| 192.0.2.1  |    | 192.0.2.2 |
'---.--------'    '-----.-----'
    |                   |
    |                   |
.---'-------------------'---.
| 192.0.2.0/24 WAN Network  |
'------.--------------------'
       |
       | WAN (igb1)
.------'------------.
| OPNsense Firewall |
| 192.0.2.3         |
| 192.168.0.1       |
'------.------------'
       | LAN (igb0)
       |
.------'---------------------.
| 192.168.0.0/24 LAN Network |
'----------------------------'


Consider that there is a firewall rule the OPNsense box, on the WAN interface which permits traffic from 192.0.2.2 to 192.0.2.3 TCP PORT 443. All other options default.

By default, OPNsense seems to create this firewall rule with a "reply-to" clause, which means that any reponse to the SYN sent by the computer is not sent back to the computer, but instead back to the default gateway on the WAN network.

This will work most of the time (except wasting sending traffic to and from the gateway), but if the gateway is itself another stateful firewall (like another OPNsense box), this causes the connection to never be established.

The solution is to check "disable reply-to" on the inbound firewall rule on the WAN.

Not sure when this became the default, it seems to have been the case since at least 2018. https://forum.opnsense.org/index.php?topic=8841.0 Although I'm not finding this global checkbox that @franco was referring to.

Now, I understand the reason for this default "reply-to" behavior, which is to ensure that in a multi-WAN scenario, you don't get triangular routing, but instead the way it's implemented, you end up with triangular traffic flow in the basic case.

A sane default would be to not enable reply-to unless the user selects a gateway on the rule. (In fact, when you just select "default" as the gateway, the help says the system routing table is used, which is NOT true.)

Another possible fix for a solution that "just works" that both works for preventing triangular routing in multi-WAN scenario and doesn't break local WAN commmunication is to install two pf rules in the back end, which disables reply-to for traffic coming from a host directly connected on the WAN network. Something like:

Code Select Expand


pass in quick on igb1                             inet proto tcp from {{igb1:network}} to {{self}} port {443} keep state
pass in quick on igb1 reply-to { igb1 192.0.2.1 } inet proto tcp from {any}            to {{self}} port {443} keep state


Of course all of this can be configured by an admin directly in the GUI, if you don't mind some doubled firewall rules.

To summarize, I don't expect Policy Based Routing to be implemented by default unless i specifically turn it off in advanced settings.

[AdSchellevis]

Hi,

You can find the general checkbox "Disable reply-to" in system_advanced_firewall.php, which disables the default as Franco mentioned in the other post.

This rule has been the default since probably pfSense 1.2.x, since many installations depend on it, it's unlikely that we're going to change it.

The toggle to disable it was introduced in https://github.com/pfsense/pfsense/commit/efefb2a1e860d082a6024b7c6b67c646b1e8aa6e more than a decade ago.


When designing such functionality now, we might make other choices, but some habits are better left alone.

Best regards,

Ad

[pv2b]

Thanks for showing me the checkbox.

I wasn't aware this was ever since pfSense 1.2. I'm surprised this hasn't bitten me yet, but then again I rarely deploy an OPNsense box behind a stateful firewall on the WAN interface. But I can definitely say that a lot of traffic has been needlessly flowing through the default gateway on some of my installed sites without me ever realizing, leaving performance on the table due to this misfeature. It just never broke anything because typically the gateway on a WAN is not a stateful firewall.

I would though also point out that the documentation text is misleading. To quote:

With Multi-WAN you generally want to ensure traffic leaves the same interface it arrives on, hence reply-to is added automatically by default. When using bridging, you must disable this behavior if the WAN gateway IP is different from the gateway IP of the hosts behind the bridged interface.

This does not mention that it also forces reply traffic bound for directly for another host on the WAN subnet is then also forced through the default gateway, which is especially surprising given the documentation on the "gateway" setting on creating a new firewall rule:

Leave as 'default' to use the system routing table. Or choose a gateway to utilize policy based routing.

In my opinion, either the help text needs to be updated to reflect the case that all traffic is forced back to the upstream gateway on the interface the packet came in (as opposed to back to the same interface, which is not exactly the same thing) or the rule generation code needs to be updated to not apply reply-to when the source IP address is on the same interface (for example using the technique I suggested in the OP). This would also avoid sending reply traffic bound for another host on the WAN network through the default gateway, and would not be likely to break many setups.

It's also not clear from the docs what's considered a "WAN" interface. Because if I make another interface for my second WAN it's going to be an OPT interface, and I'm not sure how/if reply-to is going to be applied to those rules. Based on a quick reading on the source code it seems to depend on whether you've set an upstream gateway on the interface or not, which makes sense, although it could be more explicit.

https://github.com/opnsense/core/blob/master/src/opnsense/mvc/app/library/OPNsense/Firewall/FilterRule.php#L144-L147

[AdSchellevis]

Feel free to open a PR with improvements to the help text. Wan in the context of most of these things is defined by an interface with an attached gateway (either via dhcp or statically defined).

Relevant parts of the code can be found here https://github.com/opnsense/core/blob/e690ff6fec3de90949cae42c7620ae6ec6b8beb0/src/etc/inc/filter.lib.inc#L60-L82

[ctminime]

This "reply-to" just bit me HARD. I have been testing OPNsense in a lab. My WAN gets a DHCP address from my main subnet and the OPNsense LAN is completely separate with its own test computer behind it. I set a rule to allow SSH from the WAN subnet and I beat my head against the wall for days trying to figure it out. Checking "disable reply-to" fixed my issue. I am a network engineer and have been so for well over a decade. Sending traffic to a gateway for an IP that is on the same subnet is completely stupid. You don't need routing on the same subnet. It is only layer 2.

And FYI, on pfSense with "disable reply-to" unchecked for my allow SSH from WAN subnet rule, my connection works. It apparently doesn't send local subnet traffic to the gateway.

[ctminime]

I have been thinking about this for the past day or so and I have come to the conclusion that OPNsense sending return traffic to the gateway for an IP that is on the local WAN subnet is a bug. This behavior goes against everything I have learned as a network engineer. I think if you asked other network engineers (or hell, people with their CCIE), you would get the same response every time: this behavior is wrong no matter if reply-to is enabled or disabled. Again, this should just be layer-2, not routing.

[mimugmail]

Just set in Interface config in your WAN the upstream gateway to auto-detect and it's gone.

With this settings auto nat for outbound has to be configured manually

[mimugmail]

Oh, doesnt work with DHCP. Maybe there is a chance to negate WAN net in autogenerated reply-to.

Would be better to open an issue in GitHub

[frater]

a direct IP connection is still "layer 3".The return address is then the source IP. To describe it as layer 2 traffic is a bit strange.
If I go to another town with my car it doesn't make my car the road it's traveling on, nor does it make me a car.

[mimugmail]

What he meant is that a reply packet on the same layer 2 network has to go directly and not via default gateway, which indeed is usual behavior.

[franco]

If we want to classify this as a bug, the bug exists in FreeBSD since forever. pfSense merely added a patch and we are no fans of non-standard OS modification unless they serve a higher purpose.

Any sane gateway will bounce the packet back to the destination, otherwise a checkbox to fix it is really not too much to ask from a user perspective especially since talking about the behaviour is proof that the solution has already been found.


Cheers,
Franco

[ctminime]

Unless the "sane gateway" is a firewall. Because it is a stateful device it drops the packets. Only stateless devices, like routers, will forward it on.

Also, if "a checkbox to fix it is really not too much to ask from a user perspective" then it should not be too much for this behavior to be documented on the setting since this is incorrect network behavior.

[AdSchellevis]

Luckily it has been documented for quite some time https://docs.opnsense.org/manual/firewall.html#policy-based-routing. Feel free to open PR's for useful additions on one of our Github repositories, it's time better spend than complaining how things are different than you're used to on product X,Y or Z

Best regards,
Ad

[frater]

@mimugmail

i wasn't commenting on what he meant.
I knew what he meant.
I was commenting on what he wrote.

[ctminime]

@Ad, You're right, my time might be better spent doing a PR. However, you are assuming I am a developer or programmer, which I am not. However, I am most certainly not complaining how things are different vs product x, y, or z. I am complaining that Opnsense's implementation of reply-to doesn't seem follow 802.3 standards. Standards documentation is extremely dry and I don't have the time to dig through it all to back up my claim with evidence. Franco's assertion is that this is an upstream bug with FreeBSD/pf. If he is correct, then proving that should be relatively simple. But first let's establish normal networking behavior versus Opnsense's behavior with reply-to enabled on the rule given the IPs and MACs below.

10.1.1.1/24 - aa:aa:aa:00:00:01 Default gateway
10.1.1.2/24 - aa:aa:aa:00:00:02 Computer A (anything with an SSH client)
10.1.1.3/24 - aa:aa:aa:00:00:03 Computer B (Computer with SSH server) OR OpnSense WAN interface

Normal Behavior (assuming arp tables are empty):
"A" wants to SSH to "B"
1. "A" arps who has 10.1.1.3
2. "B" replies with 10.1.1.3 is at aa:aa:aa:00:00:03
3. "A" sends SYN to "B"
4. "B" arps who has 10.1.1.2
5. "A" replies with 10.1.1.2 is at aa:aa:aa:00:00:02
6. "B" replies with SYN ACK to "A"
7. "A" replies with ACK to "B"
8. SSH connection is now established

OPNsense Behavior with reply-to enabled and a firewall as a default gateway (assuming arp tables are empty):
In this case "B" is OpnSense WAN interface. SSH is and enabled. Private IPs and bogons allowed on the WAN interface.
"A" wants to SSH to "B"
1. "A" arps who has 10.1.1.3
2. "B" replies with 10.1.1.3 is at aa:aa:aa:00:00:03
3. "A" sends SYN to "B"
4. "B" arps who has 10.1.1.1
5. Gateway replies with 10.1.1.1 is at aa:aa:aa:00:00:01
6. "B" replies with SYN ACK to "A" BUT uses the MAC address of the Gateway
7. The gateway drops the SYN ACK
8. Steps 3, 6, 7 repeats a few times.
9. "A" gives connection time-out.

To test with FreeBSD with a firewall as a gateway (I am guessing here and it may be a little simplistic):
1. Setup a freebsd VM with pf and SSH installed.
2. SSH to "B" from "A".
3. Configure pf to have a rule: allow SSH from 10.1.1.0/24 with reply-to enabled using the gateway.
4. Reboot
5a. After reboot, SSH back to "B". If connection establishes then Opnsense's behavior is not an upstream bug and a bug report should be filed on Opnsense's GitHub.
5b. If SSH session does not establish, then the behavior is indeed an upstream bug and a bug report should be filed with FreeBSD.

To be thorough while doing this testing, tcpdumps should be running to verify the behavior. I have never manually configured pf before and my only experience with any flavor of BSD is Opnsense and pfSense so I don't think I can quickly verify FreeBSD's behavior. If anyone who is familiar with FreeBSD and pf want's to perform the test, that would be great.