Messages

Bug  #3952 has been submitted.

Ad, you are right it is not 802.3. It is RFC1122 (Requirements for Internet Hosts -- Communication Layers
). Specifically in section 3.3.1.1.

https://tools.ietf.org/html/rfc1122#page-47

My interpretation is that OPNsense's implementation of the "reply-to" feature appears to be overriding the local/remote decision as described in rfc1122 section 3.3.1.1.

"""
            (b)  If the IP destination address bits extracted by the
                 address mask match the IP source address bits extracted
                 by the same mask, then the destination is on the
                 corresponding connected network, and the datagram is to
                 be transmitted directly to the destination host.
            (c)  If not, then the destination is accessible only through
                 a gateway.  Selection of a gateway is described below
                 (3.3.1.2).
"""

In section 3.5  INTERNET LAYER REQUIREMENTS SUMMARY, "Use address mask in local/remote decision" is marked as "MUST".

I will go ahead and open up a bug when I have the time.

[BUT]

@Ad, You're right, my time might be better spent doing a PR. However, you are assuming I am a developer or programmer, which I am not. However, I am most certainly not complaining how things are different vs product x, y, or z. I am complaining that Opnsense's implementation of reply-to doesn't seem follow 802.3 standards. Standards documentation is extremely dry and I don't have the time to dig through it all to back up my claim with evidence. Franco's assertion is that this is an upstream bug with FreeBSD/pf. If he is correct, then proving that should be relatively simple. But first let's establish normal networking behavior versus Opnsense's behavior with reply-to enabled on the rule given the IPs and MACs below.

10.1.1.1/24 - aa:aa:aa:00:00:01 Default gateway
10.1.1.2/24 - aa:aa:aa:00:00:02 Computer A (anything with an SSH client)
10.1.1.3/24 - aa:aa:aa:00:00:03 Computer B (Computer with SSH server) OR OpnSense WAN interface

Normal Behavior (assuming arp tables are empty):
"A" wants to SSH to "B"
1. "A" arps who has 10.1.1.3
2. "B" replies with 10.1.1.3 is at aa:aa:aa:00:00:03
3. "A" sends SYN to "B"
4. "B" arps who has 10.1.1.2
5. "A" replies with 10.1.1.2 is at aa:aa:aa:00:00:02
6. "B" replies with SYN ACK to "A"
7. "A" replies with ACK to "B"
8. SSH connection is now established

OPNsense Behavior with reply-to enabled and a firewall as a default gateway (assuming arp tables are empty):
In this case "B" is OpnSense WAN interface. SSH is and enabled. Private IPs and bogons allowed on the WAN interface.
"A" wants to SSH to "B"
1. "A" arps who has 10.1.1.3
2. "B" replies with 10.1.1.3 is at aa:aa:aa:00:00:03
3. "A" sends SYN to "B"
4. "B" arps who has 10.1.1.1
5. Gateway replies with 10.1.1.1 is at aa:aa:aa:00:00:01
6. "B" replies with SYN ACK to "A" BUT uses the MAC address of the Gateway
7. The gateway drops the SYN ACK
8. Steps 3, 6, 7 repeats a few times.
9. "A" gives connection time-out.

To test with FreeBSD with a firewall as a gateway (I am guessing here and it may be a little simplistic):
1. Setup a freebsd VM with pf and SSH installed.
2. SSH to "B" from "A".
3. Configure pf to have a rule: allow SSH from 10.1.1.0/24 with reply-to enabled using the gateway.
4. Reboot
5a. After reboot, SSH back to "B". If connection establishes then Opnsense's behavior is not an upstream bug and a bug report should be filed on Opnsense's GitHub.
5b. If SSH session does not establish, then the behavior is indeed an upstream bug and a bug report should be filed with FreeBSD.

To be thorough while doing this testing, tcpdumps should be running to verify the behavior. I have never manually configured pf before and my only experience with any flavor of BSD is Opnsense and pfSense so I don't think I can quickly verify FreeBSD's behavior. If anyone who is familiar with FreeBSD and pf want's to perform the test, that would be great.

Unless the "sane gateway" is a firewall. Because it is a stateful device it drops the packets. Only stateless devices, like routers, will forward it on.

Also, if "a checkbox to fix it is really not too much to ask from a user perspective" then it should not be too much for this behavior to be documented on the setting since this is incorrect network behavior.

I have been thinking about this for the past day or so and I have come to the conclusion that OPNsense sending return traffic to the gateway for an IP that is on the local WAN subnet is a bug. This behavior goes against everything I have learned as a network engineer. I think if you asked other network engineers (or hell, people with their CCIE), you would get the same response every time: this behavior is wrong no matter if reply-to is enabled or disabled. Again, this should just be layer-2, not routing.

This "reply-to" just bit me HARD. I have been testing OPNsense in a lab. My WAN gets a DHCP address from my main subnet and the OPNsense LAN is completely separate with its own test computer behind it. I set a rule to allow SSH from the WAN subnet and I beat my head against the wall for days trying to figure it out. Checking "disable reply-to" fixed my issue. I am a network engineer and have been so for well over a decade. Sending traffic to a gateway for an IP that is on the same subnet is completely stupid. You don't need routing on the same subnet. It is only layer 2.

And FYI, on pfSense with "disable reply-to" unchecked for my allow SSH from WAN subnet rule, my connection works. It apparently doesn't send local subnet traffic to the gateway.

UPDATE: literally 5 minutes after posting this, I read this post:
https://forum.opnsense.org/index.php?topic=15900.0
Checking "disable reply-to" fixed my issue.

Ebothe, I am running into a very similar situation.

To start off with, I have been using pf for about a decade. But there lack of updates recently has led me to some serious testing of OPNsense. I have definitely run into some odd issues (python using 50 -100% CPU in certain situations and not getting a DHCP address on the WAN interface cause the GUI to hang), however this one is a complete show stopper.

In my situation, I am on the latest version, 20.1.1, and it is currently in a lab setup. I have a pair setup and they are running on a single Proxmox server. My test machine (connected to OPNsense LAN) is also on the same Proxmox server. My WAN interfaces get IPs via DHCP (no Carp) and the IPs they gets are private IPs. Because of that, I DO have allow private and BOGON IPs enabled. The LAN interfaces are statically set, has carp enabled, and provides DHCP to my test box. Internet egress through the firewall works just fine. so does the failover. Now here is where the problem is...

I set a rule on the WAN interface to allow SSH connections from the WAN subnet to the firewall and I can NEVER get it to connect no matter what variation of rules I try. I even tried setting up a second rule on the WAN interface for the direction "out". Even the firewall logs were showing that the connection was allowed.

After beating my head against the wall for days trying to figure this out, I decided to setup a pair of pf on the same Proxmox server (with opnsense shutdown) and configured them the same as the OPNsense pair and I had no issue connecting to the WAN IP with SSH.

I have been a network engineer for over a decade and I have spent a lot of time working on firewalls. To me the primary job of a firewall is: To allow or block network traffic based off of a set of rules. If a firewall can't do its primary job, I am not going to use it.

So, to any OPNsense DEV who reads this, I would be happy to provide information to duplicate the behavior. Until this resolved, I'm done wasting my time.

OPNsense 20.1.1-amd64
FreeBSD 11.2-RELEASE-p16-HBSD
OpenSSL 1.1.1d 10 Sep 2019