Opnsense blocking web traffic automatically?

[Ebothe]

Hey Forum!

I was looking at adopting opnsense in our production environment. Before installing something new into our production network, it was installed into the test environment, some firewalls rules were created and this worked correctly so we stuck it into production!

The firewall has now been running for about a month on our production network with allow all rules (logging traffic which would have been blocked) to allow us to catch any traffic we should be allowing, today we saw that one of our sites was not working on port 80 or 443. Examining the logs on the firewall we discovered lots of denied traffic being produced. This was not deliberate. A rule to allow any to any on any port as rule 1 on the floating ruleset for all interfaces was created as a quick fix, sadly the firewall was still blocking web traffic (on 80 and 443 for internal sites and 3128 for external sites via squid web proxy). We then rebooted the server after this change was made in case something had gotten stuck in cache or a service had crashed. Unfortunately, the same thing was still occurring. We even tried to turn off packet filtering in the settings to allow all traffic to pass through, no avail even though the logs stopped showing any firewall traffic.

A decision was made to physically remove the firewall to allow users to continue operating (replace the layer 3 router which had been in place prior to the addition of the opnsense firewall). This instantly resolved the issue, that tells me that its definitely a firewall issue as nothing else was changed during this time. The firewall is still currently blocking web traffic.

Prior to this issue we did note occasional blocking of traffic to our squid server on 3128 which we assumed may have been to do with persistent connections and ageing of the state tables? Users did not seem to notice these occurrences and so we believe the browser just re-established a new connection after the traffic was blocked. I am not convinced this is related but it is note-worthy.

I cant work out what happened here so I can't tell you how to reproduce the issue. Does anybody have any ideas as to why this has occurred?

What sort of diagnostics can I do to investigate the cause of the issue (we can plug the firewall back in out of hours and hopefully replicate the issue here) and what can do to stop this from happening again (ie. why would turning off packet filtering not have worked)?

Version:
OPNsense 19.7-amd64
FreeBSD 11.2-RELEASE-p11-HBSD
OpenSSL 1.0.2s 28 May 2019

I was going to be nice and attach a copy of the logs but I can't find where to download the logs. Please let me know what you want and how to download them if required and I'll post them.

[ctminime]

UPDATE: literally 5 minutes after posting this, I read this post:
https://forum.opnsense.org/index.php?topic=15900.0
Checking "disable reply-to" fixed my issue.

Ebothe, I am running into a very similar situation.

To start off with, I have been using pf for about a decade. But there lack of updates recently has led me to some serious testing of OPNsense. I have definitely run into some odd issues (python using 50 -100% CPU in certain situations and not getting a DHCP address on the WAN interface cause the GUI to hang), however this one is a complete show stopper.

In my situation, I am on the latest version, 20.1.1, and it is currently in a lab setup. I have a pair setup and they are running on a single Proxmox server. My test machine (connected to OPNsense LAN) is also on the same Proxmox server. My WAN interfaces get IPs via DHCP (no Carp) and the IPs they gets are private IPs. Because of that, I DO have allow private and BOGON IPs enabled. The LAN interfaces are statically set, has carp enabled, and provides DHCP to my test box. Internet egress through the firewall works just fine. so does the failover. Now here is where the problem is...

I set a rule on the WAN interface to allow SSH connections from the WAN subnet to the firewall and I can NEVER get it to connect no matter what variation of rules I try. I even tried setting up a second rule on the WAN interface for the direction "out". Even the firewall logs were showing that the connection was allowed.

After beating my head against the wall for days trying to figure this out, I decided to setup a pair of pf on the same Proxmox server (with opnsense shutdown) and configured them the same as the OPNsense pair and I had no issue connecting to the WAN IP with SSH.

I have been a network engineer for over a decade and I have spent a lot of time working on firewalls. To me the primary job of a firewall is: To allow or block network traffic based off of a set of rules. If a firewall can't do its primary job, I am not going to use it.

So, to any OPNsense DEV who reads this, I would be happy to provide information to duplicate the behavior. Until this resolved, I'm done wasting my time.

OPNsense 20.1.1-amd64
FreeBSD 11.2-RELEASE-p16-HBSD
OpenSSL 1.1.1d 10 Sep 2019

[Ebothe]

I checked the "disable reply-to" checkbox and noticed some traffic was still being denied.

After reading this thread https://forum.opnsense.org/index.php?topic=8570.msg38082 I decided to see if that was indeed what was causing my issue. I changed all my rules to "none" and what do you know, no denied traffic! So this confirms the theory. But my question is what exactly is 'traffic is leaking and state tracking' and how can I make my firewall stateful again?

[Ebothe]

Can anybody please help me with this?