Show Posts
1
19.7 Legacy Series / Opnsense blocking web traffic automatically?
« on: February 17, 2020, 01:52:03 am »
Hey Forum!
I was looking at adopting opnsense in our production environment. Before installing something new into our production network, it was installed into the test environment, some firewalls rules were created and this worked correctly so we stuck it into production!
The firewall has now been running for about a month on our production network with allow all rules (logging traffic which would have been blocked) to allow us to catch any traffic we should be allowing, today we saw that one of our sites was not working on port 80 or 443. Examining the logs on the firewall we discovered lots of denied traffic being produced. This was not deliberate. A rule to allow any to any on any port as rule 1 on the floating ruleset for all interfaces was created as a quick fix, sadly the firewall was still blocking web traffic (on 80 and 443 for internal sites and 3128 for external sites via squid web proxy). We then rebooted the server after this change was made in case something had gotten stuck in cache or a service had crashed. Unfortunately, the same thing was still occurring. We even tried to turn off packet filtering in the settings to allow all traffic to pass through, no avail even though the logs stopped showing any firewall traffic.
A decision was made to physically remove the firewall to allow users to continue operating (replace the layer 3 router which had been in place prior to the addition of the opnsense firewall). This instantly resolved the issue, that tells me that its definitely a firewall issue as nothing else was changed during this time. The firewall is still currently blocking web traffic.
Prior to this issue we did note occasional blocking of traffic to our squid server on 3128 which we assumed may have been to do with persistent connections and ageing of the state tables? Users did not seem to notice these occurrences and so we believe the browser just re-established a new connection after the traffic was blocked. I am not convinced this is related but it is note-worthy.
I cant work out what happened here so I can't tell you how to reproduce the issue. Does anybody have any ideas as to why this has occurred?
What sort of diagnostics can I do to investigate the cause of the issue (we can plug the firewall back in out of hours and hopefully replicate the issue here) and what can do to stop this from happening again (ie. why would turning off packet filtering not have worked)?
Version:
OPNsense 19.7-amd64
FreeBSD 11.2-RELEASE-p11-HBSD
OpenSSL 1.0.2s 28 May 2019
I was going to be nice and attach a copy of the logs but I can't find where to download the logs. Please let me know what you want and how to download them if required and I'll post them.
I was looking at adopting opnsense in our production environment. Before installing something new into our production network, it was installed into the test environment, some firewalls rules were created and this worked correctly so we stuck it into production!
The firewall has now been running for about a month on our production network with allow all rules (logging traffic which would have been blocked) to allow us to catch any traffic we should be allowing, today we saw that one of our sites was not working on port 80 or 443. Examining the logs on the firewall we discovered lots of denied traffic being produced. This was not deliberate. A rule to allow any to any on any port as rule 1 on the floating ruleset for all interfaces was created as a quick fix, sadly the firewall was still blocking web traffic (on 80 and 443 for internal sites and 3128 for external sites via squid web proxy). We then rebooted the server after this change was made in case something had gotten stuck in cache or a service had crashed. Unfortunately, the same thing was still occurring. We even tried to turn off packet filtering in the settings to allow all traffic to pass through, no avail even though the logs stopped showing any firewall traffic.
A decision was made to physically remove the firewall to allow users to continue operating (replace the layer 3 router which had been in place prior to the addition of the opnsense firewall). This instantly resolved the issue, that tells me that its definitely a firewall issue as nothing else was changed during this time. The firewall is still currently blocking web traffic.
Prior to this issue we did note occasional blocking of traffic to our squid server on 3128 which we assumed may have been to do with persistent connections and ageing of the state tables? Users did not seem to notice these occurrences and so we believe the browser just re-established a new connection after the traffic was blocked. I am not convinced this is related but it is note-worthy.
I cant work out what happened here so I can't tell you how to reproduce the issue. Does anybody have any ideas as to why this has occurred?
What sort of diagnostics can I do to investigate the cause of the issue (we can plug the firewall back in out of hours and hopefully replicate the issue here) and what can do to stop this from happening again (ie. why would turning off packet filtering not have worked)?
Version:
OPNsense 19.7-amd64
FreeBSD 11.2-RELEASE-p11-HBSD
OpenSSL 1.0.2s 28 May 2019
I was going to be nice and attach a copy of the logs but I can't find where to download the logs. Please let me know what you want and how to download them if required and I'll post them.