Default Deny Rule - Once Again

[Gianks]

Hi,
Yesterday we upgraded both primary and backup opnsense to last release...
as a result we are suddenly experiencing the same behavior which brought us to leave pfSense for OpnSense one year ago.

Rules are now just partially applied on some interfaces. Defining the same rules again does not work, it's simply like they are not there. Some traffic is allowed if rules are defined with a broader scope but this simply makes no Sense at all.

The configuration has not been changed in weeks, just the upgrade.

Our network is partially unusable, the software is not working right. Traffic is blocked with Default Deny Rule.

[Gianks]

I can add that there are now a ton of blocked packets returning to opnsense on its VIP for DNS, NTP, etc... this is not even traffic generated by us but by the firewall itself! 

[Gianks]

For the sake of the record, I wish to make clear that is is not an OOO packet problem.
There is 100% consistency in the behavior per rule, the ones not working are ALWAYS ignored, the others keep working normally.

Also to be noticed that some connections are blocked by the firewall at packet 0, hard to believe there was a packet -1.

[Gianks]

Attempted to backup and restore the entire configuration, it did not work, nevermind just a reboot.
Disabling and/or cloning rules did not produce any effect.

Is there a way to debug how packet filter decided to fallback to default deny?
Is it possibile to obtain a dump of the actually applied rules?

[Fright]

hi
please, if its not just cry from the heart, can you give some real info?
from which to which version were updated, which rules stopped working as expected, packet detail info from Firewall: Log Files: Live View for blocked packet?
 

[Gianks]

Duplicating one of the few rules which was still working (on one of the affected interfaces), and editing the parameters to match one of the not working ones, magically fixed the issue and the traffic started flowing again, at least for that one rule.
To be noticed that creating the same rule manually did not produce the same effect (kept being ignored).

Duplicating the duplicated, the process worked again. Not sure this will make us fix all the weird behaviors we see in the logs (we can't even rule over returning traffic to opnsense...) but at least seems to be allowing us to restore the network services.

Is there still someone suggesting this is not a bug? 

[Gianks]

Hi, thanks for the reply.
I honestly I did not really know which details to provide for starters on such a problem, it's not something I did or tried to do.
We just updated the firewall from 21.7 (i hope to be correct here) to 22.1.

I wish to do some debug but i don't know how and that's why i started the thread.
Please tell me what you need to investigate the issue, I wish to provide you all the needed information to fix it!

By the way two rules that stopped working are:
pass TCP/UDP Host A:any to Host B:X
pass UDP any:any to !NetC:Y

Creating an identical rule did not help, but without port it worked, for example. The duplicated version works perfectly without any port change or else... 
Thanks

[Fright]

wish to do some debug but i don't know how

sharing actual rule parameters (with some sanitizing if needed) and dropped packet info (catch dropped packet in live view and hit "i" button in packet string) would be helpful imho.

noticed that some connections are blocked by the firewall at packet 0, hard to believe there was a packet -1.

and it should be blocked at first packet. probably the out-of-state
https://forum.opnsense.org/index.php?topic=20219.0

[Gianks]

sharing actual rule parameters (with some sanitizing if needed) and dropped packet info (catch dropped packet in live view and hit "i" button in packet string) would be helpful imho.

Not really, if I may. What you are saying would possibly help to reproduce the issue from scratch (maybe). If I am correct about this being a bug, the relevant part of code involved might have almost nothing to do with the rule definitions themselves, for ex. the ordering might be relevant, including other rules which appears innocent). To be clear, the rules are correct because they worked until yesterday. Emulating our configuration (partially moreover) might never present the problem in a test.

and it should be blocked at first packet. probably the out-of-state
https://forum.opnsense.org/index.php?topic=20219.0

Well i guess you are correct on the possibility of such case (2nd or following packet gets there first) but this does not match our case for two reasons:

• The rule misbehavior is the cause, not the network layout. Proven by our workaroud which makes everything work again flawlessly.
• The real packet 0 (SYN), once finally arrived, should by logged also as 'pass' for the same exact connection (the one just blocked), but just does not happen. I would add on this particular point: UDP is stateless, so whichever packet can be packet 0.

I might be wrong on second point, happy to hear your thoughts.

My opinion is that some rules are not loaded in pf, and I would like to be able to know if this is true or not.
Clearly the WebUI believes those rules are there, i doubt pf does (for whichever reason).

By the way I will try your hint to see if I can produce something useful for the research.

[FullyBorked]

It's hard to provide a lot of guidance without some info on network layout and rule sets.  If it's happened in the past it seems to be an environmental issue of some type with your particular configuration.  It'll be a bit hard to weed out potentially.  If it's happened more than once on different distributions my first thought would be a design issue is somehow creating the issue you are seeing.  It's possible it's a bug that's specific to your setup, but it's also possible your configuration is causing you pain and the firewall is functioning as designed (I've chased that red herring a few times). 

I had a specific issue one time NAT'ing IPSEC traffic across a Watchguard firewall between to JUNIPER VPN endpoints.  Technically what I was doing wasn't supported or even proper for that matter.  Due to our specific requirements it had to be designed this way.  It worked for a long time but at some random point the firewall would just start silently dropping packets even though the rule was in place.  A reboot of the firewall or a reapply of the rule would sometimes bring it back.  I never figured out if it was a bug or just an issue with how I was improperly configured or a combination of both. 

I give that long story just to give a representation of what I'm getting at. 

If you mange to root out a bug, it can be reported here https://github.com/opnsense/core/issues/new?assignees=&labels=&template=feature_request.md&title=

[Gianks]

Thanks for your time and story.

I agree on the difficulty of a proper external help without a full overview of the configuration.
I must admin that we pushed the overall setup to some degree of complexity using VPNs, multiWan FO & LB, HA, etc.
That's the main reason why I objected about the rule definitions, hard to imagine the problem is so easy to reproduce since I would bet it would affect many more admins.

For the sake of the discussion, I do believe that in consideration of what we said, a better set of tools for debugging should be evaluated to overcome the need of seeing/interacting with the actual device/configuration.
Providing some insight about what is going on behind the curtains to the firewall admins might be the solution to allow them to help practically to root out bugs without disclosing information.

On the other hand, similarly, integrating some export tool for debugging logs (with some mangling of IPs and other sensitive info), to allow users to provide something standardized to developers to more easily get to an answer, might conduce to a superior bug discovery process and overall support experience for the community.

Meanwhile, after replacing the various rules and a bunch of reboots (including a full power off of both units at the same time), the problem seems to be gone.

Thanks a lot!