opnsense unreachable, disk full with Filter and Suricata logs

[dinguz]

You could also enable compression in ZFS to mitigate the problem somewhat.

[9axqe]

Suricata can trigger on a lot of things if you enabled _everything_

For example, if you only have Windows laptops and iPhones  then Microsoft Exchange or Oracle Weblogic rules don't need to be enabled.

Out of 63864 entries chances are you're gonna hit enough times one or more generic rules that will quickly fill up storage

I just checked, I have not downloaded yet a single ruleset actually, I just enabled the service. I will have to fix this memory issue first before enabling anything here.

[9axqe]

I noticed that my FW tables are very large: 537079/1000000

The reason is, I have implemented some geoblocking.

Could such large table cause firewall logs to become more verbose? The only block rule leveraging the alias for geograpic IP lists is NOT logging though (small "i" is gray).

Secondly, how can flowd.log be 48GB large in less than a month if IPS is enabled but not a single ruleset has been downloaded? Is it expected? (as a reminder: small home network, single server on the network that is only used to backup computers at night) I'm starting to worry what is going to happen if I enable a couple of rulesets...

[dinguz]

In System:Settings:Logging, you can disable logging of the default pass and default block rules, have you checked this?

[9axqe]

I found the culprit I think, CPU went down and RAM usage also stabilized. I had one rule to block DoH and DoT to public DNS servers using a public list with 70k such IPs, that was causing the crazy amount of logs, at least for filter. CPU usage is also down now, I don't see the syslog-ng and filterlog processes eathing up 50% of a CPU core all the time...

It seems I needs to be careful with these large fw aliases, I didn't expect this.

[newsense]

70K DoH/DoT servers seems a bit excessive, doubt there are that many to begin with...


Block outbound Dport 853 to any, udp/443 to major providers (Google, Cloudflare, Q9, and the default one in Firefox for your country) and with a redirect of regular dns to your internal one (Unbound, AGH, Pi-hole) you've accomplished the same thing

Check in Firewall - Live view for any suspicious udp/443 traffic towards obscure DNS servers - shouldn't be that many hits - if at all

[9axqe]

Nope, I made the wrong assumption. I enabled and disabled logging for all 4 rules individually to double check and the one I assumed was the problem (using an alias with 70k IPs in it) is not causing the problem.

It's the one blocking outbound to TCP/853 (DoT).

The almost identical one, blocking outbound traffic to UDP/8853 (DoQ, DNS over quic) is NOT causing this issue.

I tested three times in a row. If I enable logging for the TCP rule, the moment I click apply I see filterlog and syslog-ng shooting up in the top I have running. If I turn it off, it goes away.

Not reproducible for the UDP rule.

If anyone is bored and want to try to reproduce this... ;) The interface I applied it is a LAN brige, and it's applied to IPv4 and IPv6.

So far, I do not understand what's so special about the TCP rule that if would cause this...

[9axqe]

70K DoH/DoT servers seems a bit excessive, doubt there are that many to begin with...

There is probably more than that actually, 70k are the known ones...

I use this list: https://public-dns.info/nameservers.txt

[newsense]

Sure, most of them are regular servers, not  DoH/DoT enabled and you wouldn't be hitting any of it with the DNS/53 intercept rules (need one for IPv6 btw)

[9axqe]

Sure, most of them are regular servers, not  DoH/DoT enabled and you wouldn't be hitting any of it with the DNS/53 intercept rules (need one for IPv6 btw)

ah, right, but they could add it any time. Ideally, I'd have the same list filtered down by DoH capability...

[CJ]

I'm using the same list to block DoH but just standard port blocking for DNS and DoT.  I have all of them set as Floating rules with logging turned on and I don't have this issue.  However, I'm not running a bridge or Suricata currently.

[9axqe]

It's on the bridge for me, and after upgrading to 23.7.2 I retested, same behaviour, the moment I enable logging on this particular rule, syslog-ng and filterlog processes go nuts.

[CJ]

It's on the bridge for me, and after upgrading to 23.7.2 I retested, same behaviour, the moment I enable logging on this particular rule, syslog-ng and filterlog processes go nuts.

Try changing it to a floating rule.  I'm curious if that will make a difference or if it's just due to the bridge.

[9axqe]

I found what the issue was. Maybe I should simply have opened the logs.

Home Assistant was sending 1-2Mbps (!) of DoT attempts to Cloudflare somehow. I changed the rule from "reject" to "block" and it stopped. That's why this specific rule was causing such a crazy cpu consumption...