Log rate limiting - any better approach?

[meyergru]

Recently, I had a problem when some of my friends told me that out voice chat was choppy. When I investigated, I found my OpnSense at 100% CPU. That turned out to be due to 7 WAN block rules for various blocklists and an ongoing DDOS attack.

I had logging enabled for those rules and the log was constantly being written to. I could not even reliably use the web UI.

This made me think about how to rate-limit the logs. Apparently, with 26.7.2, there is now max-pkg-rate from pf available.
So, now, you can create two block rules, one with logging enabled, quick block and "Max packet rate [packets]" = 1 and "Max packet rate [seconds]" = 1 to have a rule that matches at most once per second and logs and another quick block rule without logging and without rate limits immediately following that.

By using that, you still have (rate-limited) logging, but without the risk for a totally overwhelmed OpnSense.

Alas, you need two rules, because the first one will not match any more when the rate limit has been exceeded. Would there be a better way of having such a thing (i.e. eat the cake and still have it)?

P.S.: Allowing rate-limiting on block rules is a potential security hole, IMHO: When you use that without thinking, the block will not be applied when the rate is too high, i.e. block rules can be circumvented by just hitting more frequently at the target....
I guess the original idea was to do this for pass rules only and rate-limiting those on the side.

[WN1X]

It seems to me the rule should be applied regardless if the log rate limit has been reached. Obviously, without logging.

[Patrick M. Hausen]

There is no log rate limit, yet. @meyergru created a hack using the rule application limit and two consecutive rules to achieve the desired effect.

[newsense]

@meyergru Did you enable syncookies in FW -Settings? The Anti DDoS settings that is.

Try Adaptive, Start at 95% End at 70%

[Monviech (Cedrik)]

I wouldnt recommend using a syn proxy as it can silently break some services/websites from working correctly.

[meyergru]

It seems to me the rule should be applied regardless if the log rate limit has been reached. Obviously, without logging.

While that would be desirable, pf does not work that way today, so I made it work.

My question is actually three-fold:

1. Can anyone think of a cleverer way to make it work without using duplicated rules?
2. Would there be a way to have the desired behavior for block rules (i.e. "rate-limit logging without affecting rule matching"), probably as a feature request for OpnSense?
3. Should there be a more prominent web UI warning about this "unexpected" behaviour for block rules?

And no, I do not want to use syncookies for functional reasons - also they would not fix the problem: state table exhaustion is not the issue, it is the mere logging that is throttling the firewall.

[pfry]

Out of curiosity, what was your packet rate with/without (assuming the condition persisted)?

[Seimus]

2. Would there be a way to have the desired behavior for block rules (i.e. "rate-limit logging without affecting rule matching"), probably as a feature request for OpnSense?
3. Should there be a more prominent web UI warning about this "unexpected" behaviour for block rules?

I think this is more for a feature request.
But its bothersome that the DDOS came actually within when you think about it. A DDOS attack was prevented by block rules yet it was so huge that a feedback loop via logging happened that caused practically a DDOS caused by the logging rules.

So the only feasible way, is to disable logging for block rules?

Regards,
S.

[Monviech (Cedrik)]

Thats a good idea. Disable logging for default deny and create your own single default deny with rate limiting. When it falls through the logging ends.

But ofc that doesnt include allowed traffic.

And you might miss interesting info in the logs.

[meyergru]

Yes, actually your impression is correct. The limited rate after the duplicated rules obviously was limited to 9 log entries per second, because I originally had 9 rules and imposed an 1/1 limit on them. I just want to see indications as to what is happening in the logs. I still can look at the counters if I want to see how many requests come in.

I cannot say for sure how many requests per second were there at the time of attack and I also think many may have been dropped because the storage could not keep up. It is a 1 Gbps connection, but with just port probing, that may still amount to a few thousand requests per second, each of which has a significant log impact. I cannot really say what has happened, because obviously the filterlog process was out for several hours (there is also a gap into log file history on one day), such that not even remote logging shows anything. As a matter-of-fact IDK if the CPU load was only a side effect.

Logging for default deny is off here.