Roku DNS storm is impacting OPNsense

[OPNenthu]

I'm seeing this exact issue: https://github.com/FreshTomato-Project/freshtomato-arm/issues/268

My parents have a Roku box that just started flooding DNS to its telemetry endpoints which are blocked by DNSBL policy.  I'm seeing millions of requests in the reporting period (I think OPNsense keeps last 24 hrs).  The only issue is that it's causing log buildup which is overwhelming the system.  Memory use went from ~20% (baseline) to over 60%, mostly due to Unbound's logger (attached).  The Unbound reporting is taking half a minute to load.  Also seeing slowdowns in the live view and Firewall widget loading.

You cannot view this attachment.

What I did remotely was to force the WiFi client to reconnect via the UniFi console.  Unfortunately it immediately started spamming DNS again once it reconnected.  For the moment I've blocked the device from internet access.

The recommendation in the GH link is to redirect the telemetry endpoint to some blackhole IP instead of 0.0.0.0.  I think that could end up being a maintenance issue if the hostnames change, so I'm wondering if I can instead rate limit the DNS requests just from this device?  A quick forum search seems to indicate there's no way to do that, but I'm not sure.  Appreciate tips on how to best proceed (short of throwing the Roku in the trash).

[WN1X]

Why not redirect the Roku to 127.0.0.1, letting it spam itself?

[OPNenthu]

It was tried (4th comment in the ticket) and apparently only worked initially.

Worth a shot, though.

(EDIT): I think the problem is that I would have to set up an alias with the specific telemetry endpoints to use as the destination in the DNAT rule.  Roku apparently has many such endpoints.  I can't keep such a list manually updated and reliable.

For example, in my logs it's spamming "brewster.logs.roku.com" but in the logs in the linked ticket it's spamming "bayside.logs.roku.com".

[Monviech (Cedrik)]

You might be able to use a firewall overload table combined with a block rule.

If your DNS rule matches and too many requests are sent then the client will be added to the defined overload table.

Then with a block rule before the dns allow rule, that client will then be blocked for some time.

But that would block all DNS traffic of that client. So kinda moot if it should still be allowed "something" and only telemetry should be blackholed.

If the telemetry endpoints are all under some certain wildcard domains you could also use a dnsmasq ipset alias to banish them to the shadow realm.
https://docs.opnsense.org/manual/dnsmasq.html#firewall-alias-ipset

[Patrick M. Hausen]

In AdGuard Home you could do something like this:

Code Select Expand

'rewrites':
  - 'domain': *.logs.roku.com
    'answer': 127.0.0.1

And in Unbound:

[OPNenthu]

Thank you, I added the host override in DNS but it looks like the Roku doesn't want to shut up.  I tried both 127.0.0.1 and 192.168.254.254.  The log spam continues.

But that would block all DNS traffic of that client. So kinda moot if it should still be allowed "something" and only telemetry should be blackholed.

If the telemetry endpoints are all under some certain wildcard domains you could also use a dnsmasq ipset alias to banish them to the shadow realm.
https://docs.opnsense.org/manual/dnsmasq.html#firewall-alias-ipset

Yeah, I need for it to stay connected so they can watch TV and only the DNS spam should be controlled.  A firewall rule for all *.logs.roku.com would let me disable logging so that could possibly do the trick, as long as they don't change it.  Thanks, will look into this.

Long term, it would be nice for OPNsense to have a rate limiting function :)

[Monviech (Cedrik)]

Well opnsense does have a rate limiting function, its the PF overload tables and/or request limiters. But it can only limit by source and destination and other layer 3 identifiers.

Since your destination is semi dynamic, it could only rate limit a source, and that would then block the whole source, or drop requests that you might want.

A rate-limiter would do the same, it couldn't identify which traffic you want, and which you don't, because there is no metadata to use here (other than source and destination...)

If you want to rate limit on a higher OSI Layer (e.g. on the application layer by checking DNS packet contents itself) the firewall would already have to process more again which is the same as simply allowing the processing to hit the DNS daemon anway since it can handle that load if the hardware itself can handle the load.

And then we get into CDN territory, since you need beefier hardware to filter out requests that should be rate limited before it hits your own smaller hardware. Push expensive filtering/classification away from the smaller origin system. On a home firewall, that extra layer usually does not exist, so the resolver still has to receive and inspect the request before it can decide what to do with it.

[nero355]

Appreciate tips on how to best proceed (short of throwing the Roku in the trash).

IMHO that's the best thing to do after reading this : https://discourse.pi-hole.net/t/what-domain-to-whitelist-to-unblock-roku-tv-schedule/86284
And now your topic too... :'(

Long term, it would be nice for OPNsense to have a rate limiting function :)

Pi-Hole has the limit DNSmasqd has too :

-0, --dns-forward-max=<queries>

Set the maximum number of concurrent DNS queries. The default value is 150, which should be fine for most setups. The only known situation where this needs to be increased is when using web-server log file resolvers, which can generate large numbers of concurrent queries.

So in theory the one in OPNsense should behave the same :)

[OPNenthu]

Since your destination is semi dynamic, it could only rate limit a source, and that would then block the whole source, or drop requests that you might want.

Got it.  I was thinking to allow certain sources to talk to Unbound at a set rate (ray 1 req/sec) rather than blocking them after some quota, but I don't know how that would be implemented except maybe in pf itself.

Here's a different idea but it might be a dumb one...

OPNsense is already handling the blocklist policy for Unbound via python hooks.  Those policies could be enhanced to, say, have an option to silence blocked requests only from specific problematic sources rather than letting the log fill up.  Right now logging is all or none, but it needn't be.  The DNSBL is otherwise doing its job.  It's just a resource problem.

IMHO that's the best thing to do after reading this : https://discourse.pi-hole.net/t/what-domain-to-whitelist-to-unblock-roku-tv-schedule/86284

You won't find one in my home, but my parents subscribe to a streaming app on that platform for international programming.  So I try to contain it for them.

The thing that really lights my candle is recent revelations that many smart TVs have been caught capturing frequent screenshots of owners' screens and there's speculation about microphones doing the same kind of thing.

I'm perfectly happy with a "dumb" TV that I can plug what I want into, thank you very much.

BTW r/e the antenna TV EPG issue that person in your thread mentions- I don't know if my parents' Roku has the issue but I anyway set it up so that they switch to the antenna input and use the TV's own remote for channel tuning when they want to watch OTA channels.  The TV was designed to do one thing well, so might as well use it :)  However, if you have a fully integrated RokuTV I guess that limits your options.

So in theory the one in OPNsense should behave the same :)

That could be helpful if I were using Dnsmasq for DNS, but anyway that to me looks like a global limiter rather than from a specific source.  Nice that they include it.

[keeka]

Annoying isn't it. These roku boxes are infuriating from a networking POV but are a big favourite with older people for easy access to streamed TV because the interface is a masterclass in UX. Sadly I have one here :-( and I resorted to putting it on a smart plug to keep it off most of the day. I put it on its own SSID which is also on a schedule. A bit extreme but at least I no longer see 24hr DNS storms. Whilst they are known to be chatty, something updated a couple of months ago that resulted in much higher rates of queries for those same few hostnames. I have the roku querying a pihole so these requests fortunately don't hit opnsense.

[OPNenthu]

Whilst they are known to be chatty, something updated a couple of months ago that resulted in much higher rates of queries for those same few hostnames.

That jives.  It's always been the chattiest in the DNS logs, but never like this.

Maybe some middle manager vibe coded something... lol.

I have the roku querying a pihole so these requests fortunately don't hit opnsense.

That's also an option.  I've been resisting moving DNS off of the firewall because it's one more thing to host and maintain.

[Monviech (Cedrik)]

You could use NAT to forward just the Roku requests directly to 8.8.8.8 for example. Or give it a dhcp reservation with an external dns server right away.

Or to dnsmasq on port 53053, you can run multiple dns servers at the same time.

[keeka]

I don't know about the OP but IME the Roku DNS storm is a hissy fit in response to DNSBL. I only forward DNS to the pihole for select clients. Everything else, inclusing the pihole, queries opnsense unbound directly, which is not using any DNSBL.

[OPNenthu]

I only forward DNS to the pihole for select clients. Everything else, inclusing the pihole, queries opnsense unbound directly, which is not using any DNSBL.

That's effectively what the source-based Unbound policies allow you to do, since the OPNsense devs brought that into the community edition from the business edition some months ago.  It was a game changer for reducing the need for external DNS, IMO.

You could use NAT to forward just the Roku requests directly to 8.8.8.8 for example. Or give it a dhcp reservation with an external dns server right away.

Well, the desire is to actually block that telemetry, just in a way that doesn't also kill the OPNsense resources.  I feel like this is a design problem because any misbehaved actor on the network could similarly do this, no?  Just happy that this one happens to be benign.

BTW, the firewall in question has 8G of RAM.  The CPU is relatively weak (J4125) but still doesn't break a sweat as a pure firewall+router for gigabit.

[Monviech (Cedrik)]

Denial of Service can take many shapes, one of them is harddrive space exhaustion. Any service that logs can be susceptible to it, thats why logs rotate and can have upper size limits etc... but rogue clients are always an issue, its one of the main reasons CDNs have so many customers xD