Roku DNS storm is impacting OPNsense

[RobertoZ]

I have a TCL TV that has Roku OS on it.  It's constantly very chatty.  I use ControlD for DNS so I have it blocked.  I tried disconnecting the TV from the network, but then there is a bright white light on the front of the TV that constantly flashes with the intensity of a thousand suns.

I looked for a new "dumb" TV with no smart features.  I quickly found out unless you want to buy a professional display costing almost as much as a car you are stuck with this scheiße. 

</rant>

[OPNenthu]

I looked for a new "dumb" TV with no smart features.  I quickly found out unless you want to buy a professional display costing almost as much as a car you are stuck with this scheiße. 

</rant>

It almost seems like the market is rigged so the rentiers and data brokers always win...

[lilsense]

I have a TCL TV that has Roku OS on it.  It's constantly very chatty.  I use ControlD for DNS so I have it blocked.  I tried disconnecting the TV from the network, but then there is a bright white light on the front of the TV that constantly flashes with the intensity of a thousand suns.

I looked for a new "dumb" TV with no smart features.  I quickly found out unless you want to buy a professional display costing almost as much as a car you are stuck with this scheiße. 

</rant>

You can stop by your local friendly hardware store if you do not have a black electrical tape and cut a tiny piece and place it on the bright LED. :) 
No smartTV should be on anyone's network, even the world's BEST SONY Android TV's. 

[nero355]

No smartTV should be on anyone's network

MWAHAHAHA!!!!! "It's funny because it's true!" :P

[OPNenthu]

Quick update-

The DNS storm seems to have stopped overnight but I'm not sure why.  All I had done was add a host override in Unbound with the black-hole IP, but I had removed it since it wasn't helping to calm the log spam.  Now it's back to just the DNSBL policy blocking the telemetry and it's acting normally.

I guess either there's some trigger for the storm that hasn't been hit yet, or something's been fixed (hopefully).

[keeka]

Quick update-

The DNS storm seems to have stopped overnight but I'm not sure why.

I wouldn't bet on it. When I first noticed the increase in  DNS queries, I left the roku powered down for a short period. After restarting, DNS queries remained low for some time, but eventually returned to once per second for each of various hosts in logs.roku.com. I don't see any performance hits at that level but it is rather irritating and does put me off buying more such devices.

[OPNenthu]

I had almost two months of relief from this but now I got a Monit alert email that the CPU was pegged.

It's Roku again.

Code Select Expand

# top

last pid: 94206;  load averages:  12.54,  12.19,    7.00                                          up 1+00:26:27  15:38:35
75 processes:  3 running, 72 sleeping
CPU: 97.8% user,  0.0% nice,  2.3% system,  0.0% interrupt,  0.0% idle
Mem: 937M Active, 1712M Inact, 2074M Laundry, 1646M Wired, 1151M Free
ARC: 705M Total, 221M MFU, 403M MRU, 532K Anon, 14M Header, 63M Other
    566M Compressed, 6726M Uncompressed, 11.89:1 Ratio
Swap: 8192M Total, 2265M Used, 5926M Free, 27% Inuse

  PID USERNAME    THR PRI NICE  SIZE    RES STATE    C  TIME    WCPU COMMAND
 4737 root        11 113    0  470M  371M CPU1    1  2:41 191.32% python3.13
 4013 root        11 111    0  514M  396M RUN      3  2:43 180.62% python3.13
48625 unbound      4  3    0  802M  376M kqread  1 191:11  19.53% unbound
17859 root          3  0    0    81M    36M kqread  1  14:54  1.34% syslog-ng
87088 root          1  0    0    14M  1748K bpf      3  13:56  1.33% filterlog
...

You cannot view this attachment.


I went back to post #4 and re-added the host override in Unbound.

I see from the query logs that the override is in effect (it's using "Source=Local-data"), but the client is still spamming so much that it's keeping the OPNsense CPU busy and the system temps elevated.

Code Select Expand

# top

last pid: 43681;  load averages:    6.45,    2.66,    2.83                                           up 1+00:52:34  16:04:42
77 processes:  3 running, 74 sleeping
CPU: 97.9% user,  0.0% nice,  2.0% system,  0.1% interrupt,  0.0% idle
Mem: 1176M Active, 1138M Inact, 1838M Laundry, 1680M Wired, 1689M Free
ARC: 724M Total, 243M MFU, 402M MRU, 1668K Anon, 14M Header, 64M Other
     583M Compressed, 6817M Uncompressed, 11.69:1 Ratio
Swap: 8192M Total, 2147M Used, 6045M Free, 26% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
  750 root         11 111    0   443M   342M RUN      2   1:59 193.59% python3.13
   23 root         11 114    0   464M   367M CPU3     3   2:03 191.15% python3.13
 9890 unbound       4   0    0   617M   501M kqread   3   0:57   9.36% unbound
17859 root          3   0    0    85M    39M kqread   1  15:06   0.70% syslog-ng
87088 root          1   0    0    14M  1748K bpf      2  14:08   0.70% filterlog
...

Hopefully it calms down in some time but I may need to break down and install a standalone DNS there to get this load off of OPNsense.  As a last resort I'll consider disabling the telemetry blocks :(

[keeka]

Sorry to hear it's had a relapse. Mine has been furiously querying the same hosts in logs.roku.com ever since your OP. Consequently it spends 21 hours a day turned off via smart plug! From the hostnames concerned, it would seem associated with the Roku OS itself rather than some errant app. Idiotic behaviour.

[nero355]

I may need to break down and install a standalone DNS there to get this load off of OPNsense.

You could ofcourse give it a small dedicated DNS Server running on a Raspberry Pi 2B/3B or perhaps even a Pi Zero and keep everything else as is :)

What a horrible device by the way...

Has anyone ever complained about this via Roku Support or something like that ?!
That amount of DNS Requests from such a small shitty device is just INSANE!!! :(

[Monviech (Cedrik)]

You could try the new rate limits of pf out to limit how many pakets the Roku could send the opnsense dns port.

https://github.com/opnsense/core/commit/80f4affed7074a1095a3ffabe98419af2d4498af

[OPNenthu]

You could try the new rate limits of pf out to limit how many pakets the Roku could send the opnsense dns port.

https://github.com/opnsense/core/commit/80f4affed7074a1095a3ffabe98419af2d4498af

This brought the load averages down a lot and restored the system responsiveness.  Thank you for adding it!

Implementation notes below.

Code Select Expand

last pid: 73963;  load averages:    0.26,    0.18,    0.17                                           up 0+02:01:28  15:14:13
71 processes:  1 running, 70 sleeping
CPU:  0.3% user,  0.0% nice,  0.3% system,  0.0% interrupt, 99.4% idle
Mem: 265M Active, 1307M Inact, 1194M Wired, 4745M Free
ARC: 382M Total, 154M MFU, 177M MRU, 9639K Anon, 2886K Header, 39M Other
     273M Compressed, 730M Uncompressed, 2.67:1 Ratio
Swap: 8192M Total, 8192M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
17210 unbound       4   0    0   603M   500M kqread   2   1:55   0.55% unbound
50439 root          1   0    0   100M    56M nanslp   1   0:33   0.55% php
22734 root          8   5    0   411M   335M kqread   1   8:34   0.12% python3.13
73963 root          1   0    0    15M  3884K CPU0     0   0:00   0.06% top
33976 hostd        12   0    0    77M    16M uwait    2   0:04   0.04% hostwatch
54035 root          3   0    0    51M    17M kqread   0   0:24   0.03% syslog-ng
73373 root          1   0    0    14M  2524K select   3   0:01   0.03% powerd
69073 root          1   0    0    14M  3364K bpf      0   0:00   0.02% filterlog
...

I applied with "$ opnsense-patch https://github.com/opnsense/core/commit/80f4aff" on top of 26.7.1_1.

There was a trick to making it work correctly with my ruleset.  I ended up with 4 new rules in Floating (plot twist: there are actually multiple Rokus on the network so I made an alias):

You cannot view this attachment.

Code Select Expand

pass in quick inet proto {tcp udp} from $HOSTS_ROKU to {127.0.0.1} port {domain} keep state max-pkt-rate 50/10
pass in quick inet6 proto {tcp udp} from $HOSTS_ROKU to {fdff::1} port {domain} keep state max-pkt-rate 50/10
pass in quick inet proto {tcp udp} from $HOSTS_ROKU to {(self)} port {domain} keep state max-pkt-rate 50/10
pass in quick inet6 proto {tcp udp} from $HOSTS_ROKU to {(self)} port {domain} keep state max-pkt-rate 50/10
block return in quick inet proto {tcp udp} from $HOSTS_ROKU to {any} port {domain}
block return in quick inet6 proto {tcp udp} from $HOSTS_ROKU to {any} port {domain}

The first two catch any redirected requests from NAT rules.

The third catches normal queries to the Unbound listener on the interface.

The fourth (important!) catches and rejects any that were not matched by the rate limiter.  I found in testing that once the rate limiter is exceeded the queries would simply go on to match the default DNS rule at group/interface level, so they need to be explicitly dealt with.

Made all of them non-logging rules.


Now, I need to confirm any effects on Roku usability.  I'll give it a day or two to see if my parents notice any service disruptions.

[lmoore]

block return in quick inet proto {tcp udp} from $HOSTS_ROKU to {any} port {domain}
block return in quick inet6 proto {tcp udp} from $HOSTS_ROKU to {any} port {domain}

If you were to change the Action in these rules to Block, by how much would the rate of queries drop?

Code Select Expand

block drop in quick inet proto {tcp udp} from $HOSTS_ROKU to {any} port {domain}
block drop in quick inet6 proto {tcp udp} from $HOSTS_ROKU to {any} port {domain}

[OPNenthu]

The Reject action plays a critical role here.  These rules are throttling all of the DNS queries from the Roku group, not just the undesirable ones.  I need them to fail fast with feedback so that the streamer doesn't hang and has the best chance of retrying quickly when one of the necessary queries gets dropped.

[lmoore]

The Reject action plays a critical role here.  These rules are throttling all of the DNS queries from the Roku group, not just the undesirable ones.  I need them to fail fast with feedback so that the streamer doesn't hang and has the best chance of retrying quickly when one of the necessary queries gets dropped.

I see now, thanks

[DEC740airp414user]

I unhooked my Samsung tv (from iot vlan) and moved to a Apple TV because of this noise

Still has some noise but not as much as a tizen os

Appreciate the reminder on why I'll never own a Roku