Messages

PS2: I've noticed, using the top command, that Python 3.13 often takes up over 90% of the CPU (WCPU column in top).

Momentarily or constant?  Asking because python is used for several things in OPNsense and it's normal (in my experience, at least) to see it burst when a parent process or daemon is working.

If it's constantly hogging the CPU then try to narrow down the particular calling process.  Get the python PID and do:

$ ps -p <pid> -o pid,ppid,user,command

That might get some basic clues.  If lucky the command line arguments will reveal it.

Code Select Expand

root    48076  64.7  0.1   38432   22836  -  R    15:11    0:02.71 /sbin/pfctl -t Bad_Nations -T replace -f /var/db/aliastables/Bad_Nations.txt


As for the GeoIP stuff, sorry I only have a joke to offer :)

If you rename "Bad_Nations" to "My_Enemy_States" it might make things a little less ambiguous for your poor CPU!

Back with an update.

While using the actual devices for a bit, we could maybe tell there was a little bit of random delay in app and video stream startup but it was quite acceptable.  We could have just as well imagined it due to expectation bias.

The Unbound reports are overgrown now and taking long to load due to the tens of millions of accumulated queries, but this includes metrics from before the limiter was added.  I also noticed the 'syslog-ng' service had stopped and needed a restart, which is concerning.

So... partially solved.  I'll manually clear the reporting data if it doesn't rotate out soon and hopefully that will remain at manageable levels going forward.  We'll see.

Appreciate all the tips so far.

---

@yourfriendarmando: thanks for that.  Did you suggest it because there's a link between these two companies?  I wouldn't be surprised but I don't know.

@newsense: that clears it up, thank you.  As for your observation about the constant telemetry, I did notice something interesting when I was going through the Roku privacy settings that I don't recall seeing earlier (must have been added in an update?).

There were two entries in privacy settings: "Automatic Content Recognition (ACR)" and "Content Viewing Disclosure," both which can be unchecked but were enabled.  This allows visual screen scraping and sending off information to 3rd parties and advertisers.  Unfortunately, disabling them did not stop the DNS storm.

Kettlebell drop is arguably the most therapeutic solution ;)

---

I did a few quick tests and spot checked the results in Unbound reporting.  In order:

1) Using a Host Override with 127.0.0.1 as the IP and default host TTL (0).

You cannot view this attachment.

2) Using only the DNSBL policy with NXDOMAIN and the cache TTL in the DNSBL policy set to 72000:

You cannot view this attachment.

3) Using only the DBSNL policy with 0.0.0.0 (default) and the cache TTL in the DNSBL policy set to 72000:

You cannot view this attachment.

4) Reverting to Host Override but this time with host TTL set to 3600 in the override:

You cannot view this attachment.

---

I won't pretend to understand some of the things I'm seeing.  The only case where the query showed any non-zero TTL in the reporting was with the default DNSBL return value (not using NXDOMAIN) in case #3.  I don't know the source of that value though because it doesn't match anything I have configured in Unbound during that test.

Overall however, I can say that none of these has any observed effect on quieting down the Rokus.  I'm not sure that chasing TTL tweaks is going to solve this.  Remember also, I'm not trying to reduce the number of upstream resolutions.  I'm trying to limit ads & trackers, but Roku software seems relentless and uncaring about TTLs in any case I tried so far.

The pf rate limiters remain effective for keeping the CPU under control, but I have yet to confirm if the Rokus behave well enough with this rate limiting.  I plan to visit my parents this week and will see for myself how it's working then.

Another observation I had is that the python process is exercised heavily even when I'm using the Host Override for these domains.  I know that python is used for DNSBL processing in the OPNsense implementation of Unbound but I'm not sure why that would come into play when a host override is used.  Could there be a python hook somewhere that is adding to the overhead?

I think I haven't had enough coffee yet today because I'm confused by the Git ticket that @newsense posted.  Don't those cache settings already exist in the Unbound GUI?

I have them set to custom values on this firewall since long go:

You cannot view this attachment.

Code Select Expand

root@firewall:~ # cat /var/unbound/advanced.conf | grep cache
msg-cache-size: 128m
rrset-cache-size: 256m
cache-max-ttl: 86400
cache-min-ttl: 300


I agree that the Host Override I used has TTL 0, so that should probably be removed now.

The question then is: do DNSBL hits honor TTL settings?  Or do those also have TTL 0?  And for that matter, does forcing NXDOMAIN on DNSBL hits instead of 0.0.0.0 have a negative impact here as well because I use that.

[Reject]

The Reject action plays a critical role here.  These rules are throttling all of the DNS queries from the Roku group, not just the undesirable ones.  I need them to fail fast with feedback so that the streamer doesn't hang and has the best chance of retrying quickly when one of the necessary queries gets dropped.

You could try the new rate limits of pf out to limit how many pakets the Roku could send the opnsense dns port.

https://github.com/opnsense/core/commit/80f4affed7074a1095a3ffabe98419af2d4498af

This brought the load averages down a lot and restored the system responsiveness.  Thank you for adding it!

Implementation notes below.

Code Select Expand

last pid: 73963;  load averages:    0.26,    0.18,    0.17                                           up 0+02:01:28  15:14:13
71 processes:  1 running, 70 sleeping
CPU:  0.3% user,  0.0% nice,  0.3% system,  0.0% interrupt, 99.4% idle
Mem: 265M Active, 1307M Inact, 1194M Wired, 4745M Free
ARC: 382M Total, 154M MFU, 177M MRU, 9639K Anon, 2886K Header, 39M Other
     273M Compressed, 730M Uncompressed, 2.67:1 Ratio
Swap: 8192M Total, 8192M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
17210 unbound       4   0    0   603M   500M kqread   2   1:55   0.55% unbound
50439 root          1   0    0   100M    56M nanslp   1   0:33   0.55% php
22734 root          8   5    0   411M   335M kqread   1   8:34   0.12% python3.13
73963 root          1   0    0    15M  3884K CPU0     0   0:00   0.06% top
33976 hostd        12   0    0    77M    16M uwait    2   0:04   0.04% hostwatch
54035 root          3   0    0    51M    17M kqread   0   0:24   0.03% syslog-ng
73373 root          1   0    0    14M  2524K select   3   0:01   0.03% powerd
69073 root          1   0    0    14M  3364K bpf      0   0:00   0.02% filterlog
...

I applied with "$ opnsense-patch https://github.com/opnsense/core/commit/80f4aff" on top of 26.7.1_1.

There was a trick to making it work correctly with my ruleset.  I ended up with 4 new rules in Floating (plot twist: there are actually multiple Rokus on the network so I made an alias):

You cannot view this attachment.

Code Select Expand

pass in quick inet proto {tcp udp} from $HOSTS_ROKU to {127.0.0.1} port {domain} keep state max-pkt-rate 50/10
pass in quick inet6 proto {tcp udp} from $HOSTS_ROKU to {fdff::1} port {domain} keep state max-pkt-rate 50/10
pass in quick inet proto {tcp udp} from $HOSTS_ROKU to {(self)} port {domain} keep state max-pkt-rate 50/10
pass in quick inet6 proto {tcp udp} from $HOSTS_ROKU to {(self)} port {domain} keep state max-pkt-rate 50/10
block return in quick inet proto {tcp udp} from $HOSTS_ROKU to {any} port {domain}
block return in quick inet6 proto {tcp udp} from $HOSTS_ROKU to {any} port {domain}

The first two catch any redirected requests from NAT rules.

The third catches normal queries to the Unbound listener on the interface.

The fourth (important!) catches and rejects any that were not matched by the rate limiter.  I found in testing that once the rate limiter is exceeded the queries would simply go on to match the default DNS rule at group/interface level, so they need to be explicitly dealt with.

Made all of them non-logging rules.


Now, I need to confirm any effects on Roku usability.  I'll give it a day or two to see if my parents notice any service disruptions.

I was initially going to wait until I considered this resolved and then reach back out to let them know, but I'm accelerating that now.

Thanks for sharing the statement but please don't rush on account of my asking about it.  It's good to know if your issue truly is resolved by switching to AMI (and no other factors).

I'm not aware of a widespread issue with coreboot but then again I don't have a Reddit account.  Who knows what's out there :-/

It's weird that Protectli support tell people to use AMI as coreboot is one of the main selling points and something they recommend.

We don't have a clear picture.  @fornax did you get any statement from them about this?  Are they even working with you to diagnose and root cause, or just pushing you to AMI?

I had almost two months of relief from this but now I got a Monit alert email that the CPU was pegged.

It's Roku again.

Code Select Expand

# top

last pid: 94206;  load averages:  12.54,  12.19,    7.00                                          up 1+00:26:27  15:38:35
75 processes:  3 running, 72 sleeping
CPU: 97.8% user,  0.0% nice,  2.3% system,  0.0% interrupt,  0.0% idle
Mem: 937M Active, 1712M Inact, 2074M Laundry, 1646M Wired, 1151M Free
ARC: 705M Total, 221M MFU, 403M MRU, 532K Anon, 14M Header, 63M Other
    566M Compressed, 6726M Uncompressed, 11.89:1 Ratio
Swap: 8192M Total, 2265M Used, 5926M Free, 27% Inuse

  PID USERNAME    THR PRI NICE  SIZE    RES STATE    C  TIME    WCPU COMMAND
 4737 root        11 113    0  470M  371M CPU1    1  2:41 191.32% python3.13
 4013 root        11 111    0  514M  396M RUN      3  2:43 180.62% python3.13
48625 unbound      4  3    0  802M  376M kqread  1 191:11  19.53% unbound
17859 root          3  0    0    81M    36M kqread  1  14:54  1.34% syslog-ng
87088 root          1  0    0    14M  1748K bpf      3  13:56  1.33% filterlog
...

You cannot view this attachment.


I went back to post #4 and re-added the host override in Unbound.

I see from the query logs that the override is in effect (it's using "Source=Local-data"), but the client is still spamming so much that it's keeping the OPNsense CPU busy and the system temps elevated.

Code Select Expand

# top

last pid: 43681;  load averages:    6.45,    2.66,    2.83                                           up 1+00:52:34  16:04:42
77 processes:  3 running, 74 sleeping
CPU: 97.9% user,  0.0% nice,  2.0% system,  0.1% interrupt,  0.0% idle
Mem: 1176M Active, 1138M Inact, 1838M Laundry, 1680M Wired, 1689M Free
ARC: 724M Total, 243M MFU, 402M MRU, 1668K Anon, 14M Header, 64M Other
     583M Compressed, 6817M Uncompressed, 11.69:1 Ratio
Swap: 8192M Total, 2147M Used, 6045M Free, 26% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
  750 root         11 111    0   443M   342M RUN      2   1:59 193.59% python3.13
   23 root         11 114    0   464M   367M CPU3     3   2:03 191.15% python3.13
 9890 unbound       4   0    0   617M   501M kqread   3   0:57   9.36% unbound
17859 root          3   0    0    85M    39M kqread   1  15:06   0.70% syslog-ng
87088 root          1   0    0    14M  1748K bpf      2  14:08   0.70% filterlog
...

Hopefully it calms down in some time but I may need to break down and install a standalone DNS there to get this load off of OPNsense.  As a last resort I'll consider disabling the telemetry blocks :(

@cookiemonster was there a time when Unbound in OPNsense didn't support multiple DoT resolvers?  I don't remember but I'm curious what stubby is adding in that chain.

Do you specifically need different upstream resolvers?

One solution since you already have VLANs is to set up blocklist policies in Unbound (Unbound DNS->Blocklists).  Separate resolvers or DNS servers aren't needed if you are just wanting per-VLAN filtering.

The kids can circumvent your DNS blocks via e.g. browser or OS network settings.  You probably already know but worth mentioning.

D'oh! I was thinking of the VP2440. Sorry.

For context, does this only affect the SFP+ ports (Intel X710) on the VP2420?

Asking because I recall reading somewhere that the 'igc' driver does RSS by default and those tunables would not be needed for the 2.5GbE interfaces.  Can confirm that, at least on the V1410 that I own with all i226-V NICs, that is indeed the case.  I have 4 RX/TX queues active with the default tunables.

Sorry this is not answering your original question but it's my favorite aspect of the firewall that you touched on.  This mechanic makes pf more interesting, IMO.

However the last match concept has proven diffcult to grasp, so I just look at it as evaluated last!

As a purely logical construct, I think you can imagine your rules that way- as a set of 'quick' rules followed by a set of 'non-quick' rules evaluated after all the quick ones.  However, in that case you have to also think of the two sets differently:

The 'quick' set will be first-match and processing stops immediately on a match.

The 'non-quick' set will continue to be last-match and they are all processed no matter what.  The last one that matched wins.

Again, sorry for the digression.  I hope my last post helps in the discussion about evaluation counts.

P.S. If you imagine the rules this way, there may be cases where it breaks down... i.e. with tagging or something.  Interesting thought exercise.