24.7.7 dashboard widgets cause 100% CPU

[oz_djh]

Hi

We recently upgraded to 24.x and found problems with the new dashboard.  A single user with the dashboard page open would eventually make all systems behind the firewall unreachable.  Upgrading to 24.7.7 has not resolved the problem.  It looks to be related to the Services and / or Traffic Graph widget.  If I remove them from the dashboard we don't see the problem.

When those widgets are enabled on the dashboard we see a growing "backlog" of processes chewing CPU.  Output from  'top' is included below.  Over time we see the idle cpu % dropping until eventually the CPUs are maxed out and access to systems through the firewall become unstable.  We're running a very low traffic firewall on a 4 core + 8GB RAM VM.  There's nothing I can see in the logs that points to a problem.

We saw this behaviour when we upgraded our FW (that's been running for several years).  As a test I built a fresh new VM with 24.7.7 and a restore of our config and we see the same problem even when it's not in the traffic path.  Any ideas or guidance would be appreciated.


Thanks
David

---
last pid: 27517;  load averages:  7.09,  2.43,  1.53                                                                                                                       up 13+00:19:40  17:04:52
122 processes: 24 running, 98 sleeping
CPU: 93.2% user,  0.0% nice,  6.7% system,  0.1% interrupt,  0.0% idle
Mem: 1337M Active, 3297M Inact, 1037M Wired, 676M Buf, 2267M Free

  PID USERNAME    THR PRI NICE   SIZE    RES SWAP STATE    C   TIME    WCPU COMMAND
20678 root          1  92    0    91M    73M   0B RUN      2   0:01  21.62% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
24251 root          1  90    0    87M    69M   0B RUN      1   0:01  21.58% /usr/local/bin/php /usr/local/sbin/pluginctl -D
15528 root          1  94    0    93M    75M   0B CPU2     2   0:03  20.66% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
23693 root          1  89    0    85M    68M   0B RUN      0   0:00  18.54% /usr/local/bin/php /usr/local/sbin/pluginctl -S
25921 root          1  89    0    91M    67M   0B RUN      3   0:00  17.92% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
 6394 root          1  95    0    99M    76M   0B RUN      2   0:03  17.69% /usr/local/bin/php /usr/local/sbin/pluginctl -S
21554 root          1  93    0    91M    73M   0B RUN      1   0:02  17.68% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
22190 root          1  91    0    89M    71M   0B RUN      0   0:01  17.56% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
14293 root          1  96    0    95M    77M   0B RUN      2   0:03  17.49% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
20585 root          1  92    0    89M    72M   0B RUN      0   0:01  17.29% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
19862 root          1  94    0    91M    74M   0B RUN      2   0:02  16.48% /usr/local/bin/php /usr/local/sbin/pluginctl -D
 4668 root          1  96    0    97M    79M   0B RUN      3   0:04  16.23% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
13521 root          1  95    0    93M    76M   0B RUN      1   0:03  16.19% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
19250 root          1  94    0    91M    74M   0B RUN      3   0:02  16.08% /usr/local/bin/php /usr/local/sbin/pluginctl -S
 6030 root          1  95    0    95M    78M   0B RUN      3   0:03  15.64% /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
 6590 root          1  95    0    93M    76M   0B RUN      0   0:03  15.16% /usr/local/bin/php /usr/local/sbin/pluginctl -D
 5413 root          1  95    0    95M    78M   0B CPU3     3   0:04  14.98% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
 4926 root          1  68    0    97M    79M   0B piperd   2   0:04  13.94% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
15000 root          1  95    0    93M    76M   0B CPU0     0   0:03  13.78% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
24745 root          1  89    0    85M    67M   0B RUN      1   0:00  13.52% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
25533 root          1  88    0    83M    66M   0B RUN      0   0:00  12.44% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
26968 root          1  88    0    85M    66M   0B RUN      1   0:00  11.73% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
20595 root          1  20    0   123M    69M   0B select   0   0:00   4.06% /usr/local/bin/php-cgi
 7953 root          1  20    0    94M    73M   0B select   3   0:04   3.29% /usr/local/bin/php-cgi
17980 root          1  20    0    93M    68M   0B select   2   0:03   3.11% /usr/local/bin/php-cgi
68140 root         27  68    0   160M    89M   0B accept   2   1:10   2.92% /usr/local/bin/python3 /usr/local/opnsense/service/configd.py console (python3.11)
 9559 root          1  20    0    93M    70M   0B select   2   0:03   2.70% /usr/local/bin/php-cgi
20051 root          1  20    0   107M    76M   0B select   2   0:04   2.19% /usr/local/bin/php-cgi
 9757 root          1  20    0    93M    71M   0B accept   2   0:02   0.59% /usr/local/bin/php-cgi