Hi
We recently upgraded to 24.x and found problems with the new dashboard. A single user with the dashboard page open would eventually make all systems behind the firewall unreachable. Upgrading to 24.7.7 has not resolved the problem. It looks to be related to the Services and / or Traffic Graph widget. If I remove them from the dashboard we don't see the problem.
When those widgets are enabled on the dashboard we see a growing "backlog" of processes chewing CPU. Output from 'top' is included below. Over time we see the idle cpu % dropping until eventually the CPUs are maxed out and access to systems through the firewall become unstable. We're running a very low traffic firewall on a 4 core + 8GB RAM VM. There's nothing I can see in the logs that points to a problem.
We saw this behaviour when we upgraded our FW (that's been running for several years). As a test I built a fresh new VM with 24.7.7 and a restore of our config and we see the same problem even when it's not in the traffic path. Any ideas or guidance would be appreciated.
Thanks
David
---
last pid: 27517; load averages: 7.09, 2.43, 1.53 up 13+00:19:40 17:04:52
122 processes: 24 running, 98 sleeping
CPU: 93.2% user, 0.0% nice, 6.7% system, 0.1% interrupt, 0.0% idle
Mem: 1337M Active, 3297M Inact, 1037M Wired, 676M Buf, 2267M Free
PID USERNAME THR PRI NICE SIZE RES SWAP STATE C TIME WCPU COMMAND
20678 root 1 92 0 91M 73M 0B RUN 2 0:01 21.62% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
24251 root 1 90 0 87M 69M 0B RUN 1 0:01 21.58% /usr/local/bin/php /usr/local/sbin/pluginctl -D
15528 root 1 94 0 93M 75M 0B CPU2 2 0:03 20.66% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
23693 root 1 89 0 85M 68M 0B RUN 0 0:00 18.54% /usr/local/bin/php /usr/local/sbin/pluginctl -S
25921 root 1 89 0 91M 67M 0B RUN 3 0:00 17.92% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
6394 root 1 95 0 99M 76M 0B RUN 2 0:03 17.69% /usr/local/bin/php /usr/local/sbin/pluginctl -S
21554 root 1 93 0 91M 73M 0B RUN 1 0:02 17.68% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
22190 root 1 91 0 89M 71M 0B RUN 0 0:01 17.56% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
14293 root 1 96 0 95M 77M 0B RUN 2 0:03 17.49% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
20585 root 1 92 0 89M 72M 0B RUN 0 0:01 17.29% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
19862 root 1 94 0 91M 74M 0B RUN 2 0:02 16.48% /usr/local/bin/php /usr/local/sbin/pluginctl -D
4668 root 1 96 0 97M 79M 0B RUN 3 0:04 16.23% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
13521 root 1 95 0 93M 76M 0B RUN 1 0:03 16.19% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
19250 root 1 94 0 91M 74M 0B RUN 3 0:02 16.08% /usr/local/bin/php /usr/local/sbin/pluginctl -S
6030 root 1 95 0 95M 78M 0B RUN 3 0:03 15.64% /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
6590 root 1 95 0 93M 76M 0B RUN 0 0:03 15.16% /usr/local/bin/php /usr/local/sbin/pluginctl -D
5413 root 1 95 0 95M 78M 0B CPU3 3 0:04 14.98% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
4926 root 1 68 0 97M 79M 0B piperd 2 0:04 13.94% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
15000 root 1 95 0 93M 76M 0B CPU0 0 0:03 13.78% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
24745 root 1 89 0 85M 67M 0B RUN 1 0:00 13.52% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
25533 root 1 88 0 83M 66M 0B RUN 0 0:00 12.44% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
26968 root 1 88 0 85M 66M 0B RUN 1 0:00 11.73% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
20595 root 1 20 0 123M 69M 0B select 0 0:00 4.06% /usr/local/bin/php-cgi
7953 root 1 20 0 94M 73M 0B select 3 0:04 3.29% /usr/local/bin/php-cgi
17980 root 1 20 0 93M 68M 0B select 2 0:03 3.11% /usr/local/bin/php-cgi
68140 root 27 68 0 160M 89M 0B accept 2 1:10 2.92% /usr/local/bin/python3 /usr/local/opnsense/service/configd.py console (python3.11)
9559 root 1 20 0 93M 70M 0B select 2 0:03 2.70% /usr/local/bin/php-cgi
20051 root 1 20 0 107M 76M 0B select 2 0:04 2.19% /usr/local/bin/php-cgi
9757 root 1 20 0 93M 71M 0B accept 2 0:02 0.59% /usr/local/bin/php-cgi
I have a similar issue.
Did you find a solution?