Messages

Hi Ad,

Thanks for the reply.  The VM is quite capable and the config is reasonably simple so I assumed there was an issue with the config parser.  I downloaded the config to have a look at the XML and it was 80,000 lines long!  Almost all of it was BAN directives in the nginx config.  That's not something we configured so it looks like the default setting is to ban IPs (on some criteria I'm not aware of) and to never release that ban.  So by default, the list of banned IP will grow and grow until it produces we result we've been seeing.

I found the 'Autoblock TTL' setting in the advanced nginix config, which was set to 0.  I set that to 1 day, saved the config, and the issue was resolved.  The config went from 80,000 lines to 3,000 lines and the CPU problem was immediately fixed.  Perhaps the default settings need to be tweaked to avoid this situation.

Further to the above, it looks like calling the toArray() method on the config in prase_config() from config.inc is causing the problem.

function parse_config()
{
    $cnf = OPNsense\Core\Config::getInstance();
    /* make sure to write back global variable */
printf("DBG 1 : %s\n",date('h:i:s a', time()));
    $config = $cnf->toArray(listtags());
printf("DBG 2 : %s\n",date('h:i:s a', time()));
    return $config;
}

# /tmp/interfaces-broken
DBG 1 : 03:01:05 pm
DBG 2 : 03:01:12 pm
{"vtnet0":{"flags":["up","broadcast","running","simplex","multicast","lower_up"],"capabilities":["vlan_mtu","jumbo_mtu

I have a similar issue.
Did you find a solution?

No solution yet but I may have narrowed it down today.  See the forum thread below.

https://forum.opnsense.org/index.php?topic=47847

Hi

As per a previous thread I posted (https://forum.opnsense.org/index.php?topic=43858) we've been having major UI performance issues since 24.x.  CPU sitting at 100% anytime anyone is on the dashboard page.  We disabled dashboard widgets as a work-around.  After upgrading to 25.1.10 today and still experiencing the problem I thought I'd look further to see if I can at least narrow down the issue.

The problem appears to be related to the config loading functions in  /usr/local/etc/inc/config.inc.  To demonstrate I have this simple snippet from 'pluginctl' that just lists the network interfaces.  I see numerous instances of pluginctl using 100% CPU in top when we experience the performance problem.

#!/usr/local/bin/php
<?php

require_once 'config.inc';
require_once 'interfaces.inc';

echo json_encode(legacy_interfaces_details($args[0] ?? null), $jflags);

If I comment out the require_once of config.inc everything is fine.  But if it loads config.inc the script takes about 7 seconds to complete and eats 100% cpu during that time.  It looks like many PHP scripts are showing the same problem which basically brings the firewall to it's knees when anyone is on the Web UI.  Has anyone seen this before?  Any pointers of what may be going on?  This firewall has been in production for many years and just upgraded in place through lots of different releases.

# time /tmp/interfaces > /dev/null
        0.04 real         0.02 user         0.02 sys
# time /tmp/interfaces-broken > /dev/null
        6.81 real         6.71 user         0.09 sys
# diff /tmp/interfaces /tmp/interfaces-broken
3a4
> require_once 'config.inc';
#

Hi

We recently upgraded to 24.x and found problems with the new dashboard.  A single user with the dashboard page open would eventually make all systems behind the firewall unreachable.  Upgrading to 24.7.7 has not resolved the problem.  It looks to be related to the Services and / or Traffic Graph widget.  If I remove them from the dashboard we don't see the problem.

When those widgets are enabled on the dashboard we see a growing "backlog" of processes chewing CPU.  Output from  'top' is included below.  Over time we see the idle cpu % dropping until eventually the CPUs are maxed out and access to systems through the firewall become unstable.  We're running a very low traffic firewall on a 4 core + 8GB RAM VM.  There's nothing I can see in the logs that points to a problem.

We saw this behaviour when we upgraded our FW (that's been running for several years).  As a test I built a fresh new VM with 24.7.7 and a restore of our config and we see the same problem even when it's not in the traffic path.  Any ideas or guidance would be appreciated.


Thanks
David

---
last pid: 27517;  load averages:  7.09,  2.43,  1.53                                                                                                                       up 13+00:19:40  17:04:52
122 processes: 24 running, 98 sleeping
CPU: 93.2% user,  0.0% nice,  6.7% system,  0.1% interrupt,  0.0% idle
Mem: 1337M Active, 3297M Inact, 1037M Wired, 676M Buf, 2267M Free

  PID USERNAME    THR PRI NICE   SIZE    RES SWAP STATE    C   TIME    WCPU COMMAND
20678 root          1  92    0    91M    73M   0B RUN      2   0:01  21.62% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
24251 root          1  90    0    87M    69M   0B RUN      1   0:01  21.58% /usr/local/bin/php /usr/local/sbin/pluginctl -D
15528 root          1  94    0    93M    75M   0B CPU2     2   0:03  20.66% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
23693 root          1  89    0    85M    68M   0B RUN      0   0:00  18.54% /usr/local/bin/php /usr/local/sbin/pluginctl -S
25921 root          1  89    0    91M    67M   0B RUN      3   0:00  17.92% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
6394 root          1  95    0    99M    76M   0B RUN      2   0:03  17.69% /usr/local/bin/php /usr/local/sbin/pluginctl -S
21554 root          1  93    0    91M    73M   0B RUN      1   0:02  17.68% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
22190 root          1  91    0    89M    71M   0B RUN      0   0:01  17.56% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
14293 root          1  96    0    95M    77M   0B RUN      2   0:03  17.49% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
20585 root          1  92    0    89M    72M   0B RUN      0   0:01  17.29% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
19862 root          1  94    0    91M    74M   0B RUN      2   0:02  16.48% /usr/local/bin/php /usr/local/sbin/pluginctl -D
4668 root          1  96    0    97M    79M   0B RUN      3   0:04  16.23% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
13521 root          1  95    0    93M    76M   0B RUN      1   0:03  16.19% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
19250 root          1  94    0    91M    74M   0B RUN      3   0:02  16.08% /usr/local/bin/php /usr/local/sbin/pluginctl -S
6030 root          1  95    0    95M    78M   0B RUN      3   0:03  15.64% /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
6590 root          1  95    0    93M    76M   0B RUN      0   0:03  15.16% /usr/local/bin/php /usr/local/sbin/pluginctl -D
5413 root          1  95    0    95M    78M   0B CPU3     3   0:04  14.98% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
4926 root          1  68    0    97M    79M   0B piperd   2   0:04  13.94% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
15000 root          1  95    0    93M    76M   0B CPU0     0   0:03  13.78% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
24745 root          1  89    0    85M    67M   0B RUN      1   0:00  13.52% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
25533 root          1  88    0    83M    66M   0B RUN      0   0:00  12.44% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_status.php
26968 root          1  88    0    85M    66M   0B RUN      1   0:00  11.73% /usr/local/bin/php /usr/local/opnsense/scripts/interfaces/traffic_stats.php
20595 root          1  20    0   123M    69M   0B select   0   0:00   4.06% /usr/local/bin/php-cgi
7953 root          1  20    0    94M    73M   0B select   3   0:04   3.29% /usr/local/bin/php-cgi
17980 root          1  20    0    93M    68M   0B select   2   0:03   3.11% /usr/local/bin/php-cgi
68140 root         27  68    0   160M    89M   0B accept   2   1:10   2.92% /usr/local/bin/python3 /usr/local/opnsense/service/configd.py console (python3.11)
9559 root          1  20    0    93M    70M   0B select   2   0:03   2.70% /usr/local/bin/php-cgi
20051 root          1  20    0   107M    76M   0B select   2   0:04   2.19% /usr/local/bin/php-cgi
9757 root          1  20    0    93M    71M   0B accept   2   0:02   0.59% /usr/local/bin/php-cgi