Web UI and PHP performance issue - config.inc

[oz_djh]

Hi

As per a previous thread I posted (https://forum.opnsense.org/index.php?topic=43858) we've been having major UI performance issues since 24.x.  CPU sitting at 100% anytime anyone is on the dashboard page.  We disabled dashboard widgets as a work-around.  After upgrading to 25.1.10 today and still experiencing the problem I thought I'd look further to see if I can at least narrow down the issue.

The problem appears to be related to the config loading functions in  /usr/local/etc/inc/config.inc.  To demonstrate I have this simple snippet from 'pluginctl' that just lists the network interfaces.  I see numerous instances of pluginctl using 100% CPU in top when we experience the performance problem.

#!/usr/local/bin/php
<?php

require_once 'config.inc';
require_once 'interfaces.inc';

echo json_encode(legacy_interfaces_details($args[0] ?? null), $jflags);

If I comment out the require_once of config.inc everything is fine.  But if it loads config.inc the script takes about 7 seconds to complete and eats 100% cpu during that time.  It looks like many PHP scripts are showing the same problem which basically brings the firewall to it's knees when anyone is on the Web UI.  Has anyone seen this before?  Any pointers of what may be going on?  This firewall has been in production for many years and just upgraded in place through lots of different releases.

# time /tmp/interfaces > /dev/null
        0.04 real         0.02 user         0.02 sys
# time /tmp/interfaces-broken > /dev/null
        6.81 real         6.71 user         0.09 sys
# diff /tmp/interfaces /tmp/interfaces-broken
3a4
> require_once 'config.inc';
#

[oz_djh]

Further to the above, it looks like calling the toArray() method on the config in prase_config() from config.inc is causing the problem.

function parse_config()
{
    $cnf = OPNsense\Core\Config::getInstance();
    /* make sure to write back global variable */
printf("DBG 1 : %s\n",date('h:i:s a', time()));
    $config = $cnf->toArray(listtags());
printf("DBG 2 : %s\n",date('h:i:s a', time()));
    return $config;
}

# /tmp/interfaces-broken
DBG 1 : 03:01:05 pm
DBG 2 : 03:01:12 pm
{"vtnet0":{"flags":["up","broadcast","running","simplex","multicast","lower_up"],"capabilities":["vlan_mtu","jumbo_mtu

[AdSchellevis]

Hi,

Either the machine is not very capable or your config.xml file is very large (or a combination), although all new code doesn't need to serialize the file to an array (as legacy code is doing), some of the actions still depend on older code.

When there are one or two hotspots and you have identified them, it might make sense opening an issue (https://github.com/opnsense/core/issues).

In some cases rewriting the parts to direct usage of the Config instance isn't a lot of work.

Best regards,

Ad

[oz_djh]

Hi Ad,

Thanks for the reply.  The VM is quite capable and the config is reasonably simple so I assumed there was an issue with the config parser.  I downloaded the config to have a look at the XML and it was 80,000 lines long!  Almost all of it was BAN directives in the nginx config.  That's not something we configured so it looks like the default setting is to ban IPs (on some criteria I'm not aware of) and to never release that ban.  So by default, the list of banned IP will grow and grow until it produces we result we've been seeing.

I found the 'Autoblock TTL' setting in the advanced nginix config, which was set to 0.  I set that to 1 day, saved the config, and the issue was resolved.  The config went from 80,000 lines to 3,000 lines and the CPU problem was immediately fixed.  Perhaps the default settings need to be tweaked to avoid this situation.