gui/cli login extermely slow due to PHP -- HELP

[cloudsense]

Hi,

Our server is crawling. GUI login takes ages and cli also takes ages to login.
I have disabled everything from the GUI except system info and services.

server is bare metal  ..  20 cores @3.1ghz  40threads
there is no disk errors.


top

last pid: 59277;  load averages: 45.60, 37.17, 24.50                                      up 0+02:15:08  20:09:29
131 processes: 47 running, 84 sleeping
CPU: 98.8% user,  0.0% nice,  1.2% system,  0.0% interrupt,  0.0% idle
Mem: 8950M Active, 3803M Inact, 1779M Wired, 40K Buf, 172G Free
ARC: 413M Total, 124M MFU, 248M MRU, 171K Anon, 2194K Header, 39M Other
     149M Compressed, 325M Uncompressed, 2.18:1 Ratio
Swap: 16G Total, 16G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
87555 root          1 102    0   250M   198M CPU21   21   3:13 100.03% php
86056 root          1  98    0   250M   197M CPU35   35   8:01  99.72% php
37729 root          1 102    0   248M   197M CPU14   14   5:13  99.67% php
89499 root          1 100    0   234M   181M CPU9     9   2:40  99.60% php
 7784 root          1 102    0   226M   174M CPU16   16   1:54  99.59% php
 2216 root          1 101    0   228M   176M CPU13   13   2:07  99.59% php
  463 root          1 101    0   238M   187M CPU25   25   2:14  99.59% php
26232 root          1 103    0   228M   176M CPU38   38   1:25  99.59% php
90693 root          1 101    0   230M   177M CPU18   18   2:12  99.59% php
27078 root          1 100    0   244M   193M CPU34   34   5:46  99.59% php
39785 root          1 101    0   240M   189M CPU4     4   5:03  99.58% php
78831 root          1 102    0   462M   377M CPU31   31   5:34  99.57% php-cgi
 2151 root          1 101    0   252M   200M CPU23   23   7:34  99.57% php
41256 root          1 102    0   214M   161M CPU11   11   0:43  99.57% php
45343 root          1  91    0   236M   183M CPU20   20   4:53  99.57% php
77558 root          1 101    0   242M   189M CPU3     3   3:16  99.57% php
26374 root          1 102    0   238M   187M CPU33   33   6:07  99.56% php
23089 root          1 101    0   220M   169M CPU7     7   1:20  99.56% php
43177 root          1 100    0   246M   194M CPU2     2   5:12  99.56% php
60214 root          1 100    0   244M   192M CPU17   17   4:09  99.56% php
99979 root          1 102    0   228M   176M RUN     10   2:08  99.56% php
91377 root          1 100    0   238M   185M CPU26   26   2:05  99.55% php
88932 root          1 102    0   248M   195M CPU28   28   2:58  99.55% php
 3427 root          1  99    0   244M   192M CPU37   37   7:25  99.52% php
40085 root          1  97    0   218M   167M CPU32   32   0:46  99.30% php
32528 root          1 101    0   248M   196M CPU6     6   5:22  99.30% php
77016 root          1 101    0   240M   187M CPU0     0   3:27  99.29% php
78055 root          1  97    0   236M   184M CPU24   24   3:09  87.85% php
40818 root          1  95    0   234M   182M RUN     30   5:02  79.92% php
43349 root          1  91    0   206M   154M CPU12   12   0:22  77.87% php
35170 root          1  95    0   218M   165M CPU29   29   0:44  77.26% php
41873 root          1 100    0   216M   164M CPU39   39   0:42  76.67% php
42431 root          1  97    0   214M   161M RUN     36   0:32  74.81% php
32896 root          1  94    0   240M   189M CPU8     8   5:11  74.25% php
69261 root          1  98    0   244M   193M RUN      5   3:56  73.62% php
76982 root          1 101    0   258M   205M RUN     27   3:32  73.00% php
79037 root          1  93    0   236M   185M CPU15   15   3:10  71.82% php
 2075 root          1 101    0   260M   206M RUN      1   7:37  71.81% php
77777 root          1 100    0   311M   264M CPU19   19  20:52  70.88% php-cgi
88751 root          1  97    0   248M   196M CPU22   22   2:53  70.60% php
56147 root          1  89    0   204M   152M RUN     19   0:17  56.47% php
38002 root          1  93    0   216M   164M CPU5     5   0:53  51.29% php
55126 root          1  92    0   250M   199M CPU1     1   4:46  49.50% php
37552 root          1  94    0   246M   195M CPU27   27   5:01  48.89% php
70493 root          1  94    0   232M   180M CPU30   30   4:00  48.41% php
 3551 root          1  94    0   244M   192M CPU36   36   7:03  47.68% php
  288 root         43  20    0   627M   353M accept  29   2:39   1.05% python3.11
59277 root          1  20    0    14M  3900K CPU10   10   0:00   0.22% top
35395 root          4  20    0   938M    19M select  21   0:02   0.13% bgpd

# ps ax | grep php
 1122  -  Ss      0:00.19 php-fpm: master process (/usr/local/etc/php-fpm.conf) (php-fpm)
 1620  -  I       0:00.00 php-fpm: pool webgui (php-fpm)
 1748  -  I       0:00.00 php-fpm: pool webgui (php-fpm)
 1817  -  I       0:00.00 php-fpm: pool www (php-fpm)
 2166  -  I       0:00.00 php-fpm: pool www (php-fpm)
 2216  -  R       4:42.36 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-openvpn 157@lagg0_vlan224 BACKUP
 3427  -  R      10:04.91 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/50-frr 135@lagg0_vlan224 BACKUP
 3551  -  R       9:34.47 /usr/local/bin/php /usr/local/sbin/pluginctl -S
 7784  -  R       4:25.77 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
10447  -  Rs      1:14.12 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
20783  -  R       1:15.91 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-openvpn 241@lagg0_vlan224 BACKUP
21885  -  R       1:03.06 /usr/local/bin/php /usr/local/opnsense/scripts/Wireguard/wg-service-control.php -a conf
22572  -  R       0:59.95 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/50-frr 137@lagg0_vlan224 BACKUP
23038  -  R       0:54.80 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
23089  -  Rs      3:56.91 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
23495  -  R       1:00.05 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-ppp 151@lagg0_vlan224 BACKUP
24073  -  R       1:00.77 /usr/local/bin/php /usr/local/sbin/pluginctl -c crl
24759  -  R       0:42.07 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
26219  -  R       0:37.45 /usr/local/bin/php /usr/local/sbin/pluginctl -c crl
26232  -  R       4:04.92 /usr/local/bin/php /usr/local/sbin/pluginctl -S
26374  -  R       8:30.44 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-openvpn 153@lagg0_vlan224 BACKUP
26837  -  R       0:35.64 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-ppp 155@lagg0_vlan224 BACKUP
27078  -  R       8:16.72 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
28152  -  R       0:32.77 /usr/local/bin/php /usr/local/opnsense/scripts/Wireguard/wg-service-control.php -a conf
28583  -  R       0:26.58 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/50-frr 139@lagg0_vlan224 BACKUP
32140  -  Rs      0:27.05 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
32896  -  R       7:48.95 /usr/local/bin/php /usr/local/sbin/pluginctl -c crl
35170  -  R       3:04.46 /usr/local/bin/php /usr/local/sbin/pluginctl -S
37552  -  S       7:14.67 /usr/local/bin/php /usr/local/sbin/pluginctl -S
37729  -  S       6:49.08 /usr/local/bin/php /usr/local/sbin/pluginctl -S
38002  -  R       3:30.64 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
39785  -  Rs      7:43.50 /usr/local/bin/php /usr/local/etc/rc.expireaccounts
40085  -  R       3:21.19 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
40818  -  Rs      7:35.82 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
41256  -  R       3:13.19 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
41873  -  R       3:10.05 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
42431  -  R       3:01.50 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
43177  -  S       7:01.00 /usr/local/bin/php /usr/local/sbin/pluginctl -S
43349  -  Rs      2:50.71 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
43511  -  R       0:18.16 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
45343  -  R       7:23.18 /usr/local/bin/php /usr/local/sbin/pluginctl -S
55126  -  S       5:42.35 /usr/local/bin/php /usr/local/sbin/pluginctl -S
56147  -  R       2:49.63 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-openvpn 159@lagg0_vlan224 BACKUP
70493  -  R       6:31.91 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
75925  -  Is      0:00.02 /usr/local/bin/php-cgi
76048  -  R       2:10.76 /usr/local/bin/php /usr/local/sbin/pluginctl -S
76263  -  Is      0:00.02 /usr/local/bin/php-cgi
76763  -  Rs      2:02.20 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
76781  -  R      15:26.64 /usr/local/bin/php-cgi
77016  -  R       5:55.53 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
77052  -  S      17:01.47 /usr/local/bin/php-cgi
77558  -  R       5:57.04 /usr/local/bin/php /usr/local/sbin/pluginctl -S
77586  -  I       7:07.25 /usr/local/bin/php-cgi
77777  -  R      21:30.97 /usr/local/bin/php-cgi
78055  -  R       5:45.46 /usr/local/bin/php /usr/local/sbin/pluginctl -S
78274  -  I       2:53.81 /usr/local/bin/php-cgi
78831  -  R       8:06.96 /usr/local/bin/php-cgi
78999  -  R       2:04.85 /usr/local/bin/php /usr/local/sbin/pluginctl -S
79037  -  Rs      5:47.07 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
85633  -  R       1:35.71 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
86117  -  R       1:43.09 /usr/local/bin/php /usr/local/sbin/pluginctl -S
88751  -  S       3:57.07 /usr/local/bin/php /usr/local/sbin/pluginctl -S
88932  -  S       4:02.84 /usr/local/bin/php /usr/local/sbin/pluginctl -S
89499  -  R       5:09.13 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
90692  -  R       1:36.14 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
90693  -  Rs      4:48.34 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
90973  -  R       1:29.91 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
91216  -  R       1:26.67 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
99979  -  R       4:37.45 /usr/local/bin/php /usr/local/sbin/pluginctl -c crl

how to find out the root cause of whats causing this.


Thanks

[Seimus]

What the...

Did you try to restart the GUI or reboot the FW?

Regards,
S.

[cloudsense]

Hi,

reboot has no effect.. when it comes up the first time, during ssh this is shown ..

ssh
Last login: Mon Jul 22 18:32:55 2024 from x.x.x.x
----------------------------------------------
|      Hello, this is OPNsense 24.1          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website:   https://opnsense.org/        |         @@@\\\   ///@@@
| Handbook:   https://docs.opnsense.org/   |       ))))))))   ((((((((
| Forums:   https://forum.opnsense.org/  |         @@@///   \\\@@@
| Code:      https://github.com/opnsense  |        @@@@         @@@@
| Twitter:   https://twitter.com/opnsense |         @@@@@@@@@@@@@@@


then it takes at least 5 minutes minutes to finally login and show the menu options ..
ps aux after reboot just has the default nginx proces of php-fpm for www and webgui

when i load the browser, it shows the login screen, but as soon as i press the login , top/ps acts like posted above ..  dozens of processes are spawned

[cloudsense]

also what would be the reason to spawn dozens of /usr/local/sbin/pluginctl  .. can't it be done every x minute and data saved to some file which the next login picks.

also have found that /usr/local/opnsense/scripts/nginx/ngx_autoblock.php  starts another process if the first one is still running .. so sometimes there are 3-4 /usr/local/opnsense/scripts/nginx/ngx_autoblock.php processes that have not finished processing.