gui/cli login extermely slow due to PHP -- HELP

Started by cloudsense, July 22, 2024, 08:14:49 PM

Previous topic - Next topic
Print
Go Down Pages1
User actions
July 22, 2024, 08:14:49 PM
Hi,

Our server is crawling. GUI login takes ages and cli also takes ages to login.
I have disabled everything from the GUI except system info and services.

server is bare metal  ..  20 cores @3.1ghz  40threads
there is no disk errors.


top

last pid: 59277;  load averages: 45.60, 37.17, 24.50                                      up 0+02:15:08  20:09:29
131 processes: 47 running, 84 sleeping
CPU: 98.8% user,  0.0% nice,  1.2% system,  0.0% interrupt,  0.0% idle
Mem: 8950M Active, 3803M Inact, 1779M Wired, 40K Buf, 172G Free
ARC: 413M Total, 124M MFU, 248M MRU, 171K Anon, 2194K Header, 39M Other
     149M Compressed, 325M Uncompressed, 2.18:1 Ratio
Swap: 16G Total, 16G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
87555 root          1 102    0   250M   198M CPU21   21   3:13 100.03% php
86056 root          1  98    0   250M   197M CPU35   35   8:01  99.72% php
37729 root          1 102    0   248M   197M CPU14   14   5:13  99.67% php
89499 root          1 100    0   234M   181M CPU9     9   2:40  99.60% php
7784 root          1 102    0   226M   174M CPU16   16   1:54  99.59% php
2216 root          1 101    0   228M   176M CPU13   13   2:07  99.59% php
  463 root          1 101    0   238M   187M CPU25   25   2:14  99.59% php
26232 root          1 103    0   228M   176M CPU38   38   1:25  99.59% php
90693 root          1 101    0   230M   177M CPU18   18   2:12  99.59% php
27078 root          1 100    0   244M   193M CPU34   34   5:46  99.59% php
39785 root          1 101    0   240M   189M CPU4     4   5:03  99.58% php
78831 root          1 102    0   462M   377M CPU31   31   5:34  99.57% php-cgi
2151 root          1 101    0   252M   200M CPU23   23   7:34  99.57% php
41256 root          1 102    0   214M   161M CPU11   11   0:43  99.57% php
45343 root          1  91    0   236M   183M CPU20   20   4:53  99.57% php
77558 root          1 101    0   242M   189M CPU3     3   3:16  99.57% php
26374 root          1 102    0   238M   187M CPU33   33   6:07  99.56% php
23089 root          1 101    0   220M   169M CPU7     7   1:20  99.56% php
43177 root          1 100    0   246M   194M CPU2     2   5:12  99.56% php
60214 root          1 100    0   244M   192M CPU17   17   4:09  99.56% php
99979 root          1 102    0   228M   176M RUN     10   2:08  99.56% php
91377 root          1 100    0   238M   185M CPU26   26   2:05  99.55% php
88932 root          1 102    0   248M   195M CPU28   28   2:58  99.55% php
3427 root          1  99    0   244M   192M CPU37   37   7:25  99.52% php
40085 root          1  97    0   218M   167M CPU32   32   0:46  99.30% php
32528 root          1 101    0   248M   196M CPU6     6   5:22  99.30% php
77016 root          1 101    0   240M   187M CPU0     0   3:27  99.29% php
78055 root          1  97    0   236M   184M CPU24   24   3:09  87.85% php
40818 root          1  95    0   234M   182M RUN     30   5:02  79.92% php
43349 root          1  91    0   206M   154M CPU12   12   0:22  77.87% php
35170 root          1  95    0   218M   165M CPU29   29   0:44  77.26% php
41873 root          1 100    0   216M   164M CPU39   39   0:42  76.67% php
42431 root          1  97    0   214M   161M RUN     36   0:32  74.81% php
32896 root          1  94    0   240M   189M CPU8     8   5:11  74.25% php
69261 root          1  98    0   244M   193M RUN      5   3:56  73.62% php
76982 root          1 101    0   258M   205M RUN     27   3:32  73.00% php
79037 root          1  93    0   236M   185M CPU15   15   3:10  71.82% php
2075 root          1 101    0   260M   206M RUN      1   7:37  71.81% php
77777 root          1 100    0   311M   264M CPU19   19  20:52  70.88% php-cgi
88751 root          1  97    0   248M   196M CPU22   22   2:53  70.60% php
56147 root          1  89    0   204M   152M RUN     19   0:17  56.47% php
38002 root          1  93    0   216M   164M CPU5     5   0:53  51.29% php
55126 root          1  92    0   250M   199M CPU1     1   4:46  49.50% php
37552 root          1  94    0   246M   195M CPU27   27   5:01  48.89% php
70493 root          1  94    0   232M   180M CPU30   30   4:00  48.41% php
3551 root          1  94    0   244M   192M CPU36   36   7:03  47.68% php
  288 root         43  20    0   627M   353M accept  29   2:39   1.05% python3.11
59277 root          1  20    0    14M  3900K CPU10   10   0:00   0.22% top
35395 root          4  20    0   938M    19M select  21   0:02   0.13% bgpd

# ps ax | grep php
1122  -  Ss      0:00.19 php-fpm: master process (/usr/local/etc/php-fpm.conf) (php-fpm)
1620  -  I       0:00.00 php-fpm: pool webgui (php-fpm)
1748  -  I       0:00.00 php-fpm: pool webgui (php-fpm)
1817  -  I       0:00.00 php-fpm: pool www (php-fpm)
2166  -  I       0:00.00 php-fpm: pool www (php-fpm)
2216  -  R       4:42.36 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-openvpn 157@lagg0_vlan224 BACKUP
3427  -  R      10:04.91 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/50-frr 135@lagg0_vlan224 BACKUP
3551  -  R       9:34.47 /usr/local/bin/php /usr/local/sbin/pluginctl -S
7784  -  R       4:25.77 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
10447  -  Rs      1:14.12 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
20783  -  R       1:15.91 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-openvpn 241@lagg0_vlan224 BACKUP
21885  -  R       1:03.06 /usr/local/bin/php /usr/local/opnsense/scripts/Wireguard/wg-service-control.php -a conf
22572  -  R       0:59.95 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/50-frr 137@lagg0_vlan224 BACKUP
23038  -  R       0:54.80 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
23089  -  Rs      3:56.91 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
23495  -  R       1:00.05 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-ppp 151@lagg0_vlan224 BACKUP
24073  -  R       1:00.77 /usr/local/bin/php /usr/local/sbin/pluginctl -c crl
24759  -  R       0:42.07 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
26219  -  R       0:37.45 /usr/local/bin/php /usr/local/sbin/pluginctl -c crl
26232  -  R       4:04.92 /usr/local/bin/php /usr/local/sbin/pluginctl -S
26374  -  R       8:30.44 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-openvpn 153@lagg0_vlan224 BACKUP
26837  -  R       0:35.64 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-ppp 155@lagg0_vlan224 BACKUP
27078  -  R       8:16.72 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
28152  -  R       0:32.77 /usr/local/bin/php /usr/local/opnsense/scripts/Wireguard/wg-service-control.php -a conf
28583  -  R       0:26.58 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/50-frr 139@lagg0_vlan224 BACKUP
32140  -  Rs      0:27.05 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
32896  -  R       7:48.95 /usr/local/bin/php /usr/local/sbin/pluginctl -c crl
35170  -  R       3:04.46 /usr/local/bin/php /usr/local/sbin/pluginctl -S
37552  -  S       7:14.67 /usr/local/bin/php /usr/local/sbin/pluginctl -S
37729  -  S       6:49.08 /usr/local/bin/php /usr/local/sbin/pluginctl -S
38002  -  R       3:30.64 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
39785  -  Rs      7:43.50 /usr/local/bin/php /usr/local/etc/rc.expireaccounts
40085  -  R       3:21.19 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
40818  -  Rs      7:35.82 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
41256  -  R       3:13.19 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
41873  -  R       3:10.05 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
42431  -  R       3:01.50 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
43177  -  S       7:01.00 /usr/local/bin/php /usr/local/sbin/pluginctl -S
43349  -  Rs      2:50.71 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
43511  -  R       0:18.16 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
45343  -  R       7:23.18 /usr/local/bin/php /usr/local/sbin/pluginctl -S
55126  -  S       5:42.35 /usr/local/bin/php /usr/local/sbin/pluginctl -S
56147  -  R       2:49.63 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-openvpn 159@lagg0_vlan224 BACKUP
70493  -  R       6:31.91 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
75925  -  Is      0:00.02 /usr/local/bin/php-cgi
76048  -  R       2:10.76 /usr/local/bin/php /usr/local/sbin/pluginctl -S
76263  -  Is      0:00.02 /usr/local/bin/php-cgi
76763  -  Rs      2:02.20 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
76781  -  R      15:26.64 /usr/local/bin/php-cgi
77016  -  R       5:55.53 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
77052  -  S      17:01.47 /usr/local/bin/php-cgi
77558  -  R       5:57.04 /usr/local/bin/php /usr/local/sbin/pluginctl -S
77586  -  I       7:07.25 /usr/local/bin/php-cgi
77777  -  R      21:30.97 /usr/local/bin/php-cgi
78055  -  R       5:45.46 /usr/local/bin/php /usr/local/sbin/pluginctl -S
78274  -  I       2:53.81 /usr/local/bin/php-cgi
78831  -  R       8:06.96 /usr/local/bin/php-cgi
78999  -  R       2:04.85 /usr/local/bin/php /usr/local/sbin/pluginctl -S
79037  -  Rs      5:47.07 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
85633  -  R       1:35.71 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
86117  -  R       1:43.09 /usr/local/bin/php /usr/local/sbin/pluginctl -S
88751  -  S       3:57.07 /usr/local/bin/php /usr/local/sbin/pluginctl -S
88932  -  S       4:02.84 /usr/local/bin/php /usr/local/sbin/pluginctl -S
89499  -  R       5:09.13 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
90692  -  R       1:36.14 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
90693  -  Rs      4:48.34 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
90973  -  R       1:29.91 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
91216  -  R       1:26.67 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
99979  -  R       4:37.45 /usr/local/bin/php /usr/local/sbin/pluginctl -c crl

how to find out the root cause of whats causing this.


Thanks
July 22, 2024, 08:19:10 PM #1
What the...

Did you try to restart the GUI or reboot the FW?

Regards,
S.
Networking is love. You may hate it, but in the end, you always come back to it.

OPNSense HW
APU2D2 - deceased
N5105 - i226-V | Patriot 2x8G 3200 DDR4 | L 790 512G - VM HA(SOON)
N100   - i226-V | Crucial 16G  4800 DDR5 | S 980 500G - PROD
July 22, 2024, 08:41:26 PM #2
Hi,

reboot has no effect.. when it comes up the first time, during ssh this is shown ..

ssh
Last login: Mon Jul 22 18:32:55 2024 from x.x.x.x
----------------------------------------------
|      Hello, this is OPNsense 24.1          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website:   https://opnsense.org/        |         @@@\\\   ///@@@
| Handbook:   https://docs.opnsense.org/   |       ))))))))   ((((((((
| Forums:   https://forum.opnsense.org/  |         @@@///   \\\@@@
| Code:      https://github.com/opnsense  |        @@@@         @@@@
| Twitter:   https://twitter.com/opnsense |         @@@@@@@@@@@@@@@


then it takes at least 5 minutes minutes to finally login and show the menu options ..
ps aux after reboot just has the default nginx proces of php-fpm for www and webgui

when i load the browser, it shows the login screen, but as soon as i press the login , top/ps acts like posted above ..  dozens of processes are spawned



July 22, 2024, 08:50:12 PM #3
also what would be the reason to spawn dozens of /usr/local/sbin/pluginctl  .. can't it be done every x minute and data saved to some file which the next login picks.

also have found that /usr/local/opnsense/scripts/nginx/ngx_autoblock.php  starts another process if the first one is still running .. so sometimes there are 3-4 /usr/local/opnsense/scripts/nginx/ngx_autoblock.php processes that have not finished processing.

Print
Go Up Pages1
User actions