Hi,
Our server is crawling. GUI login takes ages and cli also takes ages to login.
I have disabled everything from the GUI except system info and services.
server is bare metal .. 20 cores @3.1ghz 40threads
there is no disk errors.
top
last pid: 59277; load averages: 45.60, 37.17, 24.50 up 0+02:15:08 20:09:29
131 processes: 47 running, 84 sleeping
CPU: 98.8% user, 0.0% nice, 1.2% system, 0.0% interrupt, 0.0% idle
Mem: 8950M Active, 3803M Inact, 1779M Wired, 40K Buf, 172G Free
ARC: 413M Total, 124M MFU, 248M MRU, 171K Anon, 2194K Header, 39M Other
149M Compressed, 325M Uncompressed, 2.18:1 Ratio
Swap: 16G Total, 16G Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
87555 root 1 102 0 250M 198M CPU21 21 3:13 100.03% php
86056 root 1 98 0 250M 197M CPU35 35 8:01 99.72% php
37729 root 1 102 0 248M 197M CPU14 14 5:13 99.67% php
89499 root 1 100 0 234M 181M CPU9 9 2:40 99.60% php
7784 root 1 102 0 226M 174M CPU16 16 1:54 99.59% php
2216 root 1 101 0 228M 176M CPU13 13 2:07 99.59% php
463 root 1 101 0 238M 187M CPU25 25 2:14 99.59% php
26232 root 1 103 0 228M 176M CPU38 38 1:25 99.59% php
90693 root 1 101 0 230M 177M CPU18 18 2:12 99.59% php
27078 root 1 100 0 244M 193M CPU34 34 5:46 99.59% php
39785 root 1 101 0 240M 189M CPU4 4 5:03 99.58% php
78831 root 1 102 0 462M 377M CPU31 31 5:34 99.57% php-cgi
2151 root 1 101 0 252M 200M CPU23 23 7:34 99.57% php
41256 root 1 102 0 214M 161M CPU11 11 0:43 99.57% php
45343 root 1 91 0 236M 183M CPU20 20 4:53 99.57% php
77558 root 1 101 0 242M 189M CPU3 3 3:16 99.57% php
26374 root 1 102 0 238M 187M CPU33 33 6:07 99.56% php
23089 root 1 101 0 220M 169M CPU7 7 1:20 99.56% php
43177 root 1 100 0 246M 194M CPU2 2 5:12 99.56% php
60214 root 1 100 0 244M 192M CPU17 17 4:09 99.56% php
99979 root 1 102 0 228M 176M RUN 10 2:08 99.56% php
91377 root 1 100 0 238M 185M CPU26 26 2:05 99.55% php
88932 root 1 102 0 248M 195M CPU28 28 2:58 99.55% php
3427 root 1 99 0 244M 192M CPU37 37 7:25 99.52% php
40085 root 1 97 0 218M 167M CPU32 32 0:46 99.30% php
32528 root 1 101 0 248M 196M CPU6 6 5:22 99.30% php
77016 root 1 101 0 240M 187M CPU0 0 3:27 99.29% php
78055 root 1 97 0 236M 184M CPU24 24 3:09 87.85% php
40818 root 1 95 0 234M 182M RUN 30 5:02 79.92% php
43349 root 1 91 0 206M 154M CPU12 12 0:22 77.87% php
35170 root 1 95 0 218M 165M CPU29 29 0:44 77.26% php
41873 root 1 100 0 216M 164M CPU39 39 0:42 76.67% php
42431 root 1 97 0 214M 161M RUN 36 0:32 74.81% php
32896 root 1 94 0 240M 189M CPU8 8 5:11 74.25% php
69261 root 1 98 0 244M 193M RUN 5 3:56 73.62% php
76982 root 1 101 0 258M 205M RUN 27 3:32 73.00% php
79037 root 1 93 0 236M 185M CPU15 15 3:10 71.82% php
2075 root 1 101 0 260M 206M RUN 1 7:37 71.81% php
77777 root 1 100 0 311M 264M CPU19 19 20:52 70.88% php-cgi
88751 root 1 97 0 248M 196M CPU22 22 2:53 70.60% php
56147 root 1 89 0 204M 152M RUN 19 0:17 56.47% php
38002 root 1 93 0 216M 164M CPU5 5 0:53 51.29% php
55126 root 1 92 0 250M 199M CPU1 1 4:46 49.50% php
37552 root 1 94 0 246M 195M CPU27 27 5:01 48.89% php
70493 root 1 94 0 232M 180M CPU30 30 4:00 48.41% php
3551 root 1 94 0 244M 192M CPU36 36 7:03 47.68% php
288 root 43 20 0 627M 353M accept 29 2:39 1.05% python3.11
59277 root 1 20 0 14M 3900K CPU10 10 0:00 0.22% top
35395 root 4 20 0 938M 19M select 21 0:02 0.13% bgpd
# ps ax | grep php
1122 - Ss 0:00.19 php-fpm: master process (/usr/local/etc/php-fpm.conf) (php-fpm)
1620 - I 0:00.00 php-fpm: pool webgui (php-fpm)
1748 - I 0:00.00 php-fpm: pool webgui (php-fpm)
1817 - I 0:00.00 php-fpm: pool www (php-fpm)
2166 - I 0:00.00 php-fpm: pool www (php-fpm)
2216 - R 4:42.36 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-openvpn 157@lagg0_vlan224 BACKUP
3427 - R 10:04.91 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/50-frr 135@lagg0_vlan224 BACKUP
3551 - R 9:34.47 /usr/local/bin/php /usr/local/sbin/pluginctl -S
7784 - R 4:25.77 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
10447 - Rs 1:14.12 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
20783 - R 1:15.91 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-openvpn 241@lagg0_vlan224 BACKUP
21885 - R 1:03.06 /usr/local/bin/php /usr/local/opnsense/scripts/Wireguard/wg-service-control.php -a conf
22572 - R 0:59.95 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/50-frr 137@lagg0_vlan224 BACKUP
23038 - R 0:54.80 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
23089 - Rs 3:56.91 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
23495 - R 1:00.05 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-ppp 151@lagg0_vlan224 BACKUP
24073 - R 1:00.77 /usr/local/bin/php /usr/local/sbin/pluginctl -c crl
24759 - R 0:42.07 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
26219 - R 0:37.45 /usr/local/bin/php /usr/local/sbin/pluginctl -c crl
26232 - R 4:04.92 /usr/local/bin/php /usr/local/sbin/pluginctl -S
26374 - R 8:30.44 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-openvpn 153@lagg0_vlan224 BACKUP
26837 - R 0:35.64 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-ppp 155@lagg0_vlan224 BACKUP
27078 - R 8:16.72 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
28152 - R 0:32.77 /usr/local/bin/php /usr/local/opnsense/scripts/Wireguard/wg-service-control.php -a conf
28583 - R 0:26.58 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/50-frr 139@lagg0_vlan224 BACKUP
32140 - Rs 0:27.05 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
32896 - R 7:48.95 /usr/local/bin/php /usr/local/sbin/pluginctl -c crl
35170 - R 3:04.46 /usr/local/bin/php /usr/local/sbin/pluginctl -S
37552 - S 7:14.67 /usr/local/bin/php /usr/local/sbin/pluginctl -S
37729 - S 6:49.08 /usr/local/bin/php /usr/local/sbin/pluginctl -S
38002 - R 3:30.64 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
39785 - Rs 7:43.50 /usr/local/bin/php /usr/local/etc/rc.expireaccounts
40085 - R 3:21.19 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
40818 - Rs 7:35.82 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
41256 - R 3:13.19 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
41873 - R 3:10.05 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
42431 - R 3:01.50 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
43177 - S 7:01.00 /usr/local/bin/php /usr/local/sbin/pluginctl -S
43349 - Rs 2:50.71 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
43511 - R 0:18.16 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
45343 - R 7:23.18 /usr/local/bin/php /usr/local/sbin/pluginctl -S
55126 - S 5:42.35 /usr/local/bin/php /usr/local/sbin/pluginctl -S
56147 - R 2:49.63 /usr/local/bin/php /usr/local/etc/rc.syshook.d/carp/20-openvpn 159@lagg0_vlan224 BACKUP
70493 - R 6:31.91 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
75925 - Is 0:00.02 /usr/local/bin/php-cgi
76048 - R 2:10.76 /usr/local/bin/php /usr/local/sbin/pluginctl -S
76263 - Is 0:00.02 /usr/local/bin/php-cgi
76763 - Rs 2:02.20 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
76781 - R 15:26.64 /usr/local/bin/php-cgi
77016 - R 5:55.53 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
77052 - S 17:01.47 /usr/local/bin/php-cgi
77558 - R 5:57.04 /usr/local/bin/php /usr/local/sbin/pluginctl -S
77586 - I 7:07.25 /usr/local/bin/php-cgi
77777 - R 21:30.97 /usr/local/bin/php-cgi
78055 - R 5:45.46 /usr/local/bin/php /usr/local/sbin/pluginctl -S
78274 - I 2:53.81 /usr/local/bin/php-cgi
78831 - R 8:06.96 /usr/local/bin/php-cgi
78999 - R 2:04.85 /usr/local/bin/php /usr/local/sbin/pluginctl -S
79037 - Rs 5:47.07 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
85633 - R 1:35.71 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
86117 - R 1:43.09 /usr/local/bin/php /usr/local/sbin/pluginctl -S
88751 - S 3:57.07 /usr/local/bin/php /usr/local/sbin/pluginctl -S
88932 - S 4:02.84 /usr/local/bin/php /usr/local/sbin/pluginctl -S
89499 - R 5:09.13 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
90692 - R 1:36.14 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
90693 - Rs 4:48.34 /usr/local/bin/php /usr/local/opnsense/scripts/nginx/ngx_autoblock.php
90973 - R 1:29.91 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
91216 - R 1:26.67 /usr/local/bin/php /usr/local/opnsense/scripts/ipsec/get_legacy_vti.php
99979 - R 4:37.45 /usr/local/bin/php /usr/local/sbin/pluginctl -c crl
how to find out the root cause of whats causing this.
Thanks
What the...
Did you try to restart the GUI or reboot the FW?
Regards,
S.
Hi,
reboot has no effect.. when it comes up the first time, during ssh this is shown ..
ssh
Last login: Mon Jul 22 18:32:55 2024 from x.x.x.x
----------------------------------------------
| Hello, this is OPNsense 24.1 | @@@@@@@@@@@@@@@
| | @@@@ @@@@
| Website: https://opnsense.org/ | @@@\\\ ///@@@
| Handbook: https://docs.opnsense.org/ | )))))))) ((((((((
| Forums: https://forum.opnsense.org/ | @@@/// \\\@@@
| Code: https://github.com/opnsense | @@@@ @@@@
| Twitter: https://twitter.com/opnsense | @@@@@@@@@@@@@@@
then it takes at least 5 minutes minutes to finally login and show the menu options ..
ps aux after reboot just has the default nginx proces of php-fpm for www and webgui
when i load the browser, it shows the login screen, but as soon as i press the login , top/ps acts like posted above .. dozens of processes are spawned
also what would be the reason to spawn dozens of /usr/local/sbin/pluginctl .. can't it be done every x minute and data saved to some file which the next login picks.
also have found that /usr/local/opnsense/scripts/nginx/ngx_autoblock.php starts another process if the first one is still running .. so sometimes there are 3-4 /usr/local/opnsense/scripts/nginx/ngx_autoblock.php processes that have not finished processing.