21.7.3. - high CPU and MEM usage

[chemlud]

Hi!

Updated 2 OPNsense to 21.7.3 about an hour ago. No reboot. Had to start syslog_ng manually from the Dashboard afterwards.

Now I get frequent (preconfigured...) warning emails that CPU and MEM usage is very high on both machines with no major throughput.

In Diagnostics - Activity I have

Code: [Select]

OPNsense 1

47519 root 102 0 6352 5414 CPU1 1 69:35 98.78% /usr/local/bin/python3 /usr/local/opnsense/service/configd_ctl.py -e -t 0.5 system event config_changed (python3.8)
36687 root 52 0 27 3836 RUN 1 0:45 43.36% /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid{syslog-ng}

--------------------------

OPNsense 2

92636 root 83 0 32 8724 CPU0 0 1:06 35.60% /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid{syslog-ng}
69153 root 103 0 5091 4559 CPU3 3 68:37 100.00% /usr/local/bin/python3 /usr/local/opnsense/service/configd_ctl.py -e -t 0.5 system event config_changed (python3.8)
Anyone?

[chemlud]

... did a reboot on both machines, looks better now...

[scream]

I do see a similar/same issue.

Upgraded to 27.1.3 and can see very high cpu load. python3.8 consumes around 60-90% CPU all the time. I rebooted the VM but it didn't help.

I reverted to the VM snapshot I did before the upgrade an CPU load is back to normal.

[dinguz]

I had the same issue after upgrade: high cpu usage by a python process. Was gone after a reboot.

Code: [Select]

root    48330 99.4  0.3   33936   22620  -  R    22:51      2:07.71 /usr/local/bin/python3 /usr/local/opnsense/service/configd_ctl.
I have upgraded but not rebooted my other machine which show the same issue, can I do anything helpful in investigating this?

Edit: I see lots of these in the log:

Code: [Select]

2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:

syslog-ng has high cpu load now as well.

Edit 2:

On a sidenote: this shows the power of ZFS compression:

Code: [Select]

root@haanjdj:/var/log/system # zfs get compressratio
NAME                PROPERTY       VALUE  SOURCE
zroot               compressratio  8.33x  -
zroot/ROOT          compressratio  2.22x  -
zroot/ROOT/default  compressratio  2.22x  -
zroot/tmp           compressratio  1.66x  -
zroot/usr           compressratio  1.00x  -
zroot/usr/home      compressratio  1.00x  -
zroot/usr/ports     compressratio  1.00x  -
zroot/usr/src       compressratio  1.00x  -
zroot/var           compressratio  27.41x  -
zroot/var/audit     compressratio  1.00x  -
zroot/var/crash     compressratio  1.00x  -
zroot/var/log       compressratio  27.42x  -
zroot/var/mail      compressratio  1.00x  -
zroot/var/tmp       compressratio  1.00x  -


[almodovaris]

True, python was close to 100% CPU (at one core). Rebooted and now is OK.

[Dantichrist]

I'm seeing the same issue. I rebooted and the CPU usage has dropped but it is still a good bit higher than what it is normally. The memory usage seems normal.

[33b]

I had the same issue after upgrade: high cpu usage by a python process. Was gone after a reboot.

Code: [Select]

root    48330 99.4  0.3   33936   22620  -  R    22:51      2:07.71 /usr/local/bin/python3 /usr/local/opnsense/service/configd_ctl.
I have upgraded but not rebooted my other machine which show the same issue, can I do anything helpful in investigating this?

Edit: I see lots of these in the log:

Code: [Select]

2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
2021-09-22T20:28:12 configctl[48330] event @ 1632335220.47 msg:
syslog-ng has high cpu load now as well.

If you don't reboot, those will fill up your logs pretty quickly, to the tune of over 40 GB per day of logs in my case.

https://forum.opnsense.org/index.php?topic=24148.0

Franco commented there (and also on Reddit this morning) that "Something in the combination of Python and Syslog-ng update of 21.1.9" started causing this on occasion and that they're looking into it.

[MidGe]

I had the same issue (cpu 100%) after the latest automatic update. A reboot fixed the problem.

[allebone]

Same for me. 1 core pegged but a reboot fixed it totally for me.

[pavulinux]

Reboot did not fix the issue in my case. Post upgrade to 21.7.3 Suricata consumes constantly 50%CPU (used to be 5-10% prior to upgrade). Other processes seem to be fine. I've rolled back and the CPU utilisation went back to normal.

[franco]

Reboot did not fix the issue in my case. Post upgrade to 21.7.3 Suricata consumes constantly 50%CPU (used to be 5-10% prior to upgrade). Other processes seem to be fine. I've rolled back and the CPU utilisation went back to normal.

Err, Suricata was updated from version 5 to 6. This might be unavoidable unless you want to roll back forever.


Cheers,
Franco

[tong2x]

confirmed high CPU usage
for me it was syslogNG, dont have suricata installed

reboot helped, cpu usage is not high anymore

though memory usage is abit high

[opnsenseuser]

after reboot cpu and mem is normal again

[yeraycito]

I have been formatting opnsense for a while now, every time major versions are released (summer - January). With 21.7 I did it as well. I say this because I just upgraded to 21.7.3 and have not had any problems with high cpu and memory consumption without having to reboot to get it fixed. The same has happened with other problems that have been reported since opnsense 21.7 was released. Everything has worked fine. So I recommend when major versions are released to format Opnsense.

[chemlud]

... I do a fresh install when the underlying OS get's a major update, as expected for 22.1, but not every 6 months. That would be overkill imho...