Hi,
out of my >50 opnsense boxes there is one that regularly runs out of memory over the course of around two weeks, to the point where they have to powercycle it on site.
Yesterday, I got a chance to look after it before it was to late.
top shows me its a python process:
last pid: 22346; load averages: 3.76, 2.84, 1.89 up 8+10:04:59 17:52:52
53 processes: 4 running, 49 sleeping
CPU: 30.1% user, 0.0% nice, 12.7% system, 0.2% interrupt, 56.9% idle
Mem: 5423M Active, 62M Inact, 776M Laundry, 1226M Wired, 697M Buf, 220M Free
Swap: 8192M Total, 3754M Used, 4438M Free, 45% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
65714 root 1 103 0 5667M 5136M CPU7 7 100:34 97.62% python3.7
38963 root 17 41 0 1122M 14M sigwai 7 0:02 0.00% charon
84282 root 1 20 0 1050M 4360K select 3 0:27 0.00% sshd
and here are the python processes:
root@lesc:~ # ps aux|grep python
root 65714 99.0 64.4 5803116 5258816 - R 9Dec20 99:41.55 /usr/local/bin/python3 /usr/local/opnsense/service/configd_ctl.py -e -t 0.5 system event config_chang
root 7298 0.0 0.1 20540 10716 - S 17:47 0:00.06 /usr/local/bin/python3 /usr/local/opnsense/service/configd_ctl.py -e -t 0.5 system event config_chang
root 13583 0.0 0.3 58004 21716 - I 9Dec20 0:12.39 /usr/local/bin/python3 /usr/local/opnsense/service/configd.py console (python3.7)
root 20151 0.0 0.0 31876 0 - IWs - 0:00.00 /usr/local/bin/python3 /usr/local/opnsense/service/configd.py (python3.7)
root 27623 0.0 0.1 21076 11228 - S 17:47 0:00.07 /usr/local/bin/python3 /usr/local/opnsense/scripts/syslog/lockout_handler (python3.7)
root 93887 0.0 0.2 37548 16812 - Ss 9Dec20 825:04.49 /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.7)
root 65791 0.0 0.0 1067288 2932 0 S+ 17:51 0:00.00 grep python
So, I'm wondering what "configd_ctl.py -e -t 0.5 system event config_chang" does, why there's two of them, and how I can find out what's going wrong. At the time I got to take a look, the systemlog was full off 'cannot allocate' entries.
The system has 8GB Ram + 8GB Swap. No suricata, no flowd. It is running 20.7.6 but it has shown this with earlier 20.7.x releases.
Thanks for any hints,
I share the same problem on my small APU1D. /usr/local/opnsense/service/configd_ctl.py Eating up 65-100% cpu flat out.
Type opnsense
Version 21.1.9_1
Architecture amd64
Flavour OpenSSL
Commit e10896164
Mirror http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:12:amd64/21.1
Repositories OPNsense
I need to reboot it too. The only thing I can think of might have triggered it: I updated the DNS list in System > Settings > General a few days ago. It might've since then been acting up?
Any pointers?
Cheers
M