OPNsense Forum

Archive => 21.1 Legacy Series => Topic started by: rman50 on July 27, 2021, 03:51:27 pm

Title: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: rman50 on July 27, 2021, 03:51:27 pm
After upgrading to 21.1.9 from 21.1.8 a few hours ago, the Insight Aggregator python process has been pegging the core it is running on at 100%. I tried a reboot and it didn't help. Anyone else running into this issue or have a suggestion? I am running a Netflow data repair now to see if that helps and will try a Netflow reset after that.
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: opnsenseuser on July 27, 2021, 03:54:11 pm
same problem
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: Koldnitz on July 27, 2021, 03:58:20 pm
Reboot.

It fixed it for me … or so it seemed before I left for work

Cheers,
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: gpb on July 27, 2021, 04:00:55 pm
I thought I had that problem as well (had it in the past) when I saw high CPU usage, but running htop I noticed it was a different python process "/usr/local/service/configd_ctl.py".  Manually terminated it and all was fine.

Edit: Add screenshot
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: franco on July 27, 2021, 04:09:29 pm
Check dmesg. HBSD SEGVGUARD is likely blocking syslog-ng restart as some others have reported. Reboot is an option too.


Cheers,
Franco
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: rman50 on July 27, 2021, 04:11:15 pm
I had confirmed it was this process: /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.7). A Netflow reset seems to have resolved the issue for me at the moment.
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: franco on July 27, 2021, 04:36:03 pm
Alright, Python was updated from 3.7.10 to 3.7.11 but need to look into why that would cause a hang...


Cheers,
Franco
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: Taomyn on July 27, 2021, 06:32:25 pm
Add me to the list of victims, did my upgrade to 21.1.9 about 30min ago then suddenly I could a fan going mad that turned out to be my firewall and the CPU at around 94%, usually sits around 5-10%. Tried just reloading all services, it got worse, so just rebooted and will watch it again.


Let me know if there is any particular info you need from me to diagnose.


Update: Zabbix didn't catch the CPU but it did see a huge spike in RAM usage
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: KHE on July 27, 2021, 06:41:35 pm
Hi,

in my case it was the /usr/local/bin/python3 /usr/local/opnsense/service/configd_ctl.py -e -t 0.5 system event config_changed (python3.7) process. Killing it, solved the issue for now.

KH
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: hfvk on July 27, 2021, 06:43:34 pm
I have exactly the same problem on two systems I upgraded today. However, after a reboot both seem to be working fine for now.
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: bradrel on July 27, 2021, 06:43:59 pm
Hi,

in my case it was the /usr/local/bin/python3 /usr/local/opnsense/service/configd_ctl.py -e -t 0.5 system event config_changed (python3.7) process. Killing it, solved the issue for now.

KH

Same here.
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: opnsenseuser on July 27, 2021, 06:58:25 pm
I had confirmed it was this process: /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.7). A Netflow reset seems to have resolved the issue for me at the moment.

this also works for me. thx :)
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: allebone on July 27, 2021, 10:07:24 pm
I had to kill -9 the python process. Hopefully thats alright. Restarting/resetting netflow didnt help for me.
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: mr.yx on July 28, 2021, 01:55:13 am
+add me to the list...

syslog 1 core @100%
configd 1core @100%
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: s4rs on July 28, 2021, 02:27:11 am
I upgraded two systems and both had the same issue. Killing the offending python3.7 process fixed the issue on both systems.
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: abulafia on July 28, 2021, 12:41:24 pm
roughly similar experience here, too:
1. update -> 100% CPU on one core, some Python 3.7 process.
2. reboot solved the issue instantly, normal CPU usage since.
Title: Re: 21.1.9 Upgrade & Insight Aggregator CPU Usage
Post by: Raketenmeyer on July 28, 2021, 01:01:25 pm
Same here.