SOLVED python eating CPU and RAM since 21.7.3_3 upgrade

[clarknova]

Yesterday I upgraded a CARP pair of firewalls from OPNsense 21.7.2 to 21.7.3_3 and was not prompted to reboot them. Today I notice they are both using all the RAM and swap. top indicates it is python3.8 using most of the memory. ps indicates a couple of python commands are also keeping CPU core busy each. I've attached the memory graph from the CARP master, but the graph from the backup looks the same. I imagine a reboot might fix it but since they still appear to be operating ok I thought it might be interesting to troubleshoot first.

Code Select Expand

root@rtr06b:~ # ps aux | grep python
root    81798  100.0 34.5 16186396 8638488  -  R    14Sep21   1274:24.32 /usr/local/bin/python3 /usr/local/opnsense/service/configd_ctl.py -e -t 0.5 system event config_changed (python3.8)
root    82678  100.0 19.2 15660512 4801392  -  R    14Sep21   1274:51.48 /usr/local/bin/python3 /usr/local/opnsense/service/configd_ctl.py -e -t 0.5 interface update carp service_status (python3.8)

[AdSchellevis]

Hi,

It's likely this https://github.com/opnsense/core/commit/b9de69fe440bbb144022658d439a4f34735ab338 (planned for 21.7.4), you should be able to safely kill the offensive python pid's (81798, 82678).

This sometimes happened when syslog-ng disconnected the configctl helper.

Best regards,

Ad

[clarknova]

you should be able to safely kill the offensive python pid's (81798, 82678).

That fixed it. Thanks.

[franco]

21.7.3_3 actually includes this fix, but can't change the fact that the bad code is being executed prior to update which can trigger the situation.

https://github.com/opnsense/changelog/blob/e07bf7d4a6ae62cc3e73d8c0720e805157430d78/community/21.7/21.7.3#L54

I would just raise awareness here as using opnsense-patch on 21.7.3_3 will REVERT the patch.


Cheers,
Franco