flowd_aggregate process burning high CPU in OPNsense 24.1.6-amd64

[tonys]

Ive been running OPNsense 24.1.6-amd64 since it's release. Last night, I started having loss of all networks after noticing that the CPU load was running near 100% and the Protectli VP 2420 was very hot. I couldn't access the router from any interface and ended up forcing the VP2420 to shutdown manually. After a few hours of cooling off, the system came up fine today but then I noticed high CPU usage being caused by the flowd_aggregate process. The box started heating up again and network access was very slow. Top showed a python process running at nearly 100% and I was able to isolate it to the flowd_aggregate process. I've now turned off Insight Aggregator in the GUI and will leave it off for a day or two to determine if it's the only culprit. I've seen numerous posts with similar complaints on older releases of OPNSense.

Is anyone familiar with this issue? All networks stall out completely on all interfaces and the entire VP2420 becomes inaccessible. In day to day use, I rarely see CPU usage exceed 20% and most of the time all 4 cores are essentially idle. See the attached Monitor log screenshot.

[bestboy]

I have a FW2B with a weaker dual core CPU and I also see periodic spikes of flowd_aggregate,py. However, here it's just using one of the cores for about 3 seconds, not constantly.
Maybe resetting or repairing the newflow database helps to fix your aggregation issue. Check Reporting > Settings > Reporting Database Options > Repair Netflow Data (resp. Reset Netflow Data)

[tonys]

Resetting the flowd database has indeed helped. Later today though, I noticed 238% CPU usage by the suricata process which I ended up killing manually in the console because the GUI again became unresponsive. Following this, I rebooted the router and it’s back to normal for now.

Thanks for pointing out the flowd reset/repair procedure - I hadn’t even looked at that page before.