flowd_aggregate.py very high cpu usage

[wizziLalev]

Hi all,
Just after updating to the last release I've notice that flowd_aggregate.py process start to use almost 100% of the CPU.

So I start to read different topics here and following is done:
1. All logs are removed and the machine is restarted
2. RRD is disabled
3. The process is being restarted

and still no luck - the cpu usage is still high.

-->screenshot<--

Any ideas are welcome!

Thanks!

[bartjsmit]

You can try Reporting, Netflow, clear all interfaces, then add them back in. You will lose your historic netflow information under Insight.

Bart...

[wizziLalev]

Hi Bart,
Thanks for advice, but the situation is the same, even after new restart.
Any other ideas? The same process is using ~100% of my cpu again.

Best,
Lalev

[wizziLalev]

Any other ideas? Anyone?

[wizziLalev]

up - the issue still persist and I really want to fix it - almost constantly my cpu is under full load and this slows the entire firewall and my network also.

I can provide more information if it is needed - but I need a clue for what to look...

[franco]

You have neither posted your hardware nor the traffic volume you are pushing... on the offchance that the system simply can't cope with reporting which isn't unusual because commercial reporting mostly comes as a separate appliance to not drain the processing power of the operational appliance.


Cheers,
Franco

[wizziLalev]

Hi @franco,

This is my hardware:

Athlon II x4 @3.00Ghz (I know it is old cpu but it still ticks)
8GB RAM DDR3 @1033Mhz
4x1Gbps interfaces
100Mbps internet connection
6 devices connected via switch (5 SBCs + 1AP)
3 wireless connected devices

Pure installation with very simple configuration, nothing major.

The traffic overall is not exceeding 40Mbps because most of the devices are in idle.

Just before latest update everything was perfect 1~2% CPU load, 500~700MB RAM usage and stable network.

I've already try to disconnect everything just to confirm that the connected devices are not causing the problem, but the result stay all the same.

Cheers,
Lalev

[franco]

Can you paste the output of the following commands?

# ls -lah /var/netflow/*
# ls -lah /var/log/flowd*
# df -h


Thanks,
Franco

[wizziLalev]

Hi franco,

Here it is:

Code Select Expand


root@opnsense:~ # ls -lah /var/netflow/*
-rw-r-----  1 root  wheel    96K Nov 18 05:01 /var/netflow/dst_port_000300.sqlite
-rw-r-----  1 root  wheel   684K Nov 18 05:01 /var/netflow/dst_port_003600.sqlite
-rw-r-----  1 root  wheel   1.9M Nov 18 05:01 /var/netflow/dst_port_086400.sqlite
-rw-r-----  1 root  wheel   2.5M Nov 18 05:01 /var/netflow/interface_000030.sqlite
-rw-r-----  1 root  wheel   1.1M Nov 18 05:01 /var/netflow/interface_000300.sqlite
-rw-r-----  1 root  wheel   116K Nov 18 05:01 /var/netflow/interface_003600.sqlite
-rw-r-----  1 root  wheel    16K Nov 18 05:01 /var/netflow/interface_086400.sqlite
-rw-r-----  1 root  wheel    12K Nov 10 20:57 /var/netflow/metadata.sqlite
-rw-r-----  1 root  wheel   160K Nov 18 05:01 /var/netflow/src_addr_000300.sqlite
-rw-r-----  1 root  wheel   364K Nov 18 05:01 /var/netflow/src_addr_003600.sqlite
-rw-r-----  1 root  wheel   384K Nov 18 05:01 /var/netflow/src_addr_086400.sqlite
-rw-r-----  1 root  wheel   7.8M Nov 18 05:01 /var/netflow/src_addr_details_086400.sqlite


Code Select Expand


root@opnsense:~ # ls -lah /var/log/flowd*
-rw-------  1 root  wheel   7.6M Nov 10 20:41 /var/log/flowd.log
-rw-------  1 root  wheel    11M Nov  9 03:26 /var/log/flowd.log.000001


Code Select Expand


root@opnsense:~ # df -h
Filesystem           Size    Used   Avail Capacity  Mounted on
/dev/ufs/OPNsense     27G    1.0G     24G     4%    /
devfs                1.0K    1.0K      0B   100%    /dev
devfs                1.0K    1.0K      0B   100%    /var/dhcpd/dev


[wizziLalev]

@franco, any clue?

[franco]

Unfortunately no. We have to apply more force, can you reinstall all packages and reboot?

# opnsense-update -fp


Cheers,
Franco