OPNsense Forum

Archive => 19.7 Legacy Series => Topic started by: tusc on August 29, 2019, 03:03:50 am

Title: Maltrail consuming a lot of CPU after 19.7.3
Post by: tusc on August 29, 2019, 03:03:50 am: I recently upgraded today and noticed maltrail's senor has 4 processes running and load is above 5. Is this normal?

Code: [Select]
@OPNsense:/usr/local/etc/rc.d # ps axu | grep maltrail root 21811 100.3 19.8 717304 704352 - S 19:56 4:22.30 python2.7 /usr/local/share/maltrail/sensor.py root 96015 34.4 19.8 713732 703972 - R 19:58 1:10.19 python2.7 /usr/local/share/maltrail/sensor.py root 2744 32.5 19.8 713852 704220 - S 19:58 1:11.61 python2.7 /usr/local/share/maltrail/sensor.py root 24134 32.1 19.8 713596 703792 - S 19:58 1:09.22 python2.7 /usr/local/share/maltrail/sensor.py root 55286 0.0 0.8 40668 26844 - S 19:26 0:02.11 python2.7 /usr/local/share/maltrail/server.py
Code: [Select]
root@OPNsense:/usr/local/etc/rc.d # top -bHS last pid: 78026; load averages: 4.36, 5.04, 5.42 up 0+01:31:43 20:02:23 210 processes: 12 running, 155 sleeping, 43 waiting Mem: 780M Active, 210M Inact, 538M Wired, 303M Buf, 1817M Free Swap: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 21811 root 86 0 700M 688M CPU0 0 2:01 47.46% python2.7{python2.7} 21811 root 86 0 700M 688M RUN 0 1:59 47.36% python2.7{python2.7} 96015 root 44 0 697M 687M select 2 1:26 33.40% python2.7{python2.7} 24134 root 44 0 697M 687M CPU2 2 1:25 32.67% python2.7{python2.7} 2744 root 45 0 697M 688M RUN 3 1:27 32.37% python2.7{python2.7} 11 root 155 ki31 0K 64K RUN 3 24:49 30.08% idle{idle: cpu3} 11 root 155 ki31 0K 64K RUN 2 25:12 29.49% idle{idle: cpu2} 11 root 155 ki31 0K 64K RUN 0 22:51 25.29% idle{idle: cpu0} 12 root -92 - 0K 720K CPU3 3 21:53 25.29% intr{irq265: igb1:que 3} 11 root 155 ki31 0K 64K RUN 1 19:21 20.56% idle{idle: cpu1} 12 root -92 - 0K 720K WAIT 2 16:50 18.80% intr{irq259: igb0:que 2} 12 root -92 - 0K 720K WAIT 0 3:28 2.39% intr{irq262: igb1:que 0} 12 root -92 - 0K 720K RUN 3 1:36 1.95% intr{irq260: igb0:que 3} 12 root -92 - 0K 720K WAIT 2 1:37 1.56% intr{irq264: igb1:que 2} 56999 root 22 0 35044K 22400K select 1 0:00 0.39% php-cgi
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: tusc on August 29, 2019, 04:07:47 am: Well, it looks like it has subsided. Not sure what caused it. Does it normally have that many processes active?

Code: [Select]
root@OPNsense:/usr/local/etc/rc.d # top -bHS last pid: 80886; load averages: 0.61, 0.67, 0.85 up 0+02:36:04 21:06:44 207 processes: 6 running, 156 sleeping, 45 waiting Mem: 486M Active, 501M Inact, 539M Wired, 303M Buf, 1820M Free Swap: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0K 64K CPU2 2 70:52 92.09% idle{idle: cpu2} 11 root 155 ki31 0K 64K CPU0 0 66:48 86.96% idle{idle: cpu0} 11 root 155 ki31 0K 64K RUN 3 70:55 84.47% idle{idle: cpu3} 11 root 155 ki31 0K 64K CPU1 1 62:52 84.28% idle{idle: cpu1} 24134 root 22 0 697M 688M select 0 11:00 7.28% python2.7{python2.7} 96015 root 21 0 697M 688M CPU1 1 11:00 5.76% python2.7{python2.7} 2744 root 21 0 698M 688M select 2 10:59 3.56% python2.7{python2.7} 21811 root 21 0 700M 688M bpf 2 9:46 1.86% python2.7{python2.7} 21811 root 21 0 700M 688M bpf 2 9:44 1.86% python2.7{python2.7} 81897 root 52 0 37096K 25328K accept 0 0:02 0.98% php-cgi
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: mimugmail on August 29, 2019, 06:01:30 am: Maybe downloaded all trails again
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: Northguy on November 26, 2019, 10:26:38 am: What does your /var/log/maltrail/error.log show? Mine is flooded " with Received unexpected datalink (186) ". Cannot find root cause for it, but this is definitely the rootcause for high CPU load on my device.
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: mimugmail on November 26, 2019, 03:06:53 pm: Does this appear after 0.15 -> 0.16 update?
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: Northguy on November 26, 2019, 10:27:15 pm: Not sure for my case, just noticed it. Meanwhile MT development committed a patch. @mimugmail, is there an advised way to apply the patch to the plugin? I don't think opnsense-patch can be applied to another repo?
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: mimugmail on November 27, 2019, 04:58:52 am: Which patch do you mean?
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: Northguy on November 27, 2019, 01:08:51 pm: Hi,

Please refer to

https://github.com/stamparm/maltrail/commit/f71c5c7d02d47341f009d3d0f1dbf0ccb63f86ea
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: juliocbc on January 03, 2020, 10:16:30 am: @mimugmail

Apparently changing from `PROCESS_COUNT` to `USE_MULTIPROCESSING` solved my problem with very high CPU usage.

Makes any sense to you?

ref.: https://github.com/cloudfence/plugins/commit/e878c035a465882d186e5c9181f827b5c21e177d
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: juliocbc on February 20, 2020, 06:02:52 pm: Tests with APU2e4 (quadcore / 4GB RAM) - OPNsense 19.7.8-amd64

Running with:

-100Mb + 60 Mb WANs
-4 VLANs
-1 Site to Site OpenVPN
-Plugins: Ngnix, Let's encrypt, Proxy (with Cloudfence's Webfilter), haproxy, FTPProxy;
-ARP Table: 51 entries;

Maltrail only with sensor running (We use a dedicated Maltrail Server here) with custom configuration:
Code: [Select]
PROCESS_COUNT 1 #USE_MULTIPROCESSING true DISABLE_CPU_AFFINITY false
CPU stats (every 3mins) - Load average (1 core): 94%
Memory: 15%
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: juliocbc on February 20, 2020, 06:08:15 pm: Maltrail server stats:
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: mimugmail on February 21, 2020, 01:26:38 pm: Quote from: juliocbc on February 20, 2020, 06:02:52 pm
Tests with APU2e4 (quadcore / 4GB RAM) - OPNsense 19.7.8-amd64

Running with:

-100Mb + 60 Mb WANs
-4 VLANs
-1 Site to Site OpenVPN
-Plugins: Ngnix, Let's encrypt, Proxy (with Cloudfence's Webfilter), haproxy, FTPProxy;
-ARP Table: 51 entries;

Maltrail only with sensor running (We use a dedicated Maltrail Server here) with custom configuration:
Code: [Select]
PROCESS_COUNT 1 #USE_MULTIPROCESSING true DISABLE_CPU_AFFINITY false
CPU stats (every 3mins) - Load average (1 core): 94%
Memory: 15%

Currently maltrail.conf uses fixed $CPU_CORES, shall I make this configurable?
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: juliocbc on February 24, 2020, 01:16:59 pm: Hi Michael!

I've edited jinja2 template to make some tests.
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: jds on July 07, 2020, 02:13:40 pm: This remedy to be fixed for awhile, but now I am back up to high CPU with maltrail.

8212   uwait   37.50%   python3 /usr/local/share/maltrail/sensor.py (python3.7){python3.7}
   49973   CPU3   36.38%   python3 /usr/local/share/maltrail/sensor.py (python3.7){python3.7}
   61436   CPU2   35.60%   /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf
   34952   select   34.77%   python3 /usr/local/share/maltrail/sensor.py (python3.7){python3.7}
   23576   select   33.50%   python3 /usr/local/share/maltrail/sensor.py (python3.7){python3.7}
   74367   select   23.78%   /usr/local/bin/suricata -D --netmap --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml{W#01-em3}

System is 4core APU, total memory use is only about 20%.

Forgot to mention that I am running 20.1.8
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: mimugmail on July 07, 2020, 05:20:25 pm: Usually this only happens when downloading new feeds after restart ... how long did you wait?
Title: Re: Maltrail consuming a lot of CPU after 19.7.3
Post by: jds on July 08, 2020, 04:40:23 pm: You are right, it did eventually settle down after at least 10 minutes. I had not rebooted, but it probably just downloaded new rules. Thanks.