Messages

Hi,
I've put my hack in a PDF for other to use. Please see attached. It works for me and the CPU load is now totally gone.

Judging by this output, I don't think so:

root@OPNsense:~ # crontab -l|grep update_tables
*   *   *   *   *   (/usr/local/bin/flock -n -E 0 -o /tmp/filter_update_tables.lock /usr/local/opnsense/scripts/filter/update_tables.py) > /dev/null
root@OPNsense:~ #

Hi Franco,
Thanks for the patch. Great work (as always, and I mean that in a very positive way!!!).
Would it be possible to make the execution moments of the cron job customizable?

But I guess they now occur every half hour?

OK having my 'amount' of aliases is a bit too much. Would the behavior as I have implemented the workaround using monit to trigger the alias update when a config change occurs be possible when a modification is done in the aliases screen in the UI?

Hi Franco,

You are absolutely right. Please find the answers to your questions below:

It's Protectli:
# dmidecode | grep "Product Name"|uniq
   Product Name: VP2420
According to their website: Intel Celeron® J6412 Quad Core at 2 GHz (Burst up to 2.6 GHz)

There are about 161 aliases:
# grep "alias uuid" /conf/config.xml|wc -l
     161

In total all aliases sum up to about 5.5 million. The larger ones are based on IP adresses from AbuseIP, FireHOL and about 5 large GEOIP based alias lists.

If you want me to trace anything please let me know, I will be more than happy to assist.

Thanks for the patch. Processing time reduced from somewhere between 15/20 seconds to under 7 seconds:

root@OPNsense:/usr/local/opnsense/scripts/filter # time /usr/local/opnsense/scripts/filter/update_tables.py
{"status": "ok"}
6.810u 4.850s 0:12.08 96.5%   159+171k 0+2io 0pf+0w

As for the workaround. It depends on the presence of a temporary file called: /tmp/refreshaliases
This file is created by a custom script called: /opt/local/bin/refreshaliases.sh
Contents of the script is:

#!/bin/sh

if [ $(wc -c /usr/local/opnsense/scripts/filter/update_tables.py|awk '{print $1}') -gt 100 ]
then
   mv /usr/local/opnsense/scripts/filter/update_tables.py /opt/local/bin
   cp /opt/local/bin/update_tables.py_new /usr/local/opnsense/scripts/filter/update_tables.py
   /usr/local/bin/rsync -a --delete /usr/local/opnsense/scripts/filter/lib /opt/local/bin/
fi
if [ $(drill www.google.com|grep ^www.google.com|wc -l) -ne 0 ]
then
   /usr/local/bin/flock -n -E 0 -o /tmp/filter_update_tables.lock /opt/local/bin/update_tables.py > /dev/null
   touch /tmp/refreshaliases
fi

Furthermore a new python script was created that does nothing, called: /opt/local/bin/update_tables.py_new
Contents of the script is:

#!/usr/local/bin/python3

"""
    dummy
"""

Then a monit job was created that checks whether a config change has occurred and calls the /opt/local/bin/refreshaliases.sh script.
At boot the /opt/local/bin/refreshaliases.sh script must be run as well since the /tmp/refreshaliases file is not present at boot time.

Result: no more CPU spikes but aliases are refreshed at any config change. Hope this helps.

Hi,

This behavior is present for a long time now, please see this:

https://forum.opnsense.org/index.php?topic=31662.msg153060#msg153060

I ended up with a workaround because I could not find the root of the problem.

Hi Franco,

The patch seems to work from my end.

Thanks a lot.

Just to verify. Zenarmor is checking the wireguard interface?

I see similar behavior. I disabled WG for now.

I did a 'pkg upgrade' on the command line and that seemed to fix it.


Sent from my iPhone using Tapatalk

@gpb, if this happens again, could you execute this:

/usr/local/etc/rc.configure_interface wan

Maybe that will restore things?

Hi,
I see them too. Maybe they are caused by this cron job:

*   *   *   *   *   (/usr/local/bin/flock -n -E 0 -o /tmp/filter_update_tables.lock /usr/local/opnsense/scripts/filter/update_tables.py) > /dev/null

This job  runs every minute and causes my CPU (same as yours) to spike for about 10 seconds.

Hi,
I noticed a drop in CPU load as well. I have proof in a Zabbix graph, which I'm unable to upload I'm afraid.

Franco,
No negativity from my end. You guys are doing an excellent job. Better support than many enterprises deliver these days. So keep up the fantastic work and many thanks from a former pfSense user.