Messages

I had the very exact same issue. My solution was to delete the contents of /var/db/tailscale and redo the login process and verify that the settings in the GUI where still OK (which was not the case for me).
Then I installed 25.1.9 and did a reboot (to be sure) and Tailscale is up and running without any manual intervention.

Hi,
I've put my hack in a PDF for other to use. Please see attached. It works for me and the CPU load is now totally gone.

Judging by this output, I don't think so:

root@OPNsense:~ # crontab -l|grep update_tables
*   *   *   *   *   (/usr/local/bin/flock -n -E 0 -o /tmp/filter_update_tables.lock /usr/local/opnsense/scripts/filter/update_tables.py) > /dev/null
root@OPNsense:~ #

Hi Franco,
Thanks for the patch. Great work (as always, and I mean that in a very positive way!!!).
Would it be possible to make the execution moments of the cron job customizable?

But I guess they now occur every half hour?

OK having my 'amount' of aliases is a bit too much. Would the behavior as I have implemented the workaround using monit to trigger the alias update when a config change occurs be possible when a modification is done in the aliases screen in the UI?

Hi Franco,

You are absolutely right. Please find the answers to your questions below:

It's Protectli:
# dmidecode | grep "Product Name"|uniq
   Product Name: VP2420
According to their website: Intel Celeron® J6412 Quad Core at 2 GHz (Burst up to 2.6 GHz)

There are about 161 aliases:
# grep "alias uuid" /conf/config.xml|wc -l
     161

In total all aliases sum up to about 5.5 million. The larger ones are based on IP adresses from AbuseIP, FireHOL and about 5 large GEOIP based alias lists.

If you want me to trace anything please let me know, I will be more than happy to assist.

Thanks for the patch. Processing time reduced from somewhere between 15/20 seconds to under 7 seconds:

root@OPNsense:/usr/local/opnsense/scripts/filter # time /usr/local/opnsense/scripts/filter/update_tables.py
{"status": "ok"}
6.810u 4.850s 0:12.08 96.5%   159+171k 0+2io 0pf+0w

As for the workaround. It depends on the presence of a temporary file called: /tmp/refreshaliases
This file is created by a custom script called: /opt/local/bin/refreshaliases.sh
Contents of the script is:

#!/bin/sh

if [ $(wc -c /usr/local/opnsense/scripts/filter/update_tables.py|awk '{print $1}') -gt 100 ]
then
   mv /usr/local/opnsense/scripts/filter/update_tables.py /opt/local/bin
   cp /opt/local/bin/update_tables.py_new /usr/local/opnsense/scripts/filter/update_tables.py
   /usr/local/bin/rsync -a --delete /usr/local/opnsense/scripts/filter/lib /opt/local/bin/
fi
if [ $(drill www.google.com|grep ^www.google.com|wc -l) -ne 0 ]
then
   /usr/local/bin/flock -n -E 0 -o /tmp/filter_update_tables.lock /opt/local/bin/update_tables.py > /dev/null
   touch /tmp/refreshaliases
fi

Furthermore a new python script was created that does nothing, called: /opt/local/bin/update_tables.py_new
Contents of the script is:

#!/usr/local/bin/python3

"""
    dummy
"""

Then a monit job was created that checks whether a config change has occurred and calls the /opt/local/bin/refreshaliases.sh script.
At boot the /opt/local/bin/refreshaliases.sh script must be run as well since the /tmp/refreshaliases file is not present at boot time.

Result: no more CPU spikes but aliases are refreshed at any config change. Hope this helps.

Hi,

This behavior is present for a long time now, please see this:

https://forum.opnsense.org/index.php?topic=31662.msg153060#msg153060

I ended up with a workaround because I could not find the root of the problem.

Hi Franco,

The patch seems to work from my end.

Thanks a lot.

Just to verify. Zenarmor is checking the wireguard interface?

I see similar behavior. I disabled WG for now.

I did a 'pkg upgrade' on the command line and that seemed to fix it.


Sent from my iPhone using Tapatalk

Hi all,

When an alias is changed via Firewall -> Aliases -> Apply, the script /usr/local/opnsense/scripts/filter/update_tables.py is run from the cron every minute to make sure that pf actually picks up the change, as I understand it.
However, running this script takes about 10 to 20 seconds during which the CPU gets stressed quite severly. This behavior is seen for quite some years now and I have not been able to fix this.
I did a python trace on the update_tables.py script and it is executing the /usr/local/opnsense/scripts/filter/lib/alias/pf.py file for example about 13 million times. That surely explains the CPU load, I guess. I do have some aliases (of 'URL Table (IPs)' type) some of which are quite big (more than 200,000 IP addresses). Am I doing something wrong with my firewall alias settings? I have seen some other mentions by people seeing similar behavior with the update_tables.py script, but unfortunately not with a fix.
Hopefully someone can help me with odd behavior. If I need to provide more information, please let me know. Thank you very much.

@gpb, if this happens again, could you execute this:

/usr/local/etc/rc.configure_interface wan

Maybe that will restore things?