I encountered problems while updating or adjusting the categories of the proxy blacklists. Currently, several categories of the following blacklists are active:
1. Shallalist (http://www.shallalist.de/Downloads/shallalist.tar.gz)
2. UT1 (https://dsi.ut-capitole.fr/blacklists/download/blacklists.tar.gz)
The python script for updating the above lists consumes 100% of cpu for a very long time (more than 30 minutes) which induced me to do some more investigation. I identified the following issues:
- The update process is not transactional. Multiple instances of the update script can be launched the same time which results in conflicts
- During the update process blacklists (files) are rebuilt while in use by squid :-(. I do not know whether this affects the running squid instance with its open file descriptors. But, interruption of the update process (e.g. restarting the opnsense) leaves the blacklists in an inconsistent state which prevents restarting the squid proxy (see following error message).
- Update script contains several off-by-one errors in comparison instructions (e.g. if (len(self._url) > 8 and self._url[-7:] == '.tar.gz')
Nov 8 14:11:09 opnsense-host squid[37972]: FATAL: Bungled /usr/local/etc/squid/squid.conf line 32: acl remoteblacklist_UT2 dstdomain "/usr/local/etc/squid/acl/UT2"