Remote Access Control Lists in squid not working anymore

[Benst]

Hi,

Using OPNsense 16.7.8-amd64.

I am having a problem with the Remote access control lists not being fetched or updated. It used to work before and the last one succesfully fetched was on Oct 31 15:29 CET.

I use the list from ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz. I also added the example from the documentation (http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml) but this doesn't work either. After I press Download and Apply, it returns quickly (used to take a long time), and in /var/log/system.log I see:

Nov 18 17:43:49 OPNsense configd.py: [c7615826-515d-443d-8db3-66eec2936dc3] generate template OPNsense/Proxy
Nov 18 17:43:50 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 18 17:43:52 OPNsense configd.py: [6056e2ce-722a-4966-9922-922434741223] download and reload proxy ACLs from remote locations
Nov 18 17:43:52 OPNsense configd.py: [6056e2ce-722a-4966-9922-922434741223] returned exit status 1

So it returns immediately without doing any work. I am able te retrieve both lists manually from the firewall using curl, so they are reachable.

Any ideas?

Thanks,
Ben

[Benst]

No one? Any tips on helping me debug this?

Thanks,
Ben

[pr3p]

Hi,

Using OPNsense 16.7.8-amd64.

I am having a problem with the Remote access control lists not being fetched or updated. It used to work before and the last one succesfully fetched was on Oct 31 15:29 CET.

I use the list from ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz. I also added the example from the documentation (http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml) but this doesn't work either. After I press Download and Apply, it returns quickly (used to take a long time), and in /var/log/system.log I see:

Nov 18 17:43:49 OPNsense configd.py: [c7615826-515d-443d-8db3-66eec2936dc3] generate template OPNsense/Proxy
Nov 18 17:43:50 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 18 17:43:52 OPNsense configd.py: [6056e2ce-722a-4966-9922-922434741223] download and reload proxy ACLs from remote locations
Nov 18 17:43:52 OPNsense configd.py: [6056e2ce-722a-4966-9922-922434741223] returned exit status 1

So it returns immediately without doing any work. I am able te retrieve both lists manually from the firewall using curl, so they are reachable.

Any ideas?

Thanks,
Ben

I have the same problem encountered regarding on downloading ACL's upon updating to the latest version OPNsense 16.7.8

But when i checked those and click apply even there is no category specified on the list its blocking.






https://forum.opnsense.org/index.php?topic=3967.0

[Benst]

Ok, good to know I'm not the only one!

Ben

[AdSchellevis]

The problem is with the ftp download, http(s) download works fine. We removed ftp support by switching to a different internal library.

I have prepared a patch to add ftp support again, if you like to test this, you can execute the following on a command line:

Code: [Select]

opnsense-patch c3e84685

Best regards,

Ad

[Benst]

Hi Ad,

Thanks for the patch. Fetching the ftp data works again, but in system.log I now see a timeout:

Code: [Select]

Nov 22 18:40:52 OPNsense configd.py: [b2cf595d-8d13-43a5-869e-b33dddac1949] generate template OPNsense/Proxy
Nov 22 18:40:53 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 22 18:40:55 OPNsense configd.py: [73e319a1-7595-4240-be5d-c671820f6ab3] download and reload proxy ACLs from remote locations
Nov 22 18:42:57 OPNsense configd[6698]: Timeout (120) executing : proxy fetchacls
But the data is updated in /usr/local/etc/squid/acl, and I can see/choose the categories again in the web UI. So perhaps this is only a cosmetic problem.

Kind regards,
Ben

[Benst]

Oops, it seems there is another problem after applying the download:

Code: [Select]

Nov 22 18:48:52 OPNsense configd.py: [10b51670-e81f-426e-8a60-ebd7eaa3192a] request proxy status
Nov 22 18:48:52 OPNsense configd.py: [9f07c783-7099-4f49-87c1-b7fc14f9a298] generate template OPNsense/Proxy
Nov 22 18:48:53 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 22 18:48:55 OPNsense configd.py: [f651e852-da92-4dd7-a376-2267b28ece11] reconfigure proxy
Nov 22 18:48:59 OPNsense squid: Bungled /usr/local/etc/squid/squid.conf line 38: acl remoteblacklist_UT1 dstdomain "/usr/local/etc/squid/acl/UT1"
Nov 22 18:48:59 OPNsense configd.py: [f651e852-da92-4dd7-a376-2267b28ece11] returned exit status 1
Line 38 looks like this:

Code: [Select]

acl remoteblacklist_UT1 dstdomain "/usr/local/etc/squid/acl/UT1"
And that file actually exists:

Code: [Select]

root@OPNsense:/usr/local/etc/squid # ll acl/
total 27942
-rw-r-----  1 root  squid  28580995 Nov 22 18:43 UT1
-rw-r-----  1 root  squid      1444 Nov 22 18:42 UT1.index
-rw-r-----  1 root  squid       991 Nov 22 18:43 yoyoads
-rw-r-----  1 root  squid         2 Nov 22 18:43 yoyoads.index
Kind regards,
Ben

[AdSchellevis]

Hi Ben,

The "bungled" message can't indeed be related to this fix, this part only downloads the file itself it doesn't interact with the squid config.
Strange thing is that there haven't been a lot of changes in the template area for squid.
If I'm not mistaken the "bungled" messages means that the offending line isn't used in the config.

Can you check if this rule is available in your squid.conf?

Code: [Select]

http_access deny remoteblacklist_UT1

If it is, try stopping and starting the proxy service to see if you can reproduce it.

In case it isn't solved, have you changed anything after upgrading (new packages, custom configuration hooks or specific settings)?

Best regards,

Ad

[Benst]

Hi Ad,

Yes, that line is in there. I have restarted squid and am now unable to reproduce the bungled message. The timeout is still there tough. Download ACLs gives this:

Code: [Select]

Nov 23 13:27:15 OPNsense configd.py: [8ed9d971-89dc-4d69-bb59-c99578afaccb] request proxy status
Nov 23 13:27:31 OPNsense configd.py: [776d1217-5e8f-4f66-8e3d-8aca0b8c8744] generate template OPNsense/Proxy
Nov 23 13:27:32 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 23 13:27:34 OPNsense configd.py: [c7a06f6b-5253-4251-af1a-6740ef916ed5] download proxy ACLs from remote locations
Nov 23 13:29:36 OPNsense configd[18360]: Timeout (120) executing : proxy downloadacls
The Web UI has at that point returned to normal (no spinning indicator). At that point a Python process is still chewing up 100% cpu (fetchACLs I think). When that's done I get:

Code: [Select]

Nov 23 13:30:39 OPNsense configd.py: unable to sendback response [OK ] for [proxy][downloadacls][None] {c7a06f6b-5253-4251-af1a-6740ef916ed5}, message was Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run     self.connection.sendall('%s\n' % result)   File "/usr/local/lib/python2.7/socket.py", line 228, in meth     return getattr(self._sock,name)(*args) error: [Errno 32] Broken pipe
And then I hit Apply:

Code: [Select]

Nov 23 13:33:07 OPNsense configd.py: [61d38b33-64d8-410d-86a4-dd8f13397041] request proxy status
Nov 23 13:33:07 OPNsense configd.py: [18e041fb-8f75-41b1-af92-f5b6f8c2563c] generate template OPNsense/Proxy
Nov 23 13:33:08 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 23 13:33:10 OPNsense configd.py: [63646e00-3382-4624-89c9-dfcc8f63fbd6] reconfigure proxy
Perhaps the bungled message was because I hit apply before the Python process actually ended.

Kind regards,
Ben

[pr3p]

Any update regarding on this ACL problem, im having the same problem, i temporary disabled our proxy for now.thanks and looking for solution on this bug


Regards,
pr3p

[AdSchellevis]

Hi Ben,

You can easily trigger the download from the command line to see if something strange happens, but I guess your download/process just takes more the 120 seconds (which is the timeout from the gui to wait for a response).

Code: [Select]

/usr/local/opnsense/scripts/proxy/fetchACLs.py


Best regards,

Ad

[fabian]

You can easily trigger the download from the command line to see if something strange happens, but I guess your download/process just takes more the 120 seconds (which is the timeout from the gui to wait for a response).

In this case it would be a good idea to detach the downloading and preparing of the access lists and reload the squid configuration when done asynchronously.

[franco]

Which is easier said than done. Fetching the first time reveals categories in archives, this must be presented to the user immediately.

Easy workaround: The file could be fetched and mirrored from another server, especially somewhere fast.

You can even use OPNsense for that, download from script, move file to /usr/local/www/acls and point to https://127.0.0.1/acls/file

[tillsense]

the same with shalla list here on 17.1 (http://www.shallalist.de/Downloads/shallalist.tar.gz) and the yoyo shows only encrypted on cat this

[pr3p]

Which is easier said than done. Fetching the first time reveals categories in archives, this must be presented to the user immediately.

Easy workaround: The file could be fetched and mirrored from another server, especially somewhere fast.

You can even use OPNsense for that, download from script, move file to /usr/local/www/acls and point to https://127.0.0.1/acls/file

Hi franco even i uploaded the script on localserver still not downloading and applying on ACLS