OPNsense Forum

English Forums => General Discussion => Topic started by: Benst on November 18, 2016, 05:52:32 pm

Title: Remote Access Control Lists in squid not working anymore
Post by: Benst on November 18, 2016, 05:52:32 pm
Hi,

Using OPNsense 16.7.8-amd64.

I am having a problem with the Remote access control lists not being fetched or updated. It used to work before and the last one succesfully fetched was on Oct 31 15:29 CET.

I use the list from ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz. I also added the example from the documentation (http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml) but this doesn't work either. After I press Download and Apply, it returns quickly (used to take a long time), and in /var/log/system.log I see:

Nov 18 17:43:49 OPNsense configd.py: [c7615826-515d-443d-8db3-66eec2936dc3] generate template OPNsense/Proxy
Nov 18 17:43:50 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 18 17:43:52 OPNsense configd.py: [6056e2ce-722a-4966-9922-922434741223] download and reload proxy ACLs from remote locations
Nov 18 17:43:52 OPNsense configd.py: [6056e2ce-722a-4966-9922-922434741223] returned exit status 1

So it returns immediately without doing any work. I am able te retrieve both lists manually from the firewall using curl, so they are reachable.

Any ideas?

Thanks,
Ben
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: Benst on November 22, 2016, 02:32:49 pm
No one? Any tips on helping me debug this?

Thanks,
Ben
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: pr3p on November 22, 2016, 02:59:51 pm
Hi,

Using OPNsense 16.7.8-amd64.

I am having a problem with the Remote access control lists not being fetched or updated. It used to work before and the last one succesfully fetched was on Oct 31 15:29 CET.

I use the list from ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz. I also added the example from the documentation (http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml) but this doesn't work either. After I press Download and Apply, it returns quickly (used to take a long time), and in /var/log/system.log I see:

Nov 18 17:43:49 OPNsense configd.py: [c7615826-515d-443d-8db3-66eec2936dc3] generate template OPNsense/Proxy
Nov 18 17:43:50 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 18 17:43:52 OPNsense configd.py: [6056e2ce-722a-4966-9922-922434741223] download and reload proxy ACLs from remote locations
Nov 18 17:43:52 OPNsense configd.py: [6056e2ce-722a-4966-9922-922434741223] returned exit status 1

So it returns immediately without doing any work. I am able te retrieve both lists manually from the firewall using curl, so they are reachable.

Any ideas?

Thanks,
Ben

I have the same problem encountered regarding on downloading ACL's upon updating to the latest version OPNsense 16.7.8

But when i checked those and click apply even there is no category specified on the list its blocking.

(http://image.prntscr.com/image/18fa42622c0e47fbb09a6fca441b6f3b.png)

(http://image.prntscr.com/image/8107127a69974007828e602cfac5a7f4.png)

(http://image.prntscr.com/image/f62ce90434524ddbbbafe1cef8e9ced8.png)
https://forum.opnsense.org/index.php?topic=3967.0
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: Benst on November 22, 2016, 03:01:54 pm
Ok, good to know I'm not the only one!

Ben
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: AdSchellevis on November 22, 2016, 06:39:32 pm
The problem is with the ftp download, http(s) download works fine. We removed ftp support by switching to a different internal library.

I have prepared a patch to add ftp support again, if you like to test this, you can execute the following on a command line:

Code: [Select]
opnsense-patch c3e84685

Best regards,

Ad
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: Benst on November 22, 2016, 06:50:53 pm
Hi Ad,

Thanks for the patch. Fetching the ftp data works again, but in system.log I now see a timeout:

Code: [Select]
Nov 22 18:40:52 OPNsense configd.py: [b2cf595d-8d13-43a5-869e-b33dddac1949] generate template OPNsense/Proxy
Nov 22 18:40:53 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 22 18:40:55 OPNsense configd.py: [73e319a1-7595-4240-be5d-c671820f6ab3] download and reload proxy ACLs from remote locations
Nov 22 18:42:57 OPNsense configd[6698]: Timeout (120) executing : proxy fetchacls

But the data is updated in /usr/local/etc/squid/acl, and I can see/choose the categories again in the web UI. So perhaps this is only a cosmetic problem.

Kind regards,
Ben
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: Benst on November 22, 2016, 06:57:16 pm
Oops, it seems there is another problem after applying the download:

Code: [Select]
Nov 22 18:48:52 OPNsense configd.py: [10b51670-e81f-426e-8a60-ebd7eaa3192a] request proxy status
Nov 22 18:48:52 OPNsense configd.py: [9f07c783-7099-4f49-87c1-b7fc14f9a298] generate template OPNsense/Proxy
Nov 22 18:48:53 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 22 18:48:55 OPNsense configd.py: [f651e852-da92-4dd7-a376-2267b28ece11] reconfigure proxy
Nov 22 18:48:59 OPNsense squid: Bungled /usr/local/etc/squid/squid.conf line 38: acl remoteblacklist_UT1 dstdomain "/usr/local/etc/squid/acl/UT1"
Nov 22 18:48:59 OPNsense configd.py: [f651e852-da92-4dd7-a376-2267b28ece11] returned exit status 1

Line 38 looks like this:

Code: [Select]
acl remoteblacklist_UT1 dstdomain "/usr/local/etc/squid/acl/UT1"
And that file actually exists:

Code: [Select]
root@OPNsense:/usr/local/etc/squid # ll acl/
total 27942
-rw-r-----  1 root  squid  28580995 Nov 22 18:43 UT1
-rw-r-----  1 root  squid      1444 Nov 22 18:42 UT1.index
-rw-r-----  1 root  squid       991 Nov 22 18:43 yoyoads
-rw-r-----  1 root  squid         2 Nov 22 18:43 yoyoads.index

Kind regards,
Ben
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: AdSchellevis on November 22, 2016, 07:50:52 pm
Hi Ben,

The "bungled" message can't indeed be related to this fix, this part only downloads the file itself it doesn't interact with the squid config.
Strange thing is that there haven't been a lot of changes in the template area for squid.
If I'm not mistaken the "bungled" messages means that the offending line isn't used in the config.

Can you check if this rule is available in your squid.conf?
Code: [Select]
http_access deny remoteblacklist_UT1

If it is, try stopping and starting the proxy service to see if you can reproduce it.

In case it isn't solved, have you changed anything after upgrading (new packages, custom configuration hooks or specific settings)?

Best regards,

Ad
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: Benst on November 23, 2016, 01:39:18 pm
Hi Ad,

Yes, that line is in there. I have restarted squid and am now unable to reproduce the bungled message. The timeout is still there tough. Download ACLs gives this:

Code: [Select]
Nov 23 13:27:15 OPNsense configd.py: [8ed9d971-89dc-4d69-bb59-c99578afaccb] request proxy status
Nov 23 13:27:31 OPNsense configd.py: [776d1217-5e8f-4f66-8e3d-8aca0b8c8744] generate template OPNsense/Proxy
Nov 23 13:27:32 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 23 13:27:34 OPNsense configd.py: [c7a06f6b-5253-4251-af1a-6740ef916ed5] download proxy ACLs from remote locations
Nov 23 13:29:36 OPNsense configd[18360]: Timeout (120) executing : proxy downloadacls

The Web UI has at that point returned to normal (no spinning indicator). At that point a Python process is still chewing up 100% cpu (fetchACLs I think). When that's done I get:

Code: [Select]
Nov 23 13:30:39 OPNsense configd.py: unable to sendback response [OK ] for [proxy][downloadacls][None] {c7a06f6b-5253-4251-af1a-6740ef916ed5}, message was Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run     self.connection.sendall('%s\n' % result)   File "/usr/local/lib/python2.7/socket.py", line 228, in meth     return getattr(self._sock,name)(*args) error: [Errno 32] Broken pipe
And then I hit Apply:

Code: [Select]
Nov 23 13:33:07 OPNsense configd.py: [61d38b33-64d8-410d-86a4-dd8f13397041] request proxy status
Nov 23 13:33:07 OPNsense configd.py: [18e041fb-8f75-41b1-af92-f5b6f8c2563c] generate template OPNsense/Proxy
Nov 23 13:33:08 OPNsense configd.py: generate template container OPNsense/Proxy
Nov 23 13:33:10 OPNsense configd.py: [63646e00-3382-4624-89c9-dfcc8f63fbd6] reconfigure proxy

Perhaps the bungled message was because I hit apply before the Python process actually ended.

Kind regards,
Ben
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: pr3p on November 23, 2016, 03:19:27 pm
Any update regarding on this ACL problem, im having the same problem, i temporary disabled our proxy for now.thanks and looking for solution on this bug


Regards,
pr3p
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: AdSchellevis on November 23, 2016, 05:33:32 pm
Hi Ben,

You can easily trigger the download from the command line to see if something strange happens, but I guess your download/process just takes more the 120 seconds (which is the timeout from the gui to wait for a response).

Code: [Select]
/usr/local/opnsense/scripts/proxy/fetchACLs.py


Best regards,

Ad
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: fabian on November 23, 2016, 06:29:46 pm
You can easily trigger the download from the command line to see if something strange happens, but I guess your download/process just takes more the 120 seconds (which is the timeout from the gui to wait for a response).

In this case it would be a good idea to detach the downloading and preparing of the access lists and reload the squid configuration when done asynchronously.
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: franco on November 23, 2016, 07:33:30 pm
Which is easier said than done. Fetching the first time reveals categories in archives, this must be presented to the user immediately.

Easy workaround: The file could be fetched and mirrored from another server, especially somewhere fast.

You can even use OPNsense for that, download from script, move file to /usr/local/www/acls and point to https://127.0.0.1/acls/file ;)
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: tillsense on November 23, 2016, 07:51:32 pm
the same with shalla list here on 17.1 (http://www.shallalist.de/Downloads/shallalist.tar.gz) and the yoyo shows only encrypted on cat this
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: pr3p on November 24, 2016, 02:42:13 am
Which is easier said than done. Fetching the first time reveals categories in archives, this must be presented to the user immediately.

Easy workaround: The file could be fetched and mirrored from another server, especially somewhere fast.

You can even use OPNsense for that, download from script, move file to /usr/local/www/acls and point to https://127.0.0.1/acls/file ;)

Hi franco even i uploaded the script on localserver still not downloading and applying on ACLS

(http://image.prntscr.com/image/79c8c66a32634cb290ff83f22fae2c9f.png)
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: franco on November 24, 2016, 07:34:39 am
Hmm, verify=False is required for self-signed HTTPS servers. This needs to be added. I tested HTTP downloads, with Shalla and it worked  for both with and without the patch provided by Ad.

Can confirm that it takes longer than it should, but eventually the categories show up in the entry. We'll get to the bottom of this.

In the meantime, things that could go wrong on the side are non-working resolution from the firewall itself or requiring a proxy to connect.


Cheers,
Franco
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: pr3p on November 24, 2016, 08:33:49 am
Hmm, verify=False is required for self-signed HTTPS servers. This needs to be added. I tested HTTP downloads, with Shalla and it worked  for both with and without the patch provided by Ad.

Can confirm that it takes longer than it should, but eventually the categories show up in the entry. We'll get to the bottom of this.

In the meantime, things that could go wrong on the side are non-working resolution from the firewall itself or requiring a proxy to connect.


Cheers,
Franco

Ok thanks franco, i'l check this one
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: AdSchellevis on November 24, 2016, 09:02:28 am
@franco just to be sure, my patch only adds ftp support back. Parsing large lists can take (quite) some time because of the structure needed by squid (sorting, etc.), this has been the case for quite some time.
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: franco on November 24, 2016, 11:10:45 am
Hi Ad,

Ok, so this should be back in working order, I agree.

The HTTPS verify=False is something to add eventually, maybe as an advanced option per remote list (seems the easiest to pass down without cross-referencing config.xml for it).


Cheers,
Franco
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: AdSchellevis on November 24, 2016, 11:13:27 am
Hi Franco,

Agreed, we should add verify=False as an option, we should be able to implement this in a similar way as the basic auth feature (should be easy  :) ).

Best regards,

Ad
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: Benst on November 24, 2016, 02:05:27 pm
Hi Ad,

You can easily trigger the download from the command line to see if something strange happens, but I guess your download/process just takes more the 120 seconds (which is the timeout from the gui to wait for a response).

Code: [Select]
/usr/local/opnsense/scripts/proxy/fetchACLs.py

Indeed, it looks like processing the list is taking an awful long time:

Code: [Select]
# time curl -C - -O 'ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8373k  100 8373k    0     0  1284k      0  0:00:06  0:00:06 --:--:-- 1619k
0.107u 0.071s 0:06.57 2.5% 148+120k 0+65io 0pf+0w

# time /usr/local/opnsense/scripts/proxy/fetchACLs.py
168.865u 12.138s 3:09.56 95.4% 7+167k 0+748io 12pf+0w

This is on an OPNsense A10 Quad Core SSD rack system.

Kind regards,
Ben
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: pr3p on November 25, 2016, 12:36:22 pm
Im having problem with web proxy i already disabled the exe on "Block specific MIME type reply"  blocklist and ACL but i cant still download exe files

-i restarted the squid server several times, before its working fine before updating the patch for acl's and

(http://image.prntscr.com/image/e4a652672da9405bbbc8028700ad8036.png)


Config on ACL
(http://image.prntscr.com/image/d51722f456e548b283839e9eb253ab3d.png)

Config on Remote Control List
(http://image.prntscr.com/image/62b3aca6cd874046b12ff3ca5501269a.png)


Problem even i already edited the categories on blocklist still not applying the config for example, i disabled manga's, shopping,wares and etc but some site still on block.

(http://image.prntscr.com/image/1c1d7deb6127469ea4fe9b8b83f23289.png)


Regards,
pr3p
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: AdSchellevis on November 25, 2016, 05:19:10 pm
Hi Ben,

I've taken another look at the processing code, but I don't see an easy big performance gain there. The unpacked blacklist file is approx 34MB, containing around a million lines if I saw it correctly.
Squid is very picky on the data in the lists, any duplication can lead to the proxy not starting at all, which makes sorting/de-duplication quite complex.

We could increase the timeout or detach the actual download process, but in the last case we should add some option to only download the indexes too.

Best regards,

Ad


 
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: Benst on November 25, 2016, 05:47:30 pm
Hi Ad,

I understand the problem. This also explains some odd behaviour I have seen in the past when clicking on 'Apply' and the list not sticking correctly. Perhaps the same problem as pr3p described in his latest message? For now the workaround could be to Download the list, and watch the cpu meter until all is quiet. And then hit Apply.

Increasing the timeout would help in the short run, but it is also highly dependant on the hardware of course. And what if some list suddenly takes even longer? Making it async would be best, but is probably a lot more work?

While we're on this subject: I also noticed that OPNsense currently blacklists every address in that list. But some categories are explicitly whitelisted, see for example liste_blanche. I currently have to delete that index in the GUI.

Kind regards,
Ben
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: mow4cash on November 28, 2016, 04:17:02 am
I can't get this to work at all. Maybe it's me, I've tried everything here. It's not even creating a directory.

EDIT: It finally worked. I tried it a bunch of times but I think it had something to do with enabling before downloading.
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: franco on November 28, 2016, 08:05:02 am
mow4cash, I've seen this too. I will default the new dialog to enabled.
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: tillsense on November 28, 2016, 07:11:19 pm
hi all,

use ad's workaround for download with no error:
Code: [Select]
/usr/local/opnsense/scripts/proxy/fetchACLs.py
over web interface i only selected 5 categories but all active!

the generate index look good for me...
Code: [Select]
root@mea:/usr/local/etc/squid # cat acl/shall0.index
{"finance/moneylending": "finance/moneylending", "automobile/boats": "automobile/boats", "porn": "porn", "ringtones": "ringtones", "drugs": "drugs", "socialnet": "socialnet", "dynamic": "dynamic", "anonvpn": "anonvpn", "library": "library", "science/astronomy": "science/astronomy", "costtraps": "costtraps", "finance/insurance": "finance/insurance", "chat": "chat", "politics": "politics", "searchengines": "searchengines", "shopping": "shopping", "aggressive": "aggressive", "hospitals": "hospitals", "urlshortener": "urlshortener", "adv": "adv", "weapons": "weapons", "updatesites": "updatesites", "recreation/restaurants": "recreation/restaurants", "radiotv": "radiotv", "alcohol": "alcohol", "isp": "isp", "finance/trading": "finance/trading", "webmail": "webmail", "sex/lingerie": "sex/lingerie", "religion": "religion", "tracker": "tracker", "music": "music", "automobile/planes": "automobile/planes", "hobby/gardening": "hobby/gardening", "recreation/humor": "recreation/humor", "hobby/games-misc": "hobby/games-misc", "redirector": "redirector", "gamble": "gamble", "fortunetelling": "fortunetelling", "jobsearch": "jobsearch", "finance/banking": "finance/banking", "hobby/cooking": "hobby/cooking", "webtv": "webtv", "government": "government", "models": "models", "automobile/bikes": "automobile/bikes", "downloads": "downloads", "hobby/pets": "hobby/pets", "warez": "warez", "homestyle": "homestyle", "recreation/martialarts": "recreation/martialarts", "spyware": "spyware", "recreation/wellness": "recreation/wellness", "news": "news", "hobby/games-online": "hobby/games-online", "recreation/travel": "recreation/travel", "webphone": "webphone", "sex/education": "sex/education", "finance/other": "finance/other", "automobile/cars": "automobile/cars", "dating": "dating", "remotecontrol": "remotecontrol", "forum": "forum", "violence": "violence", "imagehosting": "imagehosting", "podcasts": "podcasts", "movies": "movies", "webradio": "webradio", "military": "military", "hacking": "hacking", "finance/realestate": "finance/realestate", "science/chemistry": "science/chemistry", "education/schools": "education/schools", "recreation/sports": "recreation/sports"}
root@mea:/usr/local/etc/squid # cat externalACLs.conf
#
# Automatic generated configuration for fetching remote ACLs.
# Do not edit this file manually.
[shall0]
url:http://www.shallalist.de/Downloads/shallalist.tar.gz
enabled:1
filter:adv,hacking,spyware,tracker,warez
sslNoVerify=0
root@mea:/usr/local/etc/squid #

cheers till
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: franco on November 30, 2016, 08:30:33 am
I've moved all discussed changes to 16.7.10 and will ask Ad to look at the category hiccup Till reported just then.

EDIT: Ad said that the generated output should be checked to see if it decreases when removing categories to block. If so, everything is ok?
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: tillsense on December 02, 2016, 07:28:00 pm
hi franco,

generated output? (a reboot does not help)

cheers till
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: tillsense on December 04, 2016, 11:38:37 am
hi all,

back to 16.7.10 it's the same (all categories active) plus a error in the log:

Quote
configd.py: unable to sendback response [OK ] for [proxy][downloadacls][None] {b62421f1-b3be-4e2c-b502-366d1a140aa0}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall('%s\n' % result) File "/usr/local/lib/python2.7/socket.py", line 228, in meth return getattr(self._sock,name)(*args) error: [Errno 32] Broken pipe

cheers till
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: Benst on December 05, 2016, 02:50:38 pm
back to 16.7.10 it's the same (all categories active) plus a error in the log:

Quote
configd.py: unable to sendback response [OK ] for [proxy][downloadacls][None] {b62421f1-b3be-4e2c-b502-366d1a140aa0}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall('%s\n' % result) File "/usr/local/lib/python2.7/socket.py", line 228, in meth return getattr(self._sock,name)(*args) error: [Errno 32] Broken pipe

That's probably the same issue I have. The timeout for the GUI is set to 120 s, but the ACL script isn't finished by then.

Ben
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: tillsense on December 15, 2016, 07:41:50 pm
hi,

since the upgrade to 17.1.b also works the category selection again.
It will only be blocked what is selected.  8)

(the timeout for download i have not yet tested)

cheers till
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: tillsense on December 15, 2016, 08:31:28 pm
hi,

since the upgrade to 17.1.b also works the category selection again.
It will only be blocked what is selected.  8)

(the timeout for download i have not yet tested)

cheers till

too early. after a while all are blocked again ...
Title: Re: Remote Access Control Lists in squid not working anymore
Post by: Verasmos on September 09, 2020, 02:04:47 pm
I've taken another look at the processing code, but I don't see an easy big performance gain there.
สมัครสมาชิก12BET (https://leadershipandselfdevelopment.com/12bet/)
The unpacked blacklist file is approx 34MB, containing around a million lines if I saw it correctly.

Squid is very picky on the data in the lists, any duplication can lead to the proxy not starting at all, which makes sorting/de-duplication quite complex.

We could increase the timeout or detach the actual download process, but in the last case we should add some option to only download the indexes too.