Remote Access Control Lists in squid not working anymore

[franco]

Hmm, verify=False is required for self-signed HTTPS servers. This needs to be added. I tested HTTP downloads, with Shalla and it worked  for both with and without the patch provided by Ad.

Can confirm that it takes longer than it should, but eventually the categories show up in the entry. We'll get to the bottom of this.

In the meantime, things that could go wrong on the side are non-working resolution from the firewall itself or requiring a proxy to connect.


Cheers,
Franco

[pr3p]

Hmm, verify=False is required for self-signed HTTPS servers. This needs to be added. I tested HTTP downloads, with Shalla and it worked  for both with and without the patch provided by Ad.

Can confirm that it takes longer than it should, but eventually the categories show up in the entry. We'll get to the bottom of this.

In the meantime, things that could go wrong on the side are non-working resolution from the firewall itself or requiring a proxy to connect.


Cheers,
Franco

Ok thanks franco, i'l check this one

[AdSchellevis]

@franco just to be sure, my patch only adds ftp support back. Parsing large lists can take (quite) some time because of the structure needed by squid (sorting, etc.), this has been the case for quite some time.

[franco]

Hi Ad,

Ok, so this should be back in working order, I agree.

The HTTPS verify=False is something to add eventually, maybe as an advanced option per remote list (seems the easiest to pass down without cross-referencing config.xml for it).


Cheers,
Franco

[AdSchellevis]

Hi Franco,

Agreed, we should add verify=False as an option, we should be able to implement this in a similar way as the basic auth feature (should be easy  :) ).

Best regards,

Ad

[Benst]

Hi Ad,

You can easily trigger the download from the command line to see if something strange happens, but I guess your download/process just takes more the 120 seconds (which is the timeout from the gui to wait for a response).

Code Select Expand

/usr/local/opnsense/scripts/proxy/fetchACLs.py


Indeed, it looks like processing the list is taking an awful long time:

Code Select Expand

# time curl -C - -O 'ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8373k  100 8373k    0     0  1284k      0  0:00:06  0:00:06 --:--:-- 1619k
0.107u 0.071s 0:06.57 2.5% 148+120k 0+65io 0pf+0w

# time /usr/local/opnsense/scripts/proxy/fetchACLs.py
168.865u 12.138s 3:09.56 95.4% 7+167k 0+748io 12pf+0w

This is on an OPNsense A10 Quad Core SSD rack system.

Kind regards,
Ben

[pr3p]

Im having problem with web proxy i already disabled the exe on "Block specific MIME type reply"  blocklist and ACL but i cant still download exe files

-i restarted the squid server several times, before its working fine before updating the patch for acl's and




Config on ACL


Config on Remote Control List



Problem even i already edited the categories on blocklist still not applying the config for example, i disabled manga's, shopping,wares and etc but some site still on block.




Regards,
pr3p

[AdSchellevis]

Hi Ben,

I've taken another look at the processing code, but I don't see an easy big performance gain there. The unpacked blacklist file is approx 34MB, containing around a million lines if I saw it correctly.
Squid is very picky on the data in the lists, any duplication can lead to the proxy not starting at all, which makes sorting/de-duplication quite complex.

We could increase the timeout or detach the actual download process, but in the last case we should add some option to only download the indexes too.

Best regards,

Ad

[Benst]

Hi Ad,

I understand the problem. This also explains some odd behaviour I have seen in the past when clicking on 'Apply' and the list not sticking correctly. Perhaps the same problem as pr3p described in his latest message? For now the workaround could be to Download the list, and watch the cpu meter until all is quiet. And then hit Apply.

Increasing the timeout would help in the short run, but it is also highly dependant on the hardware of course. And what if some list suddenly takes even longer? Making it async would be best, but is probably a lot more work?

While we're on this subject: I also noticed that OPNsense currently blacklists every address in that list. But some categories are explicitly whitelisted, see for example liste_blanche. I currently have to delete that index in the GUI.

Kind regards,
Ben

[mow4cash]

I can't get this to work at all. Maybe it's me, I've tried everything here. It's not even creating a directory.

EDIT: It finally worked. I tried it a bunch of times but I think it had something to do with enabling before downloading.

[franco]

mow4cash, I've seen this too. I will default the new dialog to enabled.

[tillsense]

hi all,

use ad's workaround for download with no error:

Code Select Expand

/usr/local/opnsense/scripts/proxy/fetchACLs.py

over web interface i only selected 5 categories but all active!

the generate index look good for me...

Code Select Expand

root@mea:/usr/local/etc/squid # cat acl/shall0.index
{"finance/moneylending": "finance/moneylending", "automobile/boats": "automobile/boats", "porn": "porn", "ringtones": "ringtones", "drugs": "drugs", "socialnet": "socialnet", "dynamic": "dynamic", "anonvpn": "anonvpn", "library": "library", "science/astronomy": "science/astronomy", "costtraps": "costtraps", "finance/insurance": "finance/insurance", "chat": "chat", "politics": "politics", "searchengines": "searchengines", "shopping": "shopping", "aggressive": "aggressive", "hospitals": "hospitals", "urlshortener": "urlshortener", "adv": "adv", "weapons": "weapons", "updatesites": "updatesites", "recreation/restaurants": "recreation/restaurants", "radiotv": "radiotv", "alcohol": "alcohol", "isp": "isp", "finance/trading": "finance/trading", "webmail": "webmail", "sex/lingerie": "sex/lingerie", "religion": "religion", "tracker": "tracker", "music": "music", "automobile/planes": "automobile/planes", "hobby/gardening": "hobby/gardening", "recreation/humor": "recreation/humor", "hobby/games-misc": "hobby/games-misc", "redirector": "redirector", "gamble": "gamble", "fortunetelling": "fortunetelling", "jobsearch": "jobsearch", "finance/banking": "finance/banking", "hobby/cooking": "hobby/cooking", "webtv": "webtv", "government": "government", "models": "models", "automobile/bikes": "automobile/bikes", "downloads": "downloads", "hobby/pets": "hobby/pets", "warez": "warez", "homestyle": "homestyle", "recreation/martialarts": "recreation/martialarts", "spyware": "spyware", "recreation/wellness": "recreation/wellness", "news": "news", "hobby/games-online": "hobby/games-online", "recreation/travel": "recreation/travel", "webphone": "webphone", "sex/education": "sex/education", "finance/other": "finance/other", "automobile/cars": "automobile/cars", "dating": "dating", "remotecontrol": "remotecontrol", "forum": "forum", "violence": "violence", "imagehosting": "imagehosting", "podcasts": "podcasts", "movies": "movies", "webradio": "webradio", "military": "military", "hacking": "hacking", "finance/realestate": "finance/realestate", "science/chemistry": "science/chemistry", "education/schools": "education/schools", "recreation/sports": "recreation/sports"}
root@mea:/usr/local/etc/squid # cat externalACLs.conf
#
# Automatic generated configuration for fetching remote ACLs.
# Do not edit this file manually.
[shall0]
url:http://www.shallalist.de/Downloads/shallalist.tar.gz
enabled:1
filter:adv,hacking,spyware,tracker,warez
sslNoVerify=0
root@mea:/usr/local/etc/squid #

cheers till

[franco]

I've moved all discussed changes to 16.7.10 and will ask Ad to look at the category hiccup Till reported just then.

EDIT: Ad said that the generated output should be checked to see if it decreases when removing categories to block. If so, everything is ok?

[tillsense]

hi franco,

generated output? (a reboot does not help)

cheers till

[tillsense]

hi all,

back to 16.7.10 it's the same (all categories active) plus a error in the log:

configd.py: unable to sendback response [OK ] for [proxy][downloadacls][None] {b62421f1-b3be-4e2c-b502-366d1a140aa0}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall('%s\n' % result) File "/usr/local/lib/python2.7/socket.py", line 228, in meth return getattr(self._sock,name)(*args) error: [Errno 32] Broken pipe

cheers till