OPNsense Forum

English Forums => Web Proxy Filtering and Caching => Topic started by: t.mayer on January 07, 2020, 11:55:14 am

Title: ACL > Whitelist not not considered when using Remote ACL
Post by: t.mayer on January 07, 2020, 11:55:14 am
I have configured the OPNsense-Webproxy with shallalist as Remote ACL.
For some exceptions i always used the Whitelist under Access Control List > Whitelist.
When i try to open a domain blocked by shallalist-category but with a corresponding entry in the whitelist, the domain still will be blocked.

Version of OPNSense: 19.7.8

Forward-Proxy-Config:
- Interface: LAN
- Port: 3128 / SSL: 3129
- Transparent http-Proxy
- SSL inspection
- SNI only

Thanks for your help!

Greeds
Tom
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: t.mayer on January 23, 2020, 07:48:34 pm
May I ask again if there is anybody with an idea?
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: Amr on February 25, 2020, 07:22:58 am
Weird works for me, try adding a wild card for the domain aka add a "." before domain name ex: .whatsapp.net and stopping and restarting the service.

Check the certificate of the domain for aliases and try adding them, check logs to see if the website is trying to reach another domain for grabbing  code or something.

Since you are using sni logging only it shouldn't be a problem but try adding the domain in the no bump sites list.

Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: t.mayer on March 25, 2020, 04:21:21 pm
@Amr: Thanks for your answer.

The problem to me still exists.
I found out that it has to do something with the ssl/sni-only-settings.

Here is what i have tested:
Case 1: No Entry in ACL-Whitelist
Setting Browser to use Proxy-Port 3128
> instagram.com can't be reached
> functioning as expected

Setting Browser to not use Proxy (Proxy now transparent via SSL/SNI only)
> instagram.com can't be reached
> functioning as expected

Case 2: Entry in ACL-Whitelist: instagram.com
Setting Browser to use Proxy-Port 3128
> instagram.com can be reached
> functioning as expected

Setting Browser to not use Proxy (Proxy now transparent via SSL/SNI only)
> instagram.com can't be reached
> BUG?
> instagram.com as entry in SSL no bump sites has also no effect on this

Hopefully my description is understandable.

Greeds
Tom
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: Amr on March 26, 2020, 01:22:12 pm
From your description
Quote
Setting Browser to not use Proxy (Proxy now transparent via SSL/SNI only)
> instagram.com can't be reached
 
I assume there's a problem with NAT port forwarding so did you set it up properly? (attach a pic of your rules)

If NAT is not the problem can you access other websites? (after setting 'no proxy' in browser)

If you can access other websites what kind of error does the proxy return?
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: t.mayer on March 27, 2020, 05:49:44 pm
@Amr: Thanks to your response!

I think you got me wrong. The Proxy-NAT works as expected. I can access all websites with proxy except those that I have blocked by an active shallalist-category.

NAT-Rules:

The problem is that the whitelist (e. g. for instagram.com) is not considered  when using a remote acl (e. g. shallalist with active category socialnet) when using proxy in transparent/ssl/sni-mode.

The whitelist is only considered when the proxy is used in non-transparent mode.

Greeds
Tom
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: Amr on March 29, 2020, 09:27:45 am
Quote
The whitelist is only considered when the proxy is used in non-transparent mode.
Can you reply with the error that squid returns when Instagram gets blocked?

Also, you might want to reset cache under "Support" tab and restart the proxy
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: t.mayer on March 29, 2020, 09:35:29 am
The error is - as always when blocking https - not a squid-error but a browser-certificate-error:
NET::ERR_CERT_AUTHORITY_INVALID

I have reset the cache and restarted squid lots of times...

Can you test, if a whitelisted entry works when blocking with transparent proxy with ssl and sni on your installation?

I think its a bug!
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: Amr on March 31, 2020, 03:39:37 pm
Quote
The error is - as always when blocking https - not a squid-error but a browser-certificate-error:
NET::ERR_CERT_AUTHORITY_INVALID
weird are you using an internal certificate?

In some rare cases that I have encountered the antivirus was to blame, try disabling it.

Quote
Can you test, if a whitelisted entry works when blocking with transparent proxy with ssl and sni on your installation?
I did test it, works fine, but to be frank I didn't reset the cache.

my current setup is a transparent proxy with an internal certificate and SSL inspection with some sites in SSL no bump sites, but I don't use shallist but rather custom rules (I did use shallalist and it worked fine too)

A word of caution: using Log SNI information only won't block VPN connections made on https port.
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: t.mayer on March 31, 2020, 03:58:52 pm
@Amr

Thanks for trying to help me. But I think - sorry if this is impolite - you can not help me until you really want to recognize my problem.

You say that it is weird that i am using internal certificate - 5 lines further you explain that you are using internal certificate as well. Further you claim that you have tested the same installation, but you don't use a remote acl like shalla list.

The problem is neither the internal certificate nor antivirus (not used).
The problem is the shallalist in combination with transparent/ssl/sni-proxy: whitelisted entrys are blocked. All the rest is working as expected.

Greeds an thanks for you help.
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: Amr on April 02, 2020, 02:19:47 pm
Quote
sorry if this is impolite
no offense taken.

Quote
You say that it is weird that I am using internal certificate
well, what I meant that in your setup certificates shouldn't be used since you don't bump site at all.

Quote
Further you claim that you have tested the same installation, but you don't use a remote acl like shalla list.
I did test Log SNI information only a while ago and with shallalist and it was working, but since I found that VPN connections on https ignored all my filters I used SSL inspection only.

I'm no guru (and pretty sure you already have configured your proxy correctly) but these steps are aimed to troubleshoot your installation:
1-check Instagram's certificate to see who signed it should be DigiCert Inc, not your internal certificate.
2-check also the time of the opnsense machine (from the dashboard) and your machine.
3-check if you entered the LAN subnet in  Allowed Subnets under access control, maybe add your firewall lan address in the Unrestricted IP addresses too.
4-reboot your firewall and keep a look for squid configuration sanity check.
 
Quote
The problem is the shallalist in combination with transparent/ssl/sni-proxy: whitelisted entrys are blocked. All the rest is working as expected.
Well if the previous steps didn't work then I can think of a work around but it's not easy if you aren't familiar with linux, simply put you can edit the shallalist manually and remove instagram from the list.
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: hbc on April 02, 2020, 09:54:24 pm
@t.mayer: you are not alone with your problem. I have a similar setup. Squid, transparent, log sni, remote blacklists, local whitelists.

I tried several settings and finally had to disable transparent proxy for https. Maybe a bug in squid.

Even local domains were whitelisted, squid generated a self signed certificate in log only mode. Pretty strange

I hope it will be fixed sometime. ATM just users that have static proxy or get it via wpad or option 252 are logged by proxy.
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: t.mayer on April 05, 2020, 08:49:10 am
WPAD and static proxy are also working for me - but with mobile clients it would be much more easier to use sni.

I found a similar problem on pfsense-forum:
https://forum.netgate.com/topic/128492/we-are-trying-to-work-with-squid-proxy-squidguard-but-whitelist-dont-work

I really think it's  bug.
The question now: is it a squid-bug or an opnsense-bug?
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: Amr on April 06, 2020, 09:27:34 am
@t.mayer

Disclaimer: You should back-up your configuration before attempting to edit, Also you proceed at your own risk and I'm not responsible if you break anything attempting these changes, even though I haven't tested the mentioned configuration but theoretically they should work fine, have fun.

Well if you still want to unblock Instagram you can:
1-login into the CLI.
2- choose Shell (option 8 ).
3- install nano(editor) :
Code: [Select]
pkg install nano4- navigate to squid acl folder :
Code: [Select]
cd /usr/local/etc/squid/acl (to go back a directory use cd ..)
5-
Code: [Select]
ls will display the contents of the folder.
6- type
Code: [Select]
nano shallalist (if you entered shallalist as the name of the remote blacklist or whatever it's)
warning: be careful not to miss something by mistake .
7- navigate to Instagram entry by pressing ctrl+w and search for instagram (note there's probably a couple of entries for Instagram cdninstagram, unblockinstagram, etc, to search for them simply press ctrl+w again and enter)
8-comment or delete the Instagram entries int the list.
9-ctrl+x to exit the editor it'll ask you if you want to save changes or not (type y to accept and hit enter)
10- you can exit the shell by typing exit.
11- reload squid and test Instagram.

for advanced operations, you can check FreeBSD commands.

Hopefully, It'll work.
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: t.mayer on April 06, 2020, 10:34:15 am
Many thanks for your effort, but manually editing shallalist will not solve the bug. And when shallalist gets updated all changes are gone.

I really would like to initiate a discussion in this thread about eliminating the bug:
the whitelist ist not considered when using remote blacklist in combination with transparent-ssl-sni squid-setting

And as hbc confirms: it really seems to be a bug!
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: hbc on April 07, 2020, 03:15:29 pm
@t.mayer:

ATM I have the problem that squid randomly bumps instead of splicing. Di you have similar issues? My setup is:

Code: [Select]
# configure bump
ssl_bump peek bump_step1 all
ssl_bump splice all
ssl_bump peek bump_step2 all
ssl_bump splice bump_step3 all
ssl_bump bump

The standard "log only" setting. But I just got a calls that ssl pages cannot be retrieved and when made teamviewer sessions, I saw that there was self-signed certificates issued. But why?
Title: Re: ACL > Whitelist not not considered when using Remote ACL
Post by: t.mayer on April 15, 2020, 02:21:24 pm
I have exactly the same settings.