ACL > Whitelist not not considered when using Remote ACL

[t.mayer]

I have configured the OPNsense-Webproxy with shallalist as Remote ACL.
For some exceptions i always used the Whitelist under Access Control List > Whitelist.
When i try to open a domain blocked by shallalist-category but with a corresponding entry in the whitelist, the domain still will be blocked.

Version of OPNSense: 19.7.8

Forward-Proxy-Config:
- Interface: LAN
- Port: 3128 / SSL: 3129
- Transparent http-Proxy
- SSL inspection
- SNI only

Thanks for your help!

Greeds
Tom

[t.mayer]

May I ask again if there is anybody with an idea?

[Amr]

Weird works for me, try adding a wild card for the domain aka add a "." before domain name ex: .whatsapp.net and stopping and restarting the service.

Check the certificate of the domain for aliases and try adding them, check logs to see if the website is trying to reach another domain for grabbing  code or something.

Since you are using sni logging only it shouldn't be a problem but try adding the domain in the no bump sites list.

[t.mayer]

@Amr: Thanks for your answer.

The problem to me still exists.
I found out that it has to do something with the ssl/sni-only-settings.

Here is what i have tested:

• Remote-ACL: Shallalist with only one aktive category: socialnet
• URL for testing: instagram.com

Case 1: No Entry in ACL-Whitelist
Setting Browser to use Proxy-Port 3128
> instagram.com can't be reached
> functioning as expected

Setting Browser to not use Proxy (Proxy now transparent via SSL/SNI only)
> instagram.com can't be reached
> functioning as expected

Case 2: Entry in ACL-Whitelist: instagram.com
Setting Browser to use Proxy-Port 3128
> instagram.com can be reached
> functioning as expected

Setting Browser to not use Proxy (Proxy now transparent via SSL/SNI only)
> instagram.com can't be reached
> BUG?
> instagram.com as entry in SSL no bump sites has also no effect on this

Hopefully my description is understandable.

Greeds
Tom

[Amr]

From your description

Setting Browser to not use Proxy (Proxy now transparent via SSL/SNI only)
> instagram.com can't be reached

I assume there's a problem with NAT port forwarding so did you set it up properly? (attach a pic of your rules)

If NAT is not the problem can you access other websites? (after setting 'no proxy' in browser)

If you can access other websites what kind of error does the proxy return?

[t.mayer]

@Amr: Thanks to your response!

I think you got me wrong. The Proxy-NAT works as expected. I can access all websites with proxy except those that I have blocked by an active shallalist-category.

NAT-Rules:

• Interface: LAN / Proto: TCP / Destination-Port: 80 / NAT-IP: 127.0.0.1 / NAT-Port: 3128
• Interface: LAN / Proto: TCP / Destination-Port:443 / NAT-IP: 127.0.0.1 / NAT-Port: 3129

The problem is that the whitelist (e. g. for instagram.com) is not considered  when using a remote acl (e. g. shallalist with active category socialnet) when using proxy in transparent/ssl/sni-mode.

The whitelist is only considered when the proxy is used in non-transparent mode.

Greeds
Tom

[Amr]

The whitelist is only considered when the proxy is used in non-transparent mode.

Can you reply with the error that squid returns when Instagram gets blocked?

Also, you might want to reset cache under "Support" tab and restart the proxy

[t.mayer]

The error is - as always when blocking https - not a squid-error but a browser-certificate-error:
NET::ERR_CERT_AUTHORITY_INVALID

I have reset the cache and restarted squid lots of times...

Can you test, if a whitelisted entry works when blocking with transparent proxy with ssl and sni on your installation?

I think its a bug!

[Amr]

The error is - as always when blocking https - not a squid-error but a browser-certificate-error:
NET::ERR_CERT_AUTHORITY_INVALID

weird are you using an internal certificate?

In some rare cases that I have encountered the antivirus was to blame, try disabling it.

Can you test, if a whitelisted entry works when blocking with transparent proxy with ssl and sni on your installation?

I did test it, works fine, but to be frank I didn't reset the cache.

my current setup is a transparent proxy with an internal certificate and SSL inspection with some sites in SSL no bump sites, but I don't use shallist but rather custom rules (I did use shallalist and it worked fine too)

A word of caution: using Log SNI information only won't block VPN connections made on https port.

[t.mayer]

@Amr

Thanks for trying to help me. But I think - sorry if this is impolite - you can not help me until you really want to recognize my problem.

You say that it is weird that i am using internal certificate - 5 lines further you explain that you are using internal certificate as well. Further you claim that you have tested the same installation, but you don't use a remote acl like shalla list.

The problem is neither the internal certificate nor antivirus (not used).
The problem is the shallalist in combination with transparent/ssl/sni-proxy: whitelisted entrys are blocked. All the rest is working as expected.

Greeds an thanks for you help.

[Amr]

sorry if this is impolite

no offense taken.

You say that it is weird that I am using internal certificate

well, what I meant that in your setup certificates shouldn't be used since you don't bump site at all.

Further you claim that you have tested the same installation, but you don't use a remote acl like shalla list.

I did test Log SNI information only a while ago and with shallalist and it was working, but since I found that VPN connections on https ignored all my filters I used SSL inspection only.

I'm no guru (and pretty sure you already have configured your proxy correctly) but these steps are aimed to troubleshoot your installation:
1-check Instagram's certificate to see who signed it should be DigiCert Inc, not your internal certificate.
2-check also the time of the opnsense machine (from the dashboard) and your machine.
3-check if you entered the LAN subnet in  Allowed Subnets under access control, maybe add your firewall lan address in the Unrestricted IP addresses too.
4-reboot your firewall and keep a look for squid configuration sanity check.

The problem is the shallalist in combination with transparent/ssl/sni-proxy: whitelisted entrys are blocked. All the rest is working as expected.

Well if the previous steps didn't work then I can think of a work around but it's not easy if you aren't familiar with linux, simply put you can edit the shallalist manually and remove instagram from the list.

[hbc]

@t.mayer: you are not alone with your problem. I have a similar setup. Squid, transparent, log sni, remote blacklists, local whitelists.

I tried several settings and finally had to disable transparent proxy for https. Maybe a bug in squid.

Even local domains were whitelisted, squid generated a self signed certificate in log only mode. Pretty strange

I hope it will be fixed sometime. ATM just users that have static proxy or get it via wpad or option 252 are logged by proxy.

[t.mayer]

WPAD and static proxy are also working for me - but with mobile clients it would be much more easier to use sni.

I found a similar problem on pfsense-forum:
https://forum.netgate.com/topic/128492/we-are-trying-to-work-with-squid-proxy-squidguard-but-whitelist-dont-work

I really think it's  bug.
The question now: is it a squid-bug or an opnsense-bug?

[Amr]

@t.mayer

Disclaimer: You should back-up your configuration before attempting to edit, Also you proceed at your own risk and I'm not responsible if you break anything attempting these changes, even though I haven't tested the mentioned configuration but theoretically they should work fine, have fun.

Well if you still want to unblock Instagram you can:
1-login into the CLI.
2- choose Shell (option 8 ).
3- install nano(editor) :

Code Select Expand

pkg install nano
4- navigate to squid acl folder :

Code Select Expand

cd /usr/local/etc/squid/acl (to go back a directory use cd ..)
5-

Code Select Expand

ls will display the contents of the folder.
6- type

Code Select Expand

nano shallalist (if you entered shallalist as the name of the remote blacklist or whatever it's)
warning: be careful not to miss something by mistake .
7- navigate to Instagram entry by pressing ctrl+w and search for instagram (note there's probably a couple of entries for Instagram cdninstagram, unblockinstagram, etc, to search for them simply press ctrl+w again and enter)
8-comment or delete the Instagram entries int the list.
9-ctrl+x to exit the editor it'll ask you if you want to save changes or not (type y to accept and hit enter)
10- you can exit the shell by typing exit.
11- reload squid and test Instagram.

for advanced operations, you can check FreeBSD commands.

Hopefully, It'll work.

[t.mayer]

Many thanks for your effort, but manually editing shallalist will not solve the bug. And when shallalist gets updated all changes are gone.

I really would like to initiate a discussion in this thread about eliminating the bug:
the whitelist ist not considered when using remote blacklist in combination with transparent-ssl-sni squid-setting

And as hbc confirms: it really seems to be a bug!