Web proxy transparent filtering with SSL creates some issues

[dirkp]

Dear

I installed OpnSense 19.7.3 (I noticed today there is a 19.7.4, I will upgrade very soon). I created a transparent proxy both for HTTP and HTTPS. I strictly followed the manual and as fas as HTTP goes everything seems to be working fine.

As for the HTTPS I noticed some issues, not sure if these are the result my "misconfiguration" or expected behaviour or simply bugs.

There are a few conditions that are paramount for me in order to be able to maintain access throughout the networdk :
1) No certificates to be installed on the client machines. This is absolute horror and unmaintainable. That means that step 8 in https://docs.opnsense.org/manual/how-tos/proxytransparent.html is not executed.
[sitenote : can't I buy a certificate (not sure which kind) somewhere to avoid the creation of an own Certificate Authority (CA setting), that would be brilliant, I simply think this is a bit of a mess]
2) No maintenance in the " SSL no bump sites", again, this is not feasable from a maintenance point of view. That means that step 6 in https://docs.opnsense.org/manual/how-tos/proxytransparent.html is not an option, and hence not executed.
[sitenote, see further belwo] simply not executing step 6 results in no filtering, and I do want the filtering in SSL]
3) I want to apply (a rather stringent) Remote Access Control Lists to be applicable both in  HTTP and HTTPS (I already installed the yoyo (as suggested by the manual at https://docs.opnsense.org/manual/how-tos/cachingproxy.html#remote-black-list-ad-blocking), but I also want to use the UT1 as suggested by the manual here : https://docs.opnsense.org/manual/how-tos/proxywebfilter.html#step-2-configure-blacklist

Now these are the issues/questions I have
1) documentation : would it be possible in https://docs.opnsense.org/manual/how-tos/proxytransparent.html to indicate for step 6 how to avoid the bumping (cf my condition 2) & do enable the web filtering. I understand that you need to select the "SNI" option, but this is nowhere mentioned in the manuals. On the web some people refer to a field called "SSL/IP only" but this field no longer seem to exist.
Can someone confirm that this is the correct setting ? I might have missed something, but for my testing revealed to do transparent SSL filtering I needed to enable the "Enable SSL mode " option AND the "Log SNI information only" option right beneath it ([UPDATE: I added the correct label of the field and removed comment:ENDUPDATE], an option which is not mentioned in the manual), otherwise I get SSL errors all around the place (related to not trusted certificate), and the filtering seems simply not to take place.
2) As explained in 1 - I enabled "Enable SSL mode " option AND the "SNI". But this results in some weird behaviour :
- HTTP is keeps on working fine
- HTTPS : if I have a site in the enabled remote black list indicated by IP address, I receive an "void certifcate" & certifcation warning instead of squid redirecting me to the "not allowed page". If I access the same IP address with HTTP I get the "not allowed page"
- HTTPS : if I go to a site in the enabled remote black list indicated by a domain (style : .example.com) : I get PR_CONNECT_RESET ERROR, instead being nicely forwarded to the "not allowed page". At least the access is blocked, but I think it is not a nice handling, as the user is faced with somehting he does not understand instead of the "not allowed page" which he can interpret (and contact me about)
3) with big lists, I have the impression that not all "blocked" sites are taken into account. For instance, using the complete UT1 list as suggested by the manuel, some sites are blocked (with the PR_... error) and some are still accessible, which suggests that the list seems not be used "completely".
[UPDATE: logs from access.log with .mail.google.com in my Remote access Control List
1570783752.597      9 192.168.x.120 NONE/200 0 CONNECT 172.217.17.37:443 - HIER_NONE/- -
1570783752.599      1 192.168.x.120 TCP_DENIED/403 3721 CONNECT mail.google.com:443 - HIER_NONE/- text/html
1570783752.599      0 192.168.x.120 NONE/000 0 NONE error:transaction-end-before-headers - HIER_NONE/- -

-> I think the problem is related to the last line

:ENDUPDATE]

4) finally, when controlling the squid.conf the ssl_bump setting seems to be weird (or not consistent) if I compare this with suggestions on the squid-cache forum. But that is already another journey. I will create a separate question for this once things seem more clear.

[UPDATE: squid.conf
# setup ssl bump acl's
acl bump_step1 at_step SslBump1
acl bump_step2 at_step SslBump2
acl bump_step3 at_step SslBump3
acl bump_nobumpsites ssl::server_name "/usr/local/etc/squid/nobumpsites.acl"

# configure bump
ssl_bump peek bump_step1 all
ssl_bump splice all
ssl_bump peek bump_step2 all
ssl_bump splice bump_step3 all
ssl_bump bump

the configure of the bumps do not seem correct especially peek bump_step2 etc has no meaning

:ENDUPDATE]


Oh one more thing : is a socks proxy not an option ? And is it available by default or in the plugins ?

Happy to hear your input
rgds
dirkp

[hbc]

Oh one more thing : is a socks proxy not an option?

I think, this is a question that can only be answered by you. But socks proxy have to be explicitly configured in client software and will not work transparent like squid. No I guess no option for your zero-configuration proxy.

And is it available by default or in the plugins ?

What about just opening  "System: Firmware" and grepping plugins for 'socks'? No internet connection?

os-shadowsocks   1.0   31.9KiB   Secure socks5 proxy

[fabian]

But socks proxy have to be explicitly configured in client software and will not work transparent like squid. No I guess no option for your zero-configuration proxy.

That is not the full truth. Redsocks for example is a software that is proxying a transparently to SOCKS.

[dirkp]

Thanks for the feedback.

I understand Socks is not an option, worth of digging into, but not in the current situation

I added some updates in the original question being:
I added the result of the access.log
I added the configuration of the bump settings of the squid.conf
I updated the label of the "SNI"

tx
dirkp

[dirkp]

Hi all

upgraded to 21.7.6 and all problems around buggy config of the transparant proxy seem to have disappeared. It works smoothly and with no issues. Also error pages in case of blocking by ACL in https work OK now.

I get a SSL_ERROR_RX_RECORD_TOO_LONG on https://quad9.net but this is the only site currently resulting in this error. I do not mind. And certainly is not related to the original problem.

problem & questions closed