Transparent HTTP proxy SSL problem

[kurczak]

Hi,

I'm trying to block some https websites like facebook. I'm doing this by the book https://docs.opnsense.org/manual/how-tos/proxytransparent.html. I have enable proxy, Enable Transparent HTTP proxy, Enable SSL mode with generated certificate and created firewall rules. I've added facebook to the blacklist then export/import generated cert to windows and firefox. It works facebook is blocked but some services like windows update are blocked too and I don't know why? Is there other way to block https websites? maybe without certificate? What do I miss with configuration?

Logs:

Code: [Select]

1488923833.907 278075 192.168.8.3 TAG_NONE/409 0 CONNECT fe2.update.microsoft.com:443 - HIER_NONE/- text/html;charset=utf-8
1488923683.489 1234 192.168.8.3 TCP_TUNNEL/200 4780 CONNECT watson.telemetry.microsoft.com:443 - ORIGINAL_DST/65.55.252.202 -
1488923683.382 1128 192.168.8.3 TCP_TUNNEL/200 4796 CONNECT watson.telemetry.microsoft.com:443 - ORIGINAL_DST/65.55.252.202 -
1488923616.057 61580 192.168.8.3 TCP_TUNNEL/200 32042 CONNECT sls.update.microsoft.com:443 - ORIGINAL_DST/134.170.51.188 -
1488923381.233 0 192.168.8.3 TAG_NONE/503 0 POST https://watson.telemetry.microsoft.com/Telemetry.Request - HIER_NONE/- text/html
1488923375.749 1 192.168.8.3 TAG_NONE/503 4443 GET https://sls.update.microsoft.com/SLS/%7B7971F918-A847-4430-9279-4A52D1EFE18D%7D/x64/6.3.9600.0/0? - HIER_NONE/- text/html
1488923373.966 0 192.168.8.3 TAG_NONE/503 4443 GET https://sls.update.microsoft.com/SLS/%7B7971F918-A847-4430-9279-4A52D1EFE18D%7D/x64/6.3.9600.0/0? - HIER_NONE/- text/html
1488923350.566 61 192.168.8.3 TCP_DENIED/403 4976 GET https://ieonline.microsoft.com/ieflipahead/ie10/rules.xml? - HIER_NONE/- text/html
1488923347.482 0 192.168.8.3 TAG_NONE/503 4222 POST https://watson.telemetry.microsoft.com/Telemetry.Request - HIER_NONE/- text/html
1488923347.468 0 192.168.8.3 TAG_NONE/503 4447 POST https://watson.telemetry.microsoft.com/Telemetry.Request - HIER_NONE/- text/html
1488923341.795 0 192.168.8.3 TAG_NONE/503 389 HEAD https://fe2.update.microsoft.com/v10/3/windowsupdate/selfupdate/wuident.cab? - HIER_NONE/- text/html
1488923341.367 0 192.168.8.3 TAG_NONE/503 389 HEAD https://fe2.update.microsoft.com/v10/3/windowsupdate/selfupdate/wuident.cab? - HIER_NONE/- text/html
1488923340.940 0 192.168.8.3 TAG_NONE/503 389 HEAD https://fe2.update.microsoft.com/v10/3/windowsupdate/selfupdate/wuident.cab? - HIER_NONE/- text/html
1488923340.487 0 192.168.8.3 TAG_NONE/503 389 HEAD https://fe2.update.microsoft.com/v10/3/windowsupdate/selfupdate/wuident.cab? - HIER_NONE/- text/html
1488923315.705 134 192.168.8.3 TCP_MISS/304 498 GET https://iecvlist.microsoft.com/IE11/1434748155000/iecompatviewlist.xml - ORIGINAL_DST/93.184.221.200 -
1488922013.067 1269248 192.168.8.3 TAG_NONE/409 0 CONNECT sls.update.microsoft.com:443 - HIER_NONE/- text/html;charset=utf-8
1488922013.067 846177 192.168.8.3 TAG_NONE/409 0 CONNECT sls.update.microsoft.com:443 - HIER_NONE/- text/html;charset=utf-8
1488922013.067 968418 192.168.8.3 TAG_NONE/409 0 CONNECT sls.update.microsoft.com:443 - HIER_NONE/- text/html;charset=utf-8
1488921803.282 1070 192.168.8.3 TCP_TUNNEL/200 4796 CONNECT watson.telemetry.microsoft.com:443 - ORIGINAL_DST/65.55.252.202 -
1488921803.181 970 192.168.8.3 TCP_TUNNEL/200 4780 CONNECT watson.telemetry.microsoft.com:443 - ORIGINAL_DST/65.55.252.202 -
1488921542.401 62 192.168.8.3 TCP_MISS/200 14915 GET http://static.solvusoft.com/errors/images/logo-microsoft.png - ORIGINAL_DST/2.18.212.136 image/png
1488921542.260 103 192.168.8.3 TCP_MISS/200 52809 GET http://static.solvusoft.com/images/microsoft-award.jpg? - ORIGINAL_DST/2.18.212.136 image/jpeg
1488921542.106 92 192.168.8.3 TCP_MISS/200 43731 GET http://www.solvusoft.com/errors/images/download/pl_runtime-errors_80072EE2_80072ee2-microsoft-update-error-80072ee2_.png - ORIGINAL_DST/2.18.212.139 image/png
1488921542.102 86 192.168.8.3 TCP_MISS/200 10202 GET http://static.solvusoft.com/errors/images/microsoft-partner/pl.png? - ORIGINAL_DST/2.18.212.136 image/png
1488921541.623 214 192.168.8.3 TCP_MISS/200 21737 GET http://www.solvusoft.com/pl/errors/b%C5%82%C4%99dy-czasu-wykonania/microsoft-corporation/microsoft-update/80072ee2-microsoft-update-error-80072ee2/ - ORIGINAL_DST/2.18.212.139 text/html
1488921527.758 856 192.168.8.3 TCP_MISS/200 683 GET http://c.microsoft.com/trans_pixel.aspx? - ORIGINAL_DST/173.223.169.164 image/gif
1488921526.883 427 192.168.8.3 TCP_MISS/200 683 GET http://c.microsoft.com/trans_pixel.aspx? - ORIGINAL_DST/173.223.169.164 image/gif
1488921526.385 65 192.168.8.3 TCP_MISS/200 739 GET http://hs.windows.microsoft.com/scripts/4.2/helphub/ClientBiSettings.HelpHub.js? - ORIGINAL_DST/23.32.16.212 application/x-javascript
1488921526.309 82 192.168.8.3 TCP_MISS/404 291 GET http://hs.windows.microsoft.com/scripts/4.2/helphub/wol.hh.search.js - ORIGINAL_DST/23.32.16.212 text/html
1488921526.207 74 192.168.8.3 TCP_MISS/200 20055 GET http://hs.windows.microsoft.com/scripts/4.2/helphub/wol.common.helphub.js - ORIGINAL_DST/23.32.16.212 application/x-javascript
1488921525.842 183 192.168.8.3 TCP_MISS/200 24872 GET http://ajax.microsoft.com/ajax/4.0/4/MicrosoftAjax.js - ORIGINAL_DST/93.184.221.200 application/x-javascript
1488921525.425 62 192.168.8.3 TCP_MISS/200 378 GET http://res1.windows.microsoft.com/resbox/en/windows/main/55bf9201-0238-4ccf-8c80-44ad74319cf7_21.css - ORIGINAL_DST/23.211.158.3 text/css
1488921525.329 173 192.168.8.3 TCP_MISS/200 9057 GET http://res1.windows.microsoft.com/resources/4.2/helphub/shared/css/helphub_ltr.css - ORIGINAL_DST/23.211.158.3 text/css
1488921524.768 271 192.168.8.3 TCP_MISS/200 29099 GET http://hs.windows.microsoft.com/hhweb/content/m-pl-PL_en-US/p-6.3/id-search/? - ORIGINAL_DST/23.32.16.212 text/html
1488921428.787 60523 192.168.8.3 TCP_TUNNEL/200 4206 CONNECT settings-win.data.microsoft.com:443 - ORIGINAL_DST/40.77.226.249 -
1488921415.270 1106 192.168.8.3 TCP_TUNNEL/200 4796 CONNECT watson.telemetry.microsoft.com:443 - ORIGINAL_DST/65.55.252.202 -
1488921411.327 297 192.168.8.3 TCP_MISS/200 767 POST http://statsfe2.update.microsoft.com/ReportingWebService/ReportingWebService.asmx - ORIGINAL_DST/65.52.108.153 text/xml
1488921411.015 372 192.168.8.3 TCP_MISS/200 767 POST http://statsfe2.update.microsoft.com/ReportingWebService/ReportingWebService.asmx - ORIGINAL_DST/65.52.108.153 text/xml
1488921410.539 1503 192.168.8.3 TCP_TUNNEL/200 32074 CONNECT sls.update.microsoft.com:443 - ORIGINAL_DST/157.56.77.149 -
1488921313.153 670 192.168.8.3 TCP_MISS/200 23272 GET http://www.update.microsoft.com/windowsupdate/v6/shared/js/content.js? - ORIGINAL_DST/134.170.58.221 application/javascript
1488921312.651 169 192.168.8.3 TCP_MISS/200 3319 GET http://www.update.microsoft.com/windowsupdate/v6/shared/js/tgar.js? - ORIGINAL_DST/134.170.58.221 application/javascript
1488921312.446 337 192.168.8.3 TCP_MISS/200 4196 GET http://www.update.microsoft.com/windowsupdate/v6/thanks.aspx? - ORIGINAL_DST/134.170.58.221 text/html
1488921312.100 683 192.168.8.3 TCP_MISS_ABORTED/200 40451 GET http://windowsupdate.microsoft.com/windowsupdate/v6/shared/js/webcomtop.js? - ORIGINAL_DST/157.55.240.94 application/javascript
1488921312.095 676 192.168.8.3 TCP_MISS_ABORTED/200 17411 GET http://windowsupdate.microsoft.com/windowsupdate/v6/shared/js/resultslist.js? - ORIGINAL_DST/157.55.240.94 application/javascript
1488921312.094 677 192.168.8.3 TCP_MISS/200 38742 GET http://windowsupdate.microsoft.com/windowsupdate/v6/shared/js/commontop.js? - ORIGINAL_DST/157.55.240.94 application/javascript
1488921312.083 668 192.168.8.3 TCP_MISS/200 47126 GET http://windowsupdate.microsoft.com/windowsupdate/v6/shared/js/redirect.js? - ORIGINAL_DST/157.55.240.94 application/javascript
1488921311.929 509 192.168.8.3 TCP_MISS/200 8020 GET http://windowsupdate.microsoft.com/windowsupdate/v6/shared/js/spupdateids.js? - ORIGINAL_DST/157.55.240.94 application/javascript
1488921311.767 180 192.168.8.3 TCP_MISS/200 25596 GET http://windowsupdate.microsoft.com/windowsupdate/v6/shared/images/banners/favicon.ico - ORIGINAL_DST/157.55.240.94 image/x-icon
1488921311.584 170 192.168.8.3 TCP_MISS/200 3319 GET http://windowsupdate.microsoft.com/windowsupdate/v6/shared/js/tgar.js? - ORIGINAL_DST/157.55.240.94 application/javascript
1488921311.351 349 192.168.8.3 TCP_MISS/200 15776 GET http://windowsupdate.microsoft.com/windowsupdate/v6/default.aspx? - ORIGINAL_DST/157.55.240.94 text/html
Regards

[monstermania]

Hi,
i recommend you to add the windows update url's to the proxy whitelist!
Here you get an list whitch url's are used by windows update: https://technet.microsoft.com/en-gb/library/bb693717.aspx

best regards
Dirk

[kurczak]

Unfortunately it dosen't work. I've added domains to the white list and Windows Update still don't work. If I add .microsoft.com and .windowsupdate.microsoft.com to the  SSL no bump sites Windows Updater searching a bit longer but finally it fail.

Logs:

Code: [Select]

2017/03/08 16:14:44 kid1| SECURITY ALERT: on URL: sls.update.microsoft.com:443
2017/03/08 16:14:44 kid1| SECURITY ALERT: Host header forgery detected on local=65.55.138.149:443 remote=192.168.8.3:51091 FD 15 flags=33 (local IP does not match any domain IP)
2017/03/08 16:09:41 kid1| SECURITY ALERT: on URL: sls.update.microsoft.com:443
2017/03/08 16:09:41 kid1| SECURITY ALERT: Host header forgery detected on local=157.56.96.58:443 remote=192.168.8.3:51059 FD 12 flags=33 (local IP does not match any domain IP)
2017/03/08 16:07:39 kid1| SECURITY ALERT: on URL: sls.update.microsoft.com:443
2017/03/08 16:07:39 kid1| SECURITY ALERT: Host header forgery detected on local=157.56.96.58:443 remote=192.168.8.3:51050 FD 37 flags=33 (local IP does not match any domain IP)

1488986048.938 60487 192.168.8.3 TCP_TUNNEL/200 4215 CONNECT vortex-win.data.microsoft.com:443 - ORIGINAL_DST/40.77.226.250 -
1488985966.531 188 192.168.8.3 TCP_MISS/200 453 GET http://service.weather.microsoft.com/appex/DesktopTile/PreInstallLiveTile? - ORIGINAL_DST/2.17.22.235 -
1488985907.502 1031 192.168.8.3 TCP_TUNNEL/200 4796 CONNECT watson.telemetry.microsoft.com:443 - ORIGINAL_DST/65.55.252.202 -
1488985864.087 6827 192.168.8.3 TCP_TUNNEL/200 12084 CONNECT watson.telemetry.microsoft.com:443 - ORIGINAL_DST/65.55.252.202 -

btw I see a lot of errors with ssl in logs from many webs and services. Is there any better solution to block websites without certificates?

[fabian]

Are you using LibreSSL or OpenSSL?

[kurczak]

OpenSSL

[fabian]

This is a security feature of squid. Make sure your client and your proxy use the same DNS server, which returns the same IP.
On a mismatch, the connection is rejected.

[kurczak]

This is a security feature of squid. Make sure your client and your proxy use the same DNS server, which returns the same IP.
On a mismatch, the connection is rejected.

So if I change Proxy DNS via General Proxy Settings -> Use alternate DNS-servers to the same as Windows client (for example) the connection will not be rejected?

[fabian]

It depends, if you get the same IP, then yes.

[kurczak]

Ok, Thank You for help. I will test it. For now as workaround I'm using proxy only for http and aliases/rules for blocking https webs like facebook

[Julien]

I am facing the same issue,
just added the local dns server to the Use alternate DNS-servers however the windows updates are not working yet.
I hope someone can help.
everytime I have to delete the nat rules to get the windows update working.

[anex128]

Hello, i am facing the same issue. Can't use windows updates with SSL Proxy enables. I have imported the generated certificate onto the machine and websites work correctly. My cert is being recognized and everything works except Windows Updates. Im geetting an error (0x0801901f7). Seems like its a problem with certificate? I have added all windows updates domains into SSL no bump list. Its very problematic for me as i need to use Transparent proxy to block porn sites in my EDU organization...

[bartjsmit]

Why don't you run WSUS on an internal server and direct your workstations to that by GPO. This allows more control over updates and reduces WAN use.

Bart...

[anex128]

That will be my last resort. The problem is that i am new here and the previous IT guy had no regard for software standards... For now there are many computers with diffrent systems, W7, W8, W8.1 and W10. Aside from that Office 2010, 13 and 16. Add to that W2016 server and we are looking at more than 200GB of data just for WSUS server. For now i dont have that kind of space and no spare Windows Server licenses

[anex128]

So is this problem persistent/impossible to fix? Or maybe some OPNsense guru have some kind of a workaround/solution for this?

[BadSamaritan]

For any sites that I have that don't behave properly with the transparent SSL proxy i create an ALIAS containing the destination netblock or hosts with the issue and then create "No RDR (NOT)" rules under Firewall -> NAT -> Port forward and put the alias in the destination. It may take a while to get the alias right given this is windows update. I'd start with the names failing in the logs. I have one rule now for crashplan backups so they don't go thru the proxy, and i have another for iboss on my sons school supplied ipad and it works well.
 
You'll loose the ability to cache the updates but at least they will work.  Otherwise you are looking at forcing proxy settings using manual proxy config, or a wpad server, or sending the proxy as a dhcp option(not sure you can do this with opnsense)