HTTPS traffic not working correctly

[vico1959]

So I am having a really hard time getting the web proxy to work properly. I cannot get HTTPS traffic to work correctly no matter what I try when the firewall and NAT rules are redirecting to the proxy. With rules enabled for HTTP (80) only using a transparent setup, normal HTTP traffic works fine and directly entered HTTPS URLs work fine but any HTTP URLs that are being redirected to HTTPS sites never connect. If I try and enable transparent HTTPS rules then no HTTPS traffic works at all whether directly entered or not. If I disable proxy rules then everything works as expected. I really don't want to have to forego all web filtering just so we can have normal HTTPS redirects happening. Can somebody please help me before I pull what's left of my hair out? Thank you in advance for any assistance you can give me.

[fabian]

you are likely missing the https port forward.

[Amr]

hello vico1959,

Can you post your nat and squid(web proxy) configuration?

[vico1959]

@Fabian,

I do have it there but I have to disable it along with the firewall rules to get any HTTPS traffic to work. In fact with all the rules enabled according to the instructions on the OPNSense website here I cannot get any HTTPS traffic at all.

@Amr,

Do you want screen shots or is there a better way to export those settings for you to see?

Also, in case this makes any difference this is in front of a Windows Active Directory domain network and the main DNS is the DHCP/DNS server which is not the firewall itself.

[Amr]

Do you want screen shots or is there a better way to export those settings for you to see?

Screenshots are fine.

Also, in case this makes any difference this is in front of a Windows Active Directory domain network and the main DNS is the DHCP/DNS server which is not the firewall itself.

I'm not getting you straight just to make things clear is this what your setup looks like?
Router-->Windows server working as DHCP/DNS server-->OpnSense-->lan?

[vico1959]

No, sorry for the confusion. It is a standard setup with the OPNsense firewall connected directly to the ISP but I just wanted to let you know that the LAN is using the DHCP and DNS on the Windows server instead of the firewall providing those roles in case that brought up any other ideas and made any difference for the solution. I didn't think it should matter but just in case.

I will try to attach the appropriate screenshots. Keep in mind that for now all of the needed rules are disabled so people can work without getting blocked so just know that I had them enabled but couldn't get through as described previously. This includes the HTTPS settings in squid.

Thank you again and let me know if you need anything else.

[Amr]

It is a standard setup with the OPNsense firewall connected directly to the ISP but I just wanted to let you know that the LAN is using the DHCP and DNS on the Windows server instead of the firewall

Do you have DHCP enabled on opnsense lan interface or in relay mode?

before proceeding with transparent proxy troubleshooting first check that the caching proxy is working fine.
1- Disable transparent proxy.
2- Configure firefox proxy settings to use the proxy by :
   a- Choosing ->"Manual proxy configuration"
   b- Enter the ip in http proxy and port number
   c- Check "Also use this proxy for FTP and HTTPS"
3- Check the the proxy is working like expected with web filtering.

[lfirewall1243]

No, sorry for the confusion. It is a standard setup with the OPNsense firewall connected directly to the ISP but I just wanted to let you know that the LAN is using the DHCP and DNS on the Windows server instead of the firewall providing those roles in case that brought up any other ideas and made any difference for the solution. I didn't think it should matter but just in case.

I will try to attach the appropriate screenshots. Keep in mind that for now all of the needed rules are disabled so people can work without getting blocked so just know that I had them enabled but couldn't get through as described previously. This includes the HTTPS settings in squid.

Thank you again and let me know if you need anything else.

Don't share the screenshots in a Zip File, i think nobody is downloading them.

Take a screenshot of your NAT, Rules and Squid config and upload them as images

[vico1959]

Do you have DHCP enabled on opnsense lan interface or in relay mode?

Neither. I do not have anything DHCP related turned on at the firewall. All DHCP is being served by the Windows Domain controller. When it serves out the DHCP it is serving out the firewall address as the gateway. Also any DNS that the firewall is processing is strictly by interception. The Windows domain controller is also acting as the DNS server so again I'm not using the DNS directly from the firewall at all.

before proceeding with transparent proxy troubleshooting first check that the caching proxy is working fine.
1- Disable transparent proxy.
2- Configure firefox proxy settings to use the proxy by :
   a- Choosing ->"Manual proxy configuration"
   b- Enter the ip in http proxy and port number
   c- Check "Also use this proxy for FTP and HTTPS"
3- Check the the proxy is working like expected with web filtering.

 

I did this originally before setting up the transparent mode and HTTP worked fine but I had the same results with HTTPS traffic. Nothing got through at all so I did not have a chance to test whether it would properly redirect to HTTPS from an HTTP request. Currently I have all web filtering rules disabled because that is the only way I can get HTTPS traffic to work properly.

Don't share the screenshots in a Zip File, i think nobody is downloading them.

Take a screenshot of your NAT, Rules and Squid config and upload them as images

Okay sorry about that. Do you mean insert them into the message or just don't zip them? Is there a way to upload here on the forum inline in the message or do I need to use an external site and link to it? The insert image button only inserts the HTML code so I am assuming the latter is what you mean?

[vico1959]

Hey guys,

I finally figured out that attaching files automatically puts them in the message. I don't know how I missed that but I have posted them below. I really need to get this figured out as I need to start blocking and filtering some specific things but I can't turn it back on until I have HTTPS access working, even if it isn't being filtered I at least need it able to be accessed but it stops working every time I enable the filtering. Please help again. I'm sorry to bug you guys but I really wish some parts of this configuration were much more straight forward. As far as I know I have followed exactly what is in the instructions and I can't figure out why it is not working. Thanks again.

[vico1959]

Also, let me make one more attempt at clarifying the issue. I can successfully setup transparent web filtering for HTTP and skip HTTPS proxy altogether BUT here is what I get. HTTP filtering works great, HTTPS directly entered URLs work great. What doesn't work is HTTP to HTTPS redirects. So for instance someone enters google.com or www.google.com and Google redirects that to https://www.google.com but the browser never takes you there, it just spins around trying and then times out. IF you manually enter https://www.google.com then it works fine. I would eventually like to get the filtering for HTTPS working without inspection BUT for now if I could just get the dang HTTPS redirect to function correctly I could at least work within those limitations for now. Hopefully this plainly describes my issues. Thanks again.

P.S. - What's killing me is that I have successfully configured more complex things such as automatic failover WAN with a wireless interface connected to a LTE hotspot but this simple web filtering issue is kicking by butt!

[vico1959]

Okay so I figured out how stupid I was. Somehow when I had attached before, I never saw it come up as images in the message but now I see that is how it works. Sorry for being such an idiot. Here are the first four screenshots and I'll post another one below.

[vico1959]

Here is the final screenshot.

[vico1959]

Nobody? Anybody?

[Amr]

Sorry for the late reply,
Your NAT Rules are fine.

However, your firewall rules need tweaking, here's what I understood from your screenshots:

1-I assume that the FIREWALL LAN screen 1& 2 are pages 1 and 2 from your FW rules respectively, and you don't have other group or floating rules and that the "SSL VPN CA" is a self-signed certificate that you created.

2- In that case "your redirect traffic to proxy" rules need to be at the top of the list (Unless there's users you don't want to redirect to proxy), followed possibly by your DNS rule and finally a "block all rule" (unless of course, you want to use other ports like SMTP, IMAP, ...etc).

3-The order of FW rules is important as the first rule gets evaluated first (in your case you have block HTTP(S) rules first so nothing won't be allowed to reach the proxy, for as to why HTTP worked but not HTTPS it's probably because of the anti-lockout rule which allows HTTP traffic to the FW)

4- BTW 127.0.0.1 subnet mask should be 32, not 24 ex: 127.0.0.1/32, you might want to add your LAN address (the firewall LAN address with which you access the WEB GUI) to the unrestricted IP addresses in access control just to make sure everything works fine.

5-You can troubleshoot Firewall rules by going to Firewall> log files> Live view and type in the filter 3129 and check whether it's being blocked or denied (red) or allowed (green), you can also use ".*" for advanced filter ex: 127.0.0.1.*3129 to see all the rules associated with IP 127.0.0.1 on port 3129.


So to recap:
1- Add proper Firewall rules.
2-configure the proxy to enable SSL inspection ( and log SNI information only if you don't want to bump sites).
3- enable NAT.
4-At this point you should restart OPNsense.
5- Test by Entering Manual configuration in Firefox.