Firewall/HAProxy: configure https services [subject updated]

[mike0000]

Hello,

I've got OPNsense set up and running very well for half a year or so, OpenVPN included.

Background/status:

• Access to the admin interface is https only (HTTP Strict Transport Security enabled) and via a modified port (192.168.1.1:XX443)
• The OPNsense box is configured with Hostname opnsense and Domain mike0000.dynprovider.copm
• I have set up a dyn dns provider with my public static IP (let's say mike0000.dynprovider.com)
• I have set up and issued SSL/TLS certificates via Let's Encrypt (automated) successfully for this domain
• I've got a QNAP NAS box that has a web interface accessed via HTTPS. I can access it via internal LAN or through VPN via the local IP 192.168.1.60:8080 but not via https://nas.mike0000.dynprovicer.com:8080
• HA Proxy is running on the OPNsense box and I suspect this could also be the culprit, meaning it handles the http(s) traffic and doesnt let the request through to the right host (the OPNsense box). The HAProxy_ports list is 80 and 443. The original setup/config was done following this tutorial: https://forum.opnsense.org/index.php?topic=23339.0

Goal:
Access via the browser https://opnsense.mike0000.dynprovider.com:XX443 and be able to see and log in to the OPNsense admin web GUI. This should only be possible from a host that is connected via OpenVPN, which is configered to be the address space 10.0.7.0/24). Otherwise all inbound traffic should be rejected on not be able to access mike0000.dynprovider.com or any subdomains on any ports (I don't want to break things and have no internet though).


Issue:

At the moment the following is happening when accessing via a browser:

• Access https://mike0000.dynprovider.com (from a remote machine, with and without VPN) results in a 503 HTTP error ("503 Service Unavailable. No server is available to handle this request.")
• Access https://mike0000.dynprovider.com:XX443 (from a remote machine, with and without VPN) results in a timeout ("Safari couldn't open the page because the server stopped responding.")
• Access https://opnsense.mike0000.dynprovider.com (from a remote machine, with and without VPN) results in a 503 HTTP error ("503 Service Unavailable. No server is available to handle this request.")
• Access https://opnsense.mike0000.dynprovider.com:XX443 (from a remote machine, with and without VPN) results in a timeout ("Safari couldn't open the page because the server stopped responding.")

In case 1. and 2. above I want to prevent all access.
In case 3. above I want to prevent all access.
In case 4. above I want to allow access for LAN or VPN users (i.e. hosts with an IP from the 10.0.7.0/24 subnet).
I don't want LAN users (i.e. hosts with an IP from the 192.168.1.0/24 subnet) to have any restrictions.

Screenshot of current firewall rules:
Attached

[edit]
I suspect I need a WAN rule rule to say "allow inbound TCP/UDP traffic from subnet 10.0.7.0/24 to host opnsense on port XX443".
Looking at behaviour on point 1. and 3. above I think
[/edit]

In the meanwhile if I missed (despite searching for it) a tutorial that explains what to do I'm grateful if someone could point me to it.

Thank you
Mike

Edit: amended to make clearer and to read and added further clarifications.   

Edit 2: I have the strong suspicion that HAProxy may be relevant here and potentially needs some configuration for the local subdomains map rule and public subdomains map rule. The NAS' web interface is accessible on port 8080 (as mentioned above) and also on 443 via it's IP but also not publicly via the URL (as stated above; this is a behaviour I would only want to allow through VPN).

The weird thing is that the NAS is accessible on 8080 and 443 (NAS config allows access on both) without 503 error, but the OPNsense box throws a 503 error now that it is on port 443. Weird

Edit 17/7/23: updated subject to be more accurate as previous was misleading

[sorano]

Hi.

Regarding your TCP port the value is a bit high, maximum is 65535.

Except for that it feels like you know what you need to do:

Edit: I suspect I need a set of rules to say "allow inbound from 10.0.7.0/24 on tcp port 99443" and "deny/block pubic to all internal and all ports"? The last one assume should actually be the drop all/any rule at the end of the rule set. /end edit.

I would do it by creating an Alias for your VPN subnet then create an incoming allow rule on the correct interface with your Alias as source and your management IP as TCP port as destination.

[mike0000]

Thanks sorano. I have edited the OP above to clarify (the 99443 port was an example, I didn't put the actual port I am using here).

I have further tested and wanted to see if I could remote access the GUI when the port is set to 443. Now I obtain the 503 error as shown above (when connected via VPN both when accessing the URL as well as the LAN IP 192.168.1.1:443. The prior 192.168.1.1:XX443 is no longer available (of course, I changed it) and times out. So looks like I am locked out for now and have to physically access it when back home.

What is weird is that my NAS is also accessed through port 8080 on 192.168.1.60:8080, but listens on 443 as well. Going to 192.168.1.60:443 also gets me to the NAS' web GUI, whereas I expected a 503. I'm confused

Meanwhile would be great to get input/ideas here - I strongly suspect this is related to the HA Proxy that I am running. The original setup/config was done following this tutorial: https://forum.opnsense.org/index.php?topic=23339.0

[sorano]

Ok Mike.
So your edits clears it up a bit.

However I get the feeling that you don't fully understand what it is you are actually trying to accomplish and you seem to mix up the functionalities of DNS/reverse proxy and also which port is doing what.

What about breaking it down a bit and work in smaller steps?

If I were you I would start with being able to reach management webui (not HAProxy) over VPN via direct IP:port connection.

Then when you have that working you can move on to looking at the HAProxy settings and configure those to allow your VPN subnet to reach the management backend behind HAProxy via DNS.

Some other pointers:

Regarding your NAS not throwing a 503 that is no surprise at all since you are connecting directly to it via IP and the connection is not going through HAProxy.

503 is HAProxy saying that whatever you are trying to reach is unavailable for you. That could be due to multiple different reasons.

And when changing webui to port 443 are you sure you did not create a conflict with HAProxy also listening on 443?

And regarding being locked out you could SSH in and stop HAProxy service or try to tunnel.

[sorano]

Regarding 4. maybe a simpler way for you would be to just DNS override the FQDN on the inside of your LAN and just point it to your LAN interface where webui is listening (192.168.1.1?).

Then you just need to create fw rules and not have to bother with reconfiguring HAProxy.

[mike0000]

I am finally back home now with physical access to the box. I am at a loss though of how to log back in and make the change to the config of the HTTPS port back to the earlier xx443.

I tried connecting via the specific admin LAN port I configured, but HAProxy applies same rules there. I also tried connecting via SSH but my user, whilst an admin, does not have root privileges and cannot edit the config.xml (that would be the easiest...).

Currently a site access is captured by HAProxy and results in the 503 error.

Any tips of how to regain access to make the change?

[sorano]

I am finally back home now with physical access to the box. I am at a loss though of how to log back in and make the change to the config of the HTTPS port back to the earlier xx443.

I tried connecting via the specific admin LAN port I configured, but HAProxy applies same rules there. I also tried connecting via SSH but my user, whilst an admin, does not have root privileges and cannot edit the config.xml (that would be the easiest...).

Currently a site access is captured by HAProxy and results in the 503 error.

Any tips of how to regain access to make the change?

It's really hard to say without live troubleshooting but looking at what you wrote previously I would try to login via SSH and kill the HAPROXY service and then restart webui service. If you still cannot reach I would disable pf and try again.

If it's still not available I would verify the port that webui is bound to.

[mike0000]

Hi Sorano, thanks a lot for your fast replies and help!

I tried killing HAProxy but got errors - I believe the user I configured does not have sudo rights (something to be fixed next).

What I did now, that helped me to regain access is log in locally (plug in monitor and keyboard into the box) and via single user mode mounted the root drive, and edited the port in the config.xml directly.

I'm back in now and can connect via local LAN and VPN tunnel to the web GUI on IP:xx443.

Next step would be to fix the issue that I cant access the GUI via the subdomain of the dynamic DNS provider I configured, i.e. https://opnsense.mike0000.dynprovider.com:XX443 via VPN.

Edit: typo

[sorano]

Next step would be to fix the issue that I cant access the gui via the subdomain of the dynamic dns provider I configured...

Does that DNS record resolve to your public IP?
If so you need to activate NAT loopbackb or create a DNS override for the same hostname that resolves to an internal IP where webui (or HAProxy) is listening.

[mike0000]

Does that DNS record resolve to your public IP?

Checking with ping and tracert the dynamic dns name (mike0000.dynprovider.com) resolves to my WAN IP. Same for a subdomain (opnsense.mike0000....)

If so you need to activate NAT loopbackb or create a DNS override for the same hostname that resolves to an internal IP where webui (or HAProxy) is listening.

How can I create the DNS override to resolve that hostname to an internal IP? I went to Services --> Unbound DNS --> Overrides and added an entry under both Host Overrides as well as Domain Overrides pointing at the IP, but still does not work (after restarting the service).

Any idea or specific pointer? Thanks so far!

[sorano]

Does that DNS record resolve to your public IP?

Checking with ping and tracert the dynamic dns name (mike0000.dynprovider.com) resolves to my WAN IP. Same for a subdomain (opnsense.mike0000....)

If so you need to activate NAT loopbackb or create a DNS override for the same hostname that resolves to an internal IP where webui (or HAProxy) is listening.

How can I create the DNS override to resolve that hostname to an internal IP? I went to Services --> Unbound DNS --> Overrides and added an entry under both Host Overrides as well as Domain Overrides pointing at the IP, but still does not work (after restarting the service).

Any idea or specific pointer? Thanks so far!

Did you remember to flush your local PC DNS cache?
If it's only one record you should use Host override. So remove domain override for now and create Host with:

Host:  mike0000
Domain: dynprovider.com
Type: A

Then either flush DNS cache in your PC or use nslookup to query unbound directly and you should get a reply with whatever IP you put in.

[mike0000]

Great tip - tracking with nslookup what happens I can see that my pihole catches the DNS requests:

Code Select Expand

C:\>ipconfig /flushdns
Windows IP Configuration
Successfully flushed the DNS Resolver Cache.

C:\>nslookup opnsense.mike0000.dynprovider.com
Server:  pi.hole
Address:  192.168.5.95

Non-authoritative answer:
Name:    mike0000.dynprovider.com
Address:  [WAN IP ADDRESS]
Aliases:  opnsense.mike0000.dynprovider.com


C:\>nslookup mike0000.dynprovider.com
Server:  pi.hole
Address:  192.168.5.95

Non-authoritative answer:
Name:    mike0000.dynprovider.com
Address:  [WAN IP ADDRESS]

Now I added a DNS record in pihole pointing opnsense.mike0000.dynprovider.com to 192.168.5.1

Code Select Expand

C:\>nslookup mike0000.dynprovider.com
Server:  pi.hole
Address:  192.168.5.95

Non-authoritative answer:
Name:    mike0000.dynprovider.com
Address:  [WAN IP ADDRESS]


C:\>nslookup opnsense.mike0000.dynprovider.com
Server:  pi.hole
Address:  192.168.5.95

Name:    opnsense.mike0000.dynprovider.com
Address:  192.168.5.1

I can now access the web GUI of the OPNsense box on "opnsense.mike0000.dynprovider.com:XX443" via the browser over WIFI. This did not work before.

Testing via a different machine (using a SIM card to go online from outside of the LAN) and tunneling in via OpenVPN I cannot access the page via the same URL yet. Checking nslookup again reveals that pihole is not used for DNS, rather a 172.xx private IP:

Code Select Expand

C:\>ipconfig /flushdns
Windows IP Configuration
Successfully flushed the DNS Resolver Cache.

C:\>nslookup opnsense.mike0000.dynprovider.com
Server:  UnKnown
Address:  172.20.10.1

Non-authoritative answer:
Name:    mike0000.dynprovider.com
Address:  [WAN IP ADDRESS]
Aliases:  opnsense.mike0000.dynprovider.com


C:\>nslookup mike0000.dynprovider.com
Server:  UnKnown
Address:  172.20.10.1

Non-authoritative answer:
Name:    mike0000.dynprovider.com
Address:  [WAN IP ADDRESS]

It seems the last bit I need now is to make OpenVPN clients use the pihole DNS resolver on 192.168.5.95?

Is this a workaround / dirty way to do this, i.e. should I be fixing this via HAProxy somehow (potentially missing front/back end)?

[sorano]

Great. Looks like you are getting closer.

You can configure your OpenVPN server in OPNsense to push DNS servers so play around with that.

I don't remember if it's really needed to export a new client config file after you made DNS changes but I think it is.

Regarding your pi-hole, is it not using unbound as upstream resolver? Because if it was it should reflect whatever overrides you create.

Of course cache could be the reason since I don't know the TTL for your record.

[sorano]

Also you never mentioned anything about pi-hole in your previous posts so for the future I'll pass on a great tip from the Zen of Python:

Explicit is better than implicit

[mike0000]

Thanks for the additional thinking points.

Frankly I didn't clock this could be a DNS issue as I interpreted the 503 error rather than the request was received by a service (expected HAProxy) but misconfigured where to forward to).

I did configure pihole to use unbound, which is why I didnt think about it inititially. Pihole has upstream DNS pointing to Google DNS, enabled 'Never forward non-FQDN A and AAAA queries', 'Never forward reverse lookups for private IP ranges' and 'Use Conditional Forwarding'. I don't see where I can point it back to Unbound (would I configure Unbound as the Upstream DNS?)

Also you never mentioned anything about pi-hole in your previous posts so for the future I'll pass on a great tip from the Zen of Python:

Explicit is better than implicit

Yeah, sorry for that. I thought the same when the netstat said "pihole"... (mea culpa). thanks a lot for the support and responsiveness so far, appreciate it.