Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[vladnik]

Hey all,

first of all, thank you for the amazing guide @TheHellSite, I have set it up over a year ago and everything works like a charm.

Today, I have a small problem. I have a Mikrotik switch that can only be accessed via Port 80 and requires HTTP basic auth.
I have setup my backend and map file like always, the site is reachable fine, however I cannot login. I enter my credentials and press enter, and the dialog for entering my credentials just shows back up infinitely. The credentials are correct, if I go to the switch via IP-address I can log in normally.



I have tried adding a user and password to 'User management' & ticking the box for Basic auth in the backend and selecting my user, no luck sadly. I'm pretty sure I just have to add an option somewhere to pass the auth header, however, I can't figure it out. Googling for the problem just leads to info on how to setup HAproxy to do basic auth, which I don't need... The logs also don't seem to show anything useful.

Anyone has any ideas? Thanks in advance.

[TheHellSite]

I have tried adding a user and password to 'User management' & ticking the box for Basic auth in the backend and selecting my user, no luck sadly. I'm pretty sure I just have to add an option somewhere to pass the auth header, however, I can't figure it out. Googling for the problem just leads to info on how to setup HAproxy to do basic auth, which I don't need... The logs also don't seem to show anything useful.

Anyone has any ideas? Thanks in advance.

The user management in HAProxy has nothing to do at all with any login forms of services that are behind HAProxy! You can use this to add a login form that pops up before the client can even connect to the service that is behind HAProxy. So unrelated to your issue.

Apart from that please ask in the official HAProxy forums about your issue since it is not related to my tutorial.

[vladnik]

The user management in HAProxy has nothing to do at all with any login forms of services that are behind HAProxy! You can use this to add a login form that pops up before the client can even connect to the service that is behind HAProxy. So unrelated to your issue.

Figured as much. Was worth a try though.

Apart from that please ask in the official HAProxy forums about your issue since it is not related to my tutorial.

Alright, will do. Thanks anyways!

[EuleMitKeule]

I know this is not really part of the (great) tutorial, but I wanted to ask if it is possible to change the SSL certificate that is being used based on whether the traffic is local or public. I already have setup the rule for subdomains that are only accessible from local IPs.

The reason for this is that I want to enable Full (Strict) mode in Cloudflare. That means I have to use the Cloudflare Origin Server Certificate for public access to my HAProxy. I already uploaded the certificate to OPNsense and selected it along with the Let's Encrypt certificate for the HTTPS frontend. However it seems only the LE certificate is being used, so public access via Cloudflare fails. I looked for an HAProxy function that chooses a specific certificate, but it does not seem to exist.

Can anyone point me in the right direction?

[Bunch]

Use different set of frontends to handle it.
For example, my current setup is
WAN_TCP_frontend->WAN_SSL_frontend, with 192.168.5.1, 192.168.6.1 as listening IP
LAN_TCP_frontend->LAN_SSL_frontend, with 192.168.7.1 and 192.168.8.1 as listening IP

WAN NAT port forward to 192.169.5.1
While Unbound DNS overwrite domain name to 192.168.7.1

[DeWilde]

Hello,

and thank you for this great tutorial!
I managed to get the HAProxy running and I am able to access the desired services from the internet side.
But when i want to access them from the internal netwerk i am unable to reach them.
i tried the DNS override as explained but this does not work. I suppose i am doing something wrong here.
As far as I understood, i must create an "override" for each host i also want to reach internally and assign it the IP adres of the LAN interface of the OPNsense. Is that correct?
My default LAN interface has "192.168.10.1/24" so i created a host override f.e. "mynas.mydomain.com" pointing to 192.168.10.1.
I assume the HAProxy is also listening on the LAN interface?

thank you for your help!

[TheHellSite]

I assume the HAProxy is also listening on the LAN interface?

If configured correctly, yes. However no way to know since...

thank you for your help!

No logs, no haproxy config export, no other details...
Until provided, no help.

[DeWilde]

Hi, attached you can find my haproxy.conf and map file.
I replicated your tutorial 1:1

The OPNsense firewall LAN IP is 192.168.10.1
for the 2 items in the map file i created a DNS override for those FQDN's pointing to 192.168.10.1

access from the internet works fine. The wildcard ssl cert is being used and the port redirection works and is not visible. (ex :55443 for the firewall interface and :5000 for the NAS)

internally the browser says "ERR_CONNECTION_TIMED_OUT" and nothing is displayed.

If you need more info or details i'll be happy to provide them to you.
thank you for your help!

[guest36292]

Hey y'all,

First of all, truly appreciate the energy you put into this thorough guide @TheHellSite! Managed to make everything work, thank you!

There's one thing I can't sort out and your guide doesn't touch the topic either. HAProxy does not forward client IP to the server as expected. The server in this case is a Synology NAS which is only aware of the OPNSense IP if the connection is made via the reverse proxy.

Is this an expected behavior or could I have some settings wrong? My settings follow your guide entirely and the "X-Forwarded-For header" checkboxes are checked in both HTTP and HTTPS Frontends under the Public Services tab. Is it possible that the SNI Frontend doesn't forward the client IP "downstream" so the other two can't properly forward it?

I have found some forum posts that explain how this should be working in the HAProxy config file but I can't really translate that into the user interface as I lack the appropriate level of knowledge here.
https://access.redhat.com/solutions/3552581
https://serverfault.com/questions/722151/haproxy-how-to-append-client-ip-in-x-client-ip-and-x-forwarded-for-headers
https://community.synology.com/enu/forum/1/post/150860?reply=488269
https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/haproxy_pass_clientip_to_webserver

I was hoping you guys could help me find a solution to this that can be achieved via the OPNSense UI.

Thank you in advance!
------------------

Update:
So anyone bumping into this issue on a Synology NAS, know it can be easily resolved. Apparently the server should be aware that a reverse proxy is forwarding the connections so it can resolve the original client IP.
The solution to my problem was posted here: https://www.reddit.com/r/synology/comments/mmubnv/tip_for_using_dsm_reverse_proxy_logging_correct/

In the DSM control panel, go to security, at the bottom is "trusted proxies". Add the dsm ip address HAProxy IP address, and boom! The correct external IP address is logged at a connection attempt and you will get notified about new login behavior if you have that turned on, and ip address blocking should now work if you have that turned on.

------------------

Update 2:
Apparently this above solution comes with a caveat, which may not make it a satisfying solution at all.
In order to establish connection between the client and the Synology the proxy must be allowed in the Synology's firewall. And as it appears even though the client IP is passed on the connection itself is made through the proxy's IP address. Even if the client's IP is specifically blocked, through the the proxy it is allowed to connect to every service the proxy is allowed to.

This effectively renders the the Synology Firewall useless.

I'm going to open a new topic for this issue for better visibility.

[mintchipmadness]

Hello and thank you very much for this guide. I am definitely learning a lot as I am setting up HAProxy on my opensense for public and local access. I have hopefully a quick question. Does this setup work to access the opnsense GUI or is that a special item that I need to setup? I am still troubleshooting but I'd thought I'd ask just in case. Thank you for your help.

Edit: Addition domain name structure information for my setup and some additional troubleshooting

Public Services: service.example.com
Internal Services: service.internal.example.com

I have been troubleshooting my setup and had a small breakthrough with my unbound settings. Under the general settings of unbound I needed to check "Do not register system A/AAAA records" to get the overrides to work when I ping. Before all my interfaces were getting registered under the firewall domain name causing a random interface IP to get pulled when I pinged opnsense.local.example.com of the firewall. Only certain IPs are allowed to access the GUI from my lan. Now my override works appropriately.

[mike0000]

TheHellSite, thanks a lot for all the work that you've put into this tutorial. I have followed every step of it and almost everything is working well.

One issue I am facing is that when I ping a local domain (e.g. opnsense.mydomain.com (router/fw box), nas.mydomain.com (qnap nas)) the IP gets resolved as my external WAN IP address.

When I direct my browser at one of my internal domains I'm not getting the same result as when I simply go to the corresponding IP.

I am not sure whether there is a misconfiguration in the HAProxy setup, or whether it is in fact unrelated to your tutorial. Below are the outputs I see from accessing two different subdomains from both my phone on cell service, as well as from internal wifi:

When accessing my OPNSense webui on opnsense.mydomain.com:55443 (internal IP 192.168.5.1):

• Accessing via browser (Firefox/Chrome/Safari (iPhone) I'm getting the following error: "The connection has timed out. An error occurred during a connection to opnsense.mydomain.com:55443" (Firefox); "This site can't be reached. opnsense.mydomain.com took too long to respond." (Chrome); "Safari could not open the page because the server stopped responding." (Safari iPhone)

When accessing my QNAP NAS on nas.mydomain.com (internal IP 192.168.5.60):

• Accessing via browser (Firefox/Chrome/Safari (iPhone) I'm getting the following error: "503 Service Unavailable. No server is available to handle this request." This suggests to me the http server is actually responding to the request but not serving an SSL page, even though https://192.168.5.60 returns the NAS login page.

• Testing the server response from shell on the nas [edit1]:

Code Select Expand

user@OPNsense:~ $ wget --save-headers http://nas.mydomain.com:8080
--2023-01-08 12:47:22--  http://nas.mydomain.com:8080/
Resolving nas.mydomain.com (nas.mydomain.com)... 192.168.5.60
Connecting to nas.mydomain.com (nas.mydomain.com)|192.168.5.60|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 580 [text/html]
Saving to: 'index.html'
index.html 100%[=====================================================================================>] 580  --.-KB/s    in 0s
2023-01-08 12:47:22 (139 MB/s) - 'index.html' saved [580/580]

When checking the response on https request an error comes (with the '--no-check-certificate' parameter yields the same result/output as the http request):

Code Select Expand

user@OPNsense:~ $ wget --save-headers https://nas.mydomain.com
--2023-01-08 13:01:21--  https://nas.mydomain.com/
Resolving nas.mydomain.com (nas.mydomain.com)... 192.168.5.60
Connecting to nas.mydomain.com (nas.mydomain.com)|192.168.5.60|:443... connected.
ERROR: cannot verify nas.mydomain.com's certificate, issued by 'CN=R3,O=Let\'s Encrypt,C=US':
  Unable to locally verify the issuer's authority.
To connect to nas.mydomain.com insecurely, use `--no-check-certificate'.


There seem to be two issues here: 1) there is a certificate error (I imported the acme/LE wildcard .crt and .key into the NAS).

But 2) more generally as observed with the pings thrown at the WAN IP rather than the correct internal IP the request from the browser is also being forwarded to the WAN (mydomain.com) - this is shown by wget on mydomain.com which also returns a 503, the same that the browser does:

Code Select Expand

user@OPNsense:~ $ wget --save-headers https://mydomain.com
--2023-01-08 13:13:59--  https://mydomain.com/
Resolving mydomain.com (mydomain.com)... 185.176.xxx.xxx [WAN IP]
Connecting to mydomain.com (mydomain.com)|185.176.xxx.xxx [WAN IP]|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2023-01-08 13:13:59 ERROR 503: Service Unavailable.

Any ideas or can you recommend any tools to do further troubleshooting or does anyone spot what the issue is?

The purpose of my setup is that all subdomains should only be accessible from the LAN or through VPN (this is set up correctly, I can VPN in via OpenVPN).

-------------------------

edit1: inserted code view of wget header response for nas.mydomain.com access - seems to be working in shell but not from browser

-------------------------
Many thanks
Michael

Attached the config files requested
- HAProxy Config Export
- HAProxy errors and/or log entries
- Details about setup: above, but happy to elaborate further if unclear

[dirkscheck]

@TheHellSide
First thank you for this wonderful guide. I learned a lot of and it helps to understand everything a little bit better.
At the moment I play around with these features to understand it even better.

My setup is a bit different so I need to play around with it and see what happens.

At the moment lets encrypt is working and HAProxy is configured. And here I have some questions in general:

1. Why do I have to open port 80 and 443 when using DNS-01 challenge ? I thought that is NOT needed and that was the reason why I choose DNS01
2. I DIDNT make the internal procedure BUT I can access the configured backend internally without any problems via the name !!! BUT
a. in Safari i get an ssl certificate (it is a self signed ssl certificate from my router)
b. in firefox it is still unsecured

Why is that ? If I have to send to you some information please let me know but I think that are general questions and no files / screenshots are needed or ?

[TheHellSite]

My default LAN interface has "192.168.10.1/24" so i created a host override f.e. "mynas.mydomain.com" pointing to 192.168.10.1.
I assume the HAProxy is also listening on the LAN interface?

Yes, your OPNsense LAN IP is the correct DNS Override target, as explained in the tutorial.
Yes, HAProxy is also listening on that interface since the SNI_frontend is listening on ALL IPs:Ports (0.0.0.0:0).

Hi, attached you can find my haproxy.conf and map file.
I replicated your tutorial 1:1

The OPNsense firewall LAN IP is 192.168.10.1
for the 2 items in the map file i created a DNS override for those FQDN's pointing to 192.168.10.1

access from the internet works fine. The wildcard ssl cert is being used and the port redirection works and is not visible. (ex :55443 for the firewall interface and :5000 for the NAS)

internally the browser says "ERR_CONNECTION_TIMED_OUT" and nothing is displayed.

If you need more info or details i'll be happy to provide them to you.
thank you for your help!

If access from external network is working fine, then your DNS override isn't taking effect.
Check this using nslookup/ping from any PC within your network.
Also check if you are actually using Unbound as your DNS resolver.

Then there is your HAProxy config...
Your SFINX and MYNAS both have the exact same IP:Port in the server settings, double check this.
BTW: I would never expose my OPNsense Web UI externally, but this is up to you.
Apart from that the HAProxy config looks fine.

[TheHellSite]

TheHellSite, thanks a lot for all the work that you've put into this tutorial. I have followed every step of it and almost everything is working well.

One issue I am facing is that when I ping a local domain (e.g. opnsense.mydomain.com (router/fw box), nas.mydomain.com (qnap nas)) the IP gets resolved as my external WAN IP address.

If you also followed "Part 6 - Access from internal networks" of my guide this should be working.
If not, then check if the DNS overrides are working using nslookup/ping.

Oh and your HAProxy config looks wrong / incomplete.
There are no actual services configured (server+backend).

[TheHellSite]

@TheHellSide
First thank you for this wonderful guide. I learned a lot of and it helps to understand everything a little bit better.
At the moment I play around with these features to understand it even better.

My setup is a bit different so I need to play around with it and see what happens.

At the moment lets encrypt is working and HAProxy is configured. And here I have some questions in general:

1. Why do I have to open port 80 and 443 when using DNS-01 challenge ? I thought that is NOT needed and that was the reason why I choose DNS01
2. I DIDNT make the internal procedure BUT I can access the configured backend internally without any problems via the name !!! BUT
a. in Safari i get an ssl certificate (it is a self signed ssl certificate from my router)
b. in firefox it is still unsecured

Why is that ? If I have to send to you some information please let me know but I think that are general questions and no files / screenshots are needed or ?

1. Maybe because the internet works almost entirely over TCP ports 80(HTTP) and 443(HTTPS)?
So if we don't open WAN port 80+443 you will never be able to access your services over your WAN IP.
If you would have understood what you are doing, which you actually should have since I explained it pretty detailed in the tutorial, you would know that 80 is only open so any unencrypted traffic hitting port 80 gets redirected to the encryption required port 443.

2. I am assuming there is another router/firewall in front of your OPNsense that handles NAT, DNS and DHCP.
"My setup is a bit different so I need to play around with it and see what happens."
See my answer to your 3rd question.

3. "Why is that ? If I have to send to you some information please let me know but I think that are general questions and no files / screenshots are needed or ?"
WRONG.
You are stating that your setup is a bit different and then ask a question about some issues because you didn't follow the tutorial which is why you are probably having these issues in the first place.
I am pretty confused.