English Forums > Tutorials and FAQs

Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

(1/19) > >>

TheHellSite:
Hello,

when I started implementing HAProxy in my network I couldn't find any complete and well written guide out there. I had to puzzle everything together from various websites.
So I thought I would save many of you a lot of time and provide my ultimate HAProxy on OPNsense guide.  :)

This tutorial will show you how to configure HAProxy as a reverse proxy on OPNsense using wildcard certificates from Let's Encrypt.
It is going to be a step-by-step guide with images on how to set things up while also explaining why we set things up in a certain way.
I will try to make this as complete and detailed as possible.
If you think that there is anything wrong or missing, feel free to tell me about it and I will consider changing it.

If this guide was helpful to you then please leave me a thanks down below as it took me several days to write this down.

Kind Regards
TheHellSite


This configuration is tested to be working on OPNsense 21.7 (OpenSSL flavour) with latest updates as of 20210901.


Changelog

* 20210603

* Fixes and some enhancements
* 20210611

* Implemented @sorano's enhancements
* 20210613

* The tutorial is now using a wildcard CNAME record.
* Enabled Proxy Protocol in the "SSL_backend", "HTTPS_frontend" and "HTTP_frontend" configuration so that the IPs of clients accessing HAProxy will now no longer be overwritten with the "SSL_server" IP.
https://www.haproxy.com/blog/using-haproxy-with-the-proxy-protocol-to-better-secure-your-database/
* 20210729

* Added an alias for the HAProxy ports and updated the WAN interface firewall rule with it.
This leaves us with only one firewall rule instead of two and makes even more sense if one is using additional frontends on different ports.
Thanks @_Alchemist_ for the suggestion.
* The tutorial is now using a map file instead of "condition + rule" for service configuration.
* 20210730

* Added an explanation on how to configure local-access-only subdomains in HAProxy.
* 20210801

* Added an explanation about using self-signed certificates for internal communication to the FAQ.

Current Ciphers and Cipher Suites for a 100% A+ rating at SSLLabs
Last updated on 20210531 using Mozilla SSL Configuration Generator.
https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6

--- Code: ---Cipher List
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384

Cipher Suites
TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
--- End code ---

What will the end result look like?
We will have a wildcard SSL certificate from Let's Encrypt that gets an A+ rating with 100% score at all points from SSLLabs.
https://www.ssllabs.com/ssltest/

We will also have two levels of load balancing our services.

Level A - SSL Offloading disabled
WWW --> WAN interface --> OPNsense --> HAProxy SNI Frontend --> internal servers / services

Level B - SSL Offloading enabled
WWW --> WAN interface --> OPNsense --> HAProxy SNI Frontend --> HAProxy SSL Frontend --> internal servers / services

What are we going to do?

* We will install the necessary plugins.
* We will use a free DynDNS provider (https://desec.io/) that supports the DNS challenge which is mandatory in order to obtain a wildcard certificate.
If you already have your own domain and your hosting provider offers an API that is supported by the Let's Encrypt (ACME) plugin for OPNsense, you can use it instead.
* We will create a wildcard SSL certificate using Let's Encrypt.
* We will configure the necessary firewall rules and change some OPNsense settings in order for HAProxy to function properly.
* We will use HAProxy to do SNI (explanation below) and SSL offloading.
* We will enable access to HAProxy from the internal network.
FAQ

* Why are we using wildcard certificates and not just regular certificates?
For me the main reason is simplicity, there is no need to set up multiple certificates for multiple subdomains.
Also you don't need to add any rules in HAProxy or your firewall for the ACME plugin to function correctly as the DNS challenge doesn't need this.
If you want to read more about the differences follow the links below.
https://sectigostore.com/page/ssl-vs-wildcard-ssl-certificate/
https://comodosslstore.com/resources/whats-the-difference-wildcard-certificate-vs-regular-ssl-certificate/
* How does the DNS challenge work?
https://letsencrypt.org/docs/challenge-types/
* What is SNI?
https://en.wikipedia.org/wiki/Server_Name_Indication
As you already saw above we are going to do it in two stages and not just one.
* Why are we using a virtual IP?
You don't have to!
I am only doing it because I don't want to use the localhost as my "SSL_server". I just prefer to keep things isolated from each other.
You can of course simply use the localhost (127.0.0.1) as your "SSL_server" and let your "HTTPS_frontend" listen to that IP.
* Why are we doing 2-Level-SNI?
The main reason for this is so we can load balance services that don't require additional SSL offloading, f.e. OpenVPN over TCP.
Basically we will have two frontends listening on port 443, one with and one without SSL offloading.
* How can we load balance TCP traffic that we don't want to get SSL offloaded, f.e. OpenVPN over TCP?
In my tutorial I only explain how to "redirect+load balance SSL offloaded traffic".
This is because I myself don't have (yet) the need to actually load balance any non SSL traffic.
However balancing non SSL traffic is pretty much the same as balancing SSL traffic.
You only have to make sure that your "NOSSLservice_rule" or "NOSSLservices_map-file_rule" is placed on the "SNI_frontend" instead of the "HTTPS_frontend" and that the backend that belongs to your "NOSSLservice_server" is running in TCP mode.
* How can I add new services in HAProxy?
It is almost everytime the same procedure to configure / add a new service in HAProxy.
The long way is to...

* Create the server or servers, f.e. "NEW_server".
* Create the corresponding backend containing "NEW_server", f.e. "NEW_backend".
* Create the condition that matches the traffic of "NEW_server", f.e. "NEW_condition".
* Create the rule that redirects traffic to "NEW_backend", f.e. "NEW_rule".
* Add "NEW_rule" to your "SNI_frontend" (if you don't need SSL offloading) or to your "HTTPS_frontend" (if you need SSL offloading).
* Is there a faster way of adding new services if I have lots of subdomains and services?
Yes there is! Thank you @sorano for pointing this out.
https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/overview/
https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/syntax/
Map files are a great way of adding new services to your HAProxy configuration.
It is most of the time much faster than creating a individual condition + rule for each service and then adding each rule to your frontend.
To sum this up, the short way is to...

* Create the server or servers, f.e. "NEW_server".
* Create the corresponding backend containing "NEW_server", f.e. "NEW_backend".
* Create a map file containing all of your subdomains (for your services) with the corresponding backend.
* Create a rule that uses that map file.
* Add this rule to your desired frontend.You might be asking yourself now: Why is this faster?
Well, because next time you add a new service you will only need to create the server, the backend and add the combination of "subdomain to backend" to your map file.
* Do I need to enable "SSL" in the Real Server configuration of a service?
You need to think of a reverse proxy setup like this.

WWW ---Stage 1---> yourdomain.tld ---Stage 2---> OPNsense + HAProxy + LE ---Stage 3---> internal services

Stage 1 + 2
Public facing external traffic. Traffic in these stages is now always encrypted with a verified SSL certificate. In this case it is created and verified by Let's Encrypt.

Stage 3
Local facing internal traffic. Traffic in this stage can or can not be encrypted, depending on your service setup. This is the traffic from HAProxy to your internal service. It doesn't need to be encrypted because you can consider your internal network as trusted.
However it is still strongly advised to also run this traffic encrypted.
In HAProxy you only need to check the "SSL" box in your real server setting for this.
But then you also need to actually enable SSL encryption on that service, f.e. by installing a self-signed certificate on that service and enabling HTTPS. Even though using a self-signed certificate will give you a warning by your browser when accessing the service directly and not through the reverse proxy, the traffic is still encrypted. The certificate is just unverified, which isn't that much of an issue since we are using our reverse proxy in front of it anyway.
How to actually do this this depends on the service but this should be covered somewhere in its manual.

You can read more about this here: https://www.globalsign.com/en/ssl-information-center/dangers-self-signed-certificates



The Configuration

Part 1 - Plugin Installation

* Make sure your OPNsense is up to date and that you are using the OpenSSL firmware flavour as the LibreSSL version doesn't support TLS_1.3 as of writing this.
In your OPNsense, go to "System --> Firmware --> Updates" and install all updates.
* Go to "System --> Firmware --> Plugins" and install the following plugins: os-acme-client, os-dyndns, os-haproxy



Part 2 - DynDNS Configuration

* Visit https://desec.io/signup, create your account and verify your email address.
* Visit https://desec.io/domains and create your domain by clicking on the "round yellow + icon" in the top right corner.
Note: Your domain has to be in the form of "your_domain.dedyn.io". I will use "tutorial.dedyn.io" as an example.

* Visit https://desec.io/tokens, create a token for your domain and name it accordingly.

* Save the token somewhere secure as we will need it twice during this tutorial and you also might need it again in some time.
This token will allow the DynDNS service and the Let's Encrypt plugin of your OPNsense to authenticate to the API of deSEC.

* The next step is to add a CAA record that contains the information on who issued our SSL certificate.
This is important in order to get an A+ rating from SSLLabs.
Visit https://desec.io/domains/your_domain.dedyn.io and add a record to your domain by clicking on the "round yellow + icon" in the top right corner.

* The next step would be to add a CNAME record to your subdomain for each of your services.
If you don't need or want to do this, just skip this step. But in my opinion it is smart to do this as it makes it easier to load balance the services with HAProxy.
You can either add an individual CNAME record per service, f.e. "plex.your_subdomain.dedyn.io", "mail.your_subdomain.dedyn.io" and so on.
OR
You can do it like me and just create a single wildcard CNAME record, f.e. "*.your_subdomain.dedyn.io".
With the later "any_string.your_subdomain.dedyn.io" will resolve to "your_subdomain.dedyn.io".

* In your OPNsense go to: Services --> Dynamic DNS
Click on "Add", fill out the information accordingly and hit "Save and Force Update".
(The password is your token.)

* If everything went right "your_subdomain.dedyn.io" and "any_string.your_subdomain.dedyn.io" should resolve to your public IP.
You can test this by going to: Interfaces --> Diagnostics --> Ping
If it doesn't work wait a few minutes as it can take some time before all the DNS providers around the world have your hostname in their database.
If it still doesn't work check your configuration.
* Now that we have our DynDNS provider all set up we will want to obtain our SSL certificate.



Part 3 - Let's Encrypt

* In your OPNsense go to: Services --> Let's Encrypt --> Settings --> Settings
Change the settings according to my image. We don't need the HAProxy integration as we are obtaining our certificates using the DNS challenge.
Note: We will want to use the staging environment for now. (https://letsencrypt.org/docs/staging-environment/)

* Next go to: Services --> Let's Encrypt --> Settings --> Update Schedule
Here we will configure at which time of the day our certificates are renewed.
They won't be renewed everyday as the ACME client will first check if the certificates are close to expiration and if they are not they won't get renewed.
You want this to happen at a time of the day where there is not much load on your services as the ACME plugin restarts HAProxy so it can use your new certificates which results in a very short downtime of HAProxy.
You will also want this to NOT happen at any full hours (f.e. 3 am) because these are times when the servers of Let's Encrypt could be so busy that you certificate renewal fails.
Change the settings according to my image.

* Next go to: Services --> Let's Encrypt --> Accounts
Create a new account. You can name it whatever you like, I usually use my first level subdomain.

* Next go to: Services --> Let's Encrypt --> Automations
Create the automation to restart HAProxy after our certificates have been renewed.

* Next go to: Services --> Let's Encrypt --> Challenge Types
Add the DNS challenge for deSEC.

* Next go to: Services --> Let's Encrypt --> Certificates
Add the certificate for your domain according to the image below.
I prefer to use Elliptic-curve cryptography. (https://en.wikipedia.org/wiki/Elliptic-curve_cryptography)
But you can of course also use RSA keys just make sure to set the key length as high as possible in order to get an A+ rating from SSLLabs.

* Next go to: Services --> Let's Encrypt --> Certificates
Now we need to forcefully issue our staging certificate so we can test things out and don't have to wait for the next update schedule.
To do this click on the button marked in the image.

* Next go to: Services --> Let's Encrypt --> Log Files --> ACME Log
If your certifcate was issued successfully your log should look similiar and you can proceed to the next step.
If it doesn't check your configuration.

* Next go to: Services --> Let's Encrypt --> Settings --> Settings
Since we successfully issued our staging certificate we can now leave the test environment and issue our production certificate.
First we need to change the "Let's Encrypt Environment" back to "Production Environment [default]".
* Next go to: Services --> Let's Encrypt --> Certificates
Now again forcefully issue your certificate like we did before.
This time it you should receive a valid and trusted SSL certificate. Make sure to check the ACME log for any errors though!
* If this went well, we can now proceed to the prepare everything for HAProxy. To do this we have to configure some things in your OPNsense.



Part 4 - System Preparation

* In your OPNsense go to: System --> Settings --> Administration
You only need to "Disable web GUI redirect rule" and change the "Web GUI TCP port" to a custom one.
Otherwise HAProxy will not function correctly as you will propably want to access your services from the WWW using the default HTTPS port 443.

* Next go to: Interfaces --> Virtual IPs --> Settings
NOTE: This step is optional, see FAQ! You can safely skip this step and use the localhost instead for your "SSL_server".
But if you would like to do it my way then you will need to create a virtual IP that is in a different subnet than any of your other networks.
"192.168.64.1/32" is going to be the IP on which your "HTTPS_frontend" will be listening on.

* Next go to: Firewall --> Aliases
Now we are going to create an alias for the ports that HAProxy will be listening on.
In most setups you will probably need at least 80 and 443.

* Next go to: Firewall --> Rules --> WAN
Now we are going to allow any inbound traffic hitting our WAN interface on the ports specified in the "HAProxy_ports" alias.
Create a new firewall rule with the content below.

If you did it right it should look like this, make sure that the rule is above all other rules in the list.

* Now we can finally configure HAProxy and make our services available on WAN.



Part 5 - HAProxy Configuration

* In your OPNsense go to: Services --> HAProxy --> Settings --> Service
Change the settings according to the image below.

* Next go to: Services --> HAProxy --> Settings --> Global Parameters
Change the settings according to the image below.
Note: The number of HAProxy threads should not exceed the number of CPU threads of your OPNsense.

* Next go to: Services --> HAProxy --> Settings --> Default Parameters
Change the settings according to the image below.

* Next go to "System --> Settings --> Cron" and create a new cron job.
Because our certificate has the OCSP Must Staple extension we need to update HAProxy's OCSP data regularly.
If we don't do this clients connecting to our services will get security warnings or won't connect at all.
To do this we simply create a cron job that does exactly this.
It doesn't matter if this job runs before or after a certificate renewal as the OCSP data gets updated anyway after installing new certificates in HAProxy.
This is because the ACME plugin restarts HAProxy after installing the new certificates.

* Next go to: Services --> HAProxy --> Settings --> Real Servers
First we will add our "SSL_server" using our virtual IP (or localhost IP if you wish) to do the SSL offloading.

Then you can add all your other services according to their individual configuration.

* Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Backend Pools
Here we first create our "SSL_backend". This is the backend to which our "SNI_frontend" sends most of its traffic to.
As you can't mix HTTP mode and TCP mode in a frontend to backend relation, make sure that the "SSL_backend" is set to TCP mode since our "SNI_frontend" is also running in TCP mode.

Now we create the backend that belongs to an actual service. You will need one backend for each service.
If you have multiple servers serving the exact same content than you will want to add all servers into a single backend so HAProxy can actually balance the load between the servers.

* Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
Here we will only create a "NoSSL_condition", which is necessary in order to identify non-HTTP traffic.

* Next go to: Services --> HAProxy --> Settings --> Advanced --> Map Files
Here we will create a new map file "PUBLIC_SUBDOMAINS_map" for our public subdomains that we want to access from outside of our network.
Please read the FAQ about Map Files first!
This map file is telling HAProxy that any subdomain that starts with "plex" belongs to our "PLEX_backend" (which belongs to our "PLEX_server").

* Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Rules
Here we add the rules that decide what to do with the traffic based on our map files (or conditions if necessary).
The "HTTPtoHTTPS_rule" forwards all HTTP traffic to HTTPS so it can go to our "HTTPS_frontend".

The "PUBLIC_SUBDOMAINS_map-rule" maps our subdomains to our backends using the map file we created in the previous step.

* Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Public Services
Here we will create the frontends that are listening on our interface IPs and the virtual IP we created earlier.
At first we will create our "SNI_frontend" which will decide wether the traffic is going to be SSL offloaded or not.
You will have to place the rules for all of your services that you don't want to get SSL offloaded in here.
Our default backend in this frontend will be the "SSL_backend" that redirects all traffic to our virtual "SSL_server" which is actually our "HTTPS_frontend".

No we will create our "HTTP_frontend". Make sure to place the "HTTPtoHTTPS_rule" in this frontend!
This frontend is necessary in order to redirect HTTP traffic to HTTPS. But you could also use it to serve non SSL encrypted services on port 80.

No we will create our "HTTPS_frontend".
This will be our primary frontend which is doing the SSL offloading using our earlier created Let's Encrypt certificate.
You will have to place the rules for all of your services that you want to get SSL offloaded in here.

* Done.
Access from external networks should now already be working.
Just try to access your URL "your_service.your_subdomain.dedyn.io" from any device that is not connected to your local network, f.e. a smartphone on cellular data.
* You can now verify your SSL settings: https://www.ssllabs.com/ssltest/
* If you configured everything in HAProxy and Let's Encrypt according to my template you should get an A+ rating with a 100% score at all points.

* The last thing for us to do now is enable access from internal networks.



Part 6 - Access from internal networks
If you try to access your URL "your_service.your_subdomain.dedyn.io" from a device in your internal network, it should fail.
There are two ways of fixing this. I will cover both options but keep in mind that Option B is the suggested way of doing it.
NAT reflection is an inferior solution since you lose the ability to track originating source IP in HAProxy when going through NAT. (@sorano)

Option A - NAT Reflection (https://docs.opnsense.org/manual/nat.html)
Option B - Split DNS (https://docs.opnsense.org/manual/unbound.html#overrides)

Option A - NAT Reflection

* In your OPNsense go to: Firewall --> Rules --> WAN
* Here you will have to edit the "Allow HAProxy" rule we created in Part 4 - Step 3 of this tutorial.
At the bottom of each rule there is a setting called "NAT reflection = Use system default".
You will want to change this to "NAT reflection = Enable".
* Access from internal networks should now be working.
Option B - Split DNS (DNS Overrides)
Since you are using OPNsense you are probably also using the Unbound DNS plugin as your local DNS server.
Because of that you can easily set up DNS overrides.

* In your OPNsense go to: Services --> Unbound DNS --> Overrides
* Here you will need to create "Host Overrides" for each of your services. At least if you are using 2nd level subdomains "your_service.your_subdomain.dedyn.io" for you services.
If you are running all of your services on your 1st level subdomain "your_subdomain.dedyn.io" than you will just need to override this one.
* The IP address can be any interface IP of your OPNsense, I am using the LAN IP on which the "SNI_frontend" is also listening on because we set it to "0.0.0.0".

* Access from internal networks should now be working.



Part 7 - Advanced Configuration: local-access-only subdomains
Imagine you have a service that you would like to access / protect using your brand new reverse proxy without making it available on the internet?
Well, HAProxy has got you covered!


* In your OPNsense go to: Services --> HAProxy --> Settings --> Advanced --> Map Files
Here you need to clone the "PUBLIC_SUBDOMAINS_map", rename it to f.e. "LOCAL_SUBDOMAINS_map" and add all your local-access-only subdomains along with their corresponding backends.
Keep in mind that the content of your "PUBLIC_SUBDOMAINS_map" also has to be in this map file! I will explain why later.

* Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
Now you need a condition that detects if the source of the request is from a local IP or from a trusted FQDN!
You can of course also use the predefined "Source IP is local" condition.
I am however using only specific subnets since the predefined condition is using the entire RFC1918 IP range, which I don't want to!


You can also check for a "trusted" FQDN (hostname).
But please keep in mind that HAProxy resolves those hostnames to their IPs and then checks them. But the resolving is only done once during the start / restart of HAProxy.
So if the IP of your FQDN is changing regularly this won't work very well, except if you restart your HAProxy using a cron job like every 24 hours or so.

* Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Rules
Here you need to clone the "PUBLIC_SUBDOMAINS_map-rule", rename it to f.e. "LOCAL_SUBDOMAINS_map-rule", select your "LOCAL_SUBDOMAINS_SUBNETS_condition" condition and select your "LOCAL_SUBDOMAINS_map".
If you are also using an FQDN condition, like I do, you will need to select both your FQDN and your subnet condition together with the logical "or" operator!

* Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Public Services
The last thing left to do is to place the "LOCAL_SUBDOMAINS_map-rule" in your HTTPS_frontend.

Attention!
Remember that I told you to place your public subdomains in the local-access-only map file?
This is because HAProxy is processing the rules in the frontends based on the order they appear!
So if you place your "PUBLIC_SUBDOMAINS_map-rule" before your "LOCAL_SUBDOMAINS_map-rule" in the frontend configuration, you won't get access to your local-access-only subdomains.
And if you don't also place your public subdomains in the "LOCAL_SUBDOMAINS_map-rule", you will no longer have access to your public subdomains.

The correct way of placing both rules is like this.

* Done!
You should now still have access to your public subdomains from any network and also have access to your local-access-only subdomains from the locations you defined.

browne:
Thank you very much!
This helped me switching from regular certificates to wildcard certificates!
I now also do score 100% A+ in the SSL test.

skittlebrau:
Thank you so much for this guide! I was completely lost in the new UI layout before.

Do you mind showing in Step 5 an example of how you've configured 'PLEX_backend'? I noticed you referred to it, but there wasn't a screenshot for it.

TheHellSite:
The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different. :)

--> I will add it to the tutorial.
Reason: https://forum.opnsense.org/index.php?topic=23339.msg111143#msg111143


It is also almost everytime the same procedure to configure / add a new service in HAProxy.

* Create the server or servers, f.e. NEW_server.
* Create the corresponding backend containing NEW_server, f.e. NEW_backend.
* Create the condition that matches the traffic of NEW_server, f.e. NEW_condition.
* Create the rule that redirects traffic to NEW_backend, f.e. NEW_rule.
* Add NEW_rule to your SNI_frontend if you don't need SSL offloading or to your HTTPS_frontend if you need SSL offloading.--> I will add this to the FAQ.

ejball02:
I'm using a self-signed cert for HTTPS inspection for content filtering. I've got OPT2 configured as a guest network on my Protectli, and content filtering, using shallalist works great. Only downside, is that when I try to access am HTTPS site, Firefox/Chrome always give a warning page: "Your connection is not private" "ERR_CERT_AUTHORITY_INVALID". After much Googling, I came across an old post, that said Let's Encrypt can give public certs which would get rid of the message.

I added the LE plugin but couldn't figure out, how to create a cert for use on the Foward Proxy "CA to use" field, required for SSL inspection. Looking through this walkthrough, I'm wondering if there is something here that can help achieve creating a cert for content filtering. Anyone have any experience with this?

Navigation

[0] Message Index

[#] Next page

Go to full version