Topics

1

Tutorials and FAQs / Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

« on: May 31, 2021, 01:06:11 pm »

Hello,

when I started implementing HAProxy in my network I couldn't find any complete and well written guide out there. I had to puzzle everything together from various websites.
So I thought I would save many of you a lot of time and provide my ultimate HAProxy on OPNsense guide. 

This tutorial will show you how to configure HAProxy as a reverse proxy on OPNsense using wildcard certificates from Let's Encrypt.
It is going to be a step-by-step guide with images on how to set things up while also explaining why we set things up in a certain way.
I will try to make this as complete and detailed as possible.
If you think that there is anything wrong or missing, feel free to tell me about it and I will consider changing it.

If this guide was helpful to you then please leave me a thanks down below as it took me several days to write this down.

Kind Regards
TheHellSite

Changelog

• 20210603
  
  • Fixes and some enhancements

• 20210611
  
  • Implemented @sorano's enhancements

• 20210613
  
  • The tutorial is now using a wildcard CNAME record.
  • Enabled Proxy Protocol in the "SSL_backend", "HTTPS_frontend" and "HTTP_frontend" configuration so that the IPs of clients accessing HAProxy will now no longer be overwritten with the "SSL_server" IP.
    https://www.haproxy.com/blog/using-haproxy-with-the-proxy-protocol-to-better-secure-your-database/

Current Ciphers and Cipher Suites for a 100% A+ rating at SSLLabs
Last updated on 20210531 using Mozilla SSL Configuration Generator.
https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6

Code: [Select]

Cipher List
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384

Cipher Suites
TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
What will the end result look like?
We will have a wildcard SSL certificate from Let's Encrypt that gets an A+ rating with 100% score at all points from SSLLabs.
https://www.ssllabs.com/ssltest/

We will also have two levels of load balancing our services.

Level A - SSL Offloading disabled
WWW --> WAN interface --> OPNsense --> HAProxy SNI Frontend --> internal servers / services

Level B - SSL Offloading enabled
WWW --> WAN interface --> OPNsense --> HAProxy SNI Frontend --> HAProxy SSL Frontend --> internal servers / services

What are we going to do?

• We will install the necessary plugins.
• We will use a free DynDNS provider (https://desec.io/) that supports the DNS challenge which is mandatory in order to obtain a wildcard certificate.
  If you already have your own domain and your hosting provider offers an API that is supported by the Let's Encrypt (ACME) plugin for OPNsense, you can use it instead.
• We will create a wildcard SSL certificate using Let's Encrypt.
• We will configure the necessary firewall rules and change some OPNsense settings in order for HAProxy to function properly.
• We will use HAProxy to do SNI (explanation below) and SSL offloading.
• We will enable access to HAProxy from the internal network.

FAQ

• Why are we using wildcard certificates and not just regular certificates?
  For me the main reason is simplicity, there is no need to set up multiple certificates for multiple subdomains.
  Also you don't need to add any rules in HAProxy or your firewall for the ACME plugin to function correctly as the DNS challenge doesn't need this.
  If you want to read more about the differences follow the links below.
  https://sectigostore.com/page/ssl-vs-wildcard-ssl-certificate/
  https://comodosslstore.com/resources/whats-the-difference-wildcard-certificate-vs-regular-ssl-certificate/
• How does the DNS challenge work?
  https://letsencrypt.org/docs/challenge-types/
• What is SNI?
  https://en.wikipedia.org/wiki/Server_Name_Indication
  As you already saw above we are going to do it in two stages and not just one.
• Why are we doing 2-Level-SNI?
  The main reason for this is so we can load balance services that don't require additional SSL offloading, f.e. OpenVPN over TCP.
  Basically we will have two frontends listening on port 443, one with and one without SSL offloading.
• Why are we using a virtual IP?
  You don't have to!
  I am only doing it because I don't want to use the localhost as my "SSL_server". I just prefer to keep things isolated from each other.
  You can of course simply use the localhost (127.0.0.1) as your "SSL_server" and let your "HTTPS_frontend" listen to that IP.
• How can I add new services in HAProxy?
  It is almost everytime the same procedure to configure / add a new service in HAProxy.
  
  • Create the server or servers, f.e. "NEW_server".
  • Create the corresponding backend containing "NEW_server", f.e. "NEW_backend".
  • Create the condition that matches the traffic of "NEW_server", f.e. "NEW_condition".
  • Create the rule that redirects traffic to "NEW_backend", f.e. "NEW_rule".
  • Add "NEW_rule" to your "SNI_frontend" (if you don't need SSL offloading) or to your "HTTPS_frontend" if you need SSL offloading.
• Is there a faster way of adding new services if I have lots of subdomains and services?
  Yes there is! Thank you @sorano for pointing this out.
  https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/overview/
  https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/syntax/
  
  • Create the map file containing all of your subdomains with the corresponding backend.
  • Create a rule that uses that map file.
  • Add this rule to your desired frontend.
• How can I load balance traffic that is not SSL offloaded?
  In my tutorial I only show how to "redirect+load balance SSL offloaded traffic" and how to "redirect non SSL offloaded traffic".
  This is because I myself don't have (yet) the need to actually load balance any non SSL traffic.
  However balancing non SSL traffic is pretty much the same as balancing SSL traffic.
  You only have to make sure that your "NOSSLservice_rule" or "NOSSLservices_map-file_rule" is placed on the "SNI_frontend" instead of the "HTTPS_frontend" and that the backend that belongs to your "NOSSLservice_server" is running in TCP mode.

The Configuration
Part 1 - Plugin Installation

• Make sure your OPNsense is up to date and that you are using the OpenSSL firmware flavour as the LibreSSL version doesn't support TLS_1.3 as of writing this.
  In your OPNsense, go to "System --> Firmware --> Updates" and install all updates.
• Go to "System --> Firmware --> Plugins" and install the following plugins: os-acme-client, os-dyndns, os-haproxy

Part 2 - DynDNS Configuration

• Visit https://desec.io/signup, create your account and verify your email address.
• Visit https://desec.io/domains and create your domain by clicking on the "round yellow + icon" in the top right corner.
  Note: Your domain has to be in the form of "your_domain.dedyn.io". I will use "tutorial.dedyn.io" as an example.
  
• Visit https://desec.io/tokens, create a token for your domain and name it accordingly.
  
• Save the token somewhere secure as we will need it twice during this tutorial and you also might need it again in some time.
  This token will allow the DynDNS service and the Let's Encrypt plugin of your OPNsense to authenticate to the API of deSEC.
  
• The next step is to add a CAA record that contains the information on who issued our SSL certificate.
  This is important in order to get an A+ rating from SSLLabs.
  Visit https://desec.io/domains/your_domain.dedyn.io and add a record to your domain by clicking on the "round yellow + icon" in the top right corner.
  
• The next step would be to add a CNAME record to your subdomain for each of your services.
  If you don't need or want to do this, just skip this step. But in my opinion it is smart to do this as it makes it easier to load balance the services with HAProxy.
  You can either add an individual CNAME record per service, f.e. "plex.your_subdomain.dedyn.io", "mail.your_subdomain.dedyn.io" and so on.
  OR
  You can do it like me and just create a single wildcard CNAME record, f.e. "*.your_subdomain.dedyn.io".
  With the later "any_string.your_subdomain.dedyn.io" will resolve to "your_subdomain.dedyn.io".
  
• In your OPNsense go to: Services --> Dynamic DNS
  Click on "Add", fill out the information accordingly and hit "Save and Force Update".
  (The password is your token.)
  
• If everything went right "your_subdomain.dedyn.io" and "any_string.your_subdomain.dedyn.io" should resolve to your public IP.
  You can test this by going to: Interfaces --> Diagnostics --> Ping
  If it doesn't work wait a few minutes as it can take some time before all the DNS providers around the world have your hostname in their database.
  If it still doesn't work check your configuration.
• Now that we have our DynDNS provider all set up we will want to obtain our SSL certificate.

Part 3 - Let's Encrypt

• In your OPNsense go to: Services --> Let's Encrypt --> Settings --> Settings
  Change the settings according to my image. We don't need the HAProxy integration as we are obtaining our certificates using the DNS challenge.
  Note: We will want to use the staging environment for now. (https://letsencrypt.org/docs/staging-environment/)
  
• Next go to: Services --> Let's Encrypt --> Settings --> Update Schedule
  Here we will configure at which time of the day our certificates are renewed.
  They won't be renewed everyday as the ACME client will first check if the certificates are close to expiration and if they are not they won't get renewed.
  You want this to happen at a time of the day where there is not much load on your services as the ACME plugin restarts HAProxy so it can use your new certificates which results in a very short downtime of HAProxy.
  You will also want this to NOT happen at any full hours (f.e. 3 am) because these are times when the servers of Let's Encrypt could be so busy that you certificate renewal fails.
  Change the settings according to my image.
  
• Next go to: Services --> Let's Encrypt --> Accounts
  Create a new account. You can name it whatever you like, I usually use my first level subdomain.
  
• Next go to: Services --> Let's Encrypt --> Automations
  Create the automation to restart HAProxy after our certificates have been renewed.
  
• Next go to: Services --> Let's Encrypt --> Challenge Types
  Add the DNS challenge for deSEC.
  
• Next go to: Services --> Let's Encrypt --> Certificates
  Add the certificate for your domain according to the image below.
  I prefer to use Elliptic-curve cryptography. (https://en.wikipedia.org/wiki/Elliptic-curve_cryptography)
  But you can of course also use RSA keys just make sure to set the key length as high as possible in order to get an A+ rating from SSLLabs.
  
• Next go to: Services --> Let's Encrypt --> Certificates
  Now we need to forcefully issue our staging certificate so we can test things out and don't have to wait for the next update schedule.
  To do this click on the button marked in the image.
  
• Next go to: Services --> Let's Encrypt --> Log Files --> ACME Log
  If your certifcate was issued successfully your log should look similiar and you can proceed to the next step.
  If it doesn't check your configuration.
  
• Next go to: Services --> Let's Encrypt --> Settings --> Settings
  Since we successfully issued our staging certificate we can now leave the test environment and issue our production certificate.
  First we need to change the "Let's Encrypt Environment" back to "Production Environment [default]".
• Next go to: Services --> Let's Encrypt --> Certificates
  Now again forcefully issue your certificate like we did before.
  This time it you should receive a valid and trusted SSL certificate. Make sure to check the ACME log for any errors though!
• If this went well, we can now proceed to the prepare everything for HAProxy. To do this we have to configure some things in your OPNsense.

Part 4 - System Preparation

• In your OPNsense go to: System --> Settings --> Administration
  You only need to "Disable web GUI redirect rule" and change the "Web GUI TCP port" to a custom one.
  Otherwise HAProxy will not function correctly as you will propably want to access your services from the WWW using the default HTTPS port 443.
  
• Next go to: Interfaces --> Virtual IPs --> Settings
  NOTE: This step is optional, see FAQ! You can safely skip this step and use the localhost instead for your "SSL_server".
  But if you would like to do it my way then you will need to create a virtual IP that is in a different subnet than any of your other networks.
  "192.168.64.1/32" is going to be the IP on which your "HTTPS_frontend" will be listening on.
  
• Next go to: Firewall --> Rules --> WAN
  Now we are going to allow any inbound traffic hitting our WAN interface on port 80+443.
  You will need two rules for that, one for port 80 (HTTP) and another one for port 443 (HTTPS).
  [/url]
  If you did it right it should look like this, make sure that these rules are above all other rules in the list.
  
• Now we can finally configure HAProxy and make our services available on WAN.

Part 5 - HAProxy Configuration

• In your OPNsense go to: Services --> HAProxy --> Settings --> Service
  Change the settings according to the image below.
  
• Next go to: Services --> HAProxy --> Settings --> Global Parameters
  Change the settings according to the image below.
  Note: The number of HAProxy threads should not exceed the number of CPU threads of your OPNsense.
  
• Next go to: Services --> HAProxy --> Settings --> Default Parameters
  Change the settings according to the image below.
  
• Next go to "System --> Settings --> Cron" and create a new cron job.
  Because our certificate has the OCSP Must Staple extension we need to update HAProxy's OCSP data regularly.
  If we don't do this clients connecting to our services will get security warnings or won't connect at all.
  To do this we simply create a cron job that does exactly this.
  It doesn't matter if this job runs before or after a certificate renewal as the OCSP data gets updated anyway after installing new certificates in HAProxy.
  This is because the ACME plugin restarts HAProxy after installing the new certificates.
  
• Next go to: Services --> HAProxy --> Settings --> Real Servers
  First we will add our "SSL_server" using our virtual IP (or localhost IP if you wish) to do the SSL offloading.
  [/url]
  Then you can add all your other services according to their individual configuration.
  
• Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Backend Pools
  Here we first create our "SSL_backend". This is the backend to which our "SNI_frontend" sends most of its traffic to.
  As you can't mix HTTP mode and TCP mode in a frontend to backend relation, make sure that the "SSL_backend" is set to TCP mode since our "SNI_frontend" is also running in TCP mode.
  
  Now we create the backend that belongs to an actual service. You will need one backend for each service.
  If you have multiple servers serving the exact same content than you will want to add all servers into a single backend so HAProxy can actually balance the load between the servers.
  
• Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Rules & Checks --> Conditions
  Here we add the conditions matching our services.
  I will create a "NoSSL_condition" which is necessary to forward HTTP traffic to HTTPS ...
  
  ... and my "PLEX_condition".
  
• Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Rules & Checks --> Rules
  Here we add the rules that decide what to do with the traffic.
  The "HTTPtoHTTPS_rule" forwards all HTTP traffic to HTTPS so it can go to our "HTTPS_frontend".
  
  The "PLEX_rule" forwards traffic accessing our hostname using "plex.tutorial.dedyn.io" to our "PLEX_backend" and from there to our "PLEX_server".
  
• Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Public Services
  Here we will create the frontends that are listening on our interface IPs and the virtual IP we created earlier.
  At first we will create our "SNI_frontend" which will decide wether the traffic is going to be SSL offloaded or not.
  You will have to place the rules for all of your services that you don't want to get SSL offloaded in here.
  Our default backend in this frontend will be the "SSL_backend" that redirects all traffic to our virtual "SSL_server" which is actually our "HTTPS_frontend".
  
  No we will create our "HTTP_frontend". Make sure to place the "HTTPtoHTTPS_rule" in this frontend!
  This frontend is necessary in order to redirect HTTP traffic to HTTPS. But you could also use it to serve non SSL encrypted services on port 80.
  
  No we will create our "HTTPS_frontend".
  This will be our primary frontend which is doing the SSL offloading using our earlier created Let's Encrypt certificate.
  You will have to place the rules for all of your services that you want to get SSL offloaded in here.
  
• Done.
  Access from external networks should now already be working.
  Just try to access your URL "your_service.your_subdomain.dedyn.io" from any device that is not connected to your local network, f.e. a smartphone on cellular data.
• You can now verify your SSL settings: https://www.ssllabs.com/ssltest/
• If you configured everything in HAProxy and Let's Encrypt according to my template you should get an A+ rating with a 100% score at all points.
• The last thing for us to do now is enable access from internal networks.

Part 6 - Access from internal networks
If you try to access your URL "your_service.your_subdomain.dedyn.io" from a device in your internal network, it should fail.
There are two ways of fixing this. I will cover both options but keep in mind that Option B is the suggested way of doing it.
NAT reflection is an inferior solution since you lose the ability to track originating source IP in HAProxy when going through NAT. (@sorano)

Option A - NAT Reflection (https://docs.opnsense.org/manual/nat.html)
Option B - Split DNS (https://docs.opnsense.org/manual/unbound.html#overrides)

Option A - NAT Reflection

• In your OPNsense go to: Firewall --> Rules --> WAN
• Here you will have to edit the two rules (HAProxy HTTP and HAProxy HTTPS) we created in Part 4 - Step 3 of this tutorial.
  At the bottom of each rule there is a setting called "NAT reflection = Use system default".
  You will want to change this to "NAT reflection = Enable".
• Access from internal networks should now be working.

Option B - Split DNS (DNS Overrides)
Since you are using OPNsense you are probably also using the Unbound DNS plugin as your local DNS server.
Because of that you can easily set up DNS overrides.

• In your OPNsense go to: Services --> Unbound DNS --> Overrides
• Here you will need to create "Host Overrides" for each of your services. At least if you are using 2nd level subdomains "your_service.your_subdomain.dedyn.io" for you services.
  If you are running all of your services on your 1st level subdomain "your_subdomain.dedyn.io" than you will just need to override this one.
• The IP address can be any interface IP of your OPNsense, I am using the LAN IP on which the "SNI_frontend" is also listening on because we set it to "0.0.0.0".
  
• Access from internal networks should now be working.