Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[TheHellSite]

Hello,

when I started implementing HAProxy in my network I couldn't find any complete and well written guide out there. I had to puzzle everything together from various websites.
So I thought I would save many of you a lot of time and provide my ultimate HAProxy on OPNsense tutorial.

This tutorial will show you how to configure HAProxy as a reverse proxy on OPNsense using wildcard certificates from Let's Encrypt.
It is going to be a step-by-step guide with images on how to set things up while also explaining why we set things up in a certain way.
I will try to make this as complete and detailed as possible.
If you think that there is anything wrong or missing, feel free to tell me about it and I will consider changing it.

Consider Supporting Free Resources

If you find this tutorial helpful and it saves you a significant amount of time, please consider showing your appreciation by buying me a beer. Your contribution will be a valuable recognition of the time and effort I have put into creating this content and will help me to continue providing quality resources for others.

Thank you for your consideration,
TheHellSite

No More Free Support

Due to the increasing number of support requests I've been receiving, both directly in the topic and via DM, I regret to inform that I can no longer provide free assistance. Balancing my real job and personal life has become extremely challenging. While I genuinely want to help everyone resolve their issues to get things up and running smoothly, I find it difficult to allocate the necessary time without sacrificing my personal commitments.

In addition, it has come to my attention that some individuals seeking help are not thoroughly reading the provided tutorial or lack the fundamental knowledge of networking. This has been a recurring issue and has made the support process increasingly frustrating.

I sincerely appreciate your interest in my expertise and if you would like to receive my assistance, I am more than happy to provide you with the details via DM.

Thank you for your understanding in this matter,
TheHellSite

This configuration is tested to be working on OPNsense 24.1.x with the latest updates as of 20240207.

How To Ask For Help

When asking for help in the topic ALWAYS include the below information:

1. ALWAYS include the HAProxy Config Export.
2. ALWAYS include relevant HAProxy errors and/or log entries.
3. ALWAYS include details about your setup, your goal, the service, ... anything relevant to the issue.

Again, please note that I can no longer provide free support.

Changelog

• 20210603
  
  • Fixes and some enhancements

• 20210611
  
  • Implemented @sorano's enhancements

• 20210613
  
  • The tutorial is now using a wildcard CNAME record.
  • Enabled Proxy Protocol in the "SSL_backend", "HTTPS_frontend" and "HTTP_frontend" configuration so that the IPs of clients accessing HAProxy will now no longer be overwritten with the "SSL_server" IP.
    https://www.haproxy.com/blog/using-haproxy-with-the-proxy-protocol-to-better-secure-your-database/

• 20210729
  
  • Added an alias for the HAProxy ports and updated the WAN interface firewall rule with it.
    This leaves us with only one firewall rule instead of two and makes even more sense if one is using additional frontends on different ports.
    Thanks @_Alchemist_ for the suggestion.
  • The tutorial is now using a map file instead of "condition + rule" for service configuration.

• 20210730
  
  • Added an explanation on how to configure local-access-only subdomains in HAProxy.

• 20210801
  
  • Added an explanation about using self-signed certificates for internal communication to the FAQ.

• 20211110
  
  • Updated the Let's Encrypt part since the service has been renamed to ACME client.

• 20220411
  
  • Updated the Let's Encrypt part because of changes to the wildcard certificate generation.

• 20220604
  
  • Updated the DynDNS part to use the newer "Dynamic DNS Client" (os-ddclient) plugin.
    The previous "Dynamic DNS Support" (os-dyndns) plugin will be removed with the release of OPNsense 22.7.x.

• 20220611
  
  • Changed the IP address of the "SSL_server" to one that belongs to the localhost subnet.
  • Updated the "NoSSL_condition" based on the HAProxy docs.

• 20230223
  
  • Updated Part 7 to avoid confusion.

• 20230304
  
  • Added Part 8 to hide the certificate on access by IP.

• 20240201
  
  • Simplified and improved Part 8. The "INVALID_SNI" certificate is no longer needed, instead we use the "strict-sni" parameter to achieve an even better result.
    Thanks @mnaiman for the suggestion.
    See here for instructions on how to update your HAProxy configs.
  • Updated Part 5 - OCSP updates are now built into HAProxy. No external Cron job is necessary anymore.
    Everyone please take note of the warning from @vladnik.
    

Current Ciphers and Cipher Suites for a 100% A+ SSLLabs rating

Last updated/verified on 20230223 using Mozilla SSL Configuration Generator.
https://ssl-config.mozilla.org/#server=haproxy&version=2.4.17&config=intermediate&openssl=1.1.1o&guideline=5.6

All ciphers with a strength of 128 bit or below have been removed in order to get a 100% A+ rating at SSL Labs.

Code: [Select]

Cipher List
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384

Cipher Suites
TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

What will the end result look like?

We will have a wildcard SSL certificate from Let's Encrypt that gets an A+ rating with a score of 100% in each category in SSLLabs SSL Server Test.
https://www.ssllabs.com/ssltest/

We will also have two levels of load balancing our services.

Level 0 - SSL Offloading disabled
WWW --> WAN interface --> OPNsense --> HAProxy SNI Frontend --> internal servers / services

Level 1 - SSL Offloading enabled
WWW --> WAN interface --> OPNsense --> HAProxy SNI Frontend --> HAProxy SSL Frontend --> internal servers / services

Visual Schematic
(Idea: @cookiemonster)

What are we going to do?

• We will install the necessary plugins.
• We will use a free DynDNS provider (https://desec.io/) that supports the DNS challenge which is mandatory in order to obtain a wildcard certificate.
  If you already have your own domain and your hosting provider offers an API that is supported by the Let's Encrypt (ACME Client) plugin for OPNsense, you can use it instead.
• We will create a wildcard SSL certificate using Let's Encrypt.
• We will configure the necessary firewall rules and change some OPNsense settings in order for HAProxy to function properly.
• We will use HAProxy to do SNI (explanation below) and SSL offloading.
• We will enable access to HAProxy from the internal network.

FAQ

• Why are we using wildcard certificates and not just regular certificates?
  For me the main reason is simplicity, there is no need to set up multiple certificates for multiple subdomains.
  Also you don't need to add any rules in HAProxy or your firewall for the ACME plugin to function correctly as the DNS challenge doesn't need this.
  If you want to read more about the differences follow the links below.
  https://sectigostore.com/page/ssl-vs-wildcard-ssl-certificate/
  https://comodosslstore.com/resources/whats-the-difference-wildcard-certificate-vs-regular-ssl-certificate/
• How does the DNS challenge work?
  https://letsencrypt.org/docs/challenge-types/
• What is SNI?
  https://en.wikipedia.org/wiki/Server_Name_Indication
  As you already saw above we are going to do it in two stages and not just one.
• Why are we using a virtual IP?
  You don't have to!
  I am only doing it because I don't want to use the localhost as my "SSL_server". I just prefer to keep things isolated from each other.
  You can of course simply use the localhost (127.0.0.1) as your "SSL_server" and let your "HTTP_frontend" / "HTTPS_frontend" listen to that IP.
• Why are we doing 2-Level-SNI?
  The main reason for this is so we can load balance services that don't require additional SSL offloading, f.e. OpenVPN over TCP.
  Basically we will have two frontends listening on port 443, one with and one without SSL offloading.
• How can we load balance TCP traffic that we don't want to get SSL offloaded, f.e. OpenVPN over TCP?
  In my tutorial I only explain how to "redirect+load balance SSL offloaded traffic".
  This is because I myself don't have (yet) the need to actually load balance any non SSL traffic.
  It is perfectly possible to do this but I won't cover it in the guide. You can hit me up if you need help with this.
• How can I add new services in HAProxy?
  It is almost everytime the same procedure to configure / add a new service in HAProxy.
  The long way is to...
  
  • Create the server or servers, f.e. "NEW_server".
  • Create the corresponding backend containing "NEW_server", f.e. "NEW_backend".
  • Create the condition that matches the traffic of "NEW_server", f.e. "NEW_condition".
  • Create the rule that redirects traffic to "NEW_backend", f.e. "NEW_rule".
  • Add "NEW_rule" to your "SNI_frontend" (if you don't need SSL offloading) or to your "HTTPS_frontend" (if you need SSL offloading).
• Is there a faster way of adding new services if I have a lot of subdomains and services?
  Yes there is! Thank you @sorano for pointing this out.
  https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/overview/
  https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/syntax/
  Map files are a great way of adding new services to your HAProxy configuration.
  It is most of the time much faster than creating a individual condition + rule for each service and then adding each rule to your frontend.
  To sum this up, the short way is to...
  
  • Create the server or servers, f.e. "NEW_server".
  • Create the corresponding backend containing "NEW_server", f.e. "NEW_backend".
  • Create a map file containing all of your subdomains (of your services) with the corresponding backend.
  • Create a rule that uses that map file.
  • Add this rule to your desired frontend.
  You might be asking yourself now: Why is this faster?
  Well, because next time you add a new service you will only need to create the server, the backend and add the combination of "subdomain to backend" to your map file.
• Do I need to enable "SSL" in the Real Server configuration of a service?
  You need to think of a reverse proxy setup like this.
  
  WWW ---Stage 1---> yourdomain.tld ---Stage 2---> OPNsense + HAProxy + LE ---Stage 3---> internal services
  
  Stage 1 + 2
  Public facing external traffic. Traffic in these stages is now always encrypted with a verified SSL certificate. In this case it is created and verified by Let's Encrypt.
  
  Stage 3
  Local facing internal traffic. Traffic in this stage can or can not be encrypted, depending on your service setup. This is the traffic from HAProxy to your internal service. It doesn't need to be encrypted because you can consider your internal network as trusted.
  However it is still strongly advised to also run this traffic encrypted.
  In HAProxy you only need to check the "SSL" box in your real server setting for this.
  But then you also need to actually enable SSL encryption on that service, f.e. by installing a self-signed certificate on that service and enabling HTTPS. Even though using a self-signed certificate will give you a warning by your browser when accessing the service directly and not through the reverse proxy, the traffic is still encrypted. The certificate is just unverified, which isn't that much of an issue since we are using our reverse proxy in front of it anyway.
  How to actually do this this depends on the service but this should be covered somewhere in its manual.
  
  You can read more about this here: https://www.globalsign.com/en/ssl-information-center/dangers-self-signed-certificates
  

The Configuration

Part 1 - Plugin Installation

• Make sure your OPNsense is up to date and that you are using the OpenSSL firmware flavour as the LibreSSL version doesn't support TLS_1.3 as of writing this.
  In your OPNsense, go to "System --> Firmware --> Updates" and install all updates.
• Go to "System --> Firmware --> Plugins" and install the following plugins: os-acme-client, os-ddclient, os-haproxy

Part 2 - DynDNS configuration

• Visit https://desec.io/signup, create your account and verify your email address.
• Visit https://desec.io/domains and create your domain by clicking on the "round yellow + icon" in the top right corner.
  Your domain has to be in the form of "your_subdomain.dedyn.io". I will use "tutorial.dedyn.io" as an example.
  Note: Please don't get confused by the word "domain" here as you are not really creating a proper domain but actually just a subdomain of the domain "dedyn.io".
  
• Visit https://desec.io/tokens, create a token for your domain and name it accordingly.
  
• Save the token somewhere secure as we will need it twice during this tutorial and you also might need it again in the future.
  This token will allow the DynDNS service and the Let's Encrypt plugin of your OPNsense to authenticate to the API of deSEC.
  
• The next step is to add a CAA record containing the information on which certification authority (CA) is allowed to issue SSL certificates for your domain.
  This is important in order to get an A+ rating from SSLLabs.
  Visit https://desec.io/domains/your_subdomain.dedyn.io and add the CAA recoard by clicking on the "round yellow + icon" in the top right corner.
  
• The next step is to add a CNAME record for your services.
  If you don't need or want to do this because you only have one service, just skip this step. But in my opinion it makes sense to do this as it simplifies loadbalancing the services with HAProxy.
  You can either add an individual CNAME record per service, f.e. "plex.your_subdomain.dedyn.io", "mail.your_subdomain.dedyn.io" and so on.
  OR
  You just create a single wildcard CNAME record "*.your_subdomain.dedyn.io" for all your services.
  With the latter "any_string.your_subdomain.dedyn.io" will resolve to the IP of "your_subdomain.dedyn.io".
  
• In your OPNsense go to: Services --> Dynamic DNS --> General settings
  Check that "Enable" is ticked.
• In your OPNsense go to: Services --> Dynamic DNS --> Accounts
  Click on "Add", fill out the information accordingly, hit "Save" and then "Apply".
  (The password is your token.)
  
• If everything went right "your_subdomain.dedyn.io" and "any_string.your_subdomain.dedyn.io" should resolve to your public IP.
  You can test this by going to: Interfaces --> Diagnostics --> Ping
  If it doesn't work wait a few minutes as it can take some time before all the DNS providers around the world have your hostname in their database.
  If it still doesn't work check your configuration.
• Now that we have our DynDNS provider all set up we will want to obtain our SSL certificate.

Part 3 - Let's Encrypt (ACME Client)

• In your OPNsense go to: Services --> ACME Client --> Settings
  Change the settings according to my image. We don't need the HAProxy integration as we are obtaining our certificates using the DNS challenge.
  
• Next go to: Services --> ACME Client --> Settings --> Update Schedule
  Here we will configure at which time of the day our certificates are renewed.
  They won't be renewed everyday as the ACME client will first check if the certificates are close to expiration and if they are not they won't get renewed.
  You want this to happen at a time of the day where there is not much load on your services as the ACME plugin restarts HAProxy so it can use your new certificates which results in a very short downtime of HAProxy.
  You will also want this to NOT happen at any full hours (f.e. 3 am) because these are times when the servers of Let's Encrypt could be so busy that your certificate renewal fails.
  Change the settings according to my image.
  
• Next go to: Services --> ACME Client --> Accounts
  Create a new account. You can name it whatever you like, I usually use the domain name (if it is a real domain) or in this case the FQDN.
  Note: We will want to use the staging environment "Let's Encrypt Test CA" for now. (https://letsencrypt.org/docs/staging-environment/)
  
• Next go to: Services --> ACME Client --> Automations
  Create the automation to restart HAProxy after our certificates have been renewed.
  
• Next go to: Services --> ACME Client --> Challenge Types
  Add the DNS challenge for deSEC.
  
• Next go to: Services --> ACME Client --> Certificates
  Add the certificate for your domain according to the image below.
  I prefer to use Elliptic Curve Cryptography (ECC). (https://en.wikipedia.org/wiki/Elliptic-curve_cryptography)
  But you can of course also use RSA keys just make sure to set the key length as high as possible in order to get an A+ rating from SSLLabs.
  
• Next go to: Services --> ACME Client --> Certificates
  Now we need to forcefully issue our staging certificate so we can test things out and don't have to wait for the next update schedule.
  To do this click on the button marked in the image.
  
• Next go to: Services --> ACME Client --> Log Files --> ACME Log
  If your certificate is issued successful your log should look similiar and you can proceed to the next step.
  If it doesn't check your configuration.
  
• Next go to: Services --> ACME Client --> Accounts
  Since we successfully issued our staging certificate we can now leave the test environment and issue our production certificate.
  To do this edit your account and change the ACME CA from "Let's Encrypt Test CA" to "Let's Encrypt [default]".
• Next go to: Services --> ACME Client --> Certificates
  Now again forcefully issue your certificate like we did before.
  This time it you should receive a valid and trusted SSL certificate. Make sure to check the ACME log for any errors though!
• If this went well, we can now proceed to the prepare everything for HAProxy. To do this we have to configure some things in your OPNsense.

Part 4 - System preparation

• In your OPNsense go to: System --> Settings --> Administration
  You only need to check "Disable web GUI redirect rule" and change the "Web GUI TCP port" to a custom one.
  Otherwise HAProxy will not function correctly as you will propably want to access your services from the WWW using the default HTTPS port (TCP 443).
  
• Next go to: Interfaces --> Virtual IPs --> Settings
  NOTE: This step is optional, see FAQ! You can safely skip this step and use the localhost instead for your "SSL_server".
  But if you would like to do it my way then you will need to create a virtual IP that is in a different subnet than any of your other networks.
  Preferably you would chose an IP that belongs to the localhost subnet in order to avoid IP conflicts in your local network. (https://en.wikipedia.org/wiki/Localhost)
  In this tutorial "127.4.4.3/32" is going to be the IP on which the "HTTP_frontend" and "HTTPS_frontend" will be listening on.
  
• Next go to: Firewall --> Aliases
  Now we are going to create an alias for the ports that HAProxy will be listening on.
  In most setups you will probably need at least 80 and 443.
  
• Next go to: Firewall --> Rules --> WAN
  Now we are going to allow any inbound traffic hitting our WAN interface on the ports specified in the "HAProxy_ports" alias.
  Create a new firewall rule with the content below.
  
  If you did it right it should look like this, make sure that the rule is above all other rules in the list.
  
• Now we can finally configure HAProxy and make our services available on WAN.

Part 5 - HAProxy configuration

• In your OPNsense go to: Services --> HAProxy --> Settings --> Service
  Change the settings according to the image below.
  
• Next go to: Services --> HAProxy --> Settings --> Global Parameters
  Change the settings according to the image below.
  As our certificate has the OCSP Must Staple extension we need to update HAProxy's OCSP data regularly. If we don't do this clients connecting to our services will get security warnings or won't connect at all.
  Note: The number of HAProxy threads should not exceed the number of CPU threads of your OPNsense.
  
• Next go to: Services --> HAProxy --> Settings --> Default Parameters
  Change the settings according to the image below.
  
• Next go to: Services --> HAProxy --> Settings --> Real Servers
  First we will add our "SSL_server" using our virtual IP (or localhost IP if you wish) to do the SSL offloading.
  
  Then you can add all your other services according to their individual configuration.
  
• Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Backend Pools
  Here we first create our "SSL_backend". This is the backend to which our "SNI_frontend" sends most of its traffic to.
  As you can't mix HTTP mode and TCP mode in a frontend to backend relation, make sure that the "SSL_backend" is set to TCP mode since our "SNI_frontend" is also running in TCP mode.
  
  Now we create the backend that belongs to an actual service. You will need one backend for each service.
  If you have multiple servers serving the exact same content than you will want to add all servers into a single backend so HAProxy can actually balance the load between the servers.
  
• Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
  Here we will only create a "NoSSL_condition", which is necessary in order to identify non-HTTPS traffic.
  
• Next go to: Services --> HAProxy --> Settings --> Advanced --> Map Files
  Here we will create a new map file "PUBLIC_SUBDOMAINS_mapfile" for our public subdomains that we want to access from outside of our network.
  Please read the FAQ about Map Files first!
  This map file is telling HAProxy that any FQDN that starts with "plex" belongs to our "PLEX_backend" (which belongs to our "PLEX_server").
  
• Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Rules
  Here we add the rules that decide what to do with the traffic based on our map files (or conditions if necessary).
  The "HTTPtoHTTPS_rule" upgrades the HTTP connection to HTTPS so that the client connects to the "HTTPS_frontend" instead.
  
  The "PUBLIC_SUBDOMAINS_rule" maps our subdomains to our backends using the map file we created in the previous step.
  
• Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Public Services
  Here we will create the frontends that are listening on our interface IPs and the virtual IP we created earlier.
  At first we will create our "SNI_frontend" which will decide wether the traffic is going to be SSL offloaded or not.
  You will have to place the rules for all of your services that you don't want to get SSL offloaded in here.
  Our default backend in this frontend will be the "SSL_backend" that redirects all traffic to our virtual "SSL_server" which is actually our "HTTPS_frontend".
  
  No we will create our "HTTP_frontend". Make sure to place the "HTTPtoHTTPS_rule" in this frontend!
  This frontend is necessary in order to redirect HTTP traffic to HTTPS. But you could also use it to serve non SSL encrypted services on port 80.
  
  No we will create our "HTTPS_frontend".
  This will be our primary frontend which is doing the SSL offloading using our earlier created Let's Encrypt certificate.
  You will have to place the "PUBLIC_SUBDOMAINS_rule" and all other rules of your services that you want to get SSL offloaded in here.
  Don't forget to change the "Cipher List" and "Cipher Suites" with the ones at the top of this tutorial "Current Ciphers and Cipher Suites for a 100% A+ rating at SSLLabs".
  
• Done.
  Access from external networks should now already be working.
  Just try to access your URL "your_service.your_subdomain.dedyn.io" from any device that is not connected to your local network, f.e. a smartphone on cellular data.
• You can now also verify your SSL settings: https://www.ssllabs.com/ssltest/
• If you configured everything in HAProxy and the ACME Client according to my template you should get an A+ rating with a score of 100% in each category.
  
• The last thing for us to do now is enable access from internal networks.

Part 6 - Access from internal networks
If you try to access your URL "your_service.your_subdomain.dedyn.io" from a device in your internal network, it should fail.
There are two ways of fixing this. I will cover both options but keep in mind that Split DNS (Option A) is the suggested way of doing it.
NAT Reflection (Option B) is an inferior solution since you lose the ability to track originating source IP in HAProxy when going through NAT. (@sorano)

Option A - Split DNS (https://docs.opnsense.org/manual/unbound.html#overrides)
Option B - NAT Reflection (https://docs.opnsense.org/manual/nat.html)

Option A - Split DNS (DNS Overrides)
Since you are using OPNsense you are probably also using the Unbound DNS plugin as your local DNS server.
Because of that you can easily set up DNS overrides.

• In your OPNsense go to: Services --> Unbound DNS --> Overrides
• Here you will need to create "Host Overrides" for each of your services. At least if you are using 2nd level subdomains "your_service.your_subdomain.dedyn.io" for you services.
  If you are running all of your services on your 1st level subdomain "your_subdomain.dedyn.io" than you will just need to override this one.
• The IP address can be any LAN (or VLAN) interface IP of your OPNsense, I am using the LAN IP on which the "SNI_frontend" is also listening on because we set it to "0.0.0.0".
  
• Access from internal networks should now be working.

Option B - NAT Reflection
Please note that NAT Reflection is only applicable on port forwarding rules so you will have to change the "Allow HAProxy" rule to a port forwarding rule with the localhost (or some random virtual IP on the localhost) as target.

• In your OPNsense go to: Firewall --> NAT --> Port Forward
• Here you will have to edit the "Allow HAProxy" rule we created in Part 4 - Step 3 of this tutorial.
  At the bottom of each rule there is a setting called "NAT reflection = Use system default".
  You will want to change this to "NAT reflection = Enable".
• Access from internal networks should now be working.

Part 7 - Advanced Configuration: local-access-only subdomains
Imagine you have a service that you would like to access / protect using your brand new reverse proxy without making it available on the internet?
Well, HAProxy has got you covered!

• In your OPNsense go to: Services --> HAProxy --> Settings --> Advanced --> Map Files
  Here you need to clone the "PUBLIC_SUBDOMAINS_mapfile", rename it to f.e. "LOCAL_SUBDOMAINS_mapfile" and add all your local-access-only subdomains along with their corresponding backends.
  Keep in mind that the content of your "PUBLIC_SUBDOMAINS_mapfile" also has to be put in the "LOCAL_SUBDOMAINS_mapfile"! I will explain why later.
  
• Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
  Now you need a condition that detects if the source of the request is a local IP or a FQDN.
  You can of course also use the predefined "Source IP is local" condition.
  I am however using only specific subnets since the predefined condition is using the entire RFC1918 IP range, which I don't need!
  
  
  As I just said you can also check for a FQDN.
  But please keep in mind that HAProxy resolves those hostnames to their IPs and then checks them. But the resolving is only done once during the start / restart of HAProxy.
  So if the IP of your FQDN is changing regularly this won't work very well, except if you restart your HAProxy using a cron job like every 24 hours or so.
  
• Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Rules
  Here you need to clone the "PUBLIC_SUBDOMAINS_rule", rename it to f.e. "LOCAL_SUBDOMAINS_rule", select your "LOCAL_SUBDOMAINS_SUBNETS_condition" and select your "LOCAL_SUBDOMAINS_mapfile".
  If you are also using a FQDN condition, like I do, you will need to select both your FQDN and your subnet condition together with the logical "or" operator!
  
• Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Public Services
  The last thing left to do is to place the "LOCAL_SUBDOMAINS_rule" before your "PUBLIC_SUBDOMAINS_rule" in your "HTTPS_frontend".
  
  Attention!
  Remember that I told you to also put the content of your "PUBLIC_SUBDOMAINS_mapfile" in the "LOCAL_SUBDOMAINS_mapfile"?
  This is because HAProxy is processing the rules in the frontends based on the order they appear!
  So if you place your "PUBLIC_SUBDOMAINS_rule" before your "LOCAL_SUBDOMAINS_rule" in the frontend configuration, you won't get access to your local-access-only subdomains.
  Vice versa this will also happen and you will no longer have access to your public subdomains.
  To avoid this you have to also put the content of your "PUBLIC_SUBDOMAINS_mapfile" in the "LOCAL_SUBDOMAINS_mapfile" and place their rules in the correct order.
  
  The correct way of placing both rules is like this.
  
• Done!
  You should now still have access to your public subdomains from any network and also have access to your local-access-only subdomains from the locations you defined.

Part 8 - Advanced Configuration: Hide your certificate on access by IP
You might have noticed that if you now access your OPNsense using your public WAN IP (https://YOUR_PUBLIC_IP/) the connection will be secured and upon further inspection you will see that your Let's Encrypt certificate is beeing used.
While this is not a major security problem it still presents at least some privacy issues.
To fix this we can add the "strict-sni" parameter to the "HTTPS_frontend" so the connection will be closed when HAProxy is directly accessed using your public WAN IP (https://YOUR_PUBLIC_IP/).

• In your OPNsense go to: Services --> HAProxy --> Settings --> Virtual Services --> Public Services
  Edit your "HTTPS_frontend" and enable the "advanced mode" in the top left corner.
  You should now be able to see the "SSL option pass-through" field in the "SSL Offloading" section, here we already added the parameter "curves secp384r1" to make use of ECC.
  Change this field from "curves secp384r1" to "curves secp384r1 strict-sni", save the changes to the "HTTPS_frontend" and apply them to HAProxy.
• Done!
  You should now no longer get presented with your trusted Let's Encrypt certificate when accessing "https://YOUR_PUBLIC_IP/" but instead with a "SSL_ERROR_UNRECOGNIZED_NAME_ALERT".

[browne]

Thank you very much!
This helped me switching from regular certificates to wildcard certificates!
I now also do score 100% A+ in the SSL test.

[skittlebrau]

Thank you so much for this guide! I was completely lost in the new UI layout before.

Do you mind showing in Step 5 an example of how you've configured 'PLEX_backend'? I noticed you referred to it, but there wasn't a screenshot for it.

[TheHellSite]

The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different.

--> I will add it to the tutorial.
Reason: https://forum.opnsense.org/index.php?topic=23339.msg111143#msg111143


It is also almost everytime the same procedure to configure / add a new service in HAProxy.

• Create the server or servers, f.e. NEW_server.
• Create the corresponding backend containing NEW_server, f.e. NEW_backend.
• Create the condition that matches the traffic of NEW_server, f.e. NEW_condition.
• Create the rule that redirects traffic to NEW_backend, f.e. NEW_rule.
• Add NEW_rule to your SNI_frontend if you don't need SSL offloading or to your HTTPS_frontend if you need SSL offloading.

--> I will add this to the FAQ.

[ejball02]

I'm using a self-signed cert for HTTPS inspection for content filtering. I've got OPT2 configured as a guest network on my Protectli, and content filtering, using shallalist works great. Only downside, is that when I try to access am HTTPS site, Firefox/Chrome always give a warning page: "Your connection is not private" "ERR_CERT_AUTHORITY_INVALID". After much Googling, I came across an old post, that said Let's Encrypt can give public certs which would get rid of the message.

I added the LE plugin but couldn't figure out, how to create a cert for use on the Foward Proxy "CA to use" field, required for SSL inspection. Looking through this walkthrough, I'm wondering if there is something here that can help achieve creating a cert for content filtering. Anyone have any experience with this?

[TheHellSite]

I'm wondering if there is something here that can help achieve creating a cert for content filtering. Anyone have any experience with this?

No offense, but this is meant to be a tutorial related to HAProxy in a reverse proxy setup. 
As this is the tutorial sub-forum it is not really meant to ask for help and especially not regarding different topics.
You can of course follow this guide to the part where you get your Let's Encrypt certificate, but from there on you will be on your own.

I think you should probably open up your own thread in the help forums and ask for help there.

https://forum.opnsense.org/index.php?board=27.0
or here
https://forum.opnsense.org/index.php?board=28.0
or here
https://forum.opnsense.org/index.php?board=1.0

[skittlebrau]

The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different.

I assumed that it would be the case, so thanks for confirming 

One part that's tripping me up is in my plex_backend pool and other ones I set for my internal services like Resilio Sync, if I set the mode to TCP (Layer 4) then I get a syntax error below. Switching the mode to HTTP for those two backends resolves the syntax error, but produces blank pages. Stats area in HAProxy shows the services as being UP.

[NOTICE] 152/082107 (17536) : haproxy version is 2.2.14-a07ac36
[ALERT] 152/082107 (17536) : http frontend '1_HTTPS_frontend' (/usr/local/etc/haproxy.conf.staging:70) tries to use incompatible tcp backend 'sync1_backend' (/usr/local/etc/haproxy.conf.staging:117) in a 'use_backend' rule (see 'mode').
[ALERT] 152/082107 (17536) : http frontend '1_HTTPS_frontend' (/usr/local/etc/haproxy.conf.staging:70) tries to use incompatible tcp backend 'plex_backend' (/usr/local/etc/haproxy.conf.staging:104) in a 'use_backend' rule (see 'mode').
[ALERT] 152/082107 (17536) : Fatal errors found in configuration.

[TheHellSite]

The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different.

I assumed that it would be the case, so thanks for confirming 

One part that's tripping me up is in my plex_backend pool and other ones I set for my internal services like Resilio Sync, if I set the mode to TCP (Layer 4) then I get a syntax error below. Switching the mode to HTTP [...]
[/font]

Now I remember correctly.
The SSL_backend is different from most other backends since it is NOT running in HTTP mode.
But HTTP mode is usually the one you want for a web based service.
I changed that part in the tutorial. Please take a look at the backend creation.

My bad! I wrote the tutorial from brain memory. So I guess you are my beta tester. 

[skittlebrau]

All good, you've been so helpful.

I checked through everything again and realised that when setting up the NAT port forward, I forgot to enter the dropdown menu at the end to add the associated filter rule. I was too used to pfSense automatically selecting that by default, so no wonder it wasn't working despite changing from TCP to HTTP mode for the backend services!

For posterity sake, I've saved your guide for my own reference and archived it to PDF as well.

If there's a particular charity you support, send me a private message and I'll happily donate to one in your name, along with a donation to the OPNsense project. 

[michaelgo]

Hi,
thank you for an amazing guide.

in part 5.5 the picture is broken, can you please re-post (it's working now)
also, 5.7 "plex condition" is it a rule? (5.7 and 5.8 screenshots are vice versa)

also, i don't have "Backends" only "Backend pools" there
Services --> HAProxy --> Settings --> Virtual Services --> Backends
is it what you mean?
thanx

[TheHellSite]

Yep, those pictures are in the wrong order and yes, I meant Backend Pools.
I will change this.

[TheHellSite]

If there's a particular charity you support, send me a private message and I'll happily donate to one in your name, along with a donation to the OPNsense project. 

A donation to OPNsense would be happily welcome I guess.
I myself don't want any money for this. We are all more or less using OPNsense free of charge so the least anyone can do is help the community or donate to the project.

[sorano]

@TheHellSite Great guide! I'm sure it will help alot of people trying to get this kind of setup up and running.

I'm running a similar setup and have some suggestions for improvement to your guide.

1. You dont need to use virtual IP's.
If you bind the HAProxy frontends to 0.0.0.0:80 & 0.0.0.0:443 it will bind to you WAN interface (even if it's dynamic). And when you do there is no need for NAT forwarding to the virtual IP's so a simple firewall rule for 80/443 on the WAN interface is enough.
Then you can bind the SSL terminating frontend to 127.0.0.1:[port] and use that IP for your SSL terminating "real server".
With the added bonus of that it performs better in a CARP setup.

2. Use map files {Advanced --> Map files}
Using map files to map domains to backends will keep your config rules alot less cluttered, especially when you have many subdomains to match. With map files 1 rule is enough to map all of your domains.

Example for map file:
plex.mydomain.tld Backend_plex

Example for rule:
(Execute function: Map domains to backend pools using a map file
Map file: [name of your map file]

Then apply the map file rule to your SSL terminating frontend.

[TheHellSite]

1. You dont need to use virtual IP's.
2. Use map files {Advanced --> Map files}

1. You dont need to use virtual IP's.
I totally get your point! This makes indeed sense but I think only if you have a static WAN IP.
As it would break the access from internal networks to the external URLs "service.subdomain.mydomain.tld" if one enabled that access using DNS rewrite rules. I am not aware of a way to rewrite DNS entries in Unbound to the WAN interface address.

With NAT reflection your way of setting this up can of course work.


2. Use map files {Advanced --> Map files}
I haven't used those yet but looks very promising!
This really makes sense in a big environment with lots of subdomains.
Thank you for pointing this out! I will add it to the FAQ. 

[sorano]

1. You dont need to use virtual IP's.
I totally get your point! This makes indeed sense but I think only if you have a static WAN IP.

Well, I only have dynamic IP's for my WAN interfaces. MultiWAN consisting of a fiber primary with LTE failover, two OPNsense hosts running CARP on all interfaces except for WAN (since I cannot get proper stateful failover with dynamic WAN IP's).

As it would break the access from internal networks to the external URLs "service.subdomain.mydomain.tld" if one enabled that access using DNS rewrite rules. I am not aware of a way to rewrite DNS entries in Unbound to the WAN interface address.

With NAT reflection your way of setting this up can of course work.

The way I'm doing access from internal networks is with Split DNS (DNS override as you call it).
In my opinion NAT reflection is an inferior solution since you lose the ability to track originating source IP in HAProxy when going through NAT.

Since HAProxy is already listening on 0.0.0.0 (all available IPv4 interfaces) I resolve the Split DNS to the internal IP of my DMZ CARP IP (but any internal IPv4 interface will do as long as you allow 80/443).

I also have certain domains I don't want reachable from the Internet so I use two map file rules, one for internal domains along with a condition that checks that source is RFC1918.

And one for external domains where I also require additional authentication.