Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[bobboviking]

Hi, my setup is an Odroid with OpnSense and docker containers running on a Synology nas behind the OpnSense box. I have setup reverse proxy using this guide and everything works just fine on my PC, I can access my containers using reverse proxy (using synology.me).

However, I can't access any reverse proxies on phones (tried on both Android and iPhone). I can't find any help anywhere in any forum. Any chance anyone could guide me?

Please let me know if it helps if upload any configuration.

Thanks in advance!

BR,
Andreas

[sizzling~snitch]

Has anyone using this setup started to see failures in the LE Cert renewals DNS?

Looks like starting two weeks ago I started getting failures on all my ACME renewals that have been working for a year or more. I am not in a place right now to share log info but what I was seeing from debug log level 2 is that the TXT record set but when checks against LE with ACME it says the TXT record is not the expected one.

Ill drop more later.

[le_top]

On multiple occasions I've been observing that even thought a certificate has been renewed, the previous version is still being serviced.
I then manually trigger the HAProxy restart action directly from the "Run automations" symbol under Services>ACME Client>Certificates>Certificate Entry>Commands .
The certificate being served then becomes correct, but some time later I get the old one back.

I now find that 2 certificates listed as "Update Certificates" under Services>HAProxy>Maintenance>SSL Certificates.
There is an entry for 1_HTTPS_frontend and on that line under Commands, an "Apply Changes" icon is available.
Clicking that and confirming the operation also gets rid of the use of the old certificate.  Hopefully this stays permanent, now I only need to find a way to automate this after a certificate renewal as restarting the HAProxy does not seem to permanently affect this.

EDIT:
It is possible to create another automation (Service>ACME>Automations) for this.  When selecting "System or Plugin Command" under "Run command", "Sync SSL certificate changes into running HAProxy service" can be selected as a system command.  I think this is the permanent fix.

EDIT2:
Tested the automation on a 2nd setup where 5 was shown under "Update certificates".  Running the automations that now include the sync commande made that go away ;-).

[pottproll]

But you might be able to pull some tricks with inbound NAT port forwarding etc.

In case someone wonders: it works pretty well with selfsigned cert for internal traffic and forwarded the needed ports to jitsi.

[sizzling~snitch]

Hello All!
A question for others who use this setup. Has anyone had the need to do a pathing setup on a single domain?

i.e. domain.com & domain.com/api/v1 point to two different backend servers? I came across https://www.haproxy.com/blog/path-based-routing-with-haproxy and was playing around in OPNSense HAProxy with Conditons and Rules but when I added the api backend rule into the main domain backend rules at the bottom I got the Warning "use_backend' ignored because backend 'TESTCom_backend' has no frontend capability." so kinda at a loss tonight and thought I would post here while giving my brain a break :D

[NW4FUN]

Hello,

I've been searching both the forum and Google and I cannot find an answer, hence me posting here.

How can I redirect root domain to www? Meaning example.com pointing to www.example.com

Enriching else works as it should but for the life of me I cannot make this simple redirection working!!

[charlesnasi]

Hello all,

Great tutorial and really happy to have this setup running for more than a year. I am struggling with something I want to fix, and red a lot about what I want, but don't know where to start. More people in this topic are facing this and I am sanding in the 'complexity' of all configuration settings in HAProxy.

I have NextCloud running as internal service for webdav/caldav services over HTTPS, port 443. Running perfectly from external: signed wildcard-LE cert to subdomain. From WireGuard VPN to internal (followed OPNSense official tutorial) I can perfectly go to my subdomain with LE cert and can access my internal network. From my internal network itself, I get the self-signed certificate. I want my internal service available with an external signed certificate over my subdomain.

Is it possible to go from internal network to 'external', So I don't have to struggle with self signed certificates? And where do I have to start? There are lots of information around, but (Dutch proverb) I can't see the forest for the trees.

[NW4FUN]

Hello,

I've been searching both the forum and Google and I cannot find an answer, hence me posting here.

How can I redirect root domain to www? Meaning example.com pointing to www.example.com

Enriching else works as it should but for the life of me I cannot make this simple redirection working!!

Any clue anyone?

[par4]

Had an issue with the step where trying to forcefully issue the test and production certificate.

Seems that I am having the best of luck. The OSCP stapling just got deprecated at Lets Encrypt 5 days ago.

Resulting in:

Code Select Expand

AcmeClient: AcmeClient: The shell command returned exit code '1': '/usr/local/sbin/acme.sh --issue --syslog 9 --debug 3 --server 'letsencrypt_test' --dns 'dns_desec' --dnssleep '240' --home '/var/etc/acme-client/home' --cert-home '/var/etc/acme-client/cert-home/redacted.02853966' --certpath '/var/etc/acme-client/certs/redacted.02853966/cert.pem' --keypath '/var/etc/acme-client/keys/redacted.02853966/private.key' --capath '/var/etc/acme-client/certs/redacted.02853966/chain.pem' --fullchainpath '/var/etc/acme-client/certs/redacted.02853966/fullchain.pem' --domain '*.redacted.dedyn.io' --days '1' --force --ocsp --keylength 'ec-384' --accountconf '/var/etc/acme-client/accounts/redacted.79212996_stg/account.conf''
After turning debugging mode on it seems the following had happened:

Code Select Expand

"detail": "Error finalizing order :: OCSP must-staple extension is no longer available: see https://letsencrypt.org/2024/12/05/ending-ocsp",

Solution
Had to turn OSCP stapling off. (OSCP Must Staple setting in the certificate settings in the ACME client)

[joeko]

On multiple occasions I've been observing that even thought a certificate has been renewed, the previous version is still being serviced.
I then manually trigger the HAProxy restart action directly from the "Run automations" symbol under Services>ACME Client>Certificates>Certificate Entry>Commands .
The certificate being served then becomes correct, but some time later I get the old one back.

I now find that 2 certificates listed as "Update Certificates" under Services>HAProxy>Maintenance>SSL Certificates.
There is an entry for 1_HTTPS_frontend and on that line under Commands, an "Apply Changes" icon is available.
Clicking that and confirming the operation also gets rid of the use of the old certificate.  Hopefully this stays permanent, now I only need to find a way to automate this after a certificate renewal as restarting the HAProxy does not seem to permanently affect this.

EDIT:
It is possible to create another automation (Service>ACME>Automations) for this.  When selecting "System or Plugin Command" under "Run command", "Sync SSL certificate changes into running HAProxy service" can be selected as a system command.  I think this is the permanent fix.

EDIT2:
Tested the automation on a 2nd setup where 5 was shown under "Update certificates".  Running the automations that now include the sync commande made that go away ;-).

Do you use now both automations or only the new one to snyc only haproxy certificate without restart haproxy ?

[trevs]

Hoping someone can point me in the right direction. I've setup according to this guide and anything I DO want to offload is working perfectly. But I also have a service I do NOT want offloading and instead to just passthrough haproxy to it's own reverse proxy (nginx). But I keep getting the cert for the working offloaded service. 
I did originally put both domains into the 1 map file, but you'll notice they are now in 2. I have no issue reverting to 1 if that's how it works, but I had the same result. 
When trying the domain not working debug log shows

2025-03-13T15:37:07-06:00	Informational	haproxy	Connect from 123.123.123.123:35560 to 75.158.105.237:443 (1_HTTPS_Frontend/HTTP)
2025-03-13T15:37:07-06:00	Informational	haproxy	123.123.123.123:35488 [13/Mar/2025:15:37:06.986] 0_SNI_frontend SSL_backend/SSL_SERVER 1/0/172 3288 -- 7/4/3/3/0 0/0
2025-03-13T15:37:07-06:00	Informational	haproxy	123.123.123.123:35488 [13/Mar/2025:15:37:06.987] 1_HTTPS_Frontend/127.4.4.3:443: SSL handshake failure
2025-03-13T15:37:06-06:00	Informational	haproxy	123.123.123.123:35372 [13/Mar/2025:15:37:06.576] 0_SNI_frontend SSL_backend/SSL_SERVER 1/0/223 396 -- 5/3/2/2/0 0/0
2025-03-13T15:37:06-06:00	Informational	haproxy	123.123.123.123:35372 [13/Mar/2025:15:37:06.577] 1_HTTPS_Frontend/127.4.4.3:443: SSL handshake failure (error:0A000416:SSL routines::sslv3 alert certificate unknown)
2025-03-13T15:37:06-06:00	Informational	haproxy	123.123.123.123:35328 [13/Mar/2025:15:37:06.409] 0_SNI_frontend SSL_backend/SSL_SERVER 1/0/167 3288 -- 6/4/3/2/0 0/0
2025-03-13T15:37:06-06:00	Informational	haproxy	123.123.123.123:35328 [13/Mar/2025:15:37:06.409] 1_HTTPS_Frontend/127.4.4.3:443: SSL handshake failure

It appears to try the HTTPS front end first, fail then tries the SNI. From what I understand the SNI should then be routing the traffic according to the rule to not SSL offload but it doesn't... 

Here is my config (sanitized of course/hopefully)

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    ocsp-update.mindelay 300
    ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua
cache opnsense-haproxy-cache
    total-max-size 4
    max-age 60
    process-vary off

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 1_http_frontend ()
frontend 1_http_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive

    # logging options
    # ACL: NoSSL_condition
    acl acl_60ece619a266e9.71758723 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_60ece619a266e9.71758723

# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option tcplog
    option socket-stats

    # ACTION: PUBLIC_nooffloaddomain_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/67d34435367b99.58937721.txt)]

# Frontend: 1_HTTPS_Frontend ()
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60ed00e1c92857.09613107.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/615ce4557a4dc4.14466569.txt)]

# Backend: Plex_backend ()
backend Plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Plex 192.168.1.42:32400 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_SERVER 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Ombi_backend ()
backend Ombi_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Ombi 192.168.1.84:5055

# Backend: HomeAssist_backend ()
backend HomeAssist_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server ha 192.168.1.12:8123

# Backend: storage_backend ()
backend storage_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    option forwarded
    option forwardfor
    server storage 192.168.1.69:443 ssl alpn h2,http/1.1 verify none

# Backend: nooffloaddomain_backend (nooffloaddomain)
backend nooffloaddomain_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server nooffloaddomain 192.168.1.118 ssl verify none resolve-prefer ipv4



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED

Code Select Expand

#615ce4557a4dc4.14466569
# public access subdomains
plex Plex_backend
storage storage_backend
ha HomeAssist_backend
workingdomain.com Ombi_backend

Code Select Expand

#67d34435367b99.58937721
# public access subdomains
notworkingdomain.com notworkingdomain_backend
staticstuff notworkingdomain_backend
I have no doubt I've missed something completely, or at the very least misunderstood and would appreciate any help that can be provided. 

[dragonkeyper]

Is this guide still valid? I followed it as best I could (but found some of the screens are different, some options are now depreciated, Lets Encrypt no longer uses OSCP). I have not been able to access my service externally or locally.

[cookiemonster]

I set up my main a while ago when thehellsite was still helping. Now I mostly add new services and works so far.
So I'd say yes despite some screens perhaps (I haven't verified it) might look a bit different with OPN evolving over time.
It is a real shame that the pleads from thehellsite were not heeded and he had no choice but to stop helping on this his main thread.