Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[jwright]

Thanks for the great tutorial.

Is there a way to exclude the HTTPS force for specific Backends? (Based on the tutorial here). Background: For HomeAssistant and stupid IOT devices, i need to have my HA instance reachable over http, too (with a different domain at least so i can firewall it a lot )

Ping

If I understand what you are after, this is explained in FAQ 6: However, having tried it myself I also cannot get it to work.

How can we load balance TCP traffic that we don't want to get SSL offloaded, f.e. OpenVPN over TCP?
In my tutorial I only explain how to "redirect+load balance SSL offloaded traffic".
This is because I myself don't have (yet) the need to actually load balance any non SSL traffic.
However balancing non SSL traffic is pretty much the same as balancing SSL traffic.
You only have to make sure that your "NOSSLservice_rule" or "NOSSLservices_mapfile_rule" is placed on the "SNI_frontend" instead of the "HTTPS_frontend" and that the backend that belongs to your "NOSSLservice_server" is running in TCP mode.

[dMopp]

In theory, yes. But for me it’s still unclear what is the real todo. Even if it should be clear, a step-by-step howto would help me a lot

It’s about the public part.
The backend, the mapfile and real server are already done

[gsaudade]

Hello, so I just solved the same problem without using maps.
I have some machine ip X.X.X.X listening on 443 running nginx and doing the ssl.
For me it worked tweeking some of the option while creating the haproxy configurations:

• created real server pointing to 443 (I tried to port 80 and was not working even if nginx was supose to redirect to 443
• add a backend pool -> mode tcp layer 4 -> select your Real Server in the Server option 
•   rules -> conditions -> add a new rule with condition type "SNI TLS extension matches (TCP Request inspection)"     contains also works if you need it in some case
•   rules -> rules -> select the condition and the backend pool previously configured
• finally add to ur rule to the public SNI_fronted

Hope this helps

[spetrillo]

Hello all,

I have one question. I already have a dynamic DNS provider. Can I continue to use this and just substitute it for the one in the instructions?

Thanks,
Steve

[luckylinux]

I also have a Question ...

I am already managing my Letsencrypt Certificates for my Homelab (and Remote Servers) using a Docker/Podman Container running certbot with DNS Challenge against Cloudflare DNS Hosting (with API Key).

Typically Letsencrypt ACME will complain if you try to obtain an alteready-generated Certificate using another Method (e.g. HTTP-01) or just Certbot running somewhere else.

Isn't there a way to automatically let HAProxy / ACME retrieve the required Certificates from a given Folder ?

I can automatically upload these using SSH/SCP (maybe also Salt/Saltstack using salt-ssh in the Future), I think that would be the easiest (in my case at least).

So basically I upload the required Certificates to e.g. /usr/local/etc/letsencrypt/MYDOMAIN.TLD/{fullchain.pem,privkey.pem,chain.pem,cert.pem}, then HAProxy "just uses them" ?

[spetrillo]

I am having issues getting the staging cert process to work. I am seeing the following in my log:

024-06-08T12:05:12-04:00   acme.sh   [Sat Jun 8 12:05:12 EDT 2024] Error add txt for domain:_acme-challenge.opcotest1.regulatoryintelligence.com
2024-06-08T12:05:12-04:00   acme.sh   [Sat Jun 8 12:05:12 EDT 2024] invalid domain
2024-06-08T12:05:12-04:00   acme.sh   [Sat Jun 8 12:05:12 EDT 2024] Adding txt value: mRcRhgOxrCZ0WghQV5t0-gz6n4tVTFrX__oCwMOhIzo for domain: _acme-challenge.opcotest1.regulatoryintelligence.com
2024-06-08T12:05:12-04:00   acme.sh   [Sat Jun 8 12:05:12 EDT 2024] Getting webroot for domain='opcotest1.regulatoryintelligence.com'
2024-06-08T12:05:10-04:00   acme.sh   [Sat Jun 8 12:05:10 EDT 2024] Getting domain auth token for each domain
2024-06-08T12:05:10-04:00   acme.sh   [Sat Jun 8 12:05:10 EDT 2024] Single domain='opcotest1.regulatoryintelligence.com'
2024-06-08T12:05:10-04:00   acme.sh   [Sat Jun 8 12:05:10 EDT 2024] Using CA: https://acme-staging-v02.api.letsencrypt.org/directory

I am using Cloudflare as my DNS provider and I added a TXT record for the challenge domain and value, but it still fails. Any thoughts?

Steve

[Koda]

@dMopp have you found a solution? I don't get it to work
I have two Services
192.168.5.2:8081 --> works with https://192.168.5.2:8081 (So the HAProxy works)
192.168.5.3:8082 --> works only with http://192.168.5.3:8082

For the Service without SSL i have add this parameters:

• Real Server: Identical to the Server like Plex, Port 8082 and disabled the SSL option (Testet disabled and enabled)
• Added a Map file like the public domains
• Conditions: No new conditions
• Rules: Copy of the public domain map and change the map file
• Backend: Copy the Plex Backend and change Mode to TCP and Server to the new Real Server
• Public Service: Add the new rule to the sni frontend

I get only the 503 Service Unavailable

[jwright]

First off, I'd like to extend a huge amount of gratitude to HellSite for the superb guide. I can't begin to imagine how many hours this guide must has saved cumulatively across the community!

Regarding the non-SSL posts, I too am having this issue.

1. Created a new server "NOSSL_server", without "Verify SSL cert" checked.
2. Created a new backed "NOSSL_backend" in TCP mode.
3. Created a new mapfile "NOSSL_PUBLIC_SUBDOMAINS_mapfile" with the content nossl   NOSSL_backend
4. Created a rule "NOSSL_PUBLIC_SUBDOMAINS_rule" which maps domains to backends using the mapfile "NOSSL_PUBLIC_SUBDOMAINS_mapfile"
5. Edited the public service "0_SNI_frontend" to use the rule "NOSSL_PUBLIC_SUBDOMAINS_rule"

I've not changed any of the existing settings from the original guide provided.

It sounds like I'm getting the same result as the below posters. I get a 503, as HAproxy forces a request for http://nossl.example.com to https://nossl.example.com.

I've tried messing with a few settings which felt like they make sense to me, but I wasn't successful. I am one of those people who know enough to be dangerous, so can follow a guide to get it working and understand some/most of what I'm doing, but struggle when it goes wrong/doesn't do quite what I want.

@dMopp have you found a solution? I don't get it to work
I have two Services
192.168.5.2:8081 --> works with https://192.168.5.2:8081 (So the HAProxy works)
192.168.5.3:8082 --> works only with http://192.168.5.3:8082

For the Service without SSL i have add this parameters:

• Real Server: Identical to the Server like Plex, Port 8082 and disabled the SSL option (Testet disabled and enabled)
• Added a Map file like the public domains
• Conditions: No new conditions
• Rules: Copy of the public domain map and change the map file
• Backend: Copy the Plex Backend and change Mode to TCP and Server to the new Real Server
• Public Service: Add the new rule to the sni frontend

I get only the 503 Service Unavailable

Thanks for the great tutorial.

Is there a way to exclude the HTTPS force for specific Backends? (Based on the tutorial here). Background: For HomeAssistant and stupid IOT devices, i need to have my HA instance reachable over http, too (with a different domain at least so i can firewall it a lot )

[jwright]

Hello all,

I have one question. I already have a dynamic DNS provider. Can I continue to use this and just substitute it for the one in the instructions?

Thanks,
Steve

Yes, I am using the service provided by my domain provider, NameCheap. I had read that access to their API was prohibitive, but I guess that depends on each persons situation (I think you need to spend $50 every two years or something).

[robert199311]

First of all great tutorial and topic, it really helped me to understand how HAproxy works. That said I do need a bit of a differend setup since I don't want HAproxy to manage any of the ssl stuff.

I have three services I want to route based on SNI using HAproxy.

- Two domains/services are static (homeassistant: app1.example1.org & Nextcloud: app2.example2.org)

The third service is directadmin server with remote users, those domains are dynamic meaning that the users using the server are adding, editing and removing domains all the time.

setup should be,
IF SNI1 -> server X
IF SNI2 -> server Y
IF ANY-OTHER-SNI -> server Z

This is my current config, the two static services (homeassistant and Nextloud) are working flawless. Unfotunately the directadmin domains are not working

Code: [Select]

removed old config to cleanup
I've read every post and almost every related other topic but unfortunately I can't get it to work, pulling my hairs for days now. Happy to pay someone to help me out if needed.

update: to add, i've basically recreated the second example on this page https://www.haproxy.com/blog/enhanced-ssl-load-balancing-with-server-name-indication-sni-tls-extension#choose-a-server-using-sni-aka-ssl-routing So I believe I'm close, yet not close enough.

Update 4/7:

Got it working!

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.ocsp-update.mindelay 300
    tune.ssl.ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: Public-service-sni-listener ()
frontend Public-service-sni-listener
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:8123 name 0.0.0.0:8123
    mode tcp

    # logging options
    # ACL: homeassistant_sni
    acl acl_668517d7e34a26.66992240 req.ssl_sni -m sub -i app1.example1.org
    # ACL: nextcloud_sni
    acl acl_668517cca10095.43472848 req.ssl_sni -m sub -i app2.example2.org

    # ACTION: other_sni_rule
    use_backend directadminpool unless acl_668517d7e34a26.66992240 || acl_668517cca10095.43472848
    # ACTION: ha_sni_rule
    use_backend homeassistant-pool if acl_668517d7e34a26.66992240
    # ACTION: nextcloud_sni_rule
    use_backend nextcloudpool if acl_668517cca10095.43472848
    # ACTION: PUBLIC_DOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/667995c7e25e94.80171493.txt,directadminpool)]
    # WARNING: pass through options below this line
      tcp-request inspect-delay 5s
      tcp-request content accept if { req_ssl_hello_type 1 }

# Backend: homeassistant-pool ()
backend homeassistant-pool
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server homeassistant 192.168.1.88:8123

# Backend: nextcloudpool ()
backend nextcloudpool
    # health check: Nextcloud-Healthcheck
    mode tcp
    balance roundrobin
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server office 192.168.1.35:443 check inter 5s port 443

# Backend: directadminpool ()
backend directadminpool
    # health checking is DISABLED
    mode tcp
    balance roundrobin
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server directadmin 192.168.10.102:443
Now the only thing I would like to achieve is that HAproxy forwards the correct client ip to the real servers due to some security solutions running on their. How can I achieve that? Anyone a clue?

[smoked-proposal]

Hello,
It's been more than a year that my setup is working (for nextcloud, accessible from "outside"), thank you TheHellSite  but now I'm adding to my network a CoreOS VM to run containers (local access only, each one with its own port number) and I want to be able to type serviceX.local.
For now, I'm going simple and will use my already set domain name: serviceX.local.DOMAIN.com
I've made the changes and it seems to work except that the "real server" port (5006) is not used (443 instead). If I type: serviceX.local.DOMAIN.com:5006 it works, but the purpose is to not have to remember all the port and that serviceX direct to the correct port number.
I've tried so many things without success that I don't remember 
Do you have an idea of what can be incorrect? (maybe HAProxy can't do that? Maybe it's the CoreOS VM that must redirect based on the domain/subdomain used?)

And second thing: how can I set a .local domain name? (with container port) I thought of Unbound DNS Overrides but it does work with port.
Like:
- service1.local redirect to 192.168.60.21:5006
- service2.local redirect to 192.168.60.21:3000
etc.

Maybe .local is not a good idea: https://en.wikipedia.org/wiki/.local

[smoked-proposal]

I'm still trying to make it work and don't understand why it's not working as expected. But, just now, as I was trying to understand why there are no HTTP code from the server showing in Firefox dev tool, so I tried with Chromium and it works 
If I look at OPNsense firewall live logs, when trying to access serviceX.local.DOMAINNAME.com, with Firefox it goes to IP:443, but with Chromium it goes to IP:5006 (the correct service port) 

It's not working either with Librewolf, Mullvad browser that are also Firefox based (this also excludes a FF profile issue).
The error:

Code: [Select]

Unable to connect

An error occurred during a connection to serviceX.local.DOMAINNAME.com.

    The site could be temporarily unavailable or too busy. Try again in a few moments.
    If you are unable to load any pages, check your computer’s network connection.
    If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the web.So I guess it's not the correct topic anymore, but if anyone as a clue of what is happening...

[xkpx]

Just a head's up. In Public services picture, X-Forwarded-For Header is set but recent changes are removing it.

Code: [Select]

4.2

Added:
* add support for built-in OCSP update feature
* add support for forwarded header (RFC7239)
* add option "X-Forwarded-For Header" to backend settings
* add options for HTTP/2 performance tuning

Fixed:
* fix SSL sync cron job (bulk sync was never working properly)

Changed:
* upgrade to HAProxy 2.8 release series (#3459)
* change default for HTTP/2 to enabled (only new frontends/backends)
* add "no-alpn" option if HTTP/2 is not enabled (only TLS-enabled frontends)
* move OCSP settings from "Service" to "Global" section
* replace bundled haproxyctl library with haproxy-cli

Deprecated:
* frontend option "X-Forwarded-For Header" (the backend option should be used)


[xkpx]

Maybe tutorial will be affected from https://letsencrypt.org/2024/07/23/replacing-ocsp-with-crls.html

[Doctorcreampie]

Hi thank you for this great tutorial, but on my OPNsense i can not figure it out why it isnt working. I tried to use everything 1:1 but i can not reache my service outside my network. I pasted my config down below. When i test everything i get no connection.

The Log files tell me that the HAProxy works but something else not, so i can not reache to my subdomains. When i put the ip address of the router in the browser tab i get a error message that no server is available under this address  (i didnt add the router ip as a real server) But i can not reache to my services. When i type service_name.mydomain.com i get no error or connection even when i put the ip address in the sarchbar.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.ocsp-update.mindelay 300
    tune.ssl.ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 ; 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_Backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.20.20.1:80)
frontend 1_HTTP_frontend
    bind 192.20.20.1:80 name 192.20.20.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_66c05db3462366.82815483 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_66c05db3462366.82815483

# Frontend: 1_HTTPS_frontend (Listening on 192.20.20.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.20.20.1:443 name 192.20.20.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/668a7ba3dc2070.92150850.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/668ae413c58075.52158060.txt)]

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server 192.20.20.1 send-proxy-v2 check-send-proxy

# Backend: JELLYFIN_Backend ()
backend JELLYFIN_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server JELLYFIN_Server 172.17.10.24:8096 ssl verify none



# statistics are DISABLED