Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[TheHellSite]

Let me know if you need anymore help with your OpenVPN config.

[Lip90]

I need your help again.   ;D

Code Select Expand

root@OPNsense:~ # cat /usr/local/etc/haproxy.conf
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    option tcplog

    # ACTION: NOSSLservice_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/613b963c5f0851.94679524.txt)]

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_6138b110159553.96461818 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6138b110159553.96461818

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6138b32401a006.77997133.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6138b15d48a964.28077676.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: SEAFILE_backend ()
backend SEAFILE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server SEAFILE_server 192.168.30.16:80

# Backend: OPENVPN_backend ()
backend OPENVPN_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server OPENVPN_server 127.0.0.1:1194


Code Select Expand


2021-09-10T20:00:44 haproxy[11387] 192.168.1.231:51903 [10/Sep/2021:20:00:44.614] 0_SNI_frontend SSL_backend/SSL_server 1/0/4 0 -- 1/1/0/0/0 0/0
2021-09-10T20:00:44 haproxy[11387] 192.168.1.231:51903 [10/Sep/2021:20:00:44.615] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T20:00:40 haproxy[11387] 192.168.1.231:51902 [10/Sep/2021:20:00:40.526] 0_SNI_frontend SSL_backend/SSL_server 1/0/5 0 -- 1/1/0/0/0 0/0
2021-09-10T20:00:40 haproxy[11387] 192.168.1.231:51902 [10/Sep/2021:20:00:40.527] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 0_SNI_frontend SSL_backend/SSL_server 1/0/39 0 -- 1/1/0/0/0 0/0
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T19:59:26 haproxy[11387] xx.xx.xx.162:25707 [10/Sep/2021:19:59:26.004] 0_SNI_frontend SSL_backend/SSL_server 1/0/35 0 -- 1/1/0/0/0 0/0


MAP_file:


OPENVPN_settings:

[TheHellSite]

Okay, first you should change the listening interface of the OpenVPN plugin from "any" to "localhost".
Because this is where HAProxy is sending the traffic to since you defined your "OpenVPN real server = 127.0.0.1".

[Lip90]

Hi, I changed the interface, the problem remains the same. It looks like the VPN access is not forwarded by the SNI but by the HTTPS server, right? Unfortunately I don't know how to work around the problem.

Code Select Expand


2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 0_SNI_frontend SSL_backend/SSL_server 1/0/39 0 -- 1/1/0/0/0 0/0
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T19:59:26 haproxy[11387] xx.xx.xx.162:25707 [10/Sep/2021:19:59:26.004] 0_SNI_frontend SSL_backend/SSL_server 1/0/35 0 -- 1/1/0/0/0 0/0

[TheHellSite]

Is your OpenVPN client config correct?
- mode tcp
- port 443
- server vpn.domain.com

Some of this must be wrong.
Otherwise your SNI_frontend wouldn't pass the traffic to the configured default backend (SSL_backend) which is what it is currently doing.

[Lip90]

here is the extract of my configuration. I think that is all correct.

Code Select Expand

dev tun
persist-tun
persist-key
proto tcp-client
cipher AES-256-CBC
auth SHA256
client
resolv-retry infinite
remote vpn.xxxxx.dedyn.io 443 tcp
lport 0

[nullex]

Hello,

I'm having trouble with this guide.. Very strange trouble...
I have followed the steps to a T.. except I am using cloudflare for Dynamic DNS and ACME/let's encrypt Certs. Also, I am using localhost/127.0.0.1 instead of the virtual IP option. Although I have tried it both ways with the same results...

The problem is.. weird.. So, like I said, followed the steps.. But only a few backend servers work. I have set up Plex, Sonarr, Radarr, Ombi, Cams(blueIris) and some other stuff. However, only Plex, Cams, and Radarr work. I've been fighting with it for a while, and previously only Plex and Cams were working.. not sure what I changed to make Radarr work.. but everything else just gives me Error 503 Service Unavailable.. Even though the service is up and running and is accessible through LAN.

My opnsense is fully updated, 21.7.2_1 and os-haproxy 3.5

My HAProxy config is as follows:

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug
cache opnsense-haproxy-cache
    total-max-size 512
    max-age 60

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on localhost:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_613eabd9cb19a0.51810931 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_613eabd9cb19a0.51810931

# Frontend: 1_HTTPS_frontend (Listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/613eae5151edb0.32207081.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/613eac85c00a60.86291436.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: Radarr_backend ()
backend Radarr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Radarr_server 192.168.1.111:7878

# Backend: Plex_backend ()
backend Plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Plex_server 192.168.1.159:32400

# Backend: Cams_backend ()
backend Cams_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Cams_server 192.168.1.10:81

# Backend: Ombi_backend ()
backend Ombi_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Ombi_server 192.168.1.159:5055

# Backend: Sonarr_backend ()
backend Sonarr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Sonarr_server 192.168.1.111:8989

# Backend: Tautulli_backend ()
backend Tautulli_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Tautulli_server 192.168.1.7:8181

My map File currently looks like:

Code Select Expand

sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
Ombi Ombi_backend
Tautulli Tautulli_backend

Any advice as to what I'm doing wrong? Why would some services work and others don't when they're using the same exact config?

What's also funny is that if I reconfigure the port and IP on the one of the Real Servers that works (For example, Plex_Server has IP 192.168.1.159:32400, Ombi_server has 192.168.1.159:5055, and I replace Plex_Server port to 5055 and go to Plexi.DOMAIN.com, Ombi pops up like it should. I've cloned new Real servers and Backend Pools while updating the PUBLIC_SUBDOMAINS_map  from the working Plex one, but still no go for ALL of my services.

I've also discovered that if I modify HAProxy Rules & Checks > Rules > Public_subdomains_map-rule > Default backend pool... and change it to a service that DOESN'T work with the map file.. when I hit apply I'm able to access that service on ANY rendition of my domain, as well as the root domain.com address... And if I leave the services that DO work in the map file (Plex), plex.domain.com displays plex as it should, while the rest of the domain is showing the service that doesn't work on it's own.. which further doesn't make any sense.. The map file is working for some services but not others?

Thanks in advance, I'm stumped.

Final edit:

LMFAO I think I figured out why it wasn't working... so apparently for the map file to work you have to have the first part all lowercase, cannot use any uppercase...

So when I changed my map file to:

Code Select Expand

sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
ombi Ombi_backend
tautulli Tautulli_backend


everything started working...

[budimanjojo]

Thank you so much for this tutorial! But I have a problem and I don't know what should I do.  :'(
Maybe somebody can help me here.

I already have a Traefik reverse proxy running outside OPNSense listening on 192.168.200.244 port 80 and 443. That Traefik reverse proxying all my kubernetes cluster services including certificates handling for 2 domain names.

Now, what I want to is to have HAProxy in OPNSense to be the reverse proxy for my Traefik. For example:
- My domain names are 1stdomain.com and 2nddomain.com.
- Have a rule that: if the client go to opnsense.1stdomain.com, route it to localhost:55443 (OPNSense itself), else if the client go to *.1stdomain.com or *.2nddomain.com, route it to 192.168.200.244:443.

I don't know where to start, I tested using the tuturial but instead of SNI and HTTP frontend, I created 2 SNI frontend services but HAProxy refused to start at all.

There is an easier route for me though, that is to just create a route in Traefik to my OPNSense and be done with it. But I prefer having a firewall level reverse proxy so I can have another layer to let's say block external access like in this tutorial. Thanks before.  :)

[Lip90]

@TheHellSite

I think the problem is with the SNI frontend. Here the SSL backend is specified as the default backend. He doesn't even look at the MAP file. he forwards everything to the SSL backend. When I set the openvpn backend as default Backend for a test in the SNI frontend, openvpn work but the other things not.

do you have an idea how I can solve this?

[TheHellSite]

@TheHellSite

I think the problem is with the SNI frontend. Here the SSL backend is specified as the default backend. He doesn't even look at the MAP file. he forwards everything to the SSL backend. When I set the openvpn backend as default Backend for a test in the SNI frontend, openvpn work but the other things not.

do you have an idea how I can solve this?

I was just about to write you exactly this!  ;D
Your reply confirmed my guess.

Looking through the manual pages of HAProxy it seems that the "Default Backend" setting can only be overwritten by a "Use Backend" rule! Which a "Use map file" rule isn't able to.
https://www.haproxy.com/de/blog/the-four-essential-sections-of-an-haproxy-configuration/
But I can't imagine that this is the intended behaviour.

Anyways... you simply need to create a VPN_condition "host starts with vpn" and a "use backend OPENVPN_backend if VPN_condition=true" rule.
Add this rule to the SNI_frontend and set the default backend back to the SSL_backend.

[TheHellSite]

My map File currently looks like:

Code Select Expand

sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
Ombi Ombi_backend
Tautulli Tautulli_backend

Any advice as to what I'm doing wrong? Why would some services work and others don't when they're using the same exact config?

What's also funny is that if I reconfigure the port and IP on the one of the Real Servers that works (For example, Plex_Server has IP 192.168.1.159:32400, Ombi_server has 192.168.1.159:5055, and I replace Plex_Server port to 5055 and go to Plexi.DOMAIN.com, Ombi pops up like it should. I've cloned new Real servers and Backend Pools while updating the PUBLIC_SUBDOMAINS_map  from the working Plex one, but still no go for ALL of my services.

I've also discovered that if I modify HAProxy Rules & Checks > Rules > Public_subdomains_map-rule > Default backend pool... and change it to a service that DOESN'T work with the map file.. when I hit apply I'm able to access that service on ANY rendition of my domain, as well as the root domain.com address... And if I leave the services that DO work in the map file (Plex), plex.domain.com displays plex as it should, while the rest of the domain is showing the service that doesn't work on it's own.. which further doesn't make any sense.. The map file is working for some services but not others?

Thanks in advance, I'm stumped.

Final edit:

LMFAO I think I figured out why it wasn't working... so apparently for the map file to work you have to have the first part all lowercase, cannot use any uppercase...

So when I changed my map file to:

Code Select Expand

sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
ombi Ombi_backend
tautulli Tautulli_backend


everything started working...

Glad it is working for you know.
My first guess where some misconfigured real servers (ports, ssl, ssl-verify).

BTW: Your map file is exposing your domain name! You should remove it from the forum post.

Also I did a quick scan of your domain using https://dnsdumpster.com. It lists all your subdomains since you created a single "A Record" for each of them. Consider switching to a "Wildcard A Record" in order to hide them!
If an attacker can see what services you are running it makes it easier for them to find an attack surface.

You can then still create individual a records, f.e. www.domain.tld, since the wildcard a record is resolved after all other a records have been resolved.

This is why my tutorial is using a "Wildcard A Record / Subdomain" in the form of "*.domain.tld".

[Lip90]

Still the same  :(. Any other idea :)

[TheHellSite]

Please remove the default backend but leave the vpn_rule enabled.

I have another guess...

[Lip90]

does not work too. i don't think he matched the names

[TheHellSite]

Invert the vpn condition and name it "NoVPN_condition".
Rename the vpn rule to "NoVPN_rule" and change it to use SSL_backend.

Test it out.
1. Only the NoVPN_rule on the SNI_frontend.
2. NoVPN_rule on the SNI_frontend along with SSL_backend as default.