Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[DeWilde]

@TheHellSide

MYNAS vs SFINX was for testing purpose because i was doubting if i configured something wrong. So i created MYNAS.
About OPNsense GUI being accessible from the internet, i completely understand your concern.
I was looking for a webinterface to test the config with, but indeed, not realy a good thing to do.

Some extra troubleshooting.
Interfaces: Diagnostics: DNS Lookup

Code: [Select]

host: feniks.domain.net
server: 192.168.10.1
response: A feniks.domain.net. 3600 IN A 192.168.10.200 192.168.10.1 0 msec

Interfaces: Diagnostics: Trace Route

Code: [Select]

# /usr/sbin/traceroute -w 2 -n  -m '18'  'feniks.domain.net'
traceroute to feniks.domain.net (192.168.10.200), 18 hops max, 40 byte packets
 1  192.168.10.200  0.787 ms  0.462 ms  0.475 ms

on network client:

Code: [Select]

Pinging feniks.domain.net [192.168.10.200] with 32 bytes of data:
Reply from 192.168.10.200: bytes=32 time<1ms TTL=64


Code: [Select]

Tracing route to feniks.domain.net [192.168.10.200]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  FENIKS.domain.net [192.168.10.200]

Trace complete.

so now i'm lost 
i'll try to find/set-up another internal website to test with.

[TheHellSite]

Some extra troubleshooting.
Interfaces: Diagnostics: DNS Lookup
host: feniks.domain.net
server: 192.168.10.1
response: A   feniks.domain.net. 3600 IN A 192.168.10.200   192.168.10.1   0 msec

so now i'm lost 
i'll try to find/set-up another internal website to test with.

Please use codeboxes when posting such results. This makes it a lot easier to read them!

Your issue is mostlikely related to misconfigured DNS overwrites or another DNS resolver that is controling the DNS replies in your local network.

Question 1: Is Unbound your only DNS resolver in your network or are you running something like piHole?

Question 2: Are your client devices (f.e. iPhone, Notebook, ...) using your OPNsense as their DNS resolver inside your network or are they configured to use something like Google DNS, Cloudflare DNS, AdGuard, ...?

Also please post/attach screenshots of your configured DNS overwrites.

[DeWilde]

Hi,
as far as i am aware of, Unbound is my primary DNS resolver.
I do have Zenarmor installed on the OPNsense. But this is only a web filter, not a DNS resolver.

i checked my smartphone, laptop, kids computer, ... all of them are using OPNsense as there DNS resolver.

I attached some screenshots for you.

[DeWilde]

and some more 

[TheHellSite]

Actually I shouldn't offer this free support since you are clearly requesting it for business use.
That being said...

If I where you, I would remove "opnsense_04.jpg" asap from your post!
I just got direct access to your "opnsense.yourdomain.com" and was presented with the webinterface. Seriously don't expose it via HAProxy. Use WireGuard for this!!!

However, this screenshot also might point out your issue. Please try and write the hostnames in the host overrides in lowercase letters only.
Your "opnsense" override (lowercase) is working, but none of the others (all uppercase).

[AlexisM]

Many thank's for your tutorial. It real help me.

I'd need to throughout for my Synology Nas so I use the informations found in your topic , https://forum.opnsense.org/index.php?topic=18538.msg84958#msg84958 and https://forum.opnsense.org/index.php?topic=22630.msg118934#msg118934

I don't need to throughout admin console of my Nas but the services with port 433 (exemple https://drive.xxxx.synology.me, https://video.xxxx.synology.me etc.)

That I'm doing in completion of your tutorial (in order):

• HAProxy plugin: Create real server "nas_synology" with is local ip and port 443
• HAProxy plugin: Create backend "nas_synology_backend" with "nas_synology" with TCP (Layer 4)
• HAProxy plugin: Create "Condition" (enter name ["traffic_ssl"], condition type is "custom condition (option pass-through)" with value "req_ssl_hello_type 1")
• HAProxy plugin: Create Condition "sni_synology_me", condition "SNI TLS extension ends with (TCP request content inspection), suffixe SNI ".synologe.me"
• HAProxy plugin: Create "Rule" (enter name ["request_inspect_delay"], select no condition, function is "tcp-request inspect delay" with value "5s" or whatever suits you)
• HAProxy plugin: Create "Rule" (enter name ["request_content_accept_ssl"], select condition of 3 ["traffic_ssl"], function is "tcp-request content accept")
• HAProxy plugin: Create "Rule" (enter name ["sni_synology_me-rule"], select condition "SNI_synology_me", execute function : "use backend", Backend :"Synology_backend"
• HAProxy plugin: Modify in "Public service" (service named ["0_SNI_frontend"], select the 3 rules "request_inspect_delay", "request_content_accept_ssl" and "throughout_ssl_map_domain" and you can choose Nothing for Backend Pool by Default

Then, when I'm going with my mobile device to "plex.mydomain.com", it use backend with SSL from OpnSense
And when I use "drive.xxxx.synology.me", it throughout the ssl and use SSL from my Synology NAS

[TheHellSite]

Please provide haproxy config export

[AlexisM]

Oops. In fact, 'Create map file "throughout_ssl_map_domain" with content :' doesn't work (because SNI work on TCP).

Replace : "HAProxy plugin: Create map file "throughout_ssl_map_domain" with content : ..."
By Create Condition "SNI_synology_me", condition "SNI TLS extension ends with (TCP request content inspection), suffixe SNI ".synologe.me"

Change : HAProxy plugin: Create "Rule" (enter name ["sni_throughout_ssl-rule"], select condition "SNI_synology_me", execute function : "use backend", Backend :"Synology_backend"

*** haproxy config export :

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 500

# autogenerated entries for ACLs

# autogenerated entries for config in backends/frontends

# autogenerated entries for stats

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    # tuning options
    timeout client 30s

    # logging options
    option tcplog
    # ACL: traffic_ssl
    acl acl_63c840bdd3f440.07842774 req_ssl_hello_type 1
    # ACL: SNI_synology_me
    acl acl_63c826ed0527a7.29957165 req.ssl_sni -m end -i .synology.me

    # ACTION: request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5
    # ACTION: request_content_accept_ssl
    tcp-request content accept if acl_63c840bdd3f440.07842774
    # ACTION: sni_throughout_ssl-rule
    use_backend Synology_backend if acl_63c826ed0527a7.29957165

# Frontend: 1_HTTP_frontend (Listening on 127.74.0.0:80)
frontend 1_HTTP_frontend
    bind 127.74.0.0:80 name 127.74.0.0:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_63c813f73e3ac8.56482289 ssl_fc

    # ACTION: HTTP_to_HTTPS-rule
    http-request redirect scheme https code 301 if !acl_63c813f73e3ac8.56482289

# Frontend: 1_HTTPS_frontend (Listening on 127.74.0.0:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 127.74.0.0:443 name 127.74.0.0:443 accept-proxy ssl prefer-client-ciphers ssl-min-ver TLSv1.2 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63c817f31748b0.16739019.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63c814d7b1ebe0.58772734.txt)]

# Backend: OpnSense_backend (OpnSense Pool)
backend OpnSense_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Opnsense 192.168.74.1:444 ssl verify required ca-file /etc/ssl/cert.pem

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.74.0.0 send-proxy-v2 check-send-proxy

# Backend: Synology_backend ()
backend Synology_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server Synology 192.168.74.4:443 ##

# statistics are DISABLED

[TheHellSite]

I am unable to help here. Please ask the people that already did the things you mentioned.

[Bunch]

I am unable to help here. Please ask the people that already did the things you mentioned.

Well, I guess he is not asking question, but to update how he manage redirecting package to NAS in TCP mode by adding conditions and rules for recognizing SNI

(I have read his config and compare with mine one, and guess his config should be working flawlessly)

[AlexisM]

yes 

[brynjolm]

Hello again Mr.Hellsite the guide you provided has been rock solid for a year now. no hiccups or whatsoever. Im writing back in this post because i wanted to know exactly what you meant on NR.6 on the faq page as im interested in managing a traefik instance behind haproxy

How can we load balance TCP traffic that we don't want to get SSL offloaded, f.e. OpenVPN over TCP?
In my tutorial I only explain how to "redirect+load balance SSL offloaded traffic".
This is because I myself don't have (yet) the need to actually load balance any non SSL traffic.
However balancing non SSL traffic is pretty much the same as balancing SSL traffic.
You only have to make sure that your "NOSSLservice_rule" or "NOSSLservices_map-file_rule" is placed on the "SNI_frontend" instead of the "HTTPS_frontend" and that the backend that belongs to your

Would this kind of setup be applicable to do traefik behind haproxy? also what do you exactly mean by NOSSL_service_rule NOSSL_services_map_file_rule?

[TheHellSite]

Please refer to this post about it. Be warned I can not provide help for this since I am not using such a setup.

https://forum.opnsense.org/index.php?topic=18538.msg84958#msg84958

[tomdh76]

Thx you @TheHellSite for this tutorial. Unfortunately I cannot get it to work. I always get a "503 service unavaible status"

Here is my config

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 1_HTTP_Frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_Frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_63de5470175f22.54470191 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_63de5470175f22.54470191

# Frontend: 0_SNI_Frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_Backend

    # logging options

# Frontend: 1_HTTPS_Frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63de597c094f01.72503480.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63de5520a92049.75714996.txt)]

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: BITWARDEN_backend ()
backend BITWARDEN_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server BITWARDEN 192.168.2.55:80 ssl verify none

# Backend: CALIBRE_backend ()
backend CALIBRE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server CALIBRE 192.168.2.40:8083 ssl verify none



# statistics are DISABLED
If I use the internal addresses of the websites (192.168.2.40:8083 for example) I can login. But using the website.domain.com I get a 503 error. In the logs I see that the client is accessing the public Ip on port 443.

[TheHellSite]

Thx you @TheHellSite for this tutorial. Unfortunately I cannot get it to work. I always get a "503 service unavaible status"

If I use the internal addresses of the websites (192.168.2.40:8083 for example) I can login. But using the website.domain.com I get a 503 error. In the logs I see that the client is accessing the public Ip on port 443.

Please give further details on what is and whet it is not working.

Are you able to access your services via their domain name from a device outside of your local network?

Did you configure the DNS overrides for the local clients?

Also your Bitwarden server seems to be misconfigured are you sure it is serving SSL on the HTTP port? Also verify this for your other service.