Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[smivan]

Hi folks,

Did the recent OPNsense and Haproxy updates break anyone else? I followed this tutorial last year and everything has been flawless, but now I can't get any of my sites to load coming through HAproxy.

Logs indicate that the connections come in to HTTPS_frontend/HTTP and then get sent to SNI_frontend/TCP, but then the request seems to hang.

Checking haproxy/statistics#status I  see that all the servers and backends are up, and there are no errors in the log.

Any hints are very much appreciated!

Config:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin expose-fd listeners
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua
cache opnsense-haproxy-cache
    total-max-size 4
    max-age 60
    process-vary off

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends
userlist list_65f3b0fe7fb250.26065529
    # Origin: docker_BACKEND
    user vania insecure-password o3djfFXbsMGMoKG
    # NOTE: UserlistAddUsers called with empty group data


# autogenerated entries for stats
userlist stats_auth
    user root insecure-password lovelife
    # NOTE: UserlistAddUsers called with empty group data





# Frontend: iron-k3s-api (k3s API endpoint for new nodes)
frontend iron-k3s-api
    bind 10.3.32.1:6443 name 10.3.32.1:6443
    mode tcp
    default_backend iron-k3s

    # logging options
    option tcplog

# Frontend: SNI_frontend (Listen *:80 and *:443, this is the first public hit.)
frontend SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: HTTP_frontend (Listen on 127.4.4.3:80 (redirect to ssl))
frontend HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_65f2626661cd25.59982841 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_65f2626661cd25.59982841

# Frontend: HTTPS_frontend (Listen on 127.4.4.3:443 local bind, expands public subdomains rules)
frontend HTTPS_frontend
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets prefer-client-ciphers ssl-min-ver TLSv1.2 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/65f265db185755.94806451.certlist
    mode http
    option http-keep-alive
    option forwardfor

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/65f262a83078b7.57120343.txt)]

# Frontend (DISABLED): haproxy-stats (HAproxy Stats Page)

# Backend: iron-k3s (Iron k3s kube-apiserver)
backend iron-k3s
    option log-health-checks
    # health check: kube-api check
    mode tcp
    balance roundrobin
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server galena2 10.3.34.212:6443 check inter 30s port 6443
    server galena3 10.3.34.212:6443 check inter 30s port 6443
    server iron1-k3s 10.3.34.201:6443 check inter 30s port 6443  ssl verify none
    server iron2-k3s 10.3.34.202:6443 check inter 30s port 6443
    server iron3-k3s 10.3.34.202:6443 check inter 30s port 6443

# Backend: emby_BACKEND (emby on puma)
backend emby_BACKEND
    option log-health-checks
    # health check: TCP no ssl check 2s
    option httpchk
    http-check send meth GET uri / ver HTTP/1.1 hdr Host emby.docker
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server emby_SERVER 10.3.39.6:8096 check inter 2s no-check-ssl

# Backend: SSL_backend (haproxy ssl terminator)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: caddy_BACKEND (caddy on docker puma)
backend caddy_BACKEND
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server caddy_SERVER 10.3.39.6:11080

# Backend: hassio_BACKEND (homeassistant on pve puma)
backend hassio_BACKEND
    # health check: TCP no ssl check 2s
    option httpchk
    http-check send meth GET uri / ver HTTP/1.1 hdr Host emby.docker
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server hassio_SERVER 10.3.37.71:8123 check inter 2s no-check-ssl

# Backend: docker_BACKEND (docker on puma)
backend docker_BACKEND
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    acl auth_ok http_auth(list_65f3b0fe7fb250.26065529)
    http-request auth if !auth_ok
    http-reuse safe
    server docker_SERVER 10.3.39.6:8085

# Backend: netmaker_BACKEND (netmaker non-ssl backend)
backend netmaker_BACKEND
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server netmaker_SERVER 10.3.32.200:80

# Backend: librespeed_BACKEND (librespeed docker image on puma)
backend librespeed_BACKEND
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server librespeed_SERVER 10.3.39.6:8092

# Backend: netmaker_ssl_BACKEND (netmaker ssl backend)
backend netmaker_ssl_BACKEND
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server netmaker_ssl_SERVER 10.3.32.200:443

# Backend: traefik_BACKEND (traefik servers running on k3s via bgp)
backend traefik_BACKEND
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server traefik_SERVER 10.5.0.10:443 ssl alpn h2,http/1.1 verify required ca-file /tmp/haproxy/ssl/667798075f75f4.30922858.calist



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

listen  remote_statistics
    bind            10.3.32.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats hide-version


[MSchroeer]

Hi,

does anyone bring this to live in an active-passive Opnsense-HA combination (CARP) with active-active HAProxy? Everything works fine with this combination but when an connection comes over my backup-firewall to the client some of my services could not be reached. I could not exactly figure out why some services are reachable (portainer e.g. works over a connection from master and backup carp-device) and some other not (simple container with nginx and wordpress only works over the carp-master connection) Currently I created a carp-script which stops haproxy on backup-firewall but in some cases (reboot, cron-executed sync, ...) HAProxy will start on the backup firewall as well. When both HAProxys are active my loadbalancer will balance correct but about the half of my connections (or visitors in this case) could not establish a connection.

Thanks, Michael

[securid]

Perhaps someone can help me out.

Have a setup following the guide, with 2 internal sites with ssl termination and wildcard cert. I had a public site as well also setup via the guide using local and public mappings. I no longer use the public site but it worked well.

I have an internal gitea (alternative to gitlab) server with its own certificate. It listens on 22,80,443. 22 is for SSH and 80 is redirected to 443.

I've been reading up and changed the 0_sni_public to listen on LAN ip instead of 0.0.0.0, and attempted to add a new public service listening on wan ip and a backend server on tcp. A real server added with the ip of the internal host.

I can see traffic is allowed in firewall logging but the ha proxy logs arent showing me anything useful, even on debug.

Would appreciate the help!
Thanks.

Edit: solved! The reason HAproxy wasnt showing any relevant logging was becuase I overlooked a port forward rule. once disabled things started working.

[cpuri]

Hi smivan,

Hi folks,

Did the recent OPNsense and Haproxy updates break anyone else? I followed this tutorial last year and everything has been flawless, but now I can't get any of my sites to load coming through HAproxy.

[...]

I too followed this amazing tutorial in 2023 and yesterday (2024.09.21) I upgraded from 24.1.10 to 24.7.4 and everything is working correctly.
All my services (subdomains) are working correctly, both "internal" and "external". I'm suing the "super cool" feature of MAP FILES.

Maybe (but it's just an idea... a suggestion) taking profit of the "Snapshots" feature, you could try to create a snapshot and recreate from scratch the HAProxy config in the latest version of OPNsense, following again the tutorial....

If something goes wrong, you can go back to the saved snapshot.

I hope you figure it out ;-)

Good luck!
M-CD

[catapimba]

I think I missed something... I was able to make a service in my LAN to be accessible but others don't. One thing in common is that the ones that failed only expose a http port and the one that is working has https and in the real servers I'm using the https port.

So, I can access https://portainer.mydomain, but I can't https://ollama.mydomain.

Here's my config:

Code: [Select]

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     1000
    tune.ssl.ocsp-update.mindelay 300
    tune.ssl.ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 2000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 1_Http_frontend (Responsable to receive http request and redirect to https)
frontend 1_Http_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode http
    option http-keep-alive

    # logging options
    # ACL: NoSSL
    acl acl_66ead35d7f6255.60168678 ssl_fc

    # ACTION: HTTP to HTTPS
    http-request redirect scheme https code 301 if !acl_66ead35d7f6255.60168678

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_Https_frontend (Listening on localhost:443)
frontend 1_Https_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy ssl curves secp384r1  prefer-client-ciphers ssl-min-ver TLSv1.2 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/66ececbe6b0202.10137700.certlist
    mode http
    option http-keep-alive

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/66ec2bfa712774.56131988.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 0.0.0.0 send-proxy-v2 check-send-proxy

# Backend: Portainer (Portainer's backend)
backend Portainer
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Portainer 192.168.20.10:9443 ssl alpn h2,http/1.1 verify none

# Backend: Ollama (Ollama's backend)
backend Ollama
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Ollama 192.168.20.200:11434

# statistics are DISABLED


[olluz]

Thanks a lot for this tutorial. This is very much appreciated.

Is anybody proxying the OPNsense web interface in this way and willing to share their BACKEND_POOL settings ?

Basically, it is working, but if I make a change in the HAProxy service settings and hit the "APPLY" button it will spin indefinitely. I'm missing some parameter or setting to make it work fully.

EDIT: Maybe this is just the normal or intended behaviour: Hitting "Apply" will probably cause HAProxy to reload the setting and thus not acknowledging the button press

EDIT2: Enabling "Seamless reload" gets rid of this behaviour

[OPNsense4ever]

Hi all,

I updated to 24.7.5_3 today and HAProxy stopped being able to bind to my Virtual IP that is on my trusted interface.

It looks like the GUI is binding to the virtual IP and the trusted IP.

Has anyone else seen this?

Thanks

[Daves_nt_here]

Trying to figure out how to do subfolders.
Followed the steps in page 1 of this topic and everything works beautifully. 8 domains and multiple subdomains.

Now I would like to do this:
foo.example.com --> 192.168.1.10/
bar.example.com --> 192.168.1.10/subfolder/

I've found a handful of posts from other sites but most of them are from '95 and very vague.

[jasgg]

Hello all,
Is it possible to import a config file from a HAproxy server running on Alpine to OPNSense?

I have pretty much around 45 backends ...

Much appreciated.
JG

[Patrick M. Hausen]

You would need to come up with a script that produces OPNsense standard XML. No direct import function, sorry.

[jasgg]

You would need to come up with a script that produces OPNsense standard XML. No direct import function, sorry.

Thanks for the reply.
JG

[DaBigMoe]

Hey guys,

I followed this tutorial and it looks like i got things working.

However, one of my applications needs a port to work properly and i had to port forward that to the internet (port 1234)

If port forwarding is enabled, i can use the application and it works. But then i get an "SSL_PROTOCOL_ERROR" if i go to it on a internet browser. It has a landing page to show you it's working.

If i disable port forwarding for that port -- The landing page shows up and i get a connection secured in chrome. However, if i do subdomain.mydomain.com:1234 , i get the SSL Protocol Error and my application doesnt work because it only works on port 1234

Here is my config file

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.ocsp-update.mindelay 300
    tune.ssl.ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 1000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_67160dd8beee17.05021143 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_67160dd8beee17.05021143

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/67161100d5e833.80426093.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/67160e2ec32f48.73011383.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: BlueBubbles_backend ()
backend BlueBubbles_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server BlueBubbles_Server 192.168.50.69:1234



# statistics are DISABLED


[igloo8552]

If you're facing a 503 "Server Unavailable" error despite following the guide precisely, make sure to uncheck "SSL" (Part 5, Step 4) when adding the actual server IPs for the service you're reverse proxying.