Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[nikkon]

I am trying to use this howto only for the internal services (I have no intention to expose them out) and I can't make it work.
can anyone give me a hand with this. I tried everything I know to make this work.

[slykens]

Hello... another pfsense refugee here.

Still working on getting everything working how I want and tonight's project was wrangling haproxy. I am having a problem with https redirect so I followed the tutorial in this thread with no success.

When an https client hits haproxy, it works as expected.

When an http client hits haproxy, I get the following error in the haproxy log:

Code: [Select]

ssl_redirect/[::]:80: Received something which does not look like a PROXY protocol header
This is my present config export:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    1
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: https ()
frontend https
    bind 0.0.0.0:443 name 0.0.0.0:443 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6554226ca7c6c4.56456894.certlist
    bind [::]:443 name [::]:443 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6554226ca7c6c4.56456894.certlist
    mode http
    option http-keep-alive
    option forwardfor

    # logging options

    # ACTION: sni_translation
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/65542596a04585.83628685.txt)]

# Frontend: ssl_redirect ()
frontend ssl_redirect
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    bind [::]:80 name [::]:80 accept-proxy
    mode http
    option http-keep-alive

    # logging options

    # ACTION: ssl_redirect
    # NOTE: actions with no ACLs/conditions will always match
    http-request redirect scheme https code 301

# Backend: x_openvpn_as ()
backend x_openvpn_as
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server x_openvpn_as 10.11.23.2:443 ssl verify none

# Backend: webui ()
backend webui
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server webui 127.0.0.1:1443 ssl verify none



# statistics are DISABLED

Any ideas or guidance are welcome and appreciated. Thank you.

[nikkon]

does anyone knows what this long means:
2023-11-14T13:41:01   Warning   haproxy   Proxy <HTTPCLIENT> stopped (cumulated conns: FE: 0, BE: 0).   
2023-11-14T13:41:01   Warning   haproxy   Proxy HASS_Backend stopped (cumulated conns: FE: 0, BE: 0).   
2023-11-14T13:41:01   Warning   haproxy   Proxy PLEX_Backend stopped (cumulated conns: FE: 0, BE: 0).   
2023-11-14T13:41:01   Warning   haproxy   Proxy acme_challenge_backend stopped (cumulated conns: FE: 0, BE: 0).   
2023-11-14T13:41:01   Warning   haproxy   Proxy NAS_Backend stopped (cumulated conns: FE: 0, BE: 0).

[tadchilly]

I've been using this for months and really like it... but has anyone tried adding another domain?  What steps would I need to take?

For now the PUBLIC_SUBDOMAINS_rule is used to "Map domains to backends using a map file" and

Test Type is "IF", conditions are "Nothing selected", execute function "Map domains to backend pools using a map file".

What conditions would I use to specify one or the other?

[tadchilly]

Thank you so much for this tutorial.

I do have (hopfully) a quick question and went through 20 some pages to see if its been asked.

I have all my subdomains working perfectly, however how do I set my example.com domain?

I have all my services under service.example.com and want a website at example.com

Im sure its something I am overlooking. Like do I put something in my Map file?

Thank you again

Oops, I missed this before I posted. I'm trying to do the same.  I assume you have to make a second map file and create separate rules for it, but I have no idea what they would be.

Edit:  I think I would create a second map file, second rule pointing to that mapfile, and then a new condition for each new subdomain pointing to the new rule.  If there's a more efficient way of doing this or if anyone has any pointers, I'd really appreciate it.

I think I might try backing up my config and trying though.

[ChillinLeon]

Hello... another pfsense refugee here.

Still working on getting everything working how I want and tonight's project was wrangling haproxy. I am having a problem with https redirect so I followed the tutorial in this thread with no success.

When an https client hits haproxy, it works as expected.

When an http client hits haproxy, I get the following error in the haproxy log:

Code: [Select]

ssl_redirect/[::]:80: Received something which does not look like a PROXY protocol header
This is my present config export:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    1
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: https ()
frontend https
    bind 0.0.0.0:443 name 0.0.0.0:443 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6554226ca7c6c4.56456894.certlist
    bind [::]:443 name [::]:443 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6554226ca7c6c4.56456894.certlist
    mode http
    option http-keep-alive
    option forwardfor

    # logging options

    # ACTION: sni_translation
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/65542596a04585.83628685.txt)]

# Frontend: ssl_redirect ()
frontend ssl_redirect
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    bind [::]:80 name [::]:80 accept-proxy
    mode http
    option http-keep-alive

    # logging options

    # ACTION: ssl_redirect
    # NOTE: actions with no ACLs/conditions will always match
    http-request redirect scheme https code 301

# Backend: x_openvpn_as ()
backend x_openvpn_as
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server x_openvpn_as 10.11.23.2:443 ssl verify none

# Backend: webui ()
backend webui
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server webui 127.0.0.1:1443 ssl verify none



# statistics are DISABLED

Any ideas or guidance are welcome and appreciated. Thank you.

Unfortunately, I am stuck with this as well after following the tutorials . My goal is simply to have HA proxy act as a reverse proxy for various sub domains that I aim to make backends and frontends for. Im testing just one for my proxmox server that I'd like to reverse proxy to. Yet here is the only thing i found in the logs Everytime i access my proxmox domain

2_HTTPS_Frontend/192.168.1.43:443: Received something which does not look like a PROXY protocol header

My browser will give me this error (snippet Firefox)

PR_END_OF_FILE_ERROR

While chrome will say my connection is closed but the HAProxy logs still give the same error. I'm quite new to HAProxy so I can't initially make heads or tails of this. Any help would be appreciated.

Attached is my conf

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend (Listening on 0.0.0.0:443 and 0.0.0.0:80)
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option tcplog

# Frontend: 1_HTTP_Frontend (Listening on 127.0.0.1:80)
frontend 1_HTTP_Frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_655d4c7f77c559.77912446 ssl_fc

    # ACTION: HTTPtoHTTPS
    http-request redirect scheme https code 301 if !acl_655d4c7f77c559.77912446

# Frontend: 2_HTTPS_Frontend (Listening on WAN IP:443)
frontend 2_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 192.168.1.43:443 name 192.168.1.43:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/655d518eb205a6.14872799.certlist
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option log-separate-errors
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/655d4cef9a0796.78380664.txt)]

# Backend: SSL_backend (SSL Backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: Proxmox_backend (Proxmox Backend)
backend Proxmox_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Proxmox_server 172.16.1.1:8006



# statistics are DISABLED



[BSAfH42]

and another victim of this error here 
both when trying to connect via

Code: [Select]

http and

Code: [Select]

https

Code: [Select]

2023-11-22T16:33:22 Informational haproxy 134.xx.xx.xx:41647 [22/Nov/2023:16:33:22.341] 1_HTTP_frontend/127.4.4.3:80: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:21 Informational haproxy 134.xx.xx.xx:41645 [22/Nov/2023:16:33:21.262] 1_HTTP_frontend/127.4.4.3:80: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:18 Informational haproxy 134.xx.xx.xx:41642 [22/Nov/2023:16:33:18.847] 1_HTTPS_frontend/127.4.4.3:443: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:18 Informational haproxy 134.xx.xx.xx:41641 [22/Nov/2023:16:33:18.795] 1_HTTPS_frontend/127.4.4.3:443: Received something which does not look like a PROXY protocol header
Versions:

Code: [Select]

Name HAProxy
Version 2.6.15-446b02c
Release_date 2023/08/09

Code: [Select]

Versions OPNsense 23.7.8_1-amd64
FreeBSD 13.2-RELEASE-p5
OpenSSL 1.1.1w 11 Sep 2023
I ran out of ideas what to try 

config is:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   8192
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local1 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats


# Resolver: opnsense
resolvers 64fcd546611ba3.78740961
    nameserver 127.0.0.1:53 127.0.0.1:53
    nameserver 192.168.178.1:53 192.168.178.1:53
    nameserver 9.9.9.9:53 9.9.9.9:53
    nameserver 192.168.80.2:53 192.168.80.2:53
    parse-resolv-conf
    resolve_retries 3
    timeout resolve 1s
    timeout retry 1s


# NOTE: Mailer alert bofh ignored: not configured in any backend

# Mailer: alert CB
mailers 64fcc379c27b34.94392037
    timeout mail 30s
    mailer blah.blubb.25


# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80,  0.0.0.0:443, )
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL-backend

    # logging options

# Frontend: 1_HTTP_frontend (listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    http-request use-service prometheus-exporter if { path /metrics }

    # logging options
    # ACL: NoSSL_condition
    acl acl_6314a0aad6d518.84034638 ssl_fc
    # ACL: find_acme_challenge
    acl acl_6339cb3bd963e1.30823960 path_beg -i /.well-known/acme-challenge/

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6314a0aad6d518.84034638
    # ACTION: redirect_acme_challenges
    use_backend acme_challenge_backend if acl_6339cb3bd963e1.30823960

# Frontend: 1_HTTPS_frontend (listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6314a6a33cce38.68245567.certlist
    mode http
    option http-keep-alive
    option forwardfor
    http-request use-service prometheus-exporter if { path /metrics }
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6314a164535f16.33310179.txt)]

# Backend (DISABLED): SSL-backend-old ()

# Backend: HomeAssistant_Backend (Homeassistant)
backend HomeAssistant_Backend
    # health checking is DISABLED
    email-alert mailers 64fcc379c27b34.94392037
    email-alert from a@b.c
    email-alert to a@b.c
    email-alert level alert
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server HomeAssistant 192.168.80.21:8123 resolve-prefer ipv4

# Backend: PhotoPrism (PhotoPrism App on TrueNAS)
backend PhotoPrism
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PhotoPrism 192.168.80.30:2342

# Backend: Syncthing (Syncthing on TRueNAS)
backend Syncthing
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Syncthing 192.168.80.17:20910

# Backend: Paperless (paperless-ngx DMS)
backend Paperless
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PaperLess 192.168.80.30:8000

# Backend: FileBrowser (filebrowser on TrueNAS)
backend FileBrowser
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server FileBrowser 192.168.80.17:10187

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL-backend (SSL backend pool)
backend SSL-backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Libre_photos_backend (LibrePhotos in VM)
backend Libre_photos_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server LibrePhotos 192.168.80.30:3000

# Backend: Nextcloud_Backend (Nextcloud Backend)
backend Nextcloud_Backend
    # health checking is DISABLED
    email-alert mailers 64fcc379c27b34.94392037
    email-alert from a@b.c
    email-alert to a@b.c
    email-alert level alert
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nextcloud 192.168.80.30:80 resolve-prefer ipv4

# Backend: Jellyfin_backend (Jellyfin in VM)
backend Jellyfin_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Jellyfin 192.168.80.30:8096

# Backend: PaperMerge (papermerge DMS)
backend PaperMerge
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PaperMerge 192.168.80.17:10141



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED

frontend prometheus_exporter
   bind *:8404
   mode http
   http-request use-service prometheus-exporter if { path /metrics }

should I switch to nginx as reverse proxy

really?

[TheHellSite]

To all people above asking for help...
Or "threatening" to use NGINX instead 

Everything was and is working fine for me on 10+ instances of OPNsense (outdated and latest firmware versions) configured exactly as per tutorial.
If you are not able to follow the tutorial EXACTLY as it is, then this is totally a user error on your side.

As stated in the top of the tutorial, no more free support for a perfectly working tutorial if followed correctly.
If you still need help, feel free to PM me for paid support.

Best Regards
TheHellSite

[BSAfH42]

I was not asking YOU explicitly to help, I was asking the community. That's what forums are made for, aren't they?

And for not following the guide: this a an unmodified reloaded backup config from an OPNsense machine that broke down 2 months ago.

And thanks for the insult. I am able to follow a guide and I triple checked. Nice way to approach potential customers, really. 

And I was not "threating" to switch to nginx - but I have a problem to solve. so that would be a last resort.

Paid support is fine - send me a mesage for a quote, I might take it.

[techsolo12]

First of all: I use Opnsense with Unbound and AdGuard.

Like some other users, I had the problem that when I configured the public service "0_SNI_fronted", the HAProxy service could no longer be started. If I only set port "443" the problem did not occur.
So the problem was that port "80" was already blocked by another service.

A look at "Interfaces -> Diagnostics -> Netstat -> Socket" revealed that port "80" was already being used by AdGuard.

To change this, I adjusted the default http port in the AdGuard config. From "80" to "81". To do this, simply stop the AdGuard service in the WebUI, then go to the shell and edit the config.

Code: [Select]

nano /usr/local/AdGuardHome/AdGuardHome.yaml
Now you can edit the default port to "81".

Code: [Select]

http:
  pprof:
    port: 6060
    enabled: false
  address: 0.0.0.0:81
  session_ttl: 720h
After all you start the AdGuard service on WebUI. Now you should can configure services in haproxy.

best regards,
techsolo12

[techsolo12]

First of all, a huge thank you to TheHellSite for this detailed tutorial!

Unfortunately, I need your help. I have configured HAProxy as described in the tutorial. However, with my own domain.

All services that are to be reached externally work as desired. Only the internal service does not seem to be "noticed" by HAProxy. Unfortunately, no accesses to the "node2-ipmi" service from the source IP from the "10.10.10.0/24" network appear in the log. I cannot connect to the service "node2-ipmi".

In firefox i got this warning "SEC_ERROR_UNKNOWN_ISSUER".

Since no log entries appear in the log, I cannot attach any.

Config export:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening to 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (listening on 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_65612d875c4e55.24914702 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_65612d875c4e55.24914702

# Frontend: 1_HTTPS_frontend (listening to 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6561dfa723cb35.23136075.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_FQDN_condition
    acl acl_6563927a593ba4.09519486 src domain.tld
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_65627ea0efa5d5.95729048 src 10.10.5.0/28 10.10.10.0/24 10.10.11.0/24
    # ACL: nextcloud_caldav
    acl acl_65626936202592.20944712 path_beg -i /.well-known/caldav
    # ACL: nextcloud_carddav
    acl acl_656269439b5220.54434789 path_beg -i /.well-known/carddav

    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/656277f5815fc5.43737480.txt)] if acl_6563927a593ba4.09519486 || acl_65627ea0efa5d5.95729048
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/65612e0d931f69.06203948.txt)]
    # ACTION: nextcloud_dav
    http-request set-path /remote.php/dav if acl_65626936202592.20944712 || acl_656269439b5220.54434789

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: cloud_backend ()
backend cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server cloud_server 10.10.20.5:80

# Backend: vw_backend ()
backend vw_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server vw_server 10.10.20.7:80

# Backend: office_backend ()
backend office_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server office_server 10.10.20.8:80

# Backend: rezepte_backend ()
backend rezepte_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server rezepte_server 10.10.20.9:3000

# Backend: cash_backend ()
backend cash_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server cash_server 10.10.20.10:5006

# Backend: node2-ipmi_backend ()
backend node2-ipmi_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server node2-ipmi_server 10.10.5.6:443 ssl verify none



# statistics are DISABLED

With best regards,
techsolo12

[bucky2780]

tadchilly:

adding another domain was simple for me... easier than you might first think.... no need to change the mapfile either as subdomain names do not change with both domains.

in  1_https_frontend, simply add another wildcard cert here... and everything should work as normal.
my existing cert was (acme) *.expand.net.au, and to this I added another cert of *.expand.dedyn.io

Obviously need to ensure your wildcard dns will resolve properly. I found for local use, I can set a wildcard in unbound, the same way I do in cloudlfare, and will will resolve internally just fine.

hope this helps

I've been using this for months and really like it... but has anyone tried adding another domain?  What steps would I need to take?

For now the PUBLIC_SUBDOMAINS_rule is used to "Map domains to backends using a map file" and

Test Type is "IF", conditions are "Nothing selected", execute function "Map domains to backend pools using a map file".

What conditions would I use to specify one or the other?

[bucky2780]

techsolo12

It is probably a dns issue. You need to setup a nat reflection solution, so that internal dns names will map to the gateway, where haproxy can do its work.(much the same as when you are coming from external).

I use unbound for dns, and setup a wildcard DNS entry much the same as I did on cloudflare and desec. This wildcard entry points to the opnsense gateway, and haproxy then does its magic.

hope that helps... (worked for me)

First of all, a huge thank you to TheHellSite for this detailed tutorial!

Unfortunately, I need your help. I have configured HAProxy as described in the tutorial. However, with my own domain.

All services that are to be reached externally work as desired. Only the internal service does not seem to be "noticed" by HAProxy. Unfortunately, no accesses to the "node2-ipmi" service from the source IP from the "10.10.10.0/24" network appear in the log. I cannot connect to the service "node2-ipmi".

In firefox i got this warning "SEC_ERROR_UNKNOWN_ISSUER".

Since no log entries appear in the log, I cannot attach any.

Config export:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening to 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (listening on 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_65612d875c4e55.24914702 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_65612d875c4e55.24914702

# Frontend: 1_HTTPS_frontend (listening to 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6561dfa723cb35.23136075.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_FQDN_condition
    acl acl_6563927a593ba4.09519486 src domain.tld
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_65627ea0efa5d5.95729048 src 10.10.5.0/28 10.10.10.0/24 10.10.11.0/24
    # ACL: nextcloud_caldav
    acl acl_65626936202592.20944712 path_beg -i /.well-known/caldav
    # ACL: nextcloud_carddav
    acl acl_656269439b5220.54434789 path_beg -i /.well-known/carddav

    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/656277f5815fc5.43737480.txt)] if acl_6563927a593ba4.09519486 || acl_65627ea0efa5d5.95729048
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/65612e0d931f69.06203948.txt)]
    # ACTION: nextcloud_dav
    http-request set-path /remote.php/dav if acl_65626936202592.20944712 || acl_656269439b5220.54434789

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: cloud_backend ()
backend cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server cloud_server 10.10.20.5:80

# Backend: vw_backend ()
backend vw_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server vw_server 10.10.20.7:80

# Backend: office_backend ()
backend office_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server office_server 10.10.20.8:80

# Backend: rezepte_backend ()
backend rezepte_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server rezepte_server 10.10.20.9:3000

# Backend: cash_backend ()
backend cash_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server cash_server 10.10.20.10:5006

# Backend: node2-ipmi_backend ()
backend node2-ipmi_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server node2-ipmi_server 10.10.5.6:443 ssl verify none



# statistics are DISABLED

With best regards,
techsolo12

[DOM_EUWest]

Thank you for these detailed instructions.

It has made my life much easier.


However, I have the problem that Plex is not accessible, neither internally nor externally.


The Plex log shows me the error....


CERT: incomplete TLS handshake from xxx.xxx.xxx:28422: stream truncated

HA Log..
Informational   haproxy   xx.x.xx.xxx:21734 [18/Dec/2023:02:33:09.222] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure


The certificate is new and was requested via ACME with the exact same settings.


I am at a bit of a loss.

MTU WAN: 1500 auto
MTU LAN 9000 with 10G LAGG interface

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    8
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:443 0.0.0.0:80)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_657ed45319efa3.43352536 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_657ed45319efa3.43352536

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/657ed88b10e6c1.81075400.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/657ed57bcfd057.79414853.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    timeout connect 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Plex_DMZ_backend ()
backend Plex_DMZ_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    timeout connect 30s
    http-reuse safe
    server Plex_Server_DMZ 10.10.20.11:32400 ssl verify none resolve-prefer ipv4



# statistics are DISABLED




10.10.2.1 is the LAN IP

When i set the DNS to the Real Server ( 10.10.20.11) and go to https://plex.mydomain.eu:32400 , all work fine.

[DOM_EUWest]

Ok :-)

Now it works.

Changed the LAN MTU from 9000 to 4076 was the hack. Now all works fine