Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[Aldo]

Hi,
This a great tutorial!
I don't why my certificate is B but it helped me handle all my backends with the haproxy.
Ok, my OPNsense and network knowledge noobness made me do some mistake but everything's ok in the end.
And I'm using Cloudflare instead of deSEC with no problem at all.

[mater]

I'm using port 443 on the WAN Interface for my wireguard VPN.
So I can use it in public wifi (most have port 443 in tcp/udp not blocked).

How do I have to configure HAProxy, so it only uses this Port local?
I only use SSL + Reverse Proxy internal and over the VPN.

[Easy4Breezy]

I followed the tutorial and everything works fine, thank you so much!

But for some reason Seafile is not accessible for me and I only get error 503.
When I open the web interface with the internal IP over port 9080 it works fine.

I configured everything as it says on the website:
https://manual.seafile.com/docker/pro-edition/deploy_seafile_pro_with_docker/

Since version 10.0.x, if you want to use a reverse proxy and apply for a certificate outside docker, you can use FORCE_HTTPS_IN_CONF to force write https://<your_host> in the configuration file.

e.g.

seafile:
    ...
    environment:
        ...
        - SEAFILE_SERVER_LETSENCRYPT=false
        - SEAFILE_SERVER_HOSTNAME=seafile.example.com
        - FORCE_HTTPS_IN_CONF=true
        ...

[securid]

and another victim of this error here 
both when trying to connect via

Code: [Select]

http and

Code: [Select]

https

Code: [Select]

2023-11-22T16:33:22 Informational haproxy 134.xx.xx.xx:41647 [22/Nov/2023:16:33:22.341] 1_HTTP_frontend/127.4.4.3:80: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:21 Informational haproxy 134.xx.xx.xx:41645 [22/Nov/2023:16:33:21.262] 1_HTTP_frontend/127.4.4.3:80: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:18 Informational haproxy 134.xx.xx.xx:41642 [22/Nov/2023:16:33:18.847] 1_HTTPS_frontend/127.4.4.3:443: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:18 Informational haproxy 134.xx.xx.xx:41641 [22/Nov/2023:16:33:18.795] 1_HTTPS_frontend/127.4.4.3:443: Received something which does not look like a PROXY protocol header
Versions:

Code: [Select]

Name HAProxy
Version 2.6.15-446b02c
Release_date 2023/08/09

Code: [Select]

Versions OPNsense 23.7.8_1-amd64
FreeBSD 13.2-RELEASE-p5
OpenSSL 1.1.1w 11 Sep 2023
I ran out of ideas what to try 

config is:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   8192
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local1 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats


# Resolver: opnsense
resolvers 64fcd546611ba3.78740961
    nameserver 127.0.0.1:53 127.0.0.1:53
    nameserver 192.168.178.1:53 192.168.178.1:53
    nameserver 9.9.9.9:53 9.9.9.9:53
    nameserver 192.168.80.2:53 192.168.80.2:53
    parse-resolv-conf
    resolve_retries 3
    timeout resolve 1s
    timeout retry 1s


# NOTE: Mailer alert bofh ignored: not configured in any backend

# Mailer: alert CB
mailers 64fcc379c27b34.94392037
    timeout mail 30s
    mailer blah.blubb.25


# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80,  0.0.0.0:443, )
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL-backend

    # logging options

# Frontend: 1_HTTP_frontend (listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    http-request use-service prometheus-exporter if { path /metrics }

    # logging options
    # ACL: NoSSL_condition
    acl acl_6314a0aad6d518.84034638 ssl_fc
    # ACL: find_acme_challenge
    acl acl_6339cb3bd963e1.30823960 path_beg -i /.well-known/acme-challenge/

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6314a0aad6d518.84034638
    # ACTION: redirect_acme_challenges
    use_backend acme_challenge_backend if acl_6339cb3bd963e1.30823960

# Frontend: 1_HTTPS_frontend (listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6314a6a33cce38.68245567.certlist
    mode http
    option http-keep-alive
    option forwardfor
    http-request use-service prometheus-exporter if { path /metrics }
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6314a164535f16.33310179.txt)]

# Backend (DISABLED): SSL-backend-old ()

# Backend: HomeAssistant_Backend (Homeassistant)
backend HomeAssistant_Backend
    # health checking is DISABLED
    email-alert mailers 64fcc379c27b34.94392037
    email-alert from a@b.c
    email-alert to a@b.c
    email-alert level alert
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server HomeAssistant 192.168.80.21:8123 resolve-prefer ipv4

# Backend: PhotoPrism (PhotoPrism App on TrueNAS)
backend PhotoPrism
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PhotoPrism 192.168.80.30:2342

# Backend: Syncthing (Syncthing on TRueNAS)
backend Syncthing
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Syncthing 192.168.80.17:20910

# Backend: Paperless (paperless-ngx DMS)
backend Paperless
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PaperLess 192.168.80.30:8000

# Backend: FileBrowser (filebrowser on TrueNAS)
backend FileBrowser
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server FileBrowser 192.168.80.17:10187

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL-backend (SSL backend pool)
backend SSL-backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Libre_photos_backend (LibrePhotos in VM)
backend Libre_photos_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server LibrePhotos 192.168.80.30:3000

# Backend: Nextcloud_Backend (Nextcloud Backend)
backend Nextcloud_Backend
    # health checking is DISABLED
    email-alert mailers 64fcc379c27b34.94392037
    email-alert from a@b.c
    email-alert to a@b.c
    email-alert level alert
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nextcloud 192.168.80.30:80 resolve-prefer ipv4

# Backend: Jellyfin_backend (Jellyfin in VM)
backend Jellyfin_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Jellyfin 192.168.80.30:8096

# Backend: PaperMerge (papermerge DMS)
backend PaperMerge
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PaperMerge 192.168.80.17:10141



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED

frontend prometheus_exporter
   bind *:8404
   mode http
   http-request use-service prometheus-exporter if { path /metrics }

should I switch to nginx as reverse proxy

really?

Did you (or anyone else perhaps) ever figure this out?

I've set this up following the guide, everything works fine, but HA doesn't. I find old reddit threads and things about websockets, but it just doesn't want to work.

[TheHellSite]

Sure, fixed via PM.

[securid]

Apologies @TheHellSite.

[TheHellSite]

My post was helpful to you?

No, and I wasn't asking you either.

Yes you were, since I was that "anyone" who fixed it for him and many others that contacted me via PM.
And since you are following my guide I think it is somewhat helpful to you. But you do yours I guess.

Feel free to click [applaud] to the left underneath my profile.

Not for replies like these.

Fine with me.

Additionally you can consider donating

Only in your dreams.

Again, fine with me.

Unless you're desperate to earn something, then I can tell you these posts don't work in your advantage.

I think you are pretty smart, so if you take a look at the amount of views this guide has versus the number of people that actually donated you would be able to answer this on your own.

I even helped some people that contacted me via PM free of charge. Because they provided all the necessary information without me asking, have been very nice, friendly and just good to chat with.

Just to be clear though if it was my intention to make a living out of this I would put this whole guide behind a paywall or put it on my personal blog (which doesn't exist and hopefully never will) and fill it up with ads fishy as hell (like others do).
I even know that there are many companies out there that use this guide to bill their clients a big amount of money while I get nothing from it. Guess what, fine with me.

Instead I chose to publish this in the official forum, free of charge, available to anyone, without asking for anything.
Why? Because I believe in and support the "free knowledge for everyone" movement.
Not because I am from a third world country and can barely make a living,. I do this because it will make the world just a tiny little bit better.

But then I think perhaps you're not aware how some other people read your messages, so perhaps this makes you aware. If you want to take my advise, stop posting if you don't want to help (its fine!) but don't be like this.

Just so you know your issue and the ones of many others posting here is based on the fact that (you) misread / skipped / oversaw a tiny but very important bit of the configuration.
Additionally you didn't even include the config export, like many others. Which I clearly tell everyone to do so.
Now tell me you wouldn't be annoyed by that at this point.

But yes, you are right, I guess I am the offensive dick in this scenario.

Last but not least, the offensive dick is still happy to help.

[securid]

...

apologies.

[chemlud]

Really? Please stop, it's not worth it. If you can`t post something positive, don`t post anything at all... ;-)

[muxflash]

Hello,

I use OpnSense and Haproxy for my Portainer.

It works like a charm but I have a problem with WebSockets while opening Console of a container , It doesn't work

I try to add
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Real-IP %[src]

in Option pass-through  of my docker_backend

but this doesn't work

I try with rules ans Acl but no more success

Can someone help me ?

Thank you

[mgrunwald]

Thank you very much for the tutorial! This worked great for me until I upgraded to 24.1. This broke my setup and every connection uses the first cert in the list even if this is for a different domain. I get the NET::ERR_CERT_COMMON_NAME_INVALID error

[TheHellSite]

Attention

The latest HAProxy update has a few bugs related to SNI and OCSP updates.

This means the following:
1. If you have multiple certificates on the same frontend for different subdomains/domains then connections will fail.
2. If you have "INVALID_SNI" configured then HAProxy will now always use the INVALID_SNI certificate because of issue 1.
3. The new built-in OCSP update feature is not working as intended just yet.

The good thing is that all issues are already patched and will be fixed in the next update.

Alternatively you can patch manually.
https://github.com/opnsense/plugins/issues/3779#issuecomment-1917956814

In order to apply the patch manually, access the OPNsense shell and run the following commands.

Code: [Select]

1. Reinstall the HAProxy plugin.
pkg install -f os-haproxy
2. Apply the patch.
opnsense-patch -c plugins 404c19f6e
3. Restart HAProxy from the OPNsense dashboard or reboot OPNsense.
EDIT:
HAProxy refuses to start if a self-signed certificate is configured as (default) certificate under the SSL offloading section on a (HTTPS) frontend.
So for now it is best to remove the "INVALID_SNI" certificate as default from the HTTPS frontend.

Kind Regards
TheHellSite

[rmouthaan]

@TheHellSite, Thanks for the quick response and solution The above solution works for me!

[netnut]

EDIT:
HAProxy refuses to start if a self-signed certificate is configured as (default) certificate under the SSL offloading section on a (HTTPS) frontend.
So for now it is best to remove the "INVALID_SNI" certificate as default from the HTTPS frontend.

@TheHellSite

I'm _not_ using your plugin, but I do use HAProxy on other systems with a crt-list, default self-signed cert and ocsp updates. So a shot in the dark, not sure if this "solves" your problem: You might want to declare your "default" certificate with "!*" in a crt-list to prevent errors:

https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#5.1-crt-list

/etc/haproxy/frontend-crt-list.conf

Code: [Select]

/path/to/default.crt.pem !*
/path/to/fqdn.crt.pem [ocsp-update on alpn h2,http/1.1] foo.bar
/path/to/wildcard.crt.pem [ocsp-update on alpn h2,http/1.1] *.foo.bar


[TheHellSite]

EDIT:
HAProxy refuses to start if a self-signed certificate is configured as (default) certificate under the SSL offloading section on a (HTTPS) frontend.
So for now it is best to remove the "INVALID_SNI" certificate as default from the HTTPS frontend.

@TheHellSite

I'm _not_ using your plugin, but I do use HAProxy on other systems with a crt-list, default self-signed cert and ocsp updates. So a shot in the dark, not sure if this "solves" your problem: You might want to declare your "default" certificate with "!*" in a crt-list to prevent errors:

https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#5.1-crt-list

/etc/haproxy/frontend-crt-list.conf

Code: [Select]

/path/to/default.crt.pem !*
/path/to/fqdn.crt.pem [ocsp-update on alpn h2,http/1.1] foo.bar
/path/to/wildcard.crt.pem [ocsp-update on alpn h2,http/1.1] *.foo.bar


Thank you, this should definitely fix the issue.
However most of the people using this guide refrain from editing the OPNsense config files.
Therefore I hope we will see a fix in the OPNsense HAProxy plugin for self-signed certs.