Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[TheHellSite]

Honestly please just follow my tutorial. I will not provide support for something else here.
If you want to do it your way then just ask in the appropriate forum.
But I will say if you keep on testing your way you will need much more time.

If it is not working with my way you can simply disable the WAN firewall rule and re-enable the NAT portforward.
This way you can also test this.

I there, I followed the suggestion and at the end of the process i have this 2 issues which I can't figure out:
1) Certificate is not valid. I also run the ssllab test and I received the same answer (rating T) showing certificate not trusted
2) again error 503 Service unavailable

Checking the HAProxy log, it shows:

Informational   haproxy   public_IP:9911 [09/May/2023:17:25:07.299] 2_HTTPS_frontend/127.4.4.3:443: SSL handshake failure

which I think I solved removing the SSL tick on the  real server set up. I have the apache virtual server only listening on port 80

#1: is it possible it is because at the moment I'm using a staging cert?

#2: this is the issue I'm investigating now for few days without any luck. I'll go over your tutorial but hints are welcome

cheers

#1 yes.

#2 just follow the tutorial... You are wasting your time here.

Also if the webserver/service behind haproxy answers locally on port 80 then of course you need to untick the SSL chekbox in the associated Real Server. (as you figured correctly)

[brandorf]

Your HAProxy config looks good.
And since you get the 503 error this means connections are getting to HAProxy.
Your issue is mostlikely the SSL (yes or no) checkbox in the Real Server settings. Verify which service needs SSL and which doesn't for the local access between HAProxy and the local service.

Also unless you get your certificates using the HTTP-01 method, which I doubt, since there is no rule for it on the HTTP_frontend remove do the following:
in the HAProxy plugin: delete the acme_challenge_backend and acme_challenge_host and all other haproxy entries auto generated by the ACME plugin.

in the ACME plugin: Go the the settings and disable the "HAProxy Integration", hit Apply.

That got it, thanks!  So the SSL check on the real server setting would only be if that server was serving its own cert?

[TheHellSite]

That got it, thanks!  So the SSL check on the real server setting would only be if that server was serving its own cert?

Yes. But I already clearly explained this in the tutorial. It is also clearly explained by the hint next to the checkbox...

[mauro]

Hi there.

After last post Iand sugestion to follow the tutorial prior to ask advice, I did and I still end up with the same error when I try to browse the webapp

503 Service Unavailable
No server is available to handle this request.

So I assume:
redirect 80->443 is working fine because I start the connection on port 80
certificate is set well despite it is the staging one giving error

I'd say the issue lays between the firewall and the backend.

The backend (webserver) accept both on port 80 and 443 with a running cert for SSL connection. can this be an issue?

Considering I tried few tutorial included this one I think the issue stays withing the firewall.

The webserver is up and running and has no restrictions to the Apache ports

Any sugestion?

thanks a lot in advance

[TheHellSite]

@mauro
HAProxy config export and a basic network diagram. That is what you will have to provide now, not just error codes.

[mauro]

@mauro
HAProxy config export and a basic network diagram. That is what you will have to provide now, not just error codes.

Roger, @TheHellSite

NETWORK Topography (simplified)

LAN (IP address) <--> | FW, lo_IP (127.x.x.x) | <--> DMZ (server_ip)

IP_Address is my FW LAN Address as per OPNsense meaning
lo_IP is the equivalent of your 127.4.4.3 but customized
server_IP is the static ip of the webserver in the DMZ network

To keep the webapp available from the outside world I have the SNI Frontend based on the LAN address, port 80,443. I'm doing this way because I already tried following your tutorial using the WAN address on the SNI frontend with same result plus server unreachable. I created a Firewall rule for LAN to acceppt incoming requests on port 80,443

HAProxy set up file:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 warning
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend (listening on LAN address port 80/443)
frontend 0_SNI_Frontend
    bind lan_ip:80 name lan_ip:80
    bind lan_ip:443 name lan_ip:443
    mode tcp
    default_backend SSL_Backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on lo_ip:80)
frontend 1_HTTP_frontend
    bind lo_ip:80 name lo_ip:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option dontlognull
    option log-separate-errors
    option httplog
    # ACL: NoSSL_condition
    acl acl_6462b25dd3fc08.98092716 ssl_fc

    # ACTION: HTTP2HTTPS_r
    http-request redirect scheme https code 301 if !acl_6462b25dd3fc08.98092716

# Frontend: 2_HTTPS_frontend (Listening on lo_ip:443)
frontend 2_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind lo_ip:443 name lo_ip:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6463bbbf543239.59805119.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option dontlognull
    option log-separate-errors
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_r
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/645151c9cb3ae5.07476878.txt)]

# Backend: s1_backend (s1 server backend)
backend s1_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server s1_server server_ip

# Backend: SSL_Backend (SNI backend)
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server lo_ip send-proxy-v2 check-send-proxy



# statistics are DISABLED

Thanks a lot

[TheHellSite]

To keep the webapp available from the outside world I have the SNI Frontend based on the LAN address, port 80,443. I'm doing this way because I already tried following your tutorial using the WAN address on the SNI frontend with same result plus server unreachable.

It can not work - plain logic.
1. Your apache is listening on port 80 (no ssl) and 443 (probably with ssl).
2. My tutorial assumes that the user wants all connections to be upgraded from port 80 to 443, what you also configured by using the HTTP_frontend on port 80 with the HTTPtoHTTPS_rule.
3. The HTTPS_frontend has SSL offloading enabled, so it decrypts any connection and then forwards it to the real server based on the real server connection configuration.
4. However, in your scenario you didn't configure your apache real server correctly since you left the port blank and didn't tell haproxy if the real server expects SSL or not.

Sorry but this is yet another error of someone not reading the tutorial properly.
All your issues are explained very clearly there.

I created a Firewall rule for LAN to acceppt incoming requests on port 80,443

Shouldn't even be necessary on a fresh install since it allows anything from LAN to anything by default.

[mauro]

Thanks again for your time.

As I said I'm new to revers proxy and I'm trying to learn. Your tutorial is for a plex server and I'm working on a webserver so at some point I need to take some decision/assumptions based on my understanding and it can be wrong but in the learning curve.

Going over your remarks:

1. Your apache is listening on port 80 (no ssl) and 443 (probably with ssl).

correct, 443 with SSL

2. My tutorial assumes that the user wants all connections to be upgraded from port 80 to 443, what you also configured by using the HTTP_frontend on port 80 with the HTTPtoHTTPS_rule.

this is also apache does for my application, redirect any call on port 80 to port 443 and use encrypted communication therefore so far your set up fit the requirements

3. The HTTPS_frontend has SSL offloading enabled, so it decrypts any connection and then forwards it to the real server based on the real server connection configuration.

Reading your tutorial #9 about SSL connection with the back end I thought ur set ups should work because u use an SSL connection to the plex server.
The webserver listen on port 443 with SSL
This is the tricky part for me and also from your notes I can't follow if and how I should change my reported setup.
Could you please be more specific?

4. However, in your scenario you didn't configure your apache real server correctly since you left the port blank and didn't tell haproxy if the real server expects SSL or not.

I set up the webserver as u did for the SSL_backend. both listening to port 80,443 and both will use SSL on port 443. If the webserver backend should be different, can you please guide me how to adjust?

bottom line, I read the tutorial several time. This is the CLOSEST tutorial to my needs (u set up a ples server and I a webserver) I have found but not exactly what I need. In this difference come the potential confusion. If I knew what to do I didn't need a tutorial. I found online also other blogs and always arrived to a dead end with error 503. I mention it on previous post under this tthread already

[TheHellSite]

This is the solution.

4. However, in your scenario you didn't configure your apache real server correctly since you left the port blank and didn't tell haproxy if the real server expects SSL or not.

[moophy]

Hi!

I've followed the guide here and it has worked flawlessly for all my services internally! So thank you for that! So much better without the warning every time going into a server.

I'm struggling with nextcloud though and feel like i'm not passing something through that i should be. I've been tearing my hair out on this one! When i type in the browser nextcloud.my.domain it says site can't be reached but if i add /login to the end then it works perfectly (for internal) with padlock sign. I can also access internally via IP:11000 (This is the Apache port) which diverts me to nextcloud.my.domain/login as expected and works. I just can't get it to do this via reverse proxy so neither internal or external works with just nextcloud.my.domain.

This leads me to think there is something missing going from HAProxy to Nextcloud (Which is in Portainer). I have a bunch of other servers which are all working fine (Truenas, Proxmox etc) in HAProxy. I've tried putting Nextcloud in as a VM on Proxmox and also in Truenas to see if it was Portainer causing any issue but same problem. Code below with all my other servers removed and left Portainer and Nextcloud as they are on the same IP. Portainer works, Nextcloud doesn't...

Code: [Select]

# Automatically generated configuration.
# Do not edit this file manually.
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats

# Frontend: 0_SNI_frontend (listening on 80 and 443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option tcplog

# Frontend: 1_HTTP_frontend (listening on localhost:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_6451d6d41f14e3.72189927 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6451d6d41f14e3.72189927

# Frontend: 1_HTTPS_frontend (listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6451dd0fe21db7.63563341.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option httplog
    # ACL: cardav-endpoint
    acl acl_6461de0380c7b3.75062629 path_end -i /.well-known/carddav
    # ACL: caldav-endpoint
    acl acl_6461dde5d15634.54704624 path_end -i /.well-known/caldav
    # ACL: nc_nodeinfo
    acl acl_6466fe1b2e8ff2.47035478 path /.well-known/nodeinfo
    # ACL: nc_webfinger
    acl acl_6466fe303acb97.89104263 path /.well-known/webfinger
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6451d1c3d510f5.88841915 src 10.1.1.0/24 192.168.0.0/24

    # ACTION: cardav-endpoint
    http-request redirect code 301 location /remote.php/dav if acl_6461de0380c7b3.75062629
    # ACTION: caldav-endpoint
    http-request redirect code 301 location /remote.php/dav if acl_6461dde5d15634.54704624
    # ACTION: nc_nodeinfo
    http-request redirect code 301 location /index.php/%[capture.req.uri] if acl_6466fe1b2e8ff2.47035478
    # ACTION: nc_webfinger
    http-request redirect code 301 location /index.php/%[capture.req.uri] if acl_6466fe303acb97.89104263
    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6451d0414cb7f2.59080919.txt)] if acl_6451d1c3d510f5.88841915
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6451d7a6418a58.11785496.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: portainer_backend (portainer backend)
backend portainer_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server portainer 10.1.1.59:9443

# Backend: nextcloud_backend (nextcloud_backend)
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server nextcloud 10.1.1.59:11000
# statistics are DISABLED
 
In the HAProxy log it says:

Code: [Select]

2023-05-19T16:35:18 | Informational | haproxy | 10.1.1.59:58792 [19/May/2023:16:35:18.215] 0_SNI_frontend SSL_backend/SSL_server 1/-1/0 0 CC 2/2/1/1/0 0/0 | 
-- | -- | -- | -- | --
2023-05-19T16:35:07 | Informational | haproxy | 10.1.1.103:46386 [19/May/2023:16:34:37.775] 1_HTTPS_frontend~ 1_HTTPS_frontend/<NOSRV> -1/-1/-1/-1/30003 0 0 - - PR-- 2/2/0/0/0 0/0 "<BADREQ>" | 
2023-05-19T16:35:07 | Informational | haproxy | 10.1.1.103:46386 [19/May/2023:16:34:37.761] 0_SNI_frontend SSL_backend/SSL_server 1/0/30017 5134 cD 2/2/1/1/0 0/0 | 
2023-05-19T16:34:48 | Informational | haproxy | 10.1.1.59:45564 [19/May/2023:16:34:48.055] 0_SNI_frontend SSL_backend/SSL_server 1/0/1 0 -- 3/3/2/2/0 0/0 | 
2023-05-19T16:34:48 | Informational | haproxy | 10.1.1.59:45564 [19/May/2023:16:34:48.055] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake
I do have DNS A record in Unbound pointing at the firewall IP as with all my other services. I added the rules for Cardav/Caldav/webfinger/nodeinfo to see if that helps (was noted in other guides) but that didn't change anything.

Thanks!

[smoked-proposal]

Thanks for this tutorial TheHellSite.
The only difficulty I had was a mistake that I made: I set `Verify SSL Certificate: true` on the real server (nextcloud). Then I "couldn't connect" to my server (I remember I could with IP but not domain name, and on internal network not from outside) and waste several hours to spot what was wrong...

[moophy]

smoked-proposal would you mind posting in your HAproxy config? To my knowledge everything i've set is as per the tutorial but i'd like to see your config for Nextcloud if you have it working (I don't have SSL verify either). I've asked over in the Nextcloud forums and they do believe HAproxy is dropping something in the redirect considering hitting nextcloud directly works correctly.

[TheHellSite]

smoked-proposal would you mind posting in your HAproxy config? To my knowledge everything i've set is as per the tutorial but i'd like to see your config for Nextcloud if you have it working (I don't have SSL verify either). I've asked over in the Nextcloud forums and they do believe HAproxy is dropping something in the redirect considering hitting nextcloud directly works correctly.

I can't be of any help here as this is a service specific issue.
Search the thread, there might be some people that had similiar problems with nextcloud.

... they do believe HAproxy is dropping something in the redirect considering hitting nextcloud directly works correctly.

Wrong, HAProxy is forwarding everything hitting "nextcloud.domain.com" to your Nextcloud server, nothing gets dropped for no reason.
I recently installed Nextcloud for testing purposes on Proxmox in an LXC. No issues at all while running it through HAProxy with "nextcloud.domain.com".

The most likely cause of error is the Apache server.
Also try clearing your browser cache.

[moophy]

Thank you, i did manage to resolve part of the issue from your help with clearing my browser cache. I had been testing via incognito mode often but had forgotten last few changes. Locally I can now access nextcloud via domain name as expected.
 
Externally however, no joy. Map file entry under #public access subdomains
nextcloud nextcloud_backend in both local and public maps (in that order). Would this point to an issue somewhere on Opnsense? Whether that's firewall, HAproxy etc not sure. Considering nextcloud itself can accept connection via url locally? Happy for your guidance and if you think that issue is still the target server then i'll go deep dive further there. Halfway there

[TheHellSite]

Thank you, i did manage to resolve part of the issue from your help with clearing my browser cache. I had been testing via incognito mode often but had forgotten last few changes. Locally I can now access nextcloud via domain name as expected.
 
Externally however, no joy.

No error codes, no logs, no map file content, ...
Is it just nextcloud not working from external or all services in haproxy?

So many questions and possible solutions but no context provided, therefore ---> Unable to help.