Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[Morta]

FIRST: You should remove your personal info from your post.

SECOND: Another issue from not properly reading my guide.

Your solution is in Part 5 - Step 6.

Now we create the backend that belongs to an actual service. You will need one backend for each service.
If you have multiple servers serving the exact same content than you will want to add all servers into a single backend so HAProxy can actually balance the load between the servers.

YOU NEED: ... one backend for each service.

YOU DID: ... one backend for each server hosting individual services.

Just think about it... How should HAProxy even be able to talk to one of your services when you are only pointing him to the IP:Port of the server virtually hosting the service!? This makes no sense...
It is like telling someone "Meet me in New York in a bar." without telling him which bar.

Ok, thanks for finding the error.

English isn't my mother language so can you little bit clarify the steps.

So I have to copy the backend and adjust it for every virtual host of Apache even if they are on the same port?

I have one apache on both SERVER(192.168.1.100 and 192.168.1.118) on port 80 an only one webserver with several vhosts. I thought the map file will do the split of the to the vhosts...

For me is Apache one service on which is on both SERVER and so It's has one backend each for this two server?

[Morta]

So I have to do this for every single vhost?

So it's looking so?



So the maps file is also wrong I have to change nas_backend to nas_server_80 and 5erver_backend to 5erver_server_80 ?

[TheHellSite]

This is not a question of your mother language not being english.  ;D
This is a question of you not understanding the layout of HAProxy.

It is really hard for me to help here since you don't seem to understand the basics of a reverse proxy.
A reverse proxy is not a web server to host your sites! It is meant to route traffic to a web server.

Rule 1: You need at least ONE server for each backend.
With server I mean server created in HAProxy. A server in HAProxy represents the service (f.e. nextcloud, plex, ...) running on one of your physical servers.

Rule 2: Each backend will then present your individual services to your frontend.

HTTPS Frontend --> Mapfile --> PLEX_backend --> PLEX_server --> 192.168.2.30:32400
HTTPS Frontend --> Mapfile --> BLOG_backend --> BLOG_server --> 192.168.2.30:80
HTTPS Frontend --> Mapfile --> CLOUD_backend --> CLOUD_server --> 192.168.2.40:43569

Rule 3: A mapfile is simply mapping an access URL, f.e. www.yourdomain.com, against a backend.


To be fair it seems I misunderstood your setup/goal. But since you still didn't explain exactly what you are trying to achieve I am assuming you are trying to something like this.

Apache is your webserver. (www.yourdomain.com)
These vhost seem to provide subdirectories or individual websites. (f.e. service.yourdomain.com/subdir/)

Code Select Expand

#public subdomains mapping
flood 5erver_backend
frank 5erver_backend
www 5erver_backend
torrent 5erver_backend
grafana 5erver_backend
nas 5erver_backend
kvm 5erver_backend
monitoring 5erver_backend
speedtest 5erver_backend
sync 5erver_backend
tracker 5erver_backend
cloud NAS_backend
dav NAS_backend

Let me further explain to you the map file. I mean actually my first post does it, but you don't seem to understand or didn't read...

Code Select Expand

nas 5erver_backend
nas.yourdomain.com --> NAS_backend

Code Select Expand

www 5erver_backend
www.yourdomain.com --> 5erver_backend

Code Select Expand

grafana 5erver_backend
grafana.yourdomain.com --> 5erver_backend

[Morta]

Thanks for your help. I appreciated it!

cat haproxy.conf
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: O_SNI_fronted (Listening MYIP:80 MYIP:443)
frontend O_SNI_fronted
    bind MY IP:80 name MYIP:80
    bind MY IP:443 name MY IP:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening 192.168.1.1:80)
frontend 1_HTTP_frontend
    bind 192.168.1.1:80 name 192.168.1.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NO_SSL_Rule
    acl acl_620808a860e296.91534155 req.ssl_ver gt 0

    # ACTION: HTTP_TO_HTTPS_RULE
    http-request redirect scheme https code 301 if !acl_620808a860e296.91534155

# Frontend: 1_HTTPS_frontend (Listening 192.168.1.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.1.1:443 name 192.168.1.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6208140971a7a3.08696099.certlist
    mode http
    option http-keep-alive
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_MAP_RULE
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/620809e036a6d1.87483247.txt)]

# Backend: SSL_backend (SSL  backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 192.168.1.1 send-proxy-v2 check-send-proxy

# Backend: blog_server_backend (Server backend blog)
backend blog_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: cloud_nas_backend (cloud  backend NAS)
backend cloud_nas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nas_Server_80 192.168.1.118:80 send-proxy-v2 check-send-proxy

# Backend: dav_nas_backend (dav  backend NAS)
backend dav_nas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nas_Server_80 192.168.1.118:80 send-proxy-v2 check-send-proxy

# Backend: frank_server_backend (Server backend frank)
backend frank_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: flood_server_backend (Server backend flood)
backend flood_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: sync_server_backend (Server backend sync)
backend sync_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: monitoring_server_backend (Server backend monitoring)
backend monitoring_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: kvm_server_backend (Server backend kvm)
backend kvm_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: nas_server_backend (Server backend nas)
backend nas_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: tracker_server_backend (Server backend tracker)
backend tracker_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

and my mapfile

#public subdomains mapping
flood flood_server_backend
frank frank_server_backend
grafana grafana_server_backend
nas  nas_server_backend
kvm kvm_server_backend
monitoring monitoring_server_backend
sync sync_server_backend
tracker tracker_server_backend
cloud cloud_nas_backend
dav dav_nas_backend

You see anymore errors?

EDIT1: Do you recommend to change to nginx who can handle as web and reverse proxy
EDIT2: After studying your latest post I got it.
EDIT3: Why can't Haproxy multiple services on same port?

[TheHellSite]

You still don't understand.  ???
Your first mapfile was correct.

Just do this and tell me if it works.

1. create a mapfile with the below content.

Code Select Expand

#public subdomains mapping
flood WEBSERVER_backend
frank WEBSERVER_backend
www WEBSERVER_backend
torrent WEBSERVER_backend
grafana WEBSERVER_backend
nas WEBSERVER_backend
kvm WEBSERVER_backend
monitoring WEBSERVER_backend
speedtest WEBSERVER_backend
sync WEBSERVER_backend
tracker WEBSERVER_backend
cloud NAS_backend
dav NAS_backend

2. create the following backends and servers

Code Select Expand

WEBSERVER_backend --> contains --> WEBSERVER_server

WEBSERVER_server=192.168.1.100:80


NAS_backend --> contains --> NAS_server

NAS_server=192.168.1.118:80

3. Try to access your webserver and nas.

Code Select Expand

# WEBSERVER_backend
flood.yourdomain.com
frank.yourdomain.com
www.yourdomain.com
torrent.yourdomain.com
grafana.yourdomain.com
nas.yourdomain.com
kvm.yourdomain.com
monitoring.yourdomain.com
speedtest.yourdomain.com
sync.yourdomain.com
tracker.yourdomain.com


# NAS_backend
cloud.yourdomain.com
dav.yourdomain.com



If it still doesn't work then I can only offer you paid support. I hope you understand. It is my free time and I can only help for free up to a certain point.

[opns_neuling]

I got this error

[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

https://forum.opnsense.org/index.php?topic=27065.msg131206#msg131206

[Bunch]

I have just tried TCP mode with map file, there is a few more steps to achieve the goal instead of placing the map rule directly to 0_SNI
(I checked the package and found SNI inside, however, haproxy doesn't recognize it in TCP mode, that's why we need to force it to recognize SNI)

1. Create a "Condition" to request client hello
Name: SSL_Hello
Condition type: Custom condition (option pass-through)
Option pass-through: req_ssl_hello_type 1


2. Create a "Rule" to wait accept SSL hello
Name: tcp_request_content_accept_ssl
Select conditions: SSL_Hello
Execute function: tcp-request-content-accept


3. Create a "Rule" to wait for inspect
Name: tcp_request_inspect_delay
Optional condition: none
Execute function: tcp-request-inspect-delay
TCP inspection delay: 5s


4. Place the Rule to 0_SNI_frontend in following order
tcp_request_inspect_delay
tcp_request_content_accept_ssl
map

(hmdir_ru is my map rule)


Update according to findings in #183

5. Change the no_SSL condition to Traffic is SSL (locally deciphered)


*Remark
It is advised to use another map file for 1_HTTPS_frontend if necessary
If you really don't want to create another map file, use "SNI TLS extension matches (locally deciphered)" instead

[Bunch]

I got this error

[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

https://forum.opnsense.org/index.php?topic=27065.msg131206#msg131206

Have you Disable web GUI redirect rule in Part 4-1?
If you haven't disable it, your web GUI will keep listening to port 80.

[Morta]

I have http webui with port 4444 impossible that listen on 80

[Bunch]

I have http webui with port 4444 impossible that listen on 80

That's what I get when I try to add another frontend with 192.168.6.1:80, which already exists in my running system.
You must have another process running which listening to port 80 when you get this error

That's why I asked rather you disabled the GUI redirect.
As this redirect process is listening to port 80 of your router IP, and help you redirect any port 80 to the HTTPS port you typed.
If you already disabled, you need to figure out what other plugin you have installed and using port 80.

Guess: GUI redirect, nginx, firewall port forward

[Bunch]

Just take a look at the config file, I feel strange for some part of it
1. You don't need to type WAN IP in 0_SNI_frontend
instead, it should be 0.0.0.0:80 and 0.0.0.0:443
0.0.0.0 means any IP that points to your router.

2. What is your router IP?
If your router is 192.168.1.1, then 1_HTTP_frontend and 1_HTTPS_frontend will obviously conflict with 0_SNI_fronted
Since 0_SNI_fronted is already listening to 80 and 443 port of your router, you won't able to listen it with 192.168.1.1
Please follow Part 4-2 to create Virtual IP, and set 1_HTTP_frontend and 1_HTTPS_frontend to the virtual IP

If you don't want to create any Virtual IP, please remove 0_SNI_frontend
set 1_HTTP_frontend with 0.0.0.0:80 and 1_HTTPS_frontend 0.0.0.0:443 instead
Since all of your servers are running in http mode, it should work for having no SNI frontend

[TheHellSite]

I have just tried TCP mode with map file, there is a few more steps to achieve the goal instead of placing the map rule directly to 0_SNI
(I checked the package and found SNI inside, however, haproxy doesn't recognize it in TCP mode, that's why we need to force it to recognize SNI)

As I said in the tutorial I am not running any services that would require me to load balance / reverse proxy plain TCP traffic. This is why I never tested it and only could provide theoretical assumptions on how it could work.
So thank you very much for providing a good guide on this.

If you don't mind I will add this to the tutorial (credits to your post) when I find the time!

Just a quick question: Which service are you running in tcp mode? Does the service send the SNI header?
Or does you solution work with any TCP based service? Becuase as far as I am aware the service needs to add the SNI header otherwise the access URL is not beeing sent to HAProxy.

[TheHellSite]

Just take a look at the config file, I feel strange for some part of it
1. You don't need to type WAN IP in 0_SNI_frontend
instead, it should be 0.0.0.0:80 and 0.0.0.0:443
0.0.0.0 means any IP that points to your router.

2. What is your router IP?
If your router is 192.168.1.1, then 1_HTTP_frontend and 1_HTTPS_frontend will obviously conflict with 0_SNI_fronted
Since 0_SNI_fronted is already listening to 80 and 443 port of your router, you won't able to listen it with 192.168.1.1
Please follow Part 4-2 to create Virtual IP, and set 1_HTTP_frontend and 1_HTTPS_frontend to the virtual IP

If you don't want to create any Virtual IP, please remove 0_SNI_frontend
set 1_HTTP_frontend with 0.0.0.0:80 and 1_HTTPS_frontend 0.0.0.0:443 instead
Since all of your servers are running in http mode, it should work for having no SNI frontend

I am already on him with the fix.
Also explained to him that the HTTP(s)_frontend IPs could cause issues.
His Apache is also misconfigured badly.

[Bunch]

Sure, just add it to your tutorial if you like.
I have 2 TCP servers running. OpenVPN and v2ray
(both of them have SNI header with it)

I'm sure not all of the TCP services can use haproxy, for example minecraft server without additional tools.
(One of the ways is to add one more rule to redirect other SSL connections to SSL_backend, and set minecraft server as default backend of 0_SNI, as no conditions or rules in haproxy can catch connections that doesn't have SNI header).

[TheHellSite]

Sure, just add it to your tutorial if you like.
I have 2 TCP servers running. OpenVPN and v2ray
(both of them have SNI header with it)

I'm sure not all of the TCP services can use haproxy, for example minecraft server without additional tools.
(One of the ways is to add one more rule to redirect other SSL connections to SSL_backend, and set minecraft server as default backend of 0_SNI, as no conditions or rules in haproxy can catch connections that doesn't have SNI header).

Thank you for confirming!