Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[cookiemonster]

I wonder if I could ask this question here. I've followed the instruction to the letter, all is good.
I have an additional requirement however, to proxy TCP mode to the real server after TLS offloading at HA.
Say the real server is listening on port 8053 and we can live without tls between HA and it to begin with, is in the internal LAN. The public port for the service however is different, say 853. So HA needs to receive in one and sent to another.
I understand we can't mix tcp and http modes and the tutorial explains it in the relevant part.

My current thinking is that since 0_SNI_frontend and SSL_backend are already in TCP mode, the first part is adding the port, ending with 0.0.0.0:80, 0.0.0.0:443: 0.0.0.0:851
- Since the TLS is being terminated at the 1_HTTPS_frontend but is not TCP, that is not the right place to go next.
I thought then I needed to create a new frontend in TCP mode. I duplicated 1_HTTPS_frontend and changed name to 1_TCP_frontend. Set it to listen on the vip:851. I left the map rule on it and disabled SSL offload.

Testing gives me error:
2022-01-10T17:11:39   haproxy[81388] externalip:port [10/Jan/2022:17:11:39.859] 1_TCP_frontend 1_TCP_frontend/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0   
2022-01-10T17:11:39   haproxy[81388] Connect from externalip:port to wanip:851 (0_SNI_frontend/TCP)


Makes me think I'm not too far.
My question to you knowledgeable people is if the setup of this tutorial can be expanded to do this, or I'm trying to do something totally that'll never work, and some pointers to what I need to do would be wonderful please.

[TheHellSite]

You have to create a condition and rule for that service. And then place that rule on the 0_SNI_frontend.

However you CAN NOT do url conditions, f.e. service.domain.com ....


OR

You create a new TCP frontend that listens on that port.
Then create server + backend for the service and set this backend on the new tcp frontend as default backend.

OR

you simply use a port forwarding.....

[cookiemonster]

Option 2 was what I was hoping was possible. It is working and what I found was that I needed to add the port to the 0_SNI_frontend too, which I wasn't sure about. I had it from earlier attempts but once working, my new proxy would not work without it.
So in summary, added the port to 0_SNI_frontend; added the new default backend to the new front end I had created (that I had missing), as well as has the map file rule. I haven't tested without this map yet, that's next.
I'm going to read and re-read the config that is working against the documentation to understand how. I can't follow the new path at present due to my lack of knowledge on the HA options used.
Thank you TheHellSite.
Looks like this and allows me to put my subdomain in my android phone to get DNS over TLS on my own infra that filters domains with Adguard before going out on TLS to the open internet. That was the goal.
For anyone wondering is here:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    1
    hard-stop-after             60s
    maxconn                     10
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 10
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening on 0.0.0.0:80, 0.0.0.0:443, 0.0.0.0:853)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:853 name 0.0.0.0:853
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (listening on 192.168.5.100:80 i.e. http only)
frontend 1_HTTP_frontend
    bind 192.168.5.100:80 name 192.168.5.100:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_619439805021f2.97978352 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_619439805021f2.97978352

# Frontend: 1_HTTPS_frontend (Listening on 192.168.5.100:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.5.100:443 name 192.168.5.100:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61952b9d47d700.25962675.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Frontend: 1_TCP_frontend (Listening on 192.168.5.100:853)
frontend 1_TCP_frontend
    bind 192.168.5.100:853 name 192.168.5.100:853 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61dc51606078d9.11258474.certlist
    mode tcp
    default_backend nginx_backend-tcp
    # tuning options
    timeout client 15m

    # logging options
    option tcplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.5.100 send-proxy-v2 check-send-proxy

# Backend: nginx_backend-tcp ()
backend nginx_backend-tcp
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server nginx_1 192.168.5.152:8053

[Tattoofreak]

Thank you for this tutorial.
Unfortunately my requirements are a bit different.
Basically, what I want to do, is:

I want to set up HAProxy just for routing traffic based on URLs (https://xyz.domain.com goes to server 1 and https://abc.domain.com goes to server 2, etc...).
All SSL stuff for the destination web servers is being handled by a separate Linux certificate server and the web servers themselfes, independent from OPNsense/HAProxy. HAProxy is really only needed for routing traffic based on URLs, nothing more, nothing less.

The websites are all working when I use NAT rules, but I was not able yet to make it run with HAProxy in between.

For testing it with just one webserver, I have configured the following:

OPNsense settings:

System --> Settings --> Administration:
TCP port: 4443

Firewall --> Rules --> WAN:
Protocol: IPv4/TCP | Source: any | Port: Any | Destination: This Firewall | Port: 80 (HTTP) | Rule: Pass
Protocol: IPv4/TCP | Source: any | Port: Any | Destination: This Firewall | Port: 443 (HTTPS) | Rule: Pass

HAProxy settings (everthing not mentioned I left on default settings):

Services --> HAProxy --> Settings --> Settings:
Service:
Enable HAProxy: Checked

Services --> HAProxy --> Settings --> Real Servers:
Real Servers:
Name or Prefix: web01_http | FQDN/IP: 192.168.1.10 | Port: 80
Name or Prefix: web01_https | FQDN/IP: 192.168.1.10 | Port: 443

Services --> HAProxy --> Settings --> Virtual Services:
Backend Pools:
Name: web01_backendpool01 | Mode: TCP (Layer4) | Servers: web01_http, web01_https

Services --> HAProxy --> Settings --> Rules & Checks:
Conditions:
Name: myWebsite01_condition01 | Condition type: Host starts with | Host prefix: <mysubdomain>.<mydomain>.com

Services --> HAProxy --> Settings --> Rules & Checks:
Rules:
Name: myWebsite01_rule01 | Select conditions: myWebsite01_condition01 | Execute function: Use specific Backend Pool | Use backend pool: web01_backendpool01

Services --> HAProxy --> Settings --> Virtual Services:
Public Services:
Name: web01_publicService01 | Listen Addresses: <myPublicIP>:80, <myPublicIP>:443 | Type: TCP | Rules: myWebsite01_rule01


Any help would be very appreciated, thank you in advance!

[Tattoofreak]

Anyone?

I'm wondering if it is even possible to catch URL based packets when Public Services Type is set to TCP? Would this work?

[TheHellSite]

You have to set your backends and frontends to HTTP Mode.
Also disable SSL offloading on the frontends.

But I can't guarantee for sure that it will work.
TCP Mode will never (with a few exceptions) work because there is no header in the packets that would tell HAProxy which service to send the traffic to.

HTTP Mode could work, but you might need to create some "http header contains..." conditions.

[afall]

First of all thank you for the Guide it was great and helped me out a lot.

I just have a question as I have to use a vpn adapter call for work called Zscaler and it try to make a tunnel using port 443. How do i tell HAproxy as a default to only route only thing in the map file and dont intercept anything else. So the vpn client can create a tunnel. 

[vladnik]

Hi @TheHellSite,

just wanted to say thank you for this excellent guide, worked like a charm and thaught me much about how HAProxy works. Thank you for a job well done! 

Cheers!

[Morta]

I have two Server with several vhosts behind a OPNsense Router/Firewall

Is this also possible to have two server which need certs? Or only one?

When if it’s possible, how I have to do it?

[Tattoofreak]

@TheHellSite
Maybe you would also like on how to enable Websockets on your frontend(s) if your incoming clients are looking for such one(s). Websockets are basically used for example for streaming services over web. I was looking for so long on how to resolve my problem for making HAProxy work with Synology's DS Cam Android app which tries to connect from remote to the Synology Surveillance Station NAS behind HAProxy and I finally found out. You have to insert the following on your frontend (where you have to replace <myBackend(Pool)> with your according backend, of course):

Code: [Select]

acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend <myBackend(Pool)> if is_websocket
Here's the link to my original and solved issue:
 https://www.synoforum.com/threads/connecting-synology-ds-cam-android-app-to-synology-surveillance-station-through-opnsense-haproxy-plugin.7969/

[Morta]

I got this error

[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

[TheHellSite]

First of all thank you for the Guide it was great and helped me out a lot.

I just have a question as I have to use a vpn adapter call for work called Zscaler and it try to make a tunnel using port 443. How do i tell HAproxy as a default to only route only thing in the map file and dont intercept anything else. So the vpn client can create a tunnel.

Since your provided little to no info of the setup I am not really able to help you.
Also my guide is just to show beginners how things are done.

However if I understood your issue correctly, you can try the below. If that doesn't work you are better of asking in the HAProxy forums!

Add the following in the "Option pass-through" field of your 0_SNI_frontend.
Don't forget to create the backend along with the server for Zscaler. My example uses OpenVPN.

Code: [Select]

tcp-request inspect-delay 5s
use_backend OPENVPN_backend if req_ssl_hello_type 1
tcp-request content accept if !{ req_ssl_hello_type 1 }


[TheHellSite]

I have two Server with several vhosts behind a OPNsense Router/Firewall

Is this also possible to have two server which need certs? Or only one?

When if it’s possible, how I have to do it?

I got this error

[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

Never had this issue!

1. Go through my guide again. If you follow it step by step, you WILL HAVE a working setup. Otherwise you did something wrong.

2. Where is your config export? Nobody will be able to help you without a HAProxy config export!

[Morta]

I have following configuration:

NAS (192.168.1.118) → Apache → Nextcloud/DAV
/
ROUTER/Firewall/HAproxy (192.168.1.1)
\
Server (192.168.1.100) → Apache with severals vhost

I want to bring up one vhost as test with following configuration

<VirtualHost *:80>
     ServerAdmin mail@xxx.com
     ServerName xxx.ch
     ServerAlias www.xxxx.ch
     DocumentRoot /usr/share/webapps/blog/
     DirectoryIndex index.php
     RemoteIPProxyProtocol On
     <Directory /usr/share/webapps/blog>
        Options +Indexes +FollowSymLinks +MultiViews
        AllowOverride All
        Order allow,deny
        allow from all

    <FilesMatch \.php$>
      # For Apache version 2.4.10 and above, use SetHandler to run PHP as a fastCGI process server
      SetHandler "proxy:unix:/run/php-fpm/php-fpm.sock|fcgi://localhost"
    </FilesMatch>
    <Files "*.php">
      MultiviewsMatch Any
    </Files>
    </Directory>
     ErrorLog /var/log/httpd/blog_error.log
     CustomLog /var/log/httpd/blog_access.log combined
</VirtualHost>

My HAproxy.conf file looks like this

cat haproxy.conf

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: O_SNI_fronted (Listening 0.0.0.0:80 0.0.0.0:443)
frontend O_SNI_fronted
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NO_SSL_Rule
    acl acl_620808a860e296.91534155 req.ssl_ver gt 0

    # ACTION: HTTP_TO_HTTPS_RULE
    http-request redirect scheme https code 301 if !acl_620808a860e296.91534155

# Frontend: 1_HTTPS_frontend (Listening 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6208140971a7a3.08696099.certlist
    mode http
    option http-keep-alive
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_MAP_RULE
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/620809e036a6d1.87483247.txt)]

# Backend: SSL_backend (SSL  backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: 5erver_backend (Server backend)
backend 5erver_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80
# ERROR: server data not found (0b989d9b-eb50-4dff-8a2f-6bc56245fd74)

# Backend: NAS_backend (NAS backend)
backend NAS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nas_Server_80 192.168.1.118:80
# ERROR: server data not found (36c63574-bd94-43f7-836e-cd78c8edc6c0)

My map files looks like this

#public subdomains mapping
flood 5erver_backend
frank 5erver_backend
www 5erver_backend
torrent 5erver_backend
grafana 5erver_backend
nas 5erver_backend
kvm 5erver_backend
monitoring 5erver_backend
speedtest 5erver_backend
sync 5erver_backend
tracker 5erver_backend
cloud NAS_backend
dav NAS_backend

How I can fix this error
503 error?

Thanks for reply

[TheHellSite]

FIRST: You should remove your personal info from your post.

SECOND: Another issue from not properly reading my guide.

Your solution is in Part 5 - Step 6.

Now we create the backend that belongs to an actual service. You will need one backend for each service.
If you have multiple servers serving the exact same content than you will want to add all servers into a single backend so HAProxy can actually balance the load between the servers.

YOU NEED: ... one backend for each service.

YOU DID: ... one backend for each server hosting individual services.

Just think about it... How should HAProxy even be able to talk to one of your services when you are only pointing him to the IP:Port of the server virtually hosting the service!? This makes no sense...
It is like telling someone "Meet me in New York in a bar." without telling him which bar.