Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[Bunch]

Just checked the L6 warning a bit.
This is the code from haproxy github

Code Select Expand

int warnif_tcp_http_cond(const struct proxy *px, const struct acl_cond *cond)
{
if (!cond || px->mode != PR_MODE_HTTP)
return 0;

if (cond->use & (SMP_USE_L6REQ|SMP_USE_L6RES)) {
ha_warning("Proxy '%s': L6 sample fetches ignored on HTTP proxies (declared at %s:%d).\n",
   px->id, cond->file, cond->line);
return ERR_WARN;
}
return 0;
}

The condition: req.ssl_ver (Traffic is SSL....)
This condition might get error code, as http doesn't have tls packages.
Although negating this condition will still redirect packages to 443, haproxy will still run.
But we can simply use "Traffic is HTTP" here.
(Then, I don't know why I get such error when I create 2 port 80 frontend) ???

[TheHellSite]

Please repost your current config.

[Bunch]

You mean my working config?

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if xray)
frontend tcp_front
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_6212326a7c07e4.28981163 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_6212326a7c07e4.28981163
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6214a3ae639096.17472719.txt)]

# Frontend: http_front (Listen to VIP:80 and redirect to 443)
frontend http_front
    bind 192.168.6.1:80 name 192.168.6.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: http
    acl acl_62123bbee27260.60165685 ssl_fc

    # ACTION: http_to_https
    http-request redirect scheme https code 301 if !acl_62123bbee27260.60165685

# Frontend: ssl_front (Listen to VIP 443, SSL offload cert)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.6.1:443 name 192.168.6.1:443 accept-proxy ssl no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621244f0de5919.36753000.certlist
    mode http
    option http-keep-alive
    default_backend hkbn_back
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: local
    acl acl_62123a1cebe813.09309501 src 192.168.3.0/24 192.168.5.0/24

    # ACTION: local_ru
    use_backend opn_back if acl_62123a1cebe813.09309501

# Backend: hmdir_back (Backend of v2ray)
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.3.3:443

# Backend: hkbn_back (Backend of HKBN)
backend hkbn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hkbn 192.168.4.2:443 ssl verify none

# Backend: SSL_back (Backend to redirect SSL servers)
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server ssl 192.168.6.1 send-proxy-v2 check-send-proxy

# Backend: opn_back (Backend of opnsense with SSL)
backend opn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opn 192.168.3.1:8443 ssl verify none

# Backend: open_back (Backend of OpenVPN)
backend open_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server open 192.168.3.1:10443


BTW, I know why I will get such warning after reading the code.
Since my TCP layer already received TLS packages and inspected it already.
For HTTP and HTTPS frontend, any more "request" for SSL/TLS will probably get such warning.

Thus, for anyone using map file in 0_SNI_frontend.
They not only need to create "tcp_request_inspect_delay", "tcp_request_content_accept_ssl" and "SSL_Hello" condition
but also need to change the non-HTTP traffic condition to either
Negate of Traffic is SSL (locally deciphered) (ssl_fc), which will not request another TLS package
or
Traffic is HTTP (req.proto_http)
but not
Traffic is SSL (TCP request content inspection) (req.ssl_ver)

If Someone already used map file in 0_SNI_frontend
And they have multiple HTTP server need to offload in HTTPS_frontend, they should either use another map file
or
SNI TLS extension matches (locally deciphered) (ssl_fc_sni)
but not
SNI TLS extension ... (TCP request content inspection) (req.ssl_sni)

*For Morta's case, this might also due to multiple inspection of TLS or another port 80 listener

[Morta]

I have looked wir netsocket but no other app is on port 80 and is working so far

I have also problem that the cert redirect automatically from www.xx.ch to xx.ch

When I insert xx.ch in the map file all in existing sub domains goes to xx.ch as examples buff.xx.ch goes to xx.ch

How can I map at least www.xx.ch to a service without redirect or better how can I map xx.ch and www.xx.ch without redirecting all subdomain to xx.ch vhost

[TheHellSite]

Just set your Webserver as default backend on the https frontend.

[Morta]

I have two Webserver. So I can took only one.
Which one? WEBSERVER_backend or NAS_backend?

[TheHellSite]

Obviously your Webserver and not your NAS.......  ::)

Also you just exposed your Domain again  :-\

Please don't get me wrong but I really hope you know what you are doing by self-hosting and exposing any services from your private internet connection!
There are many risks involved and you don't seem to me like someone that is aware of everything he is doing and why it needs to be done.

[Morta]

Would be possible to adjust the tutorial with ipv6 support or a hint how to add ipv6 support to a existing configuration?

[Bunch]

Would be possible to adjust the tutorial with ipv6 support or a hint how to add ipv6 support to a existing configuration?

For the frontend that using 0.0.0.0:80, "add" [::]:80
For the frontend that using 0.0.0.0:443, "add" [::]:443

In firewall rule, add one more rule to "allow" "ipv6" "TCP" to "this firewall" in "port 80 and 443"

Note: your DDNS need to update ipv6 too
ddclient 3.9.1 should be able to update ipv6 . However, I don't know rather OPNsense had implemented to update ipv6 automatically.

P.S. dyndns plugin is too old, it doesn't support ipv6

[figadore]

When creating the virtual IP, there's a new option not included in the picture. Is there a correct setting for "Allow service binding"? Does it make a difference?

[TheHellSite]

I mean it is self-explanatory.

To answer your question: You will want this enabled.

[opns_neuling]

I got this error

[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

on Step:
Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
Here we will only create a "NoSSL_condition", which is necessary in order to identify non-HTTP traffic.


Please replace "req.ssl_ver gt 0" with "ssl_fc" also, select "custom" and on custom field enter "ssl_fc"

https://redmine.pfsense.org/issues/9261

[Morta]

Can you make a screenshot? I don't understand you instructions

I have no req.ssl_ver 0 in my configuration!

So?

[Bunch]

Mentioned in #183
Traffic is SSL (locally deciphered) is ssl_fc

And the condition need to be replaced is Part 5.7.

[huuich]

Thanks for detailed instructions, I've follow step by step to make a web hosting running nginx with https support.

My HAProxy Config:

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_622eebaf197419.36314953 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_622eebaf197419.36314953

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/622eef9a9d7268.16491040.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Backend: backend_pool_abc ()
backend backend_pool_meet_huuich_vn
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server PLEX_server 192.168.82.11:32400 ssl verify none

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

My Map File content:

Code Select Expand

# public access subdomains
plex PLEX_backend

My nginx website config

Code Select Expand

server {
        listen 32400;
        listen [::]:32400;

        root /var/www/mywebsite.com/html;
        index index.html index.htm index.nginx-debian.html;

        server_name mywebsite.com;

        location / {
                try_files $uri $uri/ =404;
        }
}

I can go my website on internal http://192.168.82.11:32400 is ok but when I access http://mywebsite.com browser go to https://mywebsite.com and show error

Code Select Expand

503 Service Unavailable
No server is available to handle this request.

How can I fix this and show my website https://mywebsite.com ok? Thanks!