Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[TheHellSite]

I can go my website on internal http://192.168.82.11:32400 is ok but when I access http://mywebsite.com browser go to https://mywebsite.com and show error

Code Select Expand

503 Service Unavailable
No server is available to handle this request.

How can I fix this and show my website https://mywebsite.com ok? Thanks!

First.
The entry "plex PLEX_backend" in the mapfile means that you will have to access it using the "plex" subdomain. --> In your case "plex.mywebsite.com"!

Alternatively just set the PLEX_backend as default backend on your HTTPS_frontend.

Second.
http will always get redirected to https. This is intended and you will most probably want this! This is configured using the HTTPtoHTTPS_rule and NoSSL_condition.

Third.
Apart from the above your config looks good. (just took a very short look at it)

Fourth.
You might have to disable the SSL checkbox in the PLEX_server settings.
But only if you are REALLY accessing it locally using http://IP:32400 and the service is NOT redirecting you to HTTPS. But I highly doubt this since Plex is running on a self-signed SSL cert by default...

[huuich]

First.
The entry "plex PLEX_backend" in the mapfile means that you will have to access it using the "plex" subdomain. --> In your case "plex.mywebsite.com"!

Alternatively just set the PLEX_backend as default backend on your HTTPS_frontend.

Second.
http will always get redirected to https. This is intended and you will most probably want this! This is configured using the HTTPtoHTTPS_rule and NoSSL_condition.

Third.
Apart from the above your config looks good. (just took a very short look at it)

Fourth.
You might have to disable the SSL checkbox in the PLEX_server settings.
But only if you are REALLY accessing it locally using http://IP:32400 and the service is NOT redirecting you to HTTPS. But I highly doubt this since Plex is running on a self-signed SSL cert by default...

You are right, mysubdomain.mywebsite.com, and follow your guide I've edit my Map File to "mysubdomain PLEX_backend" and "disable the SSL checkbox in the PLEX_server settings" and voila my subdomain website run https:// ok with A+ score, Thank you so much for quick and detailed reply!

P/s: I've follow your guide with cloudflare instead. (almost every settings are the same like your guide)

[huuich]

I want to set up HAProxy just for routing traffic based on URLs (https://xyz.domain.com goes to server 1 and https://abc.domain.com goes to server 2, etc...).
All SSL stuff for the destination web servers is being handled by a separate Linux certificate server and the web servers themselfes, independent from OPNsense/HAProxy. HAProxy is really only needed for routing traffic based on URLs, nothing more, nothing less.I want to set up HAProxy just for routing traffic based on URLs (https://xyz.domain.com goes to server 1 and https://abc.domain.com goes to server 2, etc...).
All SSL stuff for the destination web servers is being handled by a separate Linux certificate server and the web servers themselfes, independent from OPNsense/HAProxy. HAProxy is really only needed for routing traffic based on URLs, nothing more, nothing less.

Did you find solution for your purpose? I wanna do same like you.

[huuich]

You have to set your backends and frontends to HTTP Mode.
Also disable SSL offloading on the frontends.

But I can't guarantee for sure that it will work.
TCP Mode will never (with a few exceptions) work because there is no header in the packets that would tell HAProxy which service to send the traffic to.

HTTP Mode could work, but you might need to create some "http header contains..." conditions.

I can do that using SSL passthrough follow step by step this guide from @alh

Every works ok except my Nginx log can not receive real ip from client, it's only show ip  local of my OPNSense server in file /var/log/nginx/access.log

Do you think can mix SSL passthrough with your guide (edit 0_SNI_frontend with rules from SSL passthrough and Default Backend Pool) or any solution to receive real ip from client with SSL passthrough? Thanks!

Code Select Expand

This should work for any TCP-based SSL/TLS encrypted service in passthrough (HAProxy: TCP) mode... It does NOT work for STARTTLS!

In this example I use TCP port 443.

HAProxy plugin: Create "Real Server" (enter name, IP/FQDN and port number if different from 443, the rest can be left at default)
HAProxy plugin: Create "Backend Pool" (enter name, set mode to TCP and select the real server from step 1)
HAProxy plugin: Create "Condition" (enter name ["traffic_ssl"], condition type is "custom condition (option pass-through)" with value "req_ssl_hello_type 1")
HAProxy plugin: Create "condition" (enter name ["myservice_sni"], condition type is "SNI TLS extension matches (TCP request content inspection)" with value "myservice.example.com" or whatever your FQDN is)
HAProxy plugin: Create "Rule" (enter name ["request_inspect_delay"], select no condition, function is "tcp-request inspect delay" with value "5s" or whatever suits you)
HAProxy plugin: Create "Rule" (enter name ["request_content_accept_ssl"], select condition of 3 ["traffic_ssl"], function is "tcp-request content accept")
HAProxy plugin: Create "Rule" (enter name ["myservice_sni"], select condition of 4 ["myservice_sni"], function is "Use specific backend pool" with your pool from 2)
HAProxy plugin: Create "Public service" (enter name ["https_passthrough"], choose a listen address [":443" for all], type is "TCP" and select the 3 rules created earlier)
HAProxy plugin: Enable plugin or test/apply
Firewall: allow incoming traffic to WAN (address) or whatever for TCP port 443.

That works at least for me. If you have double NAT you would need to disable port randomization for the proxied port...

Does that help you?

[Bunch]

If all SSL certificate are handled by webserver themselves.
Follow #176, step 1-4 will allow 0_SNI_FRONTEND to recognize TLS package and redirect under TCP mode

Note: All backend redirect from 0_SNI_FRONTEND should be in TCP mode
HAProxy only work with server that using TLS package. For those server that doesn't contain TLS package (for example, game servers), HAProxy won't work. Although you can set default backend server to game server, but there's only one default backend server.

BTW, for backend server getting source IP, enabling X-Forwarded-For header for all frontend should work
However, this one only apply to HTTP.
reference
reference 2

[huuich]

If all SSL certificate are handled by webserver themselves.
Follow #176, step 1-4 will allow 0_SNI_FRONTEND to recognize TLS package and redirect under TCP mode

Note: All backend redirect from 0_SNI_FRONTEND should be in TCP mode
HAProxy only work with server that using TLS package. For those server that doesn't contain TLS package (for example, game servers), HAProxy won't work. Although you can set default backend server to game server, but there's only one default backend server.

BTW, for backend server getting source IP, enabling X-Forwarded-For header for all frontend should work
However, this one only apply to HTTP.
reference
reference 2

I've follow your guide but though my backend server ctl_backend has code

Code Select Expand

http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Real-IP %[src]

But it can not get source ip, here is my full config, could you show me how to get source ip? Or I have to have 2 OPNSense, one for SSL passthrough and one use Let's Encrypt Wildcard Certificates like this topic guide? Could 1  OPNSense sever run both nginx reverse proxy and HAProxy?

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    # tuning options
    timeout client 30s

    # logging options
    # ACL: traffic_ssl
    acl acl_601a842f14cee3.17646593 req_ssl_hello_type 1

    # ACTION: request_content_accept_ssl
    tcp-request content accept if acl_601a842f14cee3.17646593
    # ACTION: request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Frontend (DISABLED): 1_HTTP_frontend (Listening on 192.168.64.1:80)

# Frontend (DISABLED): 1_HTTPS_frontend (Listening on 192.168.64.1:443)

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Real-IP %[src]
    http-reuse safe
    server PLEX_server 192.168.82.11:32400

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend (DISABLED): SSL_backend ()

# Backend: ctl_backend ()
backend ctl_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Real-IP %[src]
    http-reuse safe
    server ctl_server 192.168.82.11:32401

[Bunch]

Notice that you are using TCP mode frontend only.
X-forwarded-for will not simply work under any case that using TCP mode, as X-forwarded-for will only work under HTTP mode and need to enable in all layers (TCP mode will not forward any header)

For TCP mode, you need to set all backend that wants to get real client IP and redirected from this frontend to use Proxy Protocol (either version 1 or 2, while verion 1 sending ASCII data and version 2 sending binary)
You will find "send-proxy" under that backend, which means it will send another package via proxy protocol to your real server.

Your real server need to modify config to read proxy protocol package too.
Nginx guide to set it accept, reading and logging proxy protocol address
(Not 2 lines, tones should be modified whenever there is a layer of TCP mode)

P.S. Theoretically, setting like OP having (SNI frontend->send-proxy backend->HTTPS frontend with X-forwarded-for) should also forward proxy protocol package to real server too.

Example: my v2ray server is living in TCP mode (although I don't care about the IP log)
I will get router IP in log if I don't user proxy protocol.
I cannot reach v2ray server if I use proxy protocol in haproxy but not changing any config of v2ray server (probably rejected by the server itself)
I will get correct client IP if I use proxy protocol in haproxy and changing the config of v2ray to accept proxy protocol

My HAProxy config is as follow:
1. open_back is OPENVPN live in TCP mode, and will not get any real client IP (Always 192.168.3.1)
2. hmdir_back is v2ray (another proxy server) live in TCP mode, and will get real client IP (I need to modify v2ray config to make it read and log proxyprotocol
3. opn_back is opnsense page live in HTTP mode, will not get real client IP as I don't care.
4. unifi_back is unifi controller page live in HTTP mode, I don't care rather it get real IP too
5. hkbn_back is ISP router page, which has no access right to my LAN, just a dummy to fake ISP that I'm using their device, all unwanted access will reditect to this page (Their device always open 443 to public access)
5. opn_back and unifi_back can only be accessed with source IP as 192.168.3.0/24 and 192.168.5.0/24 and I have tested no one can access to these pages except my LAN clients. Thus, real client IP should be passed to SSL_frontend successfully too.
Just need to add backend with http-request set-header X-Real-IP %[src] (Although there will be warning, you can do it in frontend, as HAProxy config manualmentioned both ways will work) and config webserver read and log X-Real-IP

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if v2ray)
frontend tcp_front
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_6212326a7c07e4.28981163 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_6212326a7c07e4.28981163
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6214a3ae639096.17472719.txt)]

# Frontend: http_front (Listen to VIP:80 and redirect to 443)
frontend http_front
    bind 192.168.6.1:80 name 192.168.6.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: http
    acl acl_62123bbee27260.60165685 ssl_fc

    # ACTION: http_to_https
    http-request redirect scheme https code 301 if !acl_62123bbee27260.60165685

# Frontend: ssl_front (Listen to VIP 443, SSL offload cert)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.6.1:443 name 192.168.6.1:443 accept-proxy ssl no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621244f0de5919.36753000.certlist
    mode http
    option http-keep-alive
    default_backend hkbn_back
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: local
    acl acl_62123a1cebe813.09309501 src 192.168.3.0/24 192.168.5.0/24

    # ACTION: local_map_ru
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623230d7bffd04.94702836.txt)] if acl_62123a1cebe813.09309501

# Backend: hmdir_back (Backend of v2ray)
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.3.3:443 send-proxy-v2 check-send-proxy

# Backend: hkbn_back (Backend of HKBN)
backend hkbn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hkbn 192.168.4.2:443 ssl verify none

# Backend: SSL_back (Backend to redirect SSL servers)
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server ssl 192.168.6.1 send-proxy-v2 check-send-proxy

# Backend: opn_back (Backend of opnsense with SSL)
backend opn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opn 192.168.3.1:8443 ssl verify none

# Backend: unifi_back (Backend of unifi with SSL)
backend unifi_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server unifi 192.168.3.4:8443 ssl verify none

# Backend: open_back (Backend of OpenVPN)
backend open_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server open 192.168.3.1:10443

[huuich]

@Bunch Thank you for detailed guide and attached config. I've tried to simulate like your server but now my server not working

This is my config

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if v2ray))
frontend tcp_front
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: traffic_ssl
    acl acl_601a842f14cee3.17646593 req_ssl_hello_type 1

    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_601a842f14cee3.17646593
    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Frontend: http_front (Listening on 192.168.64.1:80)
frontend http_front
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_622eebaf197419.36314953 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_622eebaf197419.36314953

# Frontend: ssl_front (Listening on 192.168.64.1:443)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/622eef9a9d7268.16491040.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Backend: hmdir_back ()
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.82.11:32401 send-proxy-v2 check-send-proxy

# Backend (DISABLED): PLEX_backend ()

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend: SSL_back ()
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend (DISABLED): ctl_backend ()

My hmdir_ru is link to PUBLIC_SUBDOMAINS_map map file with content:

Code Select Expand

# public access subdomains
c hmdir_back

Could you share content of map file in local_map_ru in this config

Code Select Expand

# ACTION: local_map_ru
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623230d7bffd04.94702836.txt)] if acl_62123a1cebe813.09309501

And ACL http in this config

Code Select Expand

# logging options
    # ACL: http
    acl acl_62123bbee27260.60165685 ssl_fc

Thank you!

[Bunch]

My local_map_rule is:
Test type= if
Selection condition= local
Execute function= Map domains to backend pools using a map file
Map_file= local_map

Condition local:
Condition type= Source IP matches specified IP
Source IP=192.168.3.0/24 192.168.5.0/24

Local_map:

Code Select Expand

opn opn_back
unifi unifi_back

For ssl_fc, it is mentioned in #174 Step 5
And #183

For testing why it won't work in your case,
You can try not using Proxy Protocol first. If you can access your backend, that means your webserver config is not accepting proxy protocol (It won't work both way at the same time)
You need to use Proxy Protocol and modify webserver config ( for example, /etc/nginx/sites-enabled/default) at the same time

Similar test case mentioned in previous post

Example: my v2ray server is living in TCP mode (although I don't care about the IP log)
I will get router IP in log if I don't user proxy protocol.
I cannot reach v2ray server if I use proxy protocol in haproxy but not changing any config of v2ray server (probably rejected by the server itself)
I will get correct client IP if I use proxy protocol in haproxy and changing the config of v2ray to accept proxy protocol

[huuich]

For testing why it won't work in your case,
You can try not using Proxy Protocol first. If you can access your backend, that means your webserver config is not accepting proxy protocol (It won't work both way at the same time)
You need to use Proxy Protocol and modify webserver config ( for example, /etc/nginx/sites-enabled/default) at the same time

Similar test case mentioned in previous post

Example: my v2ray server is living in TCP mode (although I don't care about the IP log)
I will get router IP in log if I don't user proxy protocol.
I cannot reach v2ray server if I use proxy protocol in haproxy but not changing any config of v2ray server (probably rejected by the server itself)
I will get correct client IP if I use proxy protocol in haproxy and changing the config of v2ray to accept proxy protocol

My /etc/nginx/sites-enabled is

Code Select Expand

server {
listen 32401;
listen [::]:32401;

server_name c.mywebsite.com;

root /var/www/html;

location / {
try_files $uri $uri/ /yourls-loader.php$is_args$args;
}

location ~ \.php$ {
include fastcgi.conf;

fastcgi_index index.php;

fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}


How could use Proxy Protocol and modify webserver config with my HAProxy config above?
Sorry for basic question. Thank you for detailed explaination.

[Bunch]

That's not basic question IMO.  :D
The modification method is mentioned in Nginx guide

Should be modified as below (haven't tested it)

Code Select Expand

server {
listen 32401 proxy_protocol;
listen [::]:32401 proxy_protocol;

server_name c.mywebsite.com;

root /var/www/html;

location / {
try_files $uri $uri/ /yourls-loader.php$is_args$args;
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr
}

location ~ \.php$ {
include fastcgi.conf;

fastcgi_index index.php;

fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}


For the log config, I forgot rather it is /etc/nginx/nginx.conf by default
You need to change the log format inside http{} to something like

Code Select Expand

log_format combined '$proxy_protocol_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent"';

[huuich]

The modification method is mentioned in Nginx guide

Should be modified as below (haven't tested it)

Your code "haven't tested but working for me with http website, but when I change to https with certbot

Code Select Expand

certbot --nginx -d c.mywebsite.com

My https website can not access though I've change my server port to 443, this is my nginx website config

Code Select Expand

server {
listen 32401 proxy_protocol;
listen [::]:32401 proxy_protocol;

listen [::]:443 ssl ipv6only=on proxy_protocol; # managed by Certbot
    listen 443 ssl proxy_protocol; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/c.mywebsite.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/c.mywebsite.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

server_name c.mywebsite.com;

root /var/www/html;

location / {
try_files $uri $uri/ /yourls-loader.php$is_args$args;
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr;
}

location ~ \.php$ {
include fastcgi.conf;

fastcgi_index index.php;

fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}
}

and my current HAProxy configure

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if v2ray))
frontend tcp_front
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: traffic_ssl
    acl acl_601a842f14cee3.17646593 req_ssl_hello_type 1

    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_601a842f14cee3.17646593
    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Frontend: http_front (Listening on 192.168.64.1:80)
frontend http_front
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_622eebaf197419.36314953 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_622eebaf197419.36314953

# Frontend: ssl_front (Listening on 192.168.64.1:443)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/622eef9a9d7268.16491040.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Backend: hmdir_back ()
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.82.11:443 ssl verify none send-proxy-v2 check-send-proxy

# Backend (DISABLED): PLEX_backend ()

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend: SSL_back ()
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend (DISABLED): ctl_backend ()

Could you take a look and guide me how to run https on my website?

For the log config, I forgot rather it is /etc/nginx/nginx.conf by default
You need to change the log format inside http{} to something like

Code Select Expand

log_format combined '$proxy_protocol_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent"';

Fire I use above code has a error, so I google and found a solution for working log, I post here for anyone need

Code Select Expand

log_format my_log '$proxy_protocol_addr - $remote_user [$time_local] '
                       '"$request" $status $body_bytes_sent '
                       '"$http_referer" "$http_user_agent"';

access_log /var/log/nginx/access.log my_log;

[Bunch]

Better check /etc/letsencrypt/options-ssl-nginx.conf
Check rather the SSL protocol is too old (is it having something like TLSv1 TLSv1.1)
If the settings are too old, try to change those settings in that file according to this link

Guess below lines need to be modified if case exists.

Code Select Expand

ssl_session_timeout 1d;
    ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
    ssl_session_tickets off;

    # intermediate configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    # HSTS (ngx_http_headers_module is required) (63072000 seconds)
    add_header Strict-Transport-Security "max-age=63072000" always;

    # OCSP stapling
    ssl_stapling on;
    ssl_stapling_verify on;

For real server in HAProxy, I have tried turning SSL on or off doesn't affect my servers under TCP mode, don't know rather it is the same in your case, please try it too.

[cookiemonster]

I'm probably out of place saying this, as is not my thread, but should't this discussion go to another thread and leave this one for it's original purpose?
It has branched off now to "how can I enable TLS on my website", from "how can I log the client ip not the proxy ip on the backend webserver" and "how do I use proxy_protocol".
What do you think?

[Bunch]

Yes, I think it is out of HAProxy scope too.
When send-proxy or X-forwarded-for part is done in HAProxy, nothing can be done in HAProxy afterwards.
All other settings are related to Nginx or some other server settings.