Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[TheHellSite]

Okay, I've been through the instructions at least 3 times and cannot find why it's not working. Can someone please take a look? Other than it being currently disabled, obviously.

Firewall rule is:
IPv4 TCP Src* Port* Dest WAN address Port AliasforHTTP/HTTPS Gateway* Schedule*

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

#
# NOTE: HAProxy is currently DISABLED
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch 1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend ()
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 15m

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_63859d8c6a7b81.10799804 ssl_fc

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_63859d8c6a7b81.10799804

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6385a4c7e68d06.81674833.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63859df5259306.89264162.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: homeassistant_backend ()
backend homeassistant_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server HomeAssistant 192.168.0.3:8123 check inter 30s port 8123

# Backend: web_backend ()
backend web_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server web 192.168.0.4:80

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: factorio_backend ()
backend factorio_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server factorio 192.168.0.17:80

# Backend: jira_backend ()
backend jira_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server jira 192.168.0.20:80

# Backend: meshcentral_backend ()
backend meshcentral_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server meshcentral 192.168.0.41:443

# Backend: nextcloud_backend ()
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nextcloud 192.168.0.38:443 ssl alpn h2,http/1.1 verify none


# statistics are DISABLED


And the mapping file, which I have tried with the full FQDN and without the periods as well.

Code Select Expand

# public access subdomains
hass. homeassistant_backend
factorio. factorio_backend
jira. jira_backend
mesh. meshcentral_backend
nextcloud. nextcloud_backend
web_backend

1. No real error description. What is the error? What is not working?

2. Your map file seems off. Remove the dots after each service, I didn't have them in my example config so why do you have them in yours?

3. At the end of your mapfile, you have "web_backend" without any matching scheme before it.

[Stingily7770]

The dots were just an attempt to see if the default backend was an issue by listing my default in the map, but I wanted to be sure nothing else would conflict with the same 'startswith'. web_backend had my domain and I missed that it was left empty when I cleaned up PII. I have went through the config again, removing the periods and defining the default on the mapping rule again.

Rejected connections every time, but I know the traffic is making it because my port forward to an internal server I am using with Nginx Proxy Manager (and am seeking to replace with this setup) works when I disable HAProxy and put back its port forward. I've completely removed the port forward and rebooted the router to ensure that it isn't interfering. My Opnsense WebGUI port was already changed to 81.

If I attempt to browse to my IP from outside my network, http shows ERR_EMPTY_RESPONSE in Chrome, https shows ERR_CONNECTION_CLOSED.

One thing I find really odd is I'm not getting anything in the log file until I disable the service, then I see the stopping messages. I setup a health check that's working and writes to the log, but that was just more troubleshooting to be sure it was able to see my internal service from the router.

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch 1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend ()
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 15m

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_63859d8c6a7b81.10799804 ssl_fc

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_63859d8c6a7b81.10799804

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6385a4c7e68d06.81674833.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63859df5259306.89264162.txt,web_backend)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: homeassistant_backend ()
backend homeassistant_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server HomeAssistant 192.168.0.3:8123

# Backend: web_backend ()
backend web_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server web 192.168.0.4:80

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: factorio_backend ()
backend factorio_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server factorio 192.168.0.17:80

# Backend: jira_backend ()
backend jira_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server jira 192.168.0.20:80

# Backend: meshcentral_backend ()
backend meshcentral_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server meshcentral 192.168.0.41:443

# Backend: nextcloud_backend ()
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nextcloud 192.168.0.38:443 ssl alpn h2,http/1.1 verify none

# Backend: gallery_backend ()
backend gallery_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server gallery 192.168.0.12:80



# statistics are DISABLED


SNI_Mapping

Code Select Expand


# public access subdomains
hass homeassistant_backend
factorio factorio_backend
jira jira_backend
mesh meshcentral_backend
nextcloud nextcloud_backend
gallery gallery_backend

[TheHellSite]

If I attempt to browse to my IP from outside my network, http shows ERR_EMPTY_RESPONSE in Chrome, https shows ERR_CONNECTION_CLOSED.

If you don't even get any 503s with a blank white Page and the HAProxy Log is not indicating any traffic, then your firewall rule is configured wrong.

Also if you are not willing to share the HAProxy log then I am unable to help. You have to set it to "Informational" in the top right corner!

[Stingily7770]

If you don't even get any 503s with a blank white Page and the HAProxy Log is not indicating any traffic, then your firewall rule is configured wrong.

Also if you are not willing to share the HAProxy log then I am unable to help. You have to set it to "Informational" in the top right corner!

Shouldn't the HAProxy log show startups as well? This is fresh after a reboot (empty log), restarting the HAProxy service from System, Diagnostics, Services (still empty), then unchecking Enable HAProxy and applying (Stop messages appear), then rechecking Enable HAProxy and applying (nothing new added).

Confirmed haproxy is listening. I've also tried the DNS redirection to my opnsense internal IP with the same results from inside the network.

Code Select Expand

root@OPNsense:~ # sockstat -l | grep '443\|80'
www      haproxy    3539  4  tcp4   *:443                 *:*
www      haproxy    3539  5  tcp4   *:80                  *:*
www      haproxy    3539  6  tcp4   127.4.4.3:80          *:*
www      haproxy    3539  7  tcp4   127.4.4.3:443         *:*


I've also compared the firewall policy once again, and it is exactly like https://postimg.cc/VS3DKGPg other than I named my alias HTTP_HTTPS.

[TheHellSite]

Please just post the overview of YOUR WAN rules page and YOUR wan rule. Troubleshooting with pictures of my tutorial won't get you any further...
If you don't expose your WAN IP or public domain name then you have nothing to worry about.

Again since your HAProxy log is empty this means that there is NO traffic reaching HAProxy and maybe not even your firewall. You would have to check the firewall logs for this though.

Also is there another firewall / router placed before your OPNsense (double NAT)?

[Stingily7770]

Ignore the HTTP_HTTPS Port Forward. I've been removing it and rebooting to ensure a clean test, but didn't want to mess with it this morning. The port forwards all work though, so the firewall is definitely receiving the traffic. There's an ATT Gateway in front of my router in passthrough mode.

Also grabbed a packet capture of the traffic, after disabling the port forward and confirming haproxy is bound to 80/443. Phone was off the network and packet capture was filtered to its public ip.

Code Select Expand

10:15:09.123024 IP PHONE.25700 > ROUTER.443: Flags [SEW], seq 642215500, win 65535, options [mss 1460,nop,wscale 12,sackOK,TS val 2478600365 ecr 0], length 0
10:15:09.123057 IP ROUTER.443 > PHONE.25700: Flags [S.E], seq 639098840, ack 642215501, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 3059434307 ecr 2478600365], length 0
10:15:09.141384 IP PHONE.25700 > ROUTER.443: Flags [.], ack 1, win 256, options [nop,nop,TS val 2478600384 ecr 3059434307], length 0
10:15:09.142758 IP PHONE.25700 > ROUTER.443: Flags [P.], seq 1:518, ack 1, win 256, options [nop,nop,TS val 2478600384 ecr 3059434307], length 517
10:15:09.142790 IP ROUTER.443 > PHONE.25700: Flags [.], ack 518, win 510, options [nop,nop,TS val 3059434328 ecr 2478600384], length 0
10:15:09.142818 IP ROUTER.443 > PHONE.25700: Flags [F.], seq 1, ack 518, win 514, options [nop,nop,TS val 3059434328 ecr 2478600384], length 0
10:15:09.161122 IP PHONE.25700 > ROUTER.443: Flags [.], ack 2, win 256, options [nop,nop,TS val 2478600404 ecr 3059434328], length 0
10:15:09.161149 IP PHONE.25700 > ROUTER.443: Flags [F.], seq 518, ack 2, win 256, options [nop,nop,TS val 2478600404 ecr 3059434328], length 0
10:15:09.161163 IP ROUTER.443 > PHONE.25700: Flags [.], ack 519, win 513, options [nop,nop,TS val 3059434346 ecr 2478600404], length 0
10:15:15.141831 IP PHONE.26438 > ROUTER.443: Flags [SEW], seq 3285634286, win 65535, options [mss 1460,nop,wscale 12,sackOK,TS val 4185299120 ecr 0], length 0
10:15:15.141883 IP ROUTER.443 > PHONE.26438: Flags [S.E], seq 4283526657, ack 3285634287, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 95186048 ecr 4185299120], length 0
10:15:15.160570 IP PHONE.26438 > ROUTER.443: Flags [.], ack 1, win 256, options [nop,nop,TS val 4185299139 ecr 95186048], length 0
10:15:15.161943 IP PHONE.26438 > ROUTER.443: Flags [P.], seq 1:518, ack 1, win 256, options [nop,nop,TS val 4185299139 ecr 95186048], length 517
10:15:15.161977 IP ROUTER.443 > PHONE.26438: Flags [.], ack 518, win 510, options [nop,nop,TS val 95186067 ecr 4185299139], length 0
10:15:15.162008 IP ROUTER.443 > PHONE.26438: Flags [F.], seq 1, ack 518, win 514, options [nop,nop,TS val 95186067 ecr 4185299139], length 0
10:15:15.181057 IP PHONE.26438 > ROUTER.443: Flags [.], ack 2, win 256, options [nop,nop,TS val 4185299159 ecr 95186067], length 0
10:15:15.181181 IP PHONE.26438 > ROUTER.443: Flags [F.], seq 518, ack 2, win 256, options [nop,nop,TS val 4185299159 ecr 95186067], length 0
10:15:15.181199 IP ROUTER.443 > PHONE.26438: Flags [.], ack 519, win 513, options [nop,nop,TS val 95186086 ecr 4185299159], length 0


[TheHellSite]

Shouldn't the HAProxy log show startups as well? This is fresh after a reboot (empty log), restarting the HAProxy service from System, Diagnostics, Services (still empty), then unchecking Enable HAProxy and applying (Stop messages appear), then rechecking Enable HAProxy and applying (nothing new added).

HAProxy shouldn't even print a stop message in the haproxy log at all. Only if there are errors, f.e. misconfiguration of your firewall.

After enabling HAProxy and hitting "Apply" then waiting for 5sec and reloading the HAProxy settings page. Is there a green Play icon in the top right corner when you are on the HAProxy Settings page?

If not, then you have something misconfigured or another service is listening on the same ip:port.

[Stingily7770]

Yes, the green play button is there.

[TheHellSite]

Please reboot the firewall, then post an updated haproxy config export and haproxy log export (after trying to access your services using an FQDN) .

[Stingily7770]

Log is empty. Green play button is present.

[TheHellSite]

I strongly believe it is not empty... AGAIN You have to set it to informational.
Post a screenshot of your PUBLIC_SUBDOMAINS_map-rule.

[Stingily7770]

Here's my log page and settings and the map-rule.

[TheHellSite]

Based on this the traffic is clearly NOT reaching HAProxy. Why? I can't tell you.
You will have to troubleshoot here on your own. Your HAProxy config and wan firewall rule looks fine.

I can only imagine that you have some leftover port forwards or whatever that are intercepting the traffic.
If you enable the logging on the WAN "HAProxy" rule and go to Firewall --> Log Files --> Live View and filter for the WAN rule... You should see the traffic beeing green.
However this does not guarantee that any old port forward rules are interfering.

[Stingily7770]

Alright. Thanks for the attempt.

[TheHellSite]

Well there has to be something wrong with either you ATT modem or you overall OPNsense settings.

Since the SNI_frontend is listening on all IPs, interfaces and 80+443 port (0.0.0.0:80+443) it should at least spit out something to the HAProxy logs when there is a connection to your OPNsense WAN IP on any of the two ports. Even if the SSL handshake or anything after it fails, the SNI_frontend will always make log entries.

If it doens't then there is something wrong with your OPNsense in general or some other network device in your WAN facing setup.

Take a look at my HAProxy log example. The first hit always goes to the SNI_frontend, from there to the SSL_server and is then catched by the HTTP(S)_frontend.

Code Select Expand


2022-12-06T00:23:39 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:34677 to OPNSENSE_WAN_IP:443 (1_HTTPS_frontend/HTTP)
2022-12-06T00:23:39 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:34677 to OPNSENSE_WAN_IP:443 (0_SNI_frontend/TCP)
2022-12-06T00:23:39 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:9659 to OPNSENSE_WAN_IP:443 (1_HTTPS_frontend/HTTP)
2022-12-06T00:23:39 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:9659 to OPNSENSE_WAN_IP:443 (0_SNI_frontend/TCP)
2022-12-06T00:23:03 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:62798 to OPNSENSE_WAN_IP:443 (1_HTTPS_frontend/HTTP)
2022-12-06T00:23:03 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:62798 to OPNSENSE_WAN_IP:443 (0_SNI_frontend/TCP)
2022-12-06T00:23:01 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:62797 to OPNSENSE_WAN_IP:443 (1_HTTPS_frontend/HTTP)
2022-12-06T00:23:01 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:62797 to OPNSENSE_WAN_IP:443 (0_SNI_frontend/TCP)


You really should consider testing if your HTTP(S) WAN rule is working properly (see my previous reply) and or if there is any other service on your OPNsense or network device in general messing with the data stream.

I am sadly unable to help here since the traffic is not hitting your firewall and or haproxy.