Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[TheHellSite]

I'm probably out of place saying this, as is not my thread, but should't this discussion go to another thread and leave this one for it's original purpose?
It has branched off now to "how can I enable TLS on my website", from "how can I log the client ip not the proxy ip on the backend webserver" and "how do I use proxy_protocol".
What do you think?

Absolutely true! 

I don't want to sound like an asshole here, but this tutorial was intended to get the basics working for new users.
This is also why I stopped answering questions about issues like "my service_abc has the requirement_xyz how work?".

If there are any questions in that regard then people should consider posting them...
here: https://forum.opnsense.org/index.php?board=28.0
or here: https://discourse.haproxy.org/

[TheHellSite]

It seems that it is the same issue as This thread
I have the same issue after update and reboot.
For temporary fix, edit the VIP, save without any changes, then apply.
You will able to start HAProxy again.

Thank you for posting the workaround!

[TheHellSite]

Just a quick notifcation for everyone following the thread.
It seems like Let's Encrypt changed something regarding wildcard certificates.

I updated the picture in Part 3 - Step 6 to reflect the changes necessary in order to obtain a certificate.

You will have to remove the alt name "*.yourdomain.tld" and change the common name to "*.yourdomain.tld".

[ElliotAlderson]

Code: [Select]

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind website.com:443 name website.com:443
    bind website.com:80 name website.com:80
    mode tcp
    default_backend SSL_backend
    timeout client 30s

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 10.10.10.1:80 name 10.10.10.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 30s

    # ACL: NoSSL_cond
    acl acl_62548efaf067e6.21908045 req.ssl_ver gt 0
    # ACTION: HTTPupgrade_rule
    http-request redirect scheme https if !acl_62548efaf067e6.21908045

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 10.10.10.1:443 name 10.10.10.1:443 accept-proxy ssl ssl-min-ver TLSv1.3 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62549082216928.65241361.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 30s

    # ACTION: PUBLIC_SUBDOMAINS_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62548f2d97ef05.80304462.txt)]

# Backend: club_backend ()
backend club_backend
website.com    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server club_host 10.0.0.94:3000 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
website.com    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    timeout connect 30s
    timeout server 30s
    server SSL_host 10.10.10.1 send-proxy-v2 check-send-proxy

I switched over from pfSense to OPNSense months ago and I had to set my side projects to the side because I simply could not replicate my HAProxy setup from before. I'm thankful for this tutorial since it's seems like the closest to what I used to have.

I'm extremely lost here. I have gone through this tutorial many times to double check my steps, I have tried changing things on my own.
At the moment, the HTTP->HTTPS redirect doesn't seem to work at all (empty response, no redirected) and the https site gives a blank response as well.
I have checked it with tcpdump directly on the OPNSense shell and could see that packets do get exchanged between my host and the virtual IP.  (TCP handshake, TLSv1 Client Hello, End connection )
The site itself is definitely working correctly internally at the host specified in the config file.

In the config I posted, I used website.com:443 in the SNI frontend.
I have tried it with 0.0.0.0:443 and my public IP with no success.

10.0.0.94 is in my LAN.
10.10.10.1 is the virtual interface I created.

My wildcard certificate seems to be working correctly.

I would really appreciate some help 


By the way, what would the process be for getting another domain and wildcard cert to work added to this setup?

[Bunch]

You are missing "code 301" in HTTPupgrade_rule
(Part 5-9)

Explanation of the code can be found here

This technique will only work when using mode http because it redirects at the HTTP layer using a 302 Found HTTP response status, which is known as a temporary redirect. Once you’re fully committed to using HTTPS and have tested it thoroughly on your website, you may wish to instruct the browser to cache the redirect, which will save one round trip between the browser and HAProxy, speeding up page load times. Set the code parameter to 301 to send a 301 Moved Permanently status back, which browsers can cache:

Although it should work with 302, just give it a try

BTW, I don't know why there is website.com in the code of both backend. There shouldn't be something like this.

[ElliotAlderson]

Oh, yeah. I actually did have the code 301 there originally. It's still the same effect with that though. That's just something I forgot to change back when I was trying new things. (Saw an older forum post that didn't use the "code 301" part.)

Regarding about the website.com,
In the config I posted, I used website.com:443 in the SNI frontend.
I have tried it with 0.0.0.0:443 and my public IP as well with no success.

I just can't get it to work so I'm trying different things.

[Bunch]

I mean

# Backend: SSL_backend ()
backend SSL_backend
website.com    mode tcp

And

# Backend: club_backend ()
backend club_backend
website.com    mode http

Usually the wouldn't have SNI or domain names hard code in backends

[keyboardDabbler]

This week i have moved away from pfSense, I had acme, cloudflare & HAProxy working prior to the switch. Installed opnsense while slowly getting my services back online I came across this well written tutorial which seems more in-depth than my old setup but run into issues while accessing the hosted web service, it is failing to load with a 522 error, the connection if timing out before a response I think?

I have not got any further in the guide than part 5, step 10. Accessing from outside of my network as this is not possible so far.

I have a static WAN IP.. in cloudflare a have [A record *.example.com > Static IP]

I have double checked all the settings in this tutorial and after some googling i came across a reddit post, suggesting they fixed the 522 error in opnsense because HAProxy wasn't listening on port 80 during the HTTPtoHTTPS redirect. Is there a way I can diagnose this issue and trace the route somehow.

Lastly

ACME do not show any error in the log files. 

Code: [Select]

2022-04-13T18:53:42 php AcmeClient: running automation (configd): Restart HAProxy
2022-04-13T18:53:42 php AcmeClient: running automations for certificate: *.example.com
2022-04-13T18:53:42 opnsense AcmeClient: updated ACME X.509 certificate: *.example.com
2022-04-13T18:53:42 opnsense AcmeClient: successfully issued/renewed certificate: *.example.com
2022-04-13T18:51:27 opnsense AcmeClient: using challenge type: CloudFlare_DNS-01
2022-04-13T18:51:27 opnsense AcmeClient: account is registered: example.com
2022-04-13T18:51:27 opnsense AcmeClient: using CA: letsencrypt
2022-04-13T18:51:27 opnsense AcmeClient: issue certificate: *.example.com

HAProxy has no errors in the log file either

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62565b172acae6.05588153 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62565b172acae6.05588153

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62565eb5d0ff12.02152772.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62565c00b116b3.27816426.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: MineOS_backend ()
backend MineOS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server MineOS_server 192.168.1.103:8443 ssl verify none


[hopey]

@TheHellSite: Thanks for your tutorial, it helped me to understand things a bit better!

I just had this problem: https://forum.opnsense.org/index.php?topic=27903.new#new. That's why I was reading your thread.

 What was strange to me, that you need to change the port of your opensense web interface! That's the point were I thought there might be a better solution 😬 I was reading and found this tutorial: https://schulnetzkonzept.de/opnsense. This guy just used a virtual IP, like you did. But instead of pointing to the lookup adress he just introduced a new adress and let haproxy listen on. And so did I - and it worked like charm! (detailes in the upper linked problem)

Maybe you wanna add it to your tutorial too?!

Have a nice day!

[pottproll]

Thanks for the tutorial! Its working great so far! I have one question: in my ACME Client log it says after renewal: php[2613]   AcmeClient: automation not supported: restart_haproxy
Is this just me or maybe because i didnt check HA-Proxy Integration?

Edit: never mind, worked perfectly the next renewal.

[Bunch]

I usually trace the session in Services: HAProxy: Statistics, Counters
When saving HAProxy setting, it will reset the session stat.
So, we can check the session stop at which server if we try to access immediately after a reset.
For example, I expect a session goes through
1. Frontend: TCP_front
2. Frontend: SSL_front
3. Backend: opn_back (usually don't have problem if you type correctly)
4. Server: opn

If all of them have session counts, but you still failed to access the website.
That means, it should be problem between browser and server, but you can access to the page when you type the server IP and port directly. Then, there might be issue about your ciphers, certs, OCSP settings, etc. You might find handshake error in server log too.

If it stop at 2. That means, your haproxy is not recognizing SNI correctly. Check your map file, or you can try to create condition with "SNI TLS extension matches (locally deciphered)" and your full SNI (the.domain.com), then create a rule to "Use specified Backend Pool" when condition matches.
Removing all rules and set default backend to test server first can also be a choice. (At least you will know rather it fails only in SNI part or more parts suffers)

If it stop at 1. There might be issue with the VIP again. (The bug similar to this)

If no session count. The listener is not working. TCP_frontend have wrong Listen Addresses. If you try it from WAN, check your firewall rules too, or maybe DNS record issue.

[keyboardDabbler]

Okay i can now access my webservices but in doing so a missed out the Virtual IP step.
After thinking about my issue some more I am listening HTTP & HTTPS traffic on 192.168.64.1 which I think is when the time out happens.

So I;
Service > HAProxy > Settings > Real Servers > SSL_server: changed FQDN or IP, from 192.168.64.1 to 192.168.1.1

Service > HAProxy > Settings > Virtual Services > 1_HTTPS_frontend: changed Listen Addresses, from 192.168.64.1:443 to 192.168.1.1:443

Service > HAProxy > Settings > Virtual Services > 1_HTTP_frontend: changed Listen Addresses, from 192.168.64.1:80 to 192.168.1.1:80

Now it is all working, What did I do wrong in setting up the Virtual IP I wonder.

0_SNI_frontend > Listen Addresses:0.0.0.0:80, 0.0.0.0:443
should this need to be the Virtual IP as opnsense runs on 192.168.1.1

^^fyi thankyou for the tips on tracing

[Bunch]

0_SNI_frontend > Listen Addresses:0.0.0.0:80, 0.0.0.0:443
should this need to be the Virtual IP as opnsense runs on 192.168.1.1

^^fyi thankyou for the tips on tracing

For this question, lets clear the package path that OP wants to do first.
Assume the following IP config:
WAN IP: 1.2.3.4
Firewall IP: 192.168.1.1
VIP: 192.168.64.1
Server IP:port and SNI: 192.168.1.2:80, the.website.com

A browser try to access https://the.website.com from internet.
It asked system to resolve from DNS server: the.website.com. DNS server replies it is 1.2.3.4
Browser try to access 1.2.3.4 with port 443, sending TLS package with SNI=the.website.com

Since haproxy SNI_frontend is listening to 0.0.0.0:443, that means it is listening to port 443 that all IP can represent the firewall. In this case, it is 1.2.3.4:443 and 192.168.1.1:443

SNI_frontend catches the TLS package in 1.2.3.4:443 and passes to SSL_backend(VIP) without changing port

Since SSL_frontend is listening to 192.168.64.1:443, it takes the TLS package and knows that it try to access 192.168.1.2:80. SSL_frontend communicate to the browser, exchanging the SSL cert and keys according to ciphers.

Session to webserver_backend start, SSL_frontend redirect remaining packages to webserver_backend.

I don't know why your VIP won't work, maybe missing opnsense update, misconfig of VIP or another new bug. (You can try to ping the VIP in LAN to check rather it reply. If the VIP is working normally, it should reply)
And SNI frontend wont use VIP unless you use hopey's method (He still need to add NAT rule to redirect packages to VIP as I mentioned in another thread)

[keyboardDabbler]

You guys are awesome, really appreciate you explaining the process. It is easy to follow along to a guide but to understand what is happening makes it that much easier down the line.

After pinging the VIP 192.168.64.1, it was timing out.

Checked the setting and all is correct to the tutorial.
Decided to changed the submask from 32 to 24.
Then I was able to ping the VIP and access my web services.
Reset the submask back to 32 and i am still able to ping the VIP and web services working with HTTP & HTTPS listening on the VIP. Very strange but it seems resolved.

[Bunch]

The issue is really similar to this one
I guess the issue should be solved in 22.1.4.
I run the patch in 22.1.3, so I don't know rather it is really fixed. But I don't have such problem anymore (currently 22.1.5)

If your are in 22.1.4 or 22.1.5, having similar issue. But general log don't have something like

Code: [Select]

/firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '192.168.64.1' -alias' failed to executePlease create another thread under 22.1 Production Series

If you have exactly the same log and you are in 22.1.4 or 22.1.5. Please try to reply to the thread