Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[TheHellSite]

I am not sure this was mentioned before but https://desec.io no longer new registrations for DynDNS.
For the German speaking audience I can highly recommend https://ipv64.net/
Many texts on the website are English, but someone not speaking German might have problems understanding everything

This is not entirely true, but I am already aware of that.
deSec had a "bot registration attack" and therefore paused all new domain registrations temporarly.

https://talk.desec.io/t/dyndns-service-down/823/3?u=thehellsite

[bunchofreeds]

Hi all,

I'm trying to get my internally hosted services to report the originating client IP when going through a proxy chain starting with Cloudflare then to HAproxy. Currently HAproxy logs shows the local CloudFlare CDN address.

Cloudflare is setup to proxy and is Full (Strict) meaning I'm using the Cloudflare origin cert offloaded at HAproxy

I've found that cloudflare do collect the Client IP within cf-connecting-ip
https://developers.cloudflare.com/support/troubleshooting/restoring-visitor-ips/restoring-original-visitor-ips/

And I have found this post that helps someone with pfSense to do what I want
https://forum.netgate.com/topic/176777/haproxy-cloudflare-restoring-original-ip/3

What I'm not sure about is how (if possible) to get HAproxy to reference the cloudflare IP address list to know what sessions to insert the cf-connecting-ip into x-forwarded-for
Ideally this is in the form of some alias or map that dynamically checks https://www.cloudflare.com/ips-v4

Thanks for any help with this, also it's not urgent at all and just for my home setup and for fun really.

[stefan21]

OPNsense is up-to-date -->

OPNsense 24.1.2_1-amd64
FreeBSD 13.2-RELEASE-p10
OpenSSL 3.0.13

Firefox: 121.0 (64-bit), archlinux (doesn't matter, latest update on windows brings up the same issue)

DEFAULT in FF: security.ssl.enable_ocsp_stapling = true

--> leads to no access on any pages with certs following the tutorial. At least if pages are secured to local access only. I assume, same error for public access.

Changing the default in FF to false gives access back.

Question: anyone else with this experiance? Something wrong in my config? Did I overlook something? I tried to follow exactly the tutorial (not using wild card certs). Did work before the last update of the certs... OCSP-server down? If so, does OCSP making sense for servers meant for internal access only? Not at all for me. Even for public access - if an OCSP server is blocked or down, no access to a server would be possible. No control to third party (man in the middle) confirm. Does this really make sense?

Opinions welcome.

regards,
stefan

[johnmcallister]

Just chiming in here --

Thanks very much doing all the work on this How-To, OP, and for keeping it updated, etc.
I successfully implemented it in my modest OPNsense instances/networks, before realizing that for small networks where there may never be more than perhaps 1 to 3 people logging in to a given OPNsense instance, in fact it's far more secure to simply shut off all HTTP listening on external network ports, and instead use SSH tunnels / port redirects, i.e. ssh -L 9450:localhost:80 my.opnsense.host to connect directly to the opnSense instance and access the webgui that way.  Then it doesn't matter at all whether HTTPS is active as the entire connection takes place inside the highly-secured SSH network connection.  With SSL tunnels there is no need for a webgui process to be listening anywhere except localhost:80.

It avoids the major overhead and ongoing maintenance complexity of getting a public/real SSL certificate issued, emplaced, and dealing with ongoing renewals, etc.

[cookiemonster]

Just chiming in here --

Thanks very much doing all the work on this How-To, OP, and for keeping it updated, etc.
I successfully implemented it in my modest OPNsense instances/networks, before realizing that for small networks where there may never be more than perhaps 1 to 3 people logging in to a given OPNsense instance, in fact it's far more secure to simply shut off all HTTP listening on external network ports, and instead use SSH tunnels / port redirects, i.e. ssh -L 9450:localhost:80 my.opnsense.host to connect directly to the opnSense instance and access the webgui that way.  Then it doesn't matter at all whether HTTPS is active as the entire connection takes place inside the highly-secured SSH network connection.  With SSL tunnels there is no need for a webgui process to be listening anywhere except localhost:80.

It avoids the major overhead and ongoing maintenance complexity of getting a public/real SSL certificate issued, emplaced, and dealing with ongoing renewals, etc.

The main purpose of the tutorial is not to to access the OPN UI, for which your method makes perfect sense, but instead to reverse proxy services that are hosted internally in a LAN.

[TheHellSite]

Just chiming in here --

Thanks very much doing all the work on this How-To, OP, and for keeping it updated, etc.
I successfully implemented it in my modest OPNsense instances/networks, before realizing that for small networks where there may never be more than perhaps 1 to 3 people logging in to a given OPNsense instance, in fact it's far more secure to simply shut off all HTTP listening on external network ports, and instead use SSH tunnels / port redirects, i.e. ssh -L 9450:localhost:80 my.opnsense.host to connect directly to the opnSense instance and access the webgui that way.  Then it doesn't matter at all whether HTTPS is active as the entire connection takes place inside the highly-secured SSH network connection.  With SSL tunnels there is no need for a webgui process to be listening anywhere except localhost:80.

It avoids the major overhead and ongoing maintenance complexity of getting a public/real SSL certificate issued, emplaced, and dealing with ongoing renewals, etc.

The main purpose of the tutorial is not to to access the OPN UI, for which your method makes perfect sense, but instead to reverse proxy services that are hosted internally in a LAN.

Exactly.
To add to this, exposing any administrative web interfaces using a reverse proxy is the worst thing you could possibly do.
Only expose those things using either your SSH tunnel method or VPN tunnels.

The whole point of a reverse proxy is to make web services available using a single port, a nice URL and a valid certificate.

It avoids the major overhead and ongoing maintenance complexity of getting a public/real SSL certificate issued, emplaced, and dealing with ongoing renewals, etc.

We are using ACME to avoid exactly all of that. It is a one time only set and forget process.

I think you didn't really understand the tutorial at all.
Also public certificates, as you can see in my tutorial, are so easy and completely free to obtain nowadays.

[bunchofreeds]

Hi all,

I'm trying to get my internally hosted services to report the originating client IP when going through a proxy chain starting with Cloudflare then to HAproxy. Currently HAproxy logs shows the local CloudFlare CDN address.

Cloudflare is setup to proxy and is Full (Strict) meaning I'm using the Cloudflare origin cert offloaded at HAproxy

I've found that cloudflare do collect the Client IP within cf-connecting-ip
https://developers.cloudflare.com/support/troubleshooting/restoring-visitor-ips/restoring-original-visitor-ips/

And I have found this post that helps someone with pfSense to do what I want
https://forum.netgate.com/topic/176777/haproxy-cloudflare-restoring-original-ip/3

What I'm not sure about is how (if possible) to get HAproxy to reference the cloudflare IP address list to know what sessions to insert the cf-connecting-ip into x-forwarded-for
Ideally this is in the form of some alias or map that dynamically checks https://www.cloudflare.com/ips-v4

Thanks for any help with this, also it's not urgent at all and just for my home setup and for fun really.

Found some answers
https://github.com/home-assistant/core/issues/40421#issuecomment-1667019787

[stefan21]

OPNsense is up-to-date -->

OPNsense 24.1.2_1-amd64
FreeBSD 13.2-RELEASE-p10
OpenSSL 3.0.13

Firefox: 121.0 (64-bit), archlinux (doesn't matter, latest update on windows brings up the same issue)

DEFAULT in FF: security.ssl.enable_ocsp_stapling = true

--> leads to no access on any pages with certs following the tutorial. At least if pages are secured to local access only. I assume, same error for public access.

Changing the default in FF to false gives access back.

Made a few changes:

Did a roll back to

OPNsense 23.7.12_5-amd64
FreeBSD 13.2-RELEASE-p7
OpenSSL 1.1.1w

For all certs I changed to key lenght 2048. OCSP must staple "off". Renewed all certs and deployed to the servers. Now on all clients FF and TB are working again with no errors.

Adding local and public subdomain rules in public services HTTPS_frontend are not working as expected in my setup. Adding in option pass through

acl lan_vpn src 192.168.x.x/24
acl lan_vpn src 10.0.x.x/24
http-request deny if ! lan_vpn

is working. Access from public is not possible - as intended. Maybe it helps someone.

@thehellsite: thx very much for this great tutorial.

regards,
stefan

[ckocank]

Edit: After a restart, every thing works again.
My HAproxy stopped working after the update and I cannot make it work again.
I tried all the latest fixes in the comment and updated part 5 and part 8.
This is my config

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.ocsp-update.mindelay 300
    tune.ssl.ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr libc,last

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening on 0000:443 0000:80)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend ssl_backend

    # logging options

# Frontend: 1_HTTP_frontend (listen on 10.1.2.3:80)
frontend 1_HTTP_frontend
    bind 10.1.2.3:80 name 10.1.2.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: no_ssl_condition
    acl acl_642ff4b1bd6b30.27652312 ssl_fc

    # ACTION: http_to_https_rule
    http-request redirect scheme https code 301 if !acl_642ff4b1bd6b30.27652312

# Frontend: 1_HTTPS_frontend (listen on 10.1.2.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 10.1.2.3:443 name 10.1.2.3:443 accept-proxy ssl curves secp384r1 strict-sni  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/642ffac3a289a1.74357812.certlist
    mode http
    option http-keep-alive
    option forwardfor

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/642ff59e3f1923.99537840.txt)]

# Backend (DISABLED): acme_challenge_backend (Added by ACME Client plugin)


# Backend: ssl_backend (ssl virtual ip backend)
backend ssl_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server ssl_server 10.1.2.3 send-proxy-v2 check-send-proxy

# Backend: test_backend (test backend pool)
backend test_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server test_server 10.1.1.100:81



# statistics are DISABLED

frontend prometheus_exporter
   bind *:8404
   mode http
   http-request use-service prometheus-exporter if { path /metrics }

[JohnnyBeGood]

TheHellSite has been incredibly helpful with both HAProxy and setting up a Postfix SMTP relay for me.
My scenario was a little bit complicated, and although I was able to follow up to a certain point, the complexity of my setup required me to reach out for assistance. Not only was the assistance top-notch, but the pricing was also quite reasonable. I highly recommend reaching out if you need any similar services. Thanks again for the great support!

[Squiggley]

Hi @HellSite

I have followed your tutorial I believe exactly, and I have almost all of it working.
For all of my services I have real servers and matching backends.
The local and public map files are working great.
It redirects http to https just fine.
I get an A+ SSL rating

All of that works great (would have taken me forever without your tutorial)

However I do not seem to be able to access any services on my LAN from a VLAN via the fqdn.

I am more than happy to work things out myself I am really just looking to try and understand how it works a bit better and perhaps get a pointer to which part is not functioning properly.

Can you tell me if these assumptions are correct.
My laptop is on vlan B and all my services are on vlan A.

1) If I can access the services via the IP address successfully then my firewall rules are allowing the correct traffic through.
2) If I do an nslookup or ping from my laptop and I get the opnsense ip address on vlan A then unbound is working correctly as traffic should be sent to haproxy for it to work out where its really going.

So it seems to me that my issue must be haproxy is not giving the correct address back when I enter the fqdn on the laptop

I have intentionally not put my configs in here as after reading all the pages in this thread I see the enormous amount of work you put into this and I would like to understand/work things out myself.

Thanks

[Squiggley]

Disregard, I don't know why but its all working fine now.

Thanks again for the tutorial

[opojomo]

Problem solved.

[WickedBone]

Hello,
  First of all i want to thank everyone in this community for the very helpful comments and descriptions and TheHellSite for the beautiful  tutorial and all the updates to it.

   That being said I'm here because I'm out of ideas and my head is about to explode. So I've followed the tutorial almost 2 years ago now and manged to get everything to work perfectly for my  environment. But 2 days ago i decided to update opnsense from 23.7.something to the newest version 24.4.4 like I've done many times in those past 2 years whenever I remembered to do it. But out of the not so many servers I have behind the haproxy, one didn't work properly and gave me the famous error 503 Service Unavailable No server is available to handle this request. For context this is a very very old server which, due to reasons, works on debian  6 and cannot be upgraded, with self signed openssl cert for its apache website which makes it inaccessible unless you downgrade your browser tls to 1 min. That's why I put this abomination behind my haproxy so the site can be reached more easily. Without any changes (at least I'm not aware of any) to the configuration it just stopped working after the update.

    I guess my question is: did the update change anything to how opnsense is connecting to real servers with ssl enabled (because the connection works if its disabled, but then the site configuration becomes a problem) for old sites and servers and in general where do i look to fix the issue (apache on the server, opnsense, haproxy)?  :'(

[oLeDfrEeZe]

Hello,

first - i will thank you so much for the tutorial!

Now my question:
I followed the instructions up to point 5 and configured everything exactly as described.
But now on Part 6: "If you try to access your URL 'your_service.your_subdomain.dedyn.io' from a device on your internal network, it should fail." but It doesn't fail... it works  ???