Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[Lip90]

Still the Same :)

Code Select Expand

2021-09-14T21:21:05 haproxy[3256] 80.187.80.8:10670 [14/Sep/2021:21:21:05.818] 0_SNI_frontend SSL_backend/SSL_server 1/0/37 0 -- 1/1/0/0/0 0/0
2021-09-14T21:21:05 haproxy[3256] 80.187.80.8:10670 [14/Sep/2021:21:21:05.818] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-14T21:21:01 haproxy[3256] 80.187.80.8:10495 [14/Sep/2021:21:21:01.658] 0_SNI_frontend SSL_backend/SSL_server 1/0/36 0 -- 1/1/0/0/0 0/0
2021-09-14T21:21:01 haproxy[3256] 80.187.80.8:10495 [14/Sep/2021:21:21:01.658] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure

[TheHellSite]

For which try?
1st or second?

If you remove the VPN_backend as default backend and place the novpn_rule in the frontend. Can you still access your seafile Server?
It should be accessible. Is it?
I want to test the sni functionality with this.

[Lip90]

Your first question:
For both.

The second:
Yes i can access the Seafile server with any config. The haproxy give every request to the ssl backend.


Gesendet von iPhone mit Tapatalk

[Lip90]

It looks like that sni don't work by OpenVPN


Gesendet von iPhone mit Tapatalk

[nullex]

My map File currently looks like:

Code Select Expand

sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
Ombi Ombi_backend
Tautulli Tautulli_backend

Any advice as to what I'm doing wrong? Why would some services work and others don't when they're using the same exact config?

What's also funny is that if I reconfigure the port and IP on the one of the Real Servers that works (For example, Plex_Server has IP 192.168.1.159:32400, Ombi_server has 192.168.1.159:5055, and I replace Plex_Server port to 5055 and go to Plexi.DOMAIN.com, Ombi pops up like it should. I've cloned new Real servers and Backend Pools while updating the PUBLIC_SUBDOMAINS_map  from the working Plex one, but still no go for ALL of my services.

I've also discovered that if I modify HAProxy Rules & Checks > Rules > Public_subdomains_map-rule > Default backend pool... and change it to a service that DOESN'T work with the map file.. when I hit apply I'm able to access that service on ANY rendition of my domain, as well as the root domain.com address... And if I leave the services that DO work in the map file (Plex), plex.domain.com displays plex as it should, while the rest of the domain is showing the service that doesn't work on it's own.. which further doesn't make any sense.. The map file is working for some services but not others?

Thanks in advance, I'm stumped.

Final edit:

LMFAO I think I figured out why it wasn't working... so apparently for the map file to work you have to have the first part all lowercase, cannot use any uppercase...

So when I changed my map file to:

Code Select Expand

sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
ombi Ombi_backend
tautulli Tautulli_backend


everything started working...

Glad it is working for you know.
My first guess where some misconfigured real servers (ports, ssl, ssl-verify).

BTW: Your map file is exposing your domain name! You should remove it from the forum post.

Also I did a quick scan of your domain using https://dnsdumpster.com. It lists all your subdomains since you created a single "A Record" for each of them. Consider switching to a "Wildcard A Record" in order to hide them!
If an attacker can see what services you are running it makes it easier for them to find an attack surface.

You can then still create individual a records, f.e. www.domain.tld, since the wildcard a record is resolved after all other a records have been resolved.

This is why my tutorial is using a "Wildcard A Record / Subdomain" in the form of "*.domain.tld".

Thanks for catching that, thought I removed all PII but apparently not lol

So I think DNSDumpster is showing those domains because I was previously using dynamic DNS on EACH of those domains and using caddy as a reverse proxy.. which all those WERE direct A records, but since following your guide I switched to wildcard domain so hopefully it should be fixed once DNSDumpster's Database is updated.. I'll keep an eye on it. Thanks again!

[Prismatic]

Hello. First of all, many thanks for your tutorial.... working like a charm, even for an (enlightened)  newbie like me....

Im using a Synology NAS + Docker  with different services. So they are available on same LAN IP adress, but different Ports....

I would llike to setup the "access from internatl network" as on your part 6. I do understand that with my setup I cannot use the unbound split DNS option, as this doesnt work with ports...

So I'm relying on option A "NAT reflexion", but I'm unable to find it, inside the created "HAproxy rule", in Firewall/WAN section..... I do find this option in Firewall/NAT section rules, but we didnt created nothing there....

Could you please help ? and update your tutorial with a screenshot of this ?

Many thanks !

[TheHellSite]

Im using a Synology NAS + Docker  with different services. So they are available on same LAN IP adress, but different Ports....

I would llike to setup the "access from internatl network" as on your part 6. I do understand that with my setup I cannot use the unbound split DNS option, as this doesnt work with ports...

Of course you can use split DNS!
haproxy is handling the port scenario.

Example:
10.1.23.55:4456 = NEXTCLOUD = cloud.yourdomain.tld
10.1.23.55:4457 = PLEX = plex.yourdomain.tld

Read the Split dns part again!
You have to rewrite ALL of your 1st/2nd-level-subdomains with same IP of your OPNsense that HAProxy is listening on, f.e. the lan ip.

[Prismatic]

Thanks for your answer !

Apologies, but I still don't understand something....

So my services IP & port are configured inHAproxy map file. (everything working fine, comming from WAN with certificate)..
.
this server IP for me on LANis 192.168.1.20
only using 1st level subdomains like:

audio.mydomain.tld
photo.mydomain.tld
video.mydomain.tld
etc...

My "LAN" port for opnsense on appliance is 192.168.1.1 and I did used a "virtual IP" as per your tutorial which is 192.168.50.1...

Using "hosts overrides in unbound" (not "domain") should my configuration look like this :

HOST : audio
DOMAIN: mydomain.tld
IP : 192.168.50.1 or 192.168.1.1  or 192.168.1.20 ??

tried all solutions, but I don't know if it works.... a "tracert" shows that I am always hitting my static public IP adress....

sorry for my approximative english and many thanks for your answer.
cheers !

[TheHellSite]

So my services IP & port are configured inHAproxy map file. (everything working fine, comming from WAN with certificate)..
.
this server IP for me on LANis 192.168.1.20
only using 1st level subdomains like:

audio.mydomain.tld
photo.mydomain.tld
video.mydomain.tld
etc...

My "LAN" port for opnsense on appliance is 192.168.1.1 and I did used a "virtual IP" as per your tutorial which is 192.168.50.1...

Using "hosts overrides in unbound" (not "domain") should my configuration look like this :

HOST : audio
DOMAIN: mydomain.tld
IP: 192.168.1.1

IP has to be the lan ip of your opnsense.
Which I very well tell to do in my tutorial.
You should probably read it word by word again!

Also post your HAProxy config export in a code box.

[N0_Klu3]

Hey all,

Is there a way to tie this in with Cloudflare?
I know ACME can use Cloudflare, but can someone add in where I need to do this for wildcards?

Also I installed the plugins and I don't see LetsEncrypt, rather I see ACME Client, and the settings are a bit different.
Any chance you can update the images?

Also I've tried to follow this to the best of my abilities.
I am getting: 503 Service Unavailable
No server is available to handle this request.

If I click on the padlock it is showing my certificate so I think its hitting right.
I'm doing things for:
homeassistant
jellyfin
radarr
sonarr

All of them say 503.

If I set a backend to maintenance I get Safari can't find the server.
So seems to be hitting it correctly.

Ok not sure what changed but now its no dice. Just server cannot be found

[TheHellSite]

Sorry but I can't help with individual domain / dns registrars, this would simply take way to much of my time. You will have to study cloudflares documentation on your own to make it work with them.
Only difference should be the Dynamic DNS and ACME configuration, to get your certificate.
HAProxy settings are the same for every registrar.

The Let's Encrypt plugin has been renamed to ACME Client in one of the recent OPNsense updates. Apart from that nothing else has changed so all pictures are still valid. No need to change them.
I will update the text when I find the time.

Your error: Post your HAProxy export in a code box. Remove sensitive information (Public IP, Domain Name).

[N0_Klu3]

Thank for taking the time to help.

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_Frontend ()
frontend SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTP_Frontend ()
frontend HTTP_Frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: No_SSL_condition
    acl acl_6160768c129757.05678189 req.ssl_ver gt 0

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_6160768c129757.05678189

# Frontend: HTTPS_Frontend ()
frontend HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6160790e8358e2.93807756.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: Public_Subdomain_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61607700a205d2.15896401.txt)]

# Backend: SSL_backend (HAProxy SSL Backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: homeassistant_backend (Home Assistant Backend)
backend homeassistant_backend
    # health check: homeassistant_tcp_check
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server homeassistant 10.0.0.9:8123 check inter 2s port 8123  ssl verify none

# Backend: jellyfin_backend (Jellyfin Backend)
backend jellyfin_backend
    # health check: jellyfin_http_check
    option httpchk
    http-check send meth OPTIONS uri / ver HTTP/1.0
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server jellyfin 10.0.0.10:8090 check inter 2s port 8090  ssl verify none


The only difference I think I made was to add health checks so I can see if service is UP.
It is showing as UP so that looks good.

And my MAP looks like:

Code Select Expand

# public access subdomains
homeassistant homeassistant_backend
jellyfin jellyfin_backend

Oh one other thing to note is I have dual WAN setup.
I had this on pfSense too and HAProxy was working perfectly fine.
I just had all the external connections go through 1 WAN only which I did again here.

[TheHellSite]

Please confirm you are accessing your servers internally using:

http://10.0.0.10:8090/
http://10.0.0.9:8123/
So no https?

Even though it shouldn't matter test the access with health checks disabled.

[N0_Klu3]

Please confirm you are accessing your servers internally using:

http://10.0.0.10:8090/
http://10.0.0.9:8123/
So no https?

Even though it shouldn't matter test the access with health checks disabled.

Correct if I hit those 2 IP's above internally I get the service correctly.
I did have it running with health checks disabled and same result 503 errors

[TheHellSite]

Is opnsense the only router / firewall in your network?
To me it looks like your servers are in a totally different subnet / vlan.

Please give a little info about your local network layout.
I think your opnsense has no access to your server network!