Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[keyboardDabbler]

I will make my way over to the other thread you linked then as it is a similar issue.

I am on 22.1.5 and after checking the general log I also have the below

Code: [Select]

2022-04-14T16:42:58 Error opnsense /firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '192.168.64.1' -alias' failed to execute


--EDIT--
After reading the linked thread, a patch was applied in 22.1.4 but you also need to untick "Allow service binding".
I can confirm this has fixed the issue. Thanks again for point me in the right direction.

---EDIT 2---
I am now interested in the last part of the tutorial, I did not have anything like this setup on my previous pfsense setup so it will be a great addition. It would make for remembering local ip/ ports easier when navigating to locally hosted services.
I have followed the steps to make these subdomains accessible only from my local network but getting a "503 service unavailable".

I think its not reading the local map file correctly. I followed the previious steps for tracing and can see the counters go up when I try to access local.website.com but nothing is being passed to the prism_backend.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62565b172acae6.05588153 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62565b172acae6.05588153

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62565eb5d0ff12.02152772.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6257dfacde7e16.43417850 src_is_local

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6257d684d34507.32920094.txt)] if acl_6257dfacde7e16.43417850
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62565c00b116b3.27816426.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: MineOS_backend ()
backend MineOS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server MineOS_server 192.168.1.103:8443 ssl verify none

# Backend: Prism_backend ()
backend Prism_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Prism_server 192.168.1.103:2342

PUBLIC_SUBDOMAINS_map

Code: [Select]

# public access subdomains
mineos MineOS_backend
LOCAL_SUBDOMAINS_map

Code: [Select]

# local access subdomains
prism Prism_backend

# public access subdomains
mineos MineOS_backend

just to confirm;
mineos.website.com > works locally and externally
prism.website.com > 503 error locally and externally

[Bunch]

mineos.website.com > works locally and externally
prism.website.com > 503 error locally and externally

Sorry, I haven't read the error 503.

Try not to use "src_is_local"
I remember that there is issue with this function, especially under proxy protocol. (It's hyproxy issue since 1.7?)
Try to use "Source IP matches a specific IP" instead

[keyboardDabbler]

Try to use "Source IP matches a specific IP" instead

I actually set this first time around which gave the same output. So I then decided to try a broader range with "IP is local".

I tried again "Source IP matches a specific IP" of the specific subnet again with no luck. have even tried isolating it to the IP of the client I am currently using.

the counter still is not being passed on the final stage of the route. After checking the HAProxy log file, as I navigate to local.website.com i am receiving a external request from cloudflare. I suspect the issue lies within the browser redirecting the request out or HAproxy not grabbing it locally before it goes out??

[Bunch]

Strange, there shouldn't be something related to cloudflare
Maybe try to create condition: "SNI TLS extension matches (locally deciphered)" with your full sni, prism.website.com
Create another rule that
Use specified Backend Pool: Prism_backend
When
Source IP matches a specific IP...
And
SNI TLS extension matches (locally deciphered)

Use this rule to replace the one with local map file first

[Bothson]

Hello and thank you for this tutorial.
It helped me alot.

My services are available from the outside.
But from inside they are not accessible.

In the logs i see:

Code: [Select]

Informational haproxy 10.10.10.206:63264 [19/Apr/2022:17:26:27.483] 1_HTTPS_frontend/10.12.0.1:443: Received something which does not look like a PROXY protocol header
I already checked the ciphers. They seem to be ok.
Does anybody know where i can search for the problem?

My config:

Code: [Select]

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 10.12.0.1:80 name 10.12.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_60d1a0c1b278f7.63252237 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_60d1a0c1b278f7.63252237

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 10.12.0.1:443 name 10.12.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6256591773a972.14047672.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/625655d89e4274.43878203.txt)]

# Backend: bitwarden_backend ()
backend bitwarden_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server bitwarden_host 10.10.10.11:8080

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 10.12.0.1 send-proxy-v2 check-send-proxy
Best
Mathias

[TheHellSite]

My services are available from the outside.
But from inside they are not accessible.

Well, there you got the point of error.
You probably configured the wrong IP in your DNS overwrites.

What is your OPNsense LAN IP, what is the DNS Overwrite IP and what is the virtual IP of the "HAProxy SSL Server"?

[Bothson]

Well, there you got the point of error.
You probably configured the wrong IP in your DNS overwrites.

What is your OPNsense LAN IP, what is the DNS Overwrite IP and what is the virtual IP of the "HAProxy SSL Server"?

My LAN IP is 10.10.10.1/24
My DNS Override points to    10.12.0.1
Which is also my Virtual IP (Loopback).

I do not get the point.

Edit: Ok, changed the Override IP to my LAN IP (10.10.10.1). Now it works.
But to be honest, i do not understand why.

[TheHellSite]

Edit: Ok, changed the Override IP to my LAN IP (10.10.10.1). Now it works.
But to be honest, i do not understand why.

You error explains why!

Code: [Select]

Informational haproxy 10.10.10.206:63264 [19/Apr/2022:17:26:27.483] 1_HTTPS_frontend/10.12.0.1:443: Received something which does not look like a PROXY protocol header
The HTTPS_frontend expects that all data sent to it has the "proxy protocol header".
Since you pointed your internal requests directly to your HTTPS_frontend (HAProxy_VIP) instead of your SNI_frontend (any of the real local IPs of your OPNsense) the data didn't get the PROXY protocol header attached by the SSL_backend.

[Bothson]

Edit: Ok, changed the Override IP to my LAN IP (10.10.10.1). Now it works.
But to be honest, i do not understand why.

You error explains why!

Code: [Select]

Informational haproxy 10.10.10.206:63264 [19/Apr/2022:17:26:27.483] 1_HTTPS_frontend/10.12.0.1:443: Received something which does not look like a PROXY protocol header
The HTTPS_frontend expects that all data sent to it has the "proxy protocol header".
Since you pointed your internal requests directly to your HTTPS_frontend (HAProxy_VIP) instead of your SNI_frontend (any of the real local IPs of your OPNsense) the data didn't get the PROXY protocol header attached by the SSL_backend.

Ah ok, i get the point.
Thank you very much for the explanation.

[keyboardDabbler]

I am not sure if this is the correct way to achieve multiple domains pointing to different backends but it seems to be working for me.
At first I ran into a issue were all domains could access the same subdomain, this is when I realized I just needed some extra conditions.

Here are the steps to achieve; service.example.com & service1.example1.com

Services --> ACME Client --> Certificates
Add the certificate for your extra domains and forcefully issue your certificate

Services --> HAProxy --> Settings --> Advanced --> Map Files
Here we will create a new map file for each domain "PUBLIC_SUBDOMAINS_map-example" & "PUBLIC_SUBDOMAINS_map-example1"

Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
Add a new condition for each domain that you have added.
Name = "example1_condition"
Description = "Traffic matches example1.com"
Condition type = "host contains"
Host Contains = "example1.com"

Services --> HAProxy --> Settings --> Rules & Checks --> Rules
Add a map rule for each domain while also selecting our newly created "example1_condition"
Name = "PUBLIC_SUBDOMAINS_map-rule-example1"
Select conditions = "example1_condition"
Map file = "PUBLIC_SUBDOMAINS_map-example1"

Services --> HAProxy --> Settings --> Virtual Services --> Public Services
Finally we edit our "1_HTTPS_frontend"
Add all extra domains in the "Certificates" input.
Scroll down and add each map-rule-example1 in the "Select Rules" input

[TheHellSite]

I am not sure if this is the correct way to achieve multiple domains pointing to different backends but it seems to be working for me.
At first I ran into a issue were all domains could access the same subdomain, this is when I realized I just needed some extra conditions.

Here are the steps to achieve; service.example.com & service1.example1.com

Services --> ACME Client --> Certificates
Add the certificate for your extra domains and forcefully issue your certificate

Services --> HAProxy --> Settings --> Virtual Services --> Public Services
Finally we edit our "1_HTTPS_frontend"
Add all extra domains in the "Certificates" input.

Just the steps above are necessary and the following step.

Then edit the "PUBLIC_SUBDOMAINS_map" and change the subdomain entries to full FQDNs.

Code: [Select]

old
===
nas NAS_backend

new
===
nas.domain1.com NAS_1_backend
nas.domain2.com NAS_2_backend
Of course you could also create dedicated map-files and rules for each domain, but in terms of functionality it is not necessary.

[keyboardDabbler]

Although it has the same outcome, The steps you provided using only 1 map file is a lot cleaner and easier to follow.

Thanks again for pointing this out.

[christian_domes]

@theHellSite
Hello
When I overide the dns Server will the haproxy still be used or not?

[Bunch]

@theHellSite
Hello
When I overide the dns Server will the haproxy still be used or not?

It depends on how you override the dns record.
Assume that you set your SNI frontend with 0.0.0.0:80 and 0.0.0.0:443

For example, you added a DNS record in Cloudflare "abc.domain.com" pointing to your WAN IP, and your tested it and found HAProxy working both locally and externally.
Then you removed the DNS record from Cloudflare, and add one in unbounded "abc.domain.com" pointing to your OpnSense IP (either LAN or WAN, doesn't metter)

Then your HAProxy should work locally, but failed externally.
*If you want to override DNS record in unbound, always point to SNI frontend. If you set it to either HTTP frontend or HTTPS frontend, it will fail.

[TheHellSite]

I just updated the tutorial with a very important change to the DynDNS part. It therefore only affects users with a dynamic WAN IP.

• 20220604
  
  • Updated the DynDNS part to use the newer "Dynamic DNS Client" (os-ddclient) plugin.
    The previous "Dynamic DNS Support" (os-dyndns) plugin will be removed with the release of OPNsense 22.7.x.