Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[TheHellSite]

Currently there is no service running on the domain name. However, when I now try to access my web server via both lan and wan I kept getting error 503 service not available.

Well, if there is no service running, then HAProxy will spit out the 503 message because it can not reach the service.  ???

Code Select Expand

# Backend: cloud_backend ()
backend cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server cloud_server 192.168.7.5:80 ssl verify required ca-file /etc/ssl/cert.pem

Code Select Expand

    server cloud_server 192.168.7.5:80 ssl verify required ca-file /etc/ssl/cert.pem
Are you really uploading the self-signed cert of the service to the OPNsense and checking it with HAProxy?
Otherwise that line is wrong.

A second question I have, single post above you talk about "You have to put the OPNsense LAN IP in the DNS overide. Not the IP of the service." I am confused about this piece, is it possible to explain a little more about this.

It is very well explained in part 6 of the post. Nothing I could really add here.
I also provided documentation links that explain the solutions.
But to maybe explain your question.
If you put the service IP in the DNS override, the client will connect to the service and therefore WON'T be able to use the Let's Encrypt - HAProxy cert.
If you put the OPNsense LAN IP in the DNS override, the client will first connect to HAProxy using a valid cert and then HAProxy talks to the client using either no or the self-signed cert of the service.

I hope you understand that I don't have the time to teach everyone how something works.
Thanks to the internet and search machines you should be able to find that out yourself.

[underwriter]

Thank you much for this amazing tutorial. I have referred a few people from reddit to this.

Questions I hope someone can help me with: If I have xdomain.com, xczxdomain.com and ltsdomain.com;

• does this support multi domain usage?
  can I use this tutorial to assign a particular domain for a given service?
  do I need to recreate the whole entries for each or at which point do I make the adjustment?

[TheHellSite]

Thank you much for this amazing tutorial. I have referred a few people from reddit to this.

Questions I hope someone can help me with: If I have xdomain.com, xczxdomain.com and ltsdomain.com;

• does this support multi domain usage?
  can I use this tutorial to assign a particular domain for a given service?
  do I need to recreate the whole entries for each or at which point do I make the adjustment?

Of course it does! Just make sure the domains point to any public IP of your OPNsense.

You will have to make three changes to the setup.

1. Let's Encrypt: Here you will have to add one certificate for each domain.
If their DNS Zones are managed at different domain registrars you will also have to create the corresponding DNS-01 challenges for each registrar or move their DNS Zones to deSEC (Managed DNS).

2. HAProxy HTTPS Frontend: Add the newly created certificates for each individual domain.

3. HAProxy Public Subdomain Map File: Change the map file content from f.e. "plex PLEX_backend" to "plex.xdomain.com PLEX_backend", "cloud.xczxdomain.com CLOUD_backend" and so on. This way HAProxy can map each subdomain to the correct domain and backend.

[underwriter]

Thank you so much. I appreciate your support.

[cookiemonster]

I was going to open a separate thread to ask for help but I think I might try and see if it is a legitimate variation to the tutorial.
First things first. My setup is exactly as per the tutorial.
I am now investigating the use of an application that uses http transport on port 80 or a custom port but without TLS. The encryption instead is done on the app layers on top. It's called rport.
I'm trying custom port 5000. Looking for the last 10 or so pages of the thread I can only see something similar in Bunch's input "Reply #171 on: February 20, 2022, 05:25:18 pm ". Not quite the same.
I've tried creating conditions matching on the hostname i.e. my subdomain part of mysubdomain.mydomain.dedyn.io and matching on http traffic. Then tried the rule on the SNI frontend, the http frontend and  https frontends. Essentially all frontends trying to make the exception there but in all cases after the sni, the http to https rule gets evaluated first, defeating any exception I've tried.

If it's not too much a deviation, could I have a suggestion on how to approach it? In sum, I'm looking for a way to route my http custom port to a back end as an exception in this Tutorial setup.

My config (you'll notice my port 5000 already in the SNI and the real server listening on that port).

Code Select Expand

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    1
    hard-stop-after             60s
    no strict-limits
    maxconn                     10
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 10
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listens on 80, 443, 853, 5000)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:853 name 0.0.0.0:853
    bind 0.0.0.0:5000 name 0.0.0.0:5000
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (listening on 192.168.5.100:80 i.e. http only)
frontend 1_HTTP_frontend
    bind 192.168.5.100:80 name 192.168.5.100:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_619439805021f2.97978352 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_619439805021f2.97978352

# Frontend: 1_HTTPS_frontend (Listening on 192.168.5.100:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.5.100:443 name 192.168.5.100:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61952b9d47d700.25962675.certlist
    bind 192.168.5.100:5000 name 192.168.5.100:5000 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61952b9d47d700.25962675.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Frontend: 1_TCP_frontend (Listening on 192.168.5.100:853)
frontend 1_TCP_frontend
    bind 192.168.5.100:853 name 192.168.5.100:853 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61dc51606078d9.11258474.certlist
    mode tcp
    default_backend nginx_backend-tcp
    # tuning options
    timeout client 15m

    # logging options
    option tcplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.5.100 send-proxy-v2 check-send-proxy

# Backend: nginx_backend-tcp ()
backend nginx_backend-tcp
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server nginx_2 192.168.5.1:8054 resolve-prefer ipv4 send-proxy check-send-proxy

# Backend: bastion_backend (bastion_backend)
backend bastion_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server bastion-1 192.168.5.157:5000

[authelia]

I am trying to follow the instructions to enable HAProxy for internal domains. However, I can't seem to get the frontend listener for the virtual ip to work. Service binding is disabled for the virtual ip.

When the frontend listener for the virtual ip is enabled:

1. haproxy cannot start (when webgui is running).
2. webgui cannot start (when haproxy is running).

I have tried various things such as assigning the virtual ip from a brand new subnet etc. However the frontend listener for virtual ip seems to conflict with lighttpd no matter what I do. The only way I can get both services to start is to remove the virtual ip from /var/etc/lighty-webConfigurator.conf.

Virtual IP in LAN Subnet (192.168.1.0/24)

Code Select Expand


root@OPNsense:~ # sockstat -4 -l | grep lighttpd
root     lighttpd   28364 6  tcp4   192.168.1.65:443      *:*
root     lighttpd   28364 8  tcp4   192.168.1.1:443       *:*
root     lighttpd   28364 10 tcp4   192.168.1.65:80       *:*
root     lighttpd   28364 12 tcp4   192.168.1.1:80        *:*
root     sshd       84263 5  tcp4   192.168.1.1:22        *:*

root@OPNsense:~ # /usr/local/etc/rc.d/haproxy start
Starting haproxy.
[ALERT]    (2036) : Starting frontend 1_HTTP_frontend: cannot bind socket (Can't assign requested address) [192.168.1.65:80]
[ALERT]    (2036) : Starting frontend 1_HTTPS_frontend: cannot bind socket (Can't assign requested address) [192.168.1.65:443]
[ALERT]    (2036) : [/usr/local/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.
/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy


Virtual IP in Brand New Subnet (192.168.10.0/32)

Code Select Expand


root@OPNsense:~ # /usr/local/etc/rc.restart_webgui
Starting web GUI...done.
Generating RRD graphs...done.

root@OPNsense:~ # sockstat -4 -l | grep lighttpd
root     lighttpd   64654 6  tcp4   192.168.10.65:443     *:*
root     lighttpd   64654 8  tcp4   192.168.1.1:443       *:*
root     lighttpd   64654 10 tcp4   192.168.10.65:80      *:*
root     lighttpd   64654 12 tcp4   192.168.1.1:80        *:*
root     sshd       84263 5  tcp4   192.168.1.1:22        *:*

root@OPNsense:~ # /usr/local/etc/rc.d/haproxy start
Starting haproxy.
[ALERT]    (18033) : Starting frontend 1_HTTP_frontend: cannot bind socket (Address already in use) [192.168.10.65:80]
[ALERT]    (18033) : Starting frontend 1_HTTPS_frontend: cannot bind socket (Address already in use) [192.168.10.65:443]
[ALERT]    (18033) : [/usr/local/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.

[TheHellSite]

I am trying to follow the instructions to enable HAProxy for internal domains. However, I can't seem to get the frontend listener for the virtual ip to work. Service binding is disabled for the virtual ip.

When the frontend listener for the virtual ip is enabled:

1. haproxy cannot start (when webgui is running).
2. webgui cannot start (when haproxy is running).

Part 4 - Step 1.

If you would have followed the tutorial STEP BY STEP you wouldn't have any issues... Just stick to the tutorial and don't skip a single step.

[TheHellSite]

Then tried the rule on the SNI frontend, the http frontend and  https frontends. Essentially all frontends trying to make the exception there but in all cases after the sni, the http to https rule gets evaluated first, defeating any exception I've tried.

If it's not too much a deviation, could I have a suggestion on how to approach it? In sum, I'm looking for a way to route my http custom port to a back end as an exception in this Tutorial setup.

The order of the rules is important! Make sure that all "http-redirect-to-backend" rules are placed BEFORE the HTTPtoHTTPS rule on the HTTP_frontend.

[boredpanda]

Hello! I've used this guide a while back and my self-hosted services are running rock solid. There hasn't been any problems at all with HAProxy - so thank you very much for this guide! I haven't exposed any of my services publicly and mainly use HAProxy to be able to use FQDN at home, and when I VPN in using my phone or personal laptop.

However, there is an instance where it would be very nice to be able to white-list one (or a couple) of specific IPs, so that I could access my services at home from my office. I am not able to install software at the office, and there are other restrictions preventing me from using a VPN.

How would I go about white-listing a single IP, allowing access to some of my internal services? Please let me know if my question is out-of-scope for this tutorial and I'll ask elsewhere. ;)

[TheHellSite]

However, there is an instance where it would be very nice to be able to white-list one (or a couple) of specific IPs, so that I could access my services at home from my office. I am not able to install software at the office, and there are other restrictions preventing me from using a VPN.

This is a very easy task, given that the IPs are static!

I don't know how you restricted local access but if you followed my tutorial you will just have to do this.
Take a look at part 7 of the tutorial.

Create the public subdomains map file, create a condition containing all the whitelist public IPs and create the corresponding redirect rule just as I did with the local access subdomains map file.

[TheEPOCH]

Hello,

my previous question is canceled. I build up my whole OPNsense from scratch and now the Tutorial worked very fine.

Thanks for writing this!

[cookiemonster]

Then tried the rule on the SNI frontend, the http frontend and  https frontends. Essentially all frontends trying to make the exception there but in all cases after the sni, the http to https rule gets evaluated first, defeating any exception I've tried.

If it's not too much a deviation, could I have a suggestion on how to approach it? In sum, I'm looking for a way to route my http custom port to a back end as an exception in this Tutorial setup.

The order of the rules is important! Make sure that all "http-redirect-to-backend" rules are placed BEFORE the HTTPtoHTTPS rule on the HTTP_frontend.

Thank you. Unfortunately I haven't been able to do this. The exact warning is:

Code Select Expand

[WARNING] (96704) : parsing [/usr/local/etc/haproxy.conf.staging:74] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.
Warnings were found.
Configuration file is valid
Any other ideas are welcome :)

[cookiemonster]

Then tried the rule on the SNI frontend, the http frontend and  https frontends. Essentially all frontends trying to make the exception there but in all cases after the sni, the http to https rule gets evaluated first, defeating any exception I've tried.

If it's not too much a deviation, could I have a suggestion on how to approach it? In sum, I'm looking for a way to route my http custom port to a back end as an exception in this Tutorial setup.

The order of the rules is important! Make sure that all "http-redirect-to-backend" rules are placed BEFORE the HTTPtoHTTPS rule on the HTTP_frontend.

Thank you. Unfortunately I haven't been able to do this. The exact warning is:

Code Select Expand

[WARNING] (96704) : parsing [/usr/local/etc/haproxy.conf.staging:74] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.
Warnings were found.
Configuration file is valid
Any other ideas are welcome :)

I've found what I think is a workaround with the service in question, leaving the haproxy setup still as per this tutorial's. Thanks for the earlier suggestion.

[brooklynmind]

I've been following this wonderfully crafted tutorial, so "THANK YOU" to the op for this.
Question (I know this might outside the scope of this tutorial):

• If I want HAProxy to handle *.my1stdomain.xyz which would be for specific services (already have this working flawlessly),
  but I would like to forward *.my2nddomain.xyz to nginx proxy manager running on docker so that nginx proxy manager will be used to manage that.

Is that something this setting can help to implement?

[TheHellSite]

I've been following this wonderfully crafted tutorial, so "THANK YOU" to the op for this.
Question (I know this might outside the scope of this tutorial):

• If I want HAProxy to handle *.my1stdomain.xyz which would be for specific services (already have this working flawlessly),
  but I would like to forward *.my2nddomain.xyz to nginx proxy manager running on docker so that nginx proxy manager will be used to manage that.

Is that something this setting can help to implement?

This has been answered 12 messages back.
https://forum.opnsense.org/index.php?topic=23339.msg143886#msg143886