Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[nmiller0113]

Part 7 - Advanced Configuration: local-access-only subdomains
Imagine you have a service that you would like to access / protect using your brand new reverse proxy without making it available on the internet?
Well, HAProxy has got you covered!

• In your OPNsense go to: Services --> HAProxy --> Settings --> Advanced --> Map Files
  Here you need to clone the "PUBLIC_SUBDOMAINS_mapfile", rename it to f.e. "LOCAL_SUBDOMAINS_mapfile" and add all your local-access-only subdomains along with their corresponding backends.
  Keep in mind that the content of your "PUBLIC_SUBDOMAINS_mapfile" also has to be put in the "LOCAL_SUBDOMAINS_mapfile"! I will explain why later.
  
• Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
  Now you need a condition that detects if the source of the request is a local IP or a FQDN.
  You can of course also use the predefined "Source IP is local" condition.
  I am however using only specific subnets since the predefined condition is using the entire RFC1918 IP range, which I don't need!
  
  
  As I just said you can also check for a FQDN.
  But please keep in mind that HAProxy resolves those hostnames to their IPs and then checks them. But the resolving is only done once during the start / restart of HAProxy.
  So if the IP of your FQDN is changing regularly this won't work very well, except if you restart your HAProxy using a cron job like every 24 hours or so.
  
• Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Rules
  Here you need to clone the "PUBLIC_SUBDOMAINS_rule", rename it to f.e. "LOCAL_SUBDOMAINS_rule", select your "LOCAL_SUBDOMAINS_SUBNETS_condition" and select your "LOCAL_SUBDOMAINS_mapfile".
  If you are also using a FQDN condition, like I do, you will need to select both your FQDN and your subnet condition together with the logical "or" operator!
  
• Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Public Services
  The last thing left to do is to place the "LOCAL_SUBDOMAINS_rule" before your in your "HTTPS_frontend".
  
  Attention!
  Remember that I told you to also put the content of your "PUBLIC_SUBDOMAINS_mapfile" in the "LOCAL_SUBDOMAINS_mapfile"?
  This is because HAProxy is processing the rules in the frontends based on the order they appear!
  So if you place your "PUBLIC_SUBDOMAINS_rule" before your "LOCAL_SUBDOMAINS_rule" in the frontend configuration, you won't get access to your local-access-only subdomains.
  Vice versa this will also happen and you will no longer have access to your public subdomains.
  To avoid this you have to also put the content of your "PUBLIC_SUBDOMAINS_mapfile" in the "LOCAL_SUBDOMAINS_mapfile" and place their rules in the correct order.
  
  The correct way of placing both rules is like this.
  
• Done!
  You should now still have access to your public subdomains from any network and also have access to your local-access-only subdomains from the locations you defined.

@TheHellSite

First of all...thank you so so much for this extensive guide! It was awesome and extremely helpful. I got everything working first time without a hitch! I sent ya *some* beer just now! Whatever you can buy with what I sent

I am writing because I saw a typo in section 4 of Part 7 I quoted above.

You wrote:

The last thing left to do is to place the "LOCAL_SUBDOMAINS_rule" before your in your "HTTPS_frontend"

And I think you meant to write:

The last thing left to do is to place the "LOCAL_SUBDOMAINS_rule" before your "PUBLIC_SUBDOMAINS_rule" in your "HTTPS_frontend"

Thanks again for everything!

[TheHellSite]

First of all...thank you so so much for this extensive guide! It was awesome and extremely helpful. I got everything working first time without a hitch! I sent ya *some* beer just now! Whatever you can buy with what I sent

I am writing because I saw a typo in section 4 of Part 7 I quoted above.

You wrote:

The last thing left to do is to place the "LOCAL_SUBDOMAINS_rule" before your in your "HTTPS_frontend"

And I think you meant to write:

The last thing left to do is to place the "LOCAL_SUBDOMAINS_rule" before your "PUBLIC_SUBDOMAINS_rule" in your "HTTPS_frontend"

Thanks again for everything!

Thank you for the beers.
Fixed, thanks for the hint!

[taker]

Hello, I tried with this tutorial to set up HA Proxy. Unfortunately, the HA proxy does not start anymore. It throws this error in the log:

Code: [Select]

Error configd.py [85b23125-6c10-4561-81f5-f28b4ca64c4e] Script action failed with Command '/usr/local/opnsense/scripts/OPNsense/HAProxy/syncCerts.py actions --output bootgrid --page-rows '10' --page '1' --search '' --sort-col '' --sort-dir ''' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 44, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/OPNsense/HAProxy/syncCerts.py actions --output bootgrid --page-rows '10' --page '1' --search '' --sort-col '' --sort-dir ''' returned non-zero exit status 1.
This is my Config

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_64b0212a904331.12997942 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_64b0212a904331.12997942

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Cloud_backend ()
backend Cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Cloud 10.7.0.206:80

# Backend: Webserver_backend ()
backend Webserver_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server webserver 10.7.0.204:80



# statistics are DISABLED
Greetings

[jellelle]

@TheHellSite
Maybe you would also like on how to enable Websockets on your frontend(s) if your incoming clients are looking for such one(s). Websockets are basically used for example for streaming services over web. I was looking for so long on how to resolve my problem for making HAProxy work with Synology's DS Cam Android app which tries to connect from remote to the Synology Surveillance Station NAS behind HAProxy and I finally found out. You have to insert the following on your frontend (where you have to replace <myBackend(Pool)> with your according backend, of course):

Code: [Select]

acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend <myBackend(Pool)> if is_websocket
Here's the link to my original and solved issue:
 https://www.synoforum.com/threads/connecting-synology-ds-cam-android-app-to-synology-surveillance-station-through-opnsense-haproxy-plugin.7969/

Could you tell us how to enable websocket on HAProxy?
Emby and Home Assistant don't work completely without them.

[Steeltitanpro]

Thank you so much for this tutorial.

I do have (hopfully) a quick question and went through 20 some pages to see if its been asked.

I have all my subdomains working perfectly, however how do I set my example.com domain?

I have all my services under service.example.com and want a website at example.com

Im sure its something I am overlooking. Like do I put something in my Map file?

Thank you again

[nikkon]

Hi all,
I'm curious if I can use this method for internal running services (jails on freenas) without exposing them outside.
I can force the DNS override so I resolve them with fqdn from LAN but I can't make HAproxy work and serve the Certificate for them. I already got certificates for all instances in acme (jail1.domain.x, jail2.domain.x)

thank you in advance

[nmiller0113]

Hi all,
I'm curious if I can use this method for internal running services (jails on freenas) without exposing them outside.
I can force the DNS override so I resolve them with fqdn from LAN but I can't make HAproxy work and serve the Certificate for them. I already got certificates for all instances in acme (jail1.domain.x, jail2.domain.x)

thank you in advance

It's all right there in Part 7 of the guide "Advanced Configuration: local-access-only subdomains"

[nikkon]

Thanks

[delicious_potatoe]

Hello,

thank you very much for this great tutorial. At some points I despaired a bit, but that was more due to my lack of attention to read correctly! 

Meanwhile everything is running, inside and outside of the network. However, there is one thing I didn't quite understand:

I didn't create DNS-Split nor NAT-Reflection rule and still I can reach all services locally through my subdomain, which shouldn't work.
The WAN rule I activate only temporarily to make a few services public for a short time, usually I am via a VPN or physically in the local network.
I see that Unbound forwards local DNS requests to the nameservers and if I set up a DNS-Split (Host Override) to keep the DNS resolution local, I have no access to my services.

Without DNS-Split or Port-Forwarding: Works
Local Device -> Pi-Hole (cache or forward) -> OPNsense -> Unbound (cache or forward) -> Root Nameserver.

With DNS-Split: Does not work
Local Device -> Pi-Hole (cache or forward) -> OPNsense -> Unbound (e.g. my-service.subdomain.dedyn.io = 192.168.1.50)

Even if everything works, I would like to understand why I do not need DNS split or it does not work.

Kind regards.

[fizykpl]

1. You dont need to use virtual IP's.
2. Use map files {Advanced --> Map files}

1. You dont need to use virtual IP's.
I totally get your point! This makes indeed sense but I think only if you have a static WAN IP.
As it would break the access from internal networks to the external URLs "service.subdomain.mydomain.tld" if one enabled that access using DNS rewrite rules. I am not aware of a way to rewrite DNS entries in Unbound to the WAN interface address.

With NAT reflection your way of setting this up can of course work.


2. Use map files {Advanced --> Map files}
I haven't used those yet but looks very promising!
This really makes sense in a big environment with lots of subdomains.
Thank you for pointing this out! I will add it to the FAQ. 

Hey there and thank you so, so much for this great tutorial! It gave me exactly what I needed!

Yet there is a reason why I'm quoting this particular post.

Configuration made basing on your tutorial was working flawlessly on version 23.7.1 (os-haproxy 4.0, haproxy26 2.6.14), but after update to 23.7.2 and haproxy26 2.6.15 HAProxy service was failing to start.

I followed sorano's suggestion to not use virtual ip and bingo! That was it (it took me hours to find out where the issue is, as there were no message in logs - just a startup failure of HAProxy).

Maybe it would be good to add adnotation or a second way to configure HTTPS_frontend?

I can confirm that it works flawlessly with dynamic WAN ip.

Once again thank you very much and @sorano too

Cheers
Paweł

[TheHellSite]

Hey there and thank you so, so much for this great tutorial! It gave me exactly what I needed!

Yet there is a reason why I'm quoting this particular post.

Configuration made basing on your tutorial was working flawlessly on version 23.7.1 (os-haproxy 4.0, haproxy26 2.6.14), but after update to 23.7.2 and haproxy26 2.6.15 HAProxy service was failing to start.

I followed sorano's suggestion to not use virtual ip and bingo! That was it (it took me hours to find out where the issue is, as there were no message in logs - just a startup failure of HAProxy).

Maybe it would be good to add adnotation or a second way to configure HTTPS_frontend?

I can confirm that it works flawlessly with dynamic WAN ip.

Once again thank you very much and @sorano too

Cheers
Paweł

Cannot reproduce! And as usual not even a config export attached...
Everything working fine here on more than 6 different OPNsense instances (with latest update 23.7.2) configured exactly as per tutorial.

Furthermore there is no reason it wouldn't work anymore with a virtual IP since the localhost is somewhat also just a virtual IP.

About your question. If you read and understood each step of the guide you can easily figure out how to configure the HTTPS_frontend directly without the SNI_frontend.

[Digital_Addict]

Ok I dont' know if I need to create a new thread for this but I'll ask here first. I've followed your guide and I've set up a service (QuasselIRC) that  uses port 4242. I've got all three public services set up with the appropriate backends, rules and conditions. Long story short I followed Part 5 to the letter. My question is for services like IRC that don't use 80 or 443 do you ahve to set up separate Virtual Services ->  Public Services for things like IRC?

[le_top]

Could you tell us how to enable websocket on HAProxy?
Emby and Home Assistant don't work completely without them.

For the record, I use the setup explained in the tutorial and I also have an issue with Home Assistant's use of websockets.

The issue is that after about 30 seconds of inactivity, the web socket is closed by haproxy, which results in a page reload as this is what Home Assistant does when the socket disconnects or reconnects (probably to ensure that the shown states are up-to-date).

I opened a github issue for that and the founder blames the haproxy configuration.

There is at least one UI for nginx that has an option for websockets:
.

I've tried amending timeout values, but none of those that I changed had a positive effect on the timeout.

I have not found the correct option yet - I guess that there is a setting to fix this.

Edit: it seems that there is a heart-beat @55 seconds and nginx times out at 60 seconds, while ha-proxy does at 30s by default.  https://github.com/home-assistant/home-assistant-js-websocket/issues/108 .

I added "timeout tunnel 60000" to Services>HAPRoxy>Settings>Settings>Default Parameters>Custom Options  which is available after enabling the advanced mode.

Then I checked Services>HAPRoxy>Config Export where I could find this setting in the same section as the one suggested by Home Assistant

[SuperMiguel]

Since last week i been getting this: MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING when trying to log into my opnsense server, any quick fix for this? Thanks

[TheHellSite]

Since last week i been getting this: MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING when trying to log into my opnsense server, any quick fix for this? Thanks

User specific issue.
Working fine here on more than 6 different instances of OPNsense + HAProxy configured as per tutorial.
And, as always, not even a HAProxy config export included, must be hard to read the first post...