Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[Grenen]

Bitwarden only works from internal since I set up HA-Proxy.

Here’s a picture of the override:
https://ibb.co/ysNPg09

[TheHellSite]

Did you read and understand  what I wrote in part 6 - option a - step 3 of my tutorial? Or did you also not bother reading? Please explain to me what I am saying there and then explain what you did there... Maybe you will spot your error.

Also post the content (in a code box) of your public and local subdomains map file.

[Grenen]

Did you read and understand  what I wrote in part 6 - option a - step 3 of my tutorial? Or did you also not bother reading? Please explain to me what I am saying there and then explain what you did there... Maybe you will spot your error.

Also post the content (in a code box) of your public and local subdomains map file.

interface IP as the IP of the gateway? I understood it as any ip in the range of the vlan. Yes I’ve read through the steps several times, English isn’t my native language so some things might be lost in translation.

Code: [Select]


# public access subdomains
bit BITWARDEN_backend
home HA_backend


[TheHellSite]

No, it has to be the the interface IP of your OPNsense that is reachable by the clients that want to use haproxy... The guide is VERY clear about that. You can't just use any IP!? Stick to the guide!!   

Please post your interface overview...

[Grenen]

No, it has to be the the interface IP of your OPNsense that is reachable by the clients that want to use haproxy... The guide is VERY clear about that. You can't just use any IP!? Stick to the guide!!   

Please post your interface overview...

As i wrote, was misinterpreting from my part. I still get a 503-error when trying to reach the services after changing IP to the gateway of each network.

My HA-server is on my LAN, the Windows server which hosts my Docker Desktop with Bitwarden is on VLAN100.

Pictures of interface overview of LAN,MGMT(vlan100) & WAN:

https://ibb.co/SDCqcrn
https://ibb.co/sKMjnv5
https://ibb.co/p0nCc4y
https://ibb.co/C8jgnf2
https://ibb.co/SxN16PS

Screenshots taken from RDP-session from my phone, so they are a bit cropped.

[TheHellSite]

And as I already told you before and in the guide!!!
The gateway of the service does not matter at all!? Why is that so hard to understand?

The client dns request needs to be overwritten, not the dns requests of any service!!! So guess what, if the client is in subnet A and wants to access ANY service in ANY subnet then what IP will the client use to connect to the service? It obviously has to be subnet A gateway address since the client is in subnet A.
If the client is in subnet B you will have to create the same override but with subnet B gateway as target.
And so on.

All of the above is however only relevant for local access from within your network.

Now answer this
Is bitwarden working from external networks (mobile data,...)?
Is bitwarden now working from internal network?

[Grenen]

And as I already told you before and in the guide!!!
The gateway of the service does not matter at all!? Why is that so hard to understand?

The client dns request needs to be overwritten, not the dns requests of any service!!! So guess what, if the client is in subnet A and wants to access ANY service in ANY subnet then what IP will the client use to connect to the service? It obviously has to be subnet A gateway address since the client is in subnet A.
If the client is in subnet B you will have to create the same override but with subnet B gateway as target.
And so on.

All of the above is however only relevant for local access from within your network.

Now answer this
Is bitwarden working from external networks (mobile data,...)?
Is bitwarden now working from internal network?

No, I can only access it from internal networks. Not from the outside.

Bitwarden works in internal networks.

[TheHellSite]

Which URL do you use to access them from inside?

How do you access them directly (ip:port) full URL?

[Grenen]

Which URL do you use to access them from inside?

How do you access them directly (ip:port) full URL?

Bitwarden is accesses via Bitwarden.mydomain.com for internal

(it works) and external (can’t reach server)
HA is accessed internally from 192.168.1.106:8123 (works) and external via homeassistant.mydomain.com (can’t reach server).

There are some settings for HA to configure proxy that I think I need to set up, but since Bitwarden doesn’t work it’s not the complete solution to my problem.


From https://www.home-assistant.io/integrations/http/

use_x_forwarded_for boolean (optional, default: false)
Enable parsing of the X-Forwarded-For header, passing on the client’s correct IP address in proxied setups. You must also whitelist trusted proxies using the trusted_proxies setting for this to work. Non-whitelisted requests with this header will be considered IP spoofing attacks, and the header will, therefore, be ignored.

trusted_proxies string | list (optional)
List of trusted proxies, consisting of IP addresses or networks, that are allowed to set the X-Forwarded-For header. This is required when using use_x_forwarded_for because all requests to Home Assistant, regardless of source, will arrive from the reverse proxy IP address. Therefore in a reverse proxy scenario, this option should be set with extreme care. If the immediate upstream proxy is not in the list, the request will be rejected. If any other intermediate proxy is not in the list, the first untrusted proxy will be considered the client.

I’m using suricata IPS/IDS and tried disabling these, but no change. Using blocklists in Unbound DNS but that should not interfere.

I’m out of ideas. I will review everything again when I get home later today.

[sapphire]

Hey everyone! Hey TheHellSite!

First of all, I'm really thankful for your effort to write this really good guide and give support to date! That's awesome and should be honoured.

A bit of a backstory before I dive into the problem.
I'm a system engineer working with Sophos Products for years at enterprises. Because of that, I was using Sophos UTM Home Editions for years for my own private  servers as well -  but wanted to make the switch to OPNsense long ago. Some months ago, I finally decided to work on the switch. Your guide supported me so hard to make this switch from Sophos UTM to OPNsense.

Fast forward
I was thinking, my haproxy on my OPNsense was working completely. I had some issues before, where I could render websites from my local network (altough not using Split DNS or similar - just public IPs), but not from the internet (tested this with my 5G connection from my phone). After some tinkering, I got it even working on my phone with 5G and thought, everything should be okay now.

My Setup
I'm using 2 OPNsense, one at home and one in a datacenter on a VPS. I have these 2 connected to eachother via a Wireguard Tunnel (OPNsense plugin) using this guide (sorry, it's German: https://www.busche.org/index.php/2021/03/21/ipv4-ueber-wireguard-von-opnsense-zu-opnsense-routen-cgnat-umgehen/). And I'm using exactly your guide with the only difference using my wireguard interface instead of WAN for firewallrules.

Wireguard is working awesome, leads all traffic via Proxy ARP virtual IP (2nd public IP on VPS) on the VPS OPNsense to my home OPNsense.

Now the problem
More often then not, it seems that my websites aren't reachable from the internet. It's working from my 5G Internet Connection, it's working for a webdeveloper, who was assisting me with a web-project from hungary. But when friends test it or when I test it from my AWS Windows Machine, it's running into a timeout. Same for multiple status checking sites like https://isitdownorjust.me/. https://www.ssllabs.com/ssltest/ on the other hand is able to reach and check my websites.

I checked the haproxy logs and in those cases where it doesn't work, I get [09/Mar/2023:09:50:14.471] 1_HTTPS_frontend/127.0.0.1:443: SSL handshake failure

I checked your tutorial like 100 times. Checked my config side by side. It doesn't make any sense in my mind, that it works in some cases, and in some other cases it doesn't. And it drives me crazy. I was trying to fix this issue myself for weeks before I decided to write a comment and ask for help here.

My Config Export looks like this (deleted most backends and only left 2 for reference):

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends


# autogenerated entries for stats



# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_Condition
    acl acl_6293a5ef2e36e8.09400894 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6293a5ef2e36e8.09400894

# Frontend: 1_HTTPS_frontend (Listening on 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets prefer-client-ciphers ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6293aa32edc294.46241266.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6293a65a1027d4.72742608.txt)]
    # WARNING: pass through options below this line
    # add X-Forwarded-Proto
    http-request set-header X-Forwarded-Proto https if { ssl_fc }

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: nextcloud-mf_backend ()
backend nextcloud-mf_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server nextcloud-mf nextcloud-mf.fuchsbau.local:443 ssl verify none

# Backend: survey_backend ()
backend survey_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server adv-survey 10.0.110.5:80

listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED


I hope someone can help me, because I don't want to use the Sophos UTM as a fallback anymore and I'm feeling like I'm losing my mind.

Thanks in advance!

[Bunch]

Which URL do you use to access them from inside?

How do you access them directly (ip:port) full URL?

Bitwarden is accesses via Bitwarden.mydomain.com for internal

(it works) and external (can’t reach server)
HA is accessed internally from 192.168.1.106:8123 (works) and external via homeassistant.mydomain.com (can’t reach server).

There are some settings for HA to configure proxy that I think I need to set up, but since Bitwarden doesn’t work it’s not the complete solution to my problem.


From https://www.home-assistant.io/integrations/http/

use_x_forwarded_for boolean (optional, default: false)
Enable parsing of the X-Forwarded-For header, passing on the client’s correct IP address in proxied setups. You must also whitelist trusted proxies using the trusted_proxies setting for this to work. Non-whitelisted requests with this header will be considered IP spoofing attacks, and the header will, therefore, be ignored.

trusted_proxies string | list (optional)
List of trusted proxies, consisting of IP addresses or networks, that are allowed to set the X-Forwarded-For header. This is required when using use_x_forwarded_for because all requests to Home Assistant, regardless of source, will arrive from the reverse proxy IP address. Therefore in a reverse proxy scenario, this option should be set with extreme care. If the immediate upstream proxy is not in the list, the request will be rejected. If any other intermediate proxy is not in the list, the first untrusted proxy will be considered the client.

I’m using suricata IPS/IDS and tried disabling these, but no change. Using blocklists in Unbound DNS but that should not interfere.

I’m out of ideas. I will review everything again when I get home later today.

# public access subdomains
bit BITWARDEN_backend
home HA_backend

From the map file you provided, you should be accessing your Bitwarden with bit.mydomain.com in local network
If you failed with bit.mydomain.com locally, check your Unbound override to see if you have override bit.mydomain.com to the IP that TCP frontend listening to (That means, LAN or VLAN gateways or Loopback IPs)

Until you can access internally with bit.mydomain.com (not any other path), then you check rather you set your WAN port forward correctly.

BTW. I have Bitwarden running in my home, can access from both LAN and WAN
Suricata and Zenarmor and dns blocklist are on for my network. They won't block your Bitwarden.
The following list is related to your issue
1. (Scope: both internal and external)Rather HAProxy setup correctly (If you follow the guide without missing any point, that should be correct)
2. (Scope: Internally) Internal DNS override bit.mydomain.com to your TCP fontend
3. (Scope: Externally) WAN port forward at least TCP 443 to TCP frontend
4. (Scope: Externally) DDNS updated bit.mydomain.com to your WAN address

[TheHellSite]

Hey everyone! Hey TheHellSite!

First of all, I'm really thankful for your effort to write this really good guide and give support to date! That's awesome and should be honoured.

A bit of a backstory before I dive into the problem.
I'm a system engineer working with Sophos Products for years at enterprises. Because of that, I was using Sophos UTM Home Editions for years for my own private  servers as well -  but wanted to make the switch to OPNsense long ago. Some months ago, I finally decided to work on the switch. Your guide supported me so hard to make this switch from Sophos UTM to OPNsense.

Fast forward
I was thinking, my haproxy on my OPNsense was working completely. I had some issues before, where I could render websites from my local network (altough not using Split DNS or similar - just public IPs), but not from the internet (tested this with my 5G connection from my phone). After some tinkering, I got it even working on my phone with 5G and thought, everything should be okay now.

My Setup
I'm using 2 OPNsense, one at home and one in a datacenter on a VPS. I have these 2 connected to eachother via a Wireguard Tunnel (OPNsense plugin) using this guide (sorry, it's German: https://www.busche.org/index.php/2021/03/21/ipv4-ueber-wireguard-von-opnsense-zu-opnsense-routen-cgnat-umgehen/). And I'm using exactly your guide with the only difference using my wireguard interface instead of WAN for firewallrules.

Wireguard is working awesome, leads all traffic via Proxy ARP virtual IP (2nd public IP on VPS) on the VPS OPNsense to my home OPNsense.

Now the problem
More often then not, it seems that my websites aren't reachable from the internet. It's working from my 5G Internet Connection, it's working for a webdeveloper, who was assisting me with a web-project from hungary. But when friends test it or when I test it from my AWS Windows Machine, it's running into a timeout. Same for multiple status checking sites like https://isitdownorjust.me/. https://www.ssllabs.com/ssltest/ on the other hand is able to reach and check my websites.

I checked the haproxy logs and in those cases where it doesn't work, I get [09/Mar/2023:09:50:14.471] 1_HTTPS_frontend/127.0.0.1:443: SSL handshake failure

I checked your tutorial like 100 times. Checked my config side by side. It doesn't make any sense in my mind, that it works in some cases, and in some other cases it doesn't. And it drives me crazy. I was trying to fix this issue myself for weeks before I decided to write a comment and ask for help here.

You'r welcome!

This error usually indicates that the client (os/browser/app) does not support the SSL settings of haproxy (min tls version, cipher list, cipher suites). In your config I can see that you are still using the default cipher list + suites in your https frontend, those include "weak" 128 bit ciphers which are not really necessary anymore today except if you have very ancient devices connecting to your haproxy. So better use the ciphers I provide in the beginning of the tutorial.

In combination with the above it is also important to know that the ciphers are mostly used in a server (here haproxy) preferred order! (if I remember correctly) Sometimes this is enough to let the handshake fail if the client connecting has very strict SSL requirements. Again, you are using 128 bit ciphers!

Are you scoring a 100% A+ in the SSL Labs test? (please post a screenshot of the result diagram)

Another thing worth checking is packet fragmentation between your two OPNsense instances since you have a wireguard tunnel involved. I also had very strange https timeouts/bugs with wireguard until I configured the correct mtu+mss on my wireguard interface to my VPN provider. I am using VDSL though so PPPoE on the WAN interface resulting in a WAN MTU of 1492. Those 8 Bytes are enough to seriously mess with wireguard tunnels.



Here is a good explanation of SSL negotiation:

During the SSL/TLS handshake, the client sends a list of supported ciphers to the server, and the server responds with a list of ciphers it supports and their preferred order. The client then selects the first cipher in its list that is also present in the server's list, and the server and client agree to use that cipher for the remainder of the SSL/TLS session.

If there is no cipher that both the client and server support, the SSL/TLS handshake fails and the connection is terminated.

Therefore, it is important to configure the server's cipher suite list in a way that prioritizes the strongest ciphers, while still ensuring compatibility with a broad range of clients. This can help to ensure that the SSL/TLS connection is using the strongest possible encryption while maintaining maximum compatibility.

[sapphire]

Good Morning HellSite!

thank you for your very detailed reply.

I was using your cipher list and cipher suits, before I tried everything to fix the issue. I got the list and suits from https://ssl-config.mozilla.org/ and used "intermediate" because it made somehow sense to me.

I now switched it back to your list and suits and this is the outcome:


I've got a FibreChannel Connection from Deutsche Glasfaser at home (therefore I need the VPS OPNsense to get a static public IP at home). I didn't change the MTU for Wireguard because the mentioned guide used values for VDSL. So I'm running on default 1500 MTU.

With a "wrong" MTU, shouldn't the VPN connection be shaky with every device from every "outside" network? It's working 100% all the time on my mobile internet (and my girlfriend's as well), and it's working 100% all the time for ssllabs but only some "Is it down or just me"-kinda sites. But for at least 2 friends (one using the newest Opera Browser on Windows), there are timeouts while trying to connect to my websites. Same for a Windows VPS hosted on AWS - can't get a handshake there either (using newest Chrome browser).

I got no problems with my mailserver/proxmox mail gateway at home. I got no hickups with SSH via NAT. I got no issues with gaming servers at home (friends can connect to it). So I think, something isn't working correctly with my haproxy, sadly

Edit: Home MTU is at the Wireguard default value = 1420. In the VPS it's set to 1340 (nowhere changed though - so it must have decided to use this MTU automatically on the Wireguard interface). And the VPS Wireguard Interace has some error (in/out - 1 / 688).

Changed the VPS MTU to 1420 as well. Errors now 5/3 since tunnel restart. But nothing else changed. I still get "ERR_TIMED_OUT" on my websites from AWS VPS.

[coreyinoz]

First off, thank you TheHellSite for this amazing tutorial. I highly appreciate your level of attention to detail and the  after post support.

The purpose of my design is to expose my Jellyfin Server for remote access from a 2nd home, but hopefully in a secure fashion. As far as I can tell, I'm trying to match your network diagram from the first post aside from JF vs Plex.

I almost feel bad asking for help today because I sense it's something stone deaf obvious I've missed. I even slept on it and went back through each step from the very start and I can't see any fat finger errors.

So with apologies in advance, I'm hoping you can offer some troubleshooting for instances where the SSL Server Test comes back as T / Certificate name mismatch. I've made it to the end of Step 5.

Protocol Support, Key Exchange, and Cipher Strength are all top marks, but SSL Test is marking me T because of the invalid cert.

Here's a link to my HAProxy Config - https://pastebin.com/P5QtYPUt

My ACME Client log looks identical to your screenshot in the tutorial. I have redone the issue/renew procedure and the log comes back looking happy again.

I'm happy to post screenshots that would help diagnose. Appreciate any redirect on where to start looking.

[meyergru]

With a "wrong" MTU, shouldn't the VPN connection be shaky with every device from every "outside" network? It's working 100% all the time on my mobile internet (and my girlfriend's as well), and it's working 100% all the time for ssllabs but only some "Is it down or just me"-kinda sites. But for at least 2 friends (one using the newest Opera Browser on Windows), there are timeouts while trying to connect to my websites. Same for a Windows VPS hosted on AWS - can't get a handshake there either (using newest Chrome browser).

I got no problems with my mailserver/proxmox mail gateway at home. I got no hickups with SSH via NAT. I got no issues with gaming servers at home (friends can connect to it). So I think, something isn't working correctly with my haproxy, sadly

That depends on PMTU discovery, so not every connection has to fail. Try reducing MTU/MSS just for the wireguard interface group like so: