OPNsense Forum

English Forums => Tutorials and FAQs => Topic started by: TheHellSite on May 31, 2021, 01:06:11 pm

Title: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 31, 2021, 01:06:11 pm
Hello,

when I started implementing HAProxy in my network I couldn't find any complete and well written guide out there. I had to puzzle everything together from various websites.
So I thought I would save many of you a lot of time and provide my ultimate HAProxy on OPNsense guide.  :)

This tutorial will show you how to configure HAProxy as a reverse proxy on OPNsense using wildcard certificates from Let's Encrypt.
It is going to be a step-by-step guide with images on how to set things up while also explaining why we set things up in a certain way.
I will try to make this as complete and detailed as possible.
If you think that there is anything wrong or missing, feel free to tell me about it and I will consider changing it.

If this guide was helpful to you then please leave me a thanks down below as it took me several days to write this down.

Kind Regards
TheHellSite



This configuration is tested to be working on OPNsense 22.1.x (OpenSSL flavour) with latest updates as of 20220611.



How to ask for help.

From now on I will no longer provide support to people asking for help without giving any additional information about their setup.

1. ALWAYS include the HAProxy Config Export.
2. ALWAYS include relevant HAProxy errors and/or log entries.
3. ALWAYS include details about your setup, your goal, the service, ... anything relevant to the issue.




Changelog



Current Ciphers and Cipher Suites for a 100% A+ rating at SSLLabs
Last updated on 20220615 using Mozilla SSL Configuration Generator.
https://ssl-config.mozilla.org/#server=haproxy&version=2.4.17&config=intermediate&openssl=1.1.1o&guideline=5.6

All ciphers with a strength of 128 bit or below have been removed in order to get a 100% A+ rating at SSL Labs.

Code: [Select]
Cipher List
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384

Cipher Suites
TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

What will the end result look like?
We will have a wildcard SSL certificate from Let's Encrypt that gets an A+ rating with 100% score at all points from SSLLabs.
https://www.ssllabs.com/ssltest/

We will also have two levels of load balancing our services.

Level 0 - SSL Offloading disabled
WWW --> WAN interface --> OPNsense --> HAProxy SNI Frontend --> internal servers / services

Level 1 - SSL Offloading enabled
WWW --> WAN interface --> OPNsense --> HAProxy SNI Frontend --> HAProxy SSL Frontend --> internal servers / services

Visual Schematic
(Idea: @cookiemonster (https://forum.opnsense.org/index.php?topic=23339.msg123295#msg123295))
(https://i.postimg.cc/MM2dj6zQ/P000-001-Schematic.png) (https://postimg.cc/MM2dj6zQ)


What are we going to do?

FAQ



The Configuration

Part 1 - Plugin Installation




Part 2 - DynDNS Configuration




Part 3 - Let's Encrypt (ACME Client)




Part 4 - System Preparation




Part 5 - HAProxy Configuration




Part 6 - Access from internal networks
If you try to access your URL "your_service.your_subdomain.dedyn.io" from a device in your internal network, it should fail.
There are two ways of fixing this. I will cover both options but keep in mind that Split DNS (Option A) is the suggested way of doing it.
NAT Reflection (Option B) is an inferior solution since you lose the ability to track originating source IP in HAProxy when going through NAT. (@sorano (https://forum.opnsense.org/index.php?topic=23339.msg111439#msg111439))

Option A - Split DNS (https://docs.opnsense.org/manual/unbound.html#overrides)
Option B - NAT Reflection (https://docs.opnsense.org/manual/nat.html)

Option A - Split DNS (DNS Overrides)
Since you are using OPNsense you are probably also using the Unbound DNS plugin as your local DNS server.
Because of that you can easily set up DNS overrides.

Option B - NAT Reflection
Please note that NAT Reflection is only applicable on port forwarding rules so you will have to change the "Allow HAProxy" rule to a port forwarding rule with the localhost (or some random virtual IP on the localhost) as target.




Part 7 - Advanced Configuration: local-access-only subdomains
Imagine you have a service that you would like to access / protect using your brand new reverse proxy without making it available on the internet?
Well, HAProxy has got you covered!

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: browne on June 01, 2021, 10:35:23 am
Thank you very much!
This helped me switching from regular certificates to wildcard certificates!
I now also do score 100% A+ in the SSL test.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: skittlebrau on June 01, 2021, 04:23:25 pm
Thank you so much for this guide! I was completely lost in the new UI layout before.

Do you mind showing in Step 5 an example of how you've configured 'PLEX_backend'? I noticed you referred to it, but there wasn't a screenshot for it.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 01, 2021, 06:52:28 pm
The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different. :)

--> I will add it to the tutorial.

Reason: https://forum.opnsense.org/index.php?topic=23339.msg111143#msg111143


It is also almost everytime the same procedure to configure / add a new service in HAProxy.
--> I will add this to the FAQ.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: ejball02 on June 01, 2021, 08:55:28 pm
I'm using a self-signed cert for HTTPS inspection for content filtering. I've got OPT2 configured as a guest network on my Protectli, and content filtering, using shallalist works great. Only downside, is that when I try to access am HTTPS site, Firefox/Chrome always give a warning page: "Your connection is not private" "ERR_CERT_AUTHORITY_INVALID". After much Googling, I came across an old post, that said Let's Encrypt can give public certs which would get rid of the message.

I added the LE plugin but couldn't figure out, how to create a cert for use on the Foward Proxy "CA to use" field, required for SSL inspection. Looking through this walkthrough, I'm wondering if there is something here that can help achieve creating a cert for content filtering. Anyone have any experience with this?
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 01, 2021, 11:36:13 pm
I'm wondering if there is something here that can help achieve creating a cert for content filtering. Anyone have any experience with this?
No offense, but this is meant to be a tutorial related to HAProxy in a reverse proxy setup.  :)
As this is the tutorial sub-forum it is not really meant to ask for help and especially not regarding different topics.
You can of course follow this guide to the part where you get your Let's Encrypt certificate, but from there on you will be on your own.

I think you should probably open up your own thread in the help forums and ask for help there.

https://forum.opnsense.org/index.php?board=27.0
or here
https://forum.opnsense.org/index.php?board=28.0
or here
https://forum.opnsense.org/index.php?board=1.0
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: skittlebrau on June 02, 2021, 02:24:48 am
The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different. :)

I assumed that it would be the case, so thanks for confirming  ;D

One part that's tripping me up is in my plex_backend pool and other ones I set for my internal services like Resilio Sync, if I set the mode to TCP (Layer 4) then I get a syntax error below. Switching the mode to HTTP for those two backends resolves the syntax error, but produces blank pages. Stats area in HAProxy shows the services as being UP.

[NOTICE] 152/082107 (17536) : haproxy version is 2.2.14-a07ac36
[ALERT] 152/082107 (17536) : http frontend '1_HTTPS_frontend' (/usr/local/etc/haproxy.conf.staging:70) tries to use incompatible tcp backend 'sync1_backend' (/usr/local/etc/haproxy.conf.staging:117) in a 'use_backend' rule (see 'mode').
[ALERT] 152/082107 (17536) : http frontend '1_HTTPS_frontend' (/usr/local/etc/haproxy.conf.staging:70) tries to use incompatible tcp backend 'plex_backend' (/usr/local/etc/haproxy.conf.staging:104) in a 'use_backend' rule (see 'mode').
[ALERT] 152/082107 (17536) : Fatal errors found in configuration.

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 02, 2021, 08:08:18 am
The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different. :)

I assumed that it would be the case, so thanks for confirming  ;D

One part that's tripping me up is in my plex_backend pool and other ones I set for my internal services like Resilio Sync, if I set the mode to TCP (Layer 4) then I get a syntax error below. Switching the mode to HTTP [...]
[/font]

Now I remember correctly.
The SSL_backend is different from most other backends since it is NOT running in HTTP mode.
But HTTP mode is usually the one you want for a web based service.
I changed that part in the tutorial. Please take a look at the backend creation.

My bad! I wrote the tutorial from brain memory. So I guess you are my beta tester.  ;D
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: skittlebrau on June 02, 2021, 08:54:51 am
All good, you've been so helpful.

I checked through everything again and realised that when setting up the NAT port forward, I forgot to enter the dropdown menu at the end to add the associated filter rule. I was too used to pfSense automatically selecting that by default, so no wonder it wasn't working despite changing from TCP to HTTP mode for the backend services!

For posterity sake, I've saved your guide for my own reference and archived it to PDF as well.

If there's a particular charity you support, send me a private message and I'll happily donate to one in your name, along with a donation to the OPNsense project.  :D
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: michaelgo on June 02, 2021, 03:19:36 pm
Hi,
thank you for an amazing guide.

in part 5.5 the picture is broken, can you please re-post (it's working now)
also, 5.7 "plex condition" is it a rule? (5.7 and 5.8 screenshots are vice versa)

also, i don't have "Backends" only "Backend pools" there
Services --> HAProxy --> Settings --> Virtual Services --> Backends
is it what you mean?
thanx
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 03, 2021, 01:01:13 am
Yep, those pictures are in the wrong order and yes, I meant Backend Pools.
I will change this.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 03, 2021, 01:24:05 pm
If there's a particular charity you support, send me a private message and I'll happily donate to one in your name, along with a donation to the OPNsense project.  :D
A donation to OPNsense would be happily welcome I guess. :)
I myself don't want any money for this. We are all more or less using OPNsense free of charge so the least anyone can do is help the community or donate to the project.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: sorano on June 05, 2021, 03:00:21 pm
@TheHellSite Great guide! I'm sure it will help alot of people trying to get this kind of setup up and running.

I'm running a similar setup and have some suggestions for improvement to your guide.

1. You dont need to use virtual IP's.
If you bind the HAProxy frontends to 0.0.0.0:80 & 0.0.0.0:443 it will bind to you WAN interface (even if it's dynamic). And when you do there is no need for NAT forwarding to the virtual IP's so a simple firewall rule for 80/443 on the WAN interface is enough.
Then you can bind the SSL terminating frontend to 127.0.0.1:[port] and use that IP for your SSL terminating "real server".
With the added bonus of that it performs better in a CARP setup.

2. Use map files {Advanced --> Map files}
Using map files to map domains to backends will keep your config rules alot less cluttered, especially when you have many subdomains to match. With map files 1 rule is enough to map all of your domains.

Example for map file:
plex.mydomain.tld Backend_plex

Example for rule:
(Execute function: Map domains to backend pools using a map file
Map file: [name of your map file]

Then apply the map file rule to your SSL terminating frontend.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 07, 2021, 11:29:22 am
1. You dont need to use virtual IP's.
2. Use map files {Advanced --> Map files}

1. You dont need to use virtual IP's.
I totally get your point! This makes indeed sense but I think only if you have a static WAN IP.
As it would break the access from internal networks to the external URLs "service.subdomain.mydomain.tld" if one enabled that access using DNS rewrite rules. I am not aware of a way to rewrite DNS entries in Unbound to the WAN interface address.

With NAT reflection your way of setting this up can of course work.


2. Use map files {Advanced --> Map files}
I haven't used those yet but looks very promising!
This really makes sense in a big environment with lots of subdomains.
Thank you for pointing this out! I will add it to the FAQ.  :)
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: sorano on June 07, 2021, 02:21:02 pm
1. You dont need to use virtual IP's.
I totally get your point! This makes indeed sense but I think only if you have a static WAN IP.

Well, I only have dynamic IP's for my WAN interfaces. MultiWAN consisting of a fiber primary with LTE failover, two OPNsense hosts running CARP on all interfaces except for WAN (since I cannot get proper stateful failover with dynamic WAN IP's).

As it would break the access from internal networks to the external URLs "service.subdomain.mydomain.tld" if one enabled that access using DNS rewrite rules. I am not aware of a way to rewrite DNS entries in Unbound to the WAN interface address.

With NAT reflection your way of setting this up can of course work.

The way I'm doing access from internal networks is with Split DNS (DNS override as you call it).
In my opinion NAT reflection is an inferior solution since you lose the ability to track originating source IP in HAProxy when going through NAT.

Since HAProxy is already listening on 0.0.0.0 (all available IPv4 interfaces) I resolve the Split DNS to the internal IP of my DMZ CARP IP (but any internal IPv4 interface will do as long as you allow 80/443).

I also have certain domains I don't want reachable from the Internet so I use two map file rules, one for internal domains along with a condition that checks that source is RFC1918.

And one for external domains where I also require additional authentication.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 09, 2021, 12:03:57 pm
Since HAProxy is already listening on 0.0.0.0 (all available IPv4 interfaces) I resolve the Split DNS to the internal IP of my DMZ CARP IP (but any internal IPv4 interface will do as long as you allow 80/443).
So this means you are actually also using sort of a virtual IP.  :D
Okay so you say the easier way is like this:
- Firewall rule on WAN allowing 443+80
- SNI_frontend listening on 0.0.0.0:80+443 (Which covers all interfaces)
- SSL_frontend listening on 127.0.0.1:443
- Split DNS pointing to f.e. the LAN IP of OPNsense.

You are right, with that you could skip the virtual IPs.
I would however still prefer to use a virtual IP for the SSL_frontend as I don't want to hijack the localhost for it. I just prefer to keep things seperated.

I also have certain domains I don't want reachable from the Internet so I use two map file rules, one for internal domains along with a condition that checks that source is RFC1918.
For this I guess you have to use 0.0.0.0 on the SNI_frontend otherwise you would need another NAT rule forwarding 443-LAN traffic to the virtual IP.

EDIT 20210611:
Implemented @sorano (https://forum.opnsense.org/index.php?topic=23339.msg111339#msg111339)'s enhancements  :)
Title: Re: Tutorial: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ SSLLabs Rating
Post by: TheHellSite on June 13, 2021, 04:13:00 pm
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: pintu1228 on July 08, 2021, 04:49:36 pm
This is exactly what I was looking for, have had trouble coming from pfsense to opnsense to setup haproxy/let's encrypt.  I have cloudflare setup to use DNS.

I am using google domain, how do I go about setting up the 1st part (Dynamic DNS), do I need to create 3 custom records:

domain.com (A type)
*.domain.com (A type)
www.domain.com (CNAME)

And also I created separate dynamicDNS for plex.domain.com to use for part 7 (configure Dynamic DNS on opnsense).

Thanks
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 09, 2021, 12:28:11 pm
I am using google domain, how do I go about setting up the 1st part (Dynamic DNS), do I need to create 3 custom records:

domain.com (A type)
*.domain.com (A type)
www.domain.com (CNAME)

It all depends on what you would like to achieve!
You can set your DynDNS to update the IP of "domain.com" and then have a CNAME record in the form of "subdomain.domain.com" pointing to the IP of "domain.com".

However since you might also want to host a website on "domain.com / www.domain.com", you will probably want a subdomain for your domain first!
I myself always create individual 1st-level-subdomains for each physical location that is sitting in a different building / city / country having there own public IP.

f.e. location.domain.tld = earth.thehellsite.com / moon.thehellsite.com / mars.thehellsite.com and so on

And then (as I already explained in my tutorial) a wildcard 2nd-level-subdomain.
Which means that any string in the 2nd-level-subdomain is pointing to my 1st-level-subdomain.

f.e. any_string.location.domain.tld = *.earth.thehellsite.com = abcxyz.earth.thehellsite.com / 123.earth.thehellsite.com


If you do it my way then you only need to set up three things in your Google DNS zone.


Your DynDNS should then be updating the "A Record".




But again, there are many different ways to achieve this.
You just need to find out how you would like to do it.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: _Alchemist_ on July 14, 2021, 01:45:11 pm
Thanks a lot for the write up, I will try this out as soon as I can :)

The only thing that could be added on Part 4.3 is to use an Alias for Port 80 and 443 to only use one Firewall Rule ;)
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 14, 2021, 03:40:05 pm
Thanks a lot for the write up, I will try this out as soon as I can :)

The only thing that could be added on Part 4.3 is to use an Alias for Port 80 and 443 to only use one Firewall Rule ;)

Good Idea!
Will add this in my next update.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: saarko on July 27, 2021, 11:44:25 pm

I also have certain domains I don't want reachable from the Internet so I use two map file rules, one for internal domains along with a condition that checks that source is RFC1918.

And one for external domains where I also require additional authentication.

Dear @TheHellSite,
thanks for the great tutorial! It works well.

Dear @sorano,

thanks for your input. The hint with map file works well. However, I am unable to create a rule with multiple "OR" conditions for various sub-domains to match and check it with an "AND" condition to test if it is an internal IP. It shall cover your described rule.

Currently I try to create a rule like:
use map file 1
IF
condition 1 "subdomain1" OR condition 2 "subdomain2"
AND condition 3 "local IP (RFC1918)" is matched


How did you solve this with the conditions and rules within OPNsense HAProxy plugin?

thanks in advance for your help and reply.
Saarko
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 28, 2021, 09:18:12 pm
Thanks for the feedback!

I had a chat with @sorano and changed my config a little bit in the meantime.

I will update my post in a few days, hopefully. Very busy with work atm.
This will also solve your problem @saarko.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 29, 2021, 07:34:44 pm
I followed this, however, decided against using the LE and now not getting 100% A+. is there something I am missing...
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 29, 2021, 08:07:09 pm
I followed this, however, decided against using the LE and now not getting 100% A+. is there something I am missing...

Since you didn't use "LE"? (I guess you mean Let's Encrypt certificates) you should probably check out this page.
https://github.com/ssllabs/research/wiki/SSL-Server-Rating-Guide

So if I would have had to guess it is most likely due to the fact that the key length of your certificate is too short!

If you followed the rest of my guide (OPNsense settings + HAProxy settings) accordingly and didn't miss any configuration steps, you should be getting 100% A+ at SSLLabs!
Again, provided that your certificate has a long enough key length...



I hope you can understand that it is very hard to help you out, since you didn't share any more details about your config, the SSLLabs test results, the certificate you are using ...
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 29, 2021, 10:23:29 pm


@saarko
I didn't manage to finish part 7 "local-access-only subdomains" today.
I will try to do this tomorrow!

In the meantime you can already change your config over to map files.  :)
See part 5, step 7 to step 10. This is the only thing that has changed for using map files.
Map files will also make part 7 very easy for you!
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 30, 2021, 05:43:22 pm
ggetting this from hap:

[WARNING] 210/114212 (27105) : Proxy '1_HTTPS_Frontend': no-sslv3/no-tlsv1x are ignored for bind '192.168.1.50:443' at [/usr/local/etc/haproxy.conf.staging:71]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
Warnings were found.
Configuration file is valid
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 30, 2021, 06:21:48 pm
ggetting this from hap:

[WARNING] 210/114212 (27105) : Proxy '1_HTTPS_Frontend': no-sslv3/no-tlsv1x are ignored for bind '192.168.1.50:443' at [/usr/local/etc/haproxy.conf.staging:71]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
Warnings were found.
Configuration file is valid

Please post a screenshot of your entire HTTPS_frontend config, including advanced.
You can redact your rules.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 30, 2021, 06:51:11 pm


Code: [Select]
# Frontend: 1_HTTPS_Frontend
frontend 1_HTTPS_Frontend
    # WARNING: ciphersuites cannot be used with flavour libressl.
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 10.10.1.50:5555 name 10.10.1.50:555 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-min-ver TLSv1.2 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60f9db54XXX3488.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 30, 2021, 07:08:35 pm


Code: [Select]
# Frontend: 1_HTTPS_Frontend
frontend 1_HTTPS_Frontend
    # WARNING: ciphersuites cannot be used with flavour libressl.
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 10.10.1.50:5555 name 10.10.1.50:555 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-min-ver TLSv1.2 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60f9db54XXX3488.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

I was already assuming what you did (wrong) and your config is confirming it.  ;D

First of all. This is not an error but only a warning which won't make HAProxy refuse to start.
Your setup should still be working if you did everything else correctly.

This warning is appearing because you set "ssl-min-version = TLS1.2" in your HTTPS_frontend config.
Which is pretty much pointless since my config example is already blocking "sslv3 tls1.0 tls1.1" using the "Bind options" in the HTTPS_frontend field. This is why HAProxy is giving you a warning.
You can of course also remove my bind options and only use "min-ssl-version = TLS1.2", this is up to you.
Either way will work and allow only TLS1.3 connections.


Code: [Select]
# Frontend: 1_HTTPS_Frontend
frontend 1_HTTPS_Frontend
    # WARNING: ciphersuites cannot be used with flavour libressl.
...
What you should be aware of is that I specifically mentioned that my guide might not work well on LibreSSL due to the fact that not all features of the last HAProxy version is supported by the LibreSSL firmware variant of OPNsense, yet!
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 30, 2021, 07:13:04 pm
Dear @TheHellSite,
thanks for the great tutorial! It works well.

Dear @sorano,

thanks for your input. The hint with map file works well. However, I am unable to create a rule with multiple "OR" conditions for various sub-domains to match and check it with an "AND" condition to test if it is an internal IP. It shall cover your described rule.

Currently I try to create a rule like:
use map file 1
IF
condition 1 "subdomain1" OR condition 2 "subdomain2"
AND condition 3 "local IP (RFC1918)" is matched


How did you solve this with the conditions and rules within OPNsense HAProxy plugin?

thanks in advance for your help and reply.
Saarko

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 30, 2021, 08:08:34 pm
so not sure what you are referring to...

I used the cloudflare... what option is needed for the cert that's different from the  norm.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 30, 2021, 09:26:40 pm
So not sure if you understand what I am trying to tell you.........

I followed this, however, decided against using the LE and now not getting 100% A+. is there something I am missing...
Firstly, when you made your one-liner about the bad SSLLabs result, I asked you for details about your config and the SSLLabs result.
You gave NONE.
How could anyone be able to help you with that little information given?!  :o
This makes me even a bit angry.

Secondly, you asked why HAProxy is giving you a warning, so I explained to you in a very detailed manner why that is.
And now you are saying that you don't know what I am referring to?
Like, I even quoted the issue.

Thirdly, I am really willing to help anyone that follows my guide.
I am doing this in my free time free of charge.
But in return the least you could do is to say thank you, before asking for help about an issue that is most likely due to the fact that you didn't read my tutorial correctly and that you are not using Let's Encrypt certificates.


So with that being sad.
Maybe you should think about what I just said.
And then, if you are willing to, share just a tiny little bit more details about: your certificate, your SLLLabs result, ... like anything that could be of help and not just one-liners without any context.
Your issue could have already been solved if you had provided those information in the first place.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 30, 2021, 09:47:41 pm
here you go...


Protocols
TLS 1.3   Yes
TLS 1.2   Yes
TLS 1.1   Yes
TLS 1.0   Yes
SSL 3   No
SSL 2   No


Cipher Suites
# TLS 1.3 (server has no preference)
TLS_AES_128_GCM_SHA256 (0x1301)   ECDH x25519 (eq. 3072 bits RSA)   FS   128
TLS_AES_256_GCM_SHA384 (0x1302)   ECDH x25519 (eq. 3072 bits RSA)   FS   256
TLS_CHACHA20_POLY1305_SHA256 (0x1303)   ECDH x25519 (eq. 3072 bits RSA)   FS   256
# TLS 1.2 (suites in server-preferred order)
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)   ECDH x25519 (eq. 3072 bits RSA)   FS   128
OLD_TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcc14)   ECDH x25519 (eq. 3072 bits RSA)   FS   256P
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)   ECDH x25519 (eq. 3072 bits RSA)   FS   256P
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)   ECDH x25519 (eq. 3072 bits RSA)   FS   WEAK   128
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)   ECDH x25519 (eq. 3072 bits RSA)   FS   WEAK   128
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)   ECDH x25519 (eq. 3072 bits RSA)   FS   256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)   ECDH x25519 (eq. 3072 bits RSA)   FS   WEAK   256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)   ECDH x25519 (eq. 3072 bits RSA)   FS   WEAK   256
# TLS 1.1 (suites in server-preferred order)
# TLS 1.0 (suites in server-preferred order)
(P) This server prefers ChaCha20 suites with clients that don't have AES-NI (e.g., Android devices)   
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 31, 2021, 12:40:03 am
Please don't get me wrong, but...

You know what it's like trying to help you?
It's like trying to teach a stone how to sing.
You can try as long as you want and still fail.
Because...
A) He doesn't understand you.
and
B) He is not answering your questions.




This is the last time I am asking you.
Give a screenshot of your ENTIRE SSLLabs result.
A SCREENSHOT. But with your domain name and IP blacked out.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: saarko on July 31, 2021, 02:47:32 pm
  • 20210730
    • Added an explanation on how to configure local-access-only subdomains in HAProxy.

thanks, works like a charm.
My "mistake" was that I thought to need a condition to trigger a map rule. Since it is not necessary, it is even easier, except for the RFC1918 condition of course. :)
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Crappysauce on August 01, 2021, 07:26:04 am
First off, thank you so much for this guide. Really helped.

I was having issues connecting to my server due to handshake errors which I think got fixed after generating new ciphers using the Mozilla SSL Config generator and changing the HAProxy and OpenSSL versions to match my setup.

After that, HAProxy seemed to refuse to redirect me to my Vaultwarden server, unless I turned off the SSL option in my Real Server setting. It still shows that I'm secured with the proper (wildcard cert from Let's Encrypt).

Do I need the SSL option enabled? The SSL test still gave me an A+...
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 01, 2021, 10:05:57 am
I was having issues connecting to my server due to handshake errors which I think got fixed after generating new ciphers using the Mozilla SSL Config generator and changing the HAProxy and OpenSSL versions to match my setup.

Are you on the latest version of OPNsense and are the installed plugins up to date?
Just out of interest, which versions of OPNsense, HAProxy and Let's Encrypt are you running?

After that, HAProxy seemed to refuse to redirect me to my Vaultwarden server, unless I turned off the SSL option in my Real Server setting. It still shows that I'm secured with the proper (wildcard cert from Let's Encrypt).

Do I need the SSL option enabled? The SSL test still gave me an A+...

The reason you couldn't connect was due to a misconfiguration in your real server, as you figured out yourself.
You enabled the "SSL -  Enable or disable SSL communication with this server. " checkbox in your real server for Vaultwarden even though the port used to connect doesn't offer SSL encryption.
https://github.com/dani-garcia/vaultwarden/wiki/Enabling-HTTPS
https://github.com/dani-garcia/vaultwarden/wiki/Private-CA-and-self-signed-certs-that-work-with-Chrome



You need to think of a reverse proxy setup like this.

WWW ---Stage 1---> yourdomain.tld ---Stage 2---> OPNsense + HAProxy + LE ---Stage 3---> internal services

Stage 1 + 2
Public facing external traffic. Traffic in these stages is now always encrypted with a verified SSL certificate. In this case it is created and verified by Let's Encrypt.

Stage 3
Local facing internal traffic. Traffic in this stage can or can not be encrypted, depending on your service setup. This is the traffic from HAProxy to your internal service. It doesn't need to be encrypt because you can consider your internal network as trusted.
However it is still strongly advised to also run this traffic encrypted.
In HAProxy you only need to check the "SSL" box in your real server setting for this.
But then you also need to actually enable SSL encryption on that service, f.e. by installing a self-signed certificate on that service and enabling HTTPS. Even though using a self-signed certificate will give you a warning by your browser when accessing the service directly and not through the reverse proxy, the traffic is still encrypted, the certificate is just unverified.
How to actually do this this depends on the service but this should be covered somewhere in its manual.

You can read more about this here: https://www.globalsign.com/en/ssl-information-center/dangers-self-signed-certificates



I will add this explanation to the FAQ.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Crappysauce on August 02, 2021, 07:37:30 am
Are you on the latest version of OPNsense and are the installed plugins up to date?
Just out of interest, which versions of OPNsense, HAProxy and Let's Encrypt are you running?

Everything is at the latest version:
OPNSense: 21.7
HAProxy Plugin: 3.4
Let's Encrypt: 2.6


The reason you couldn't connect was due to a misconfiguration in your real server, as you figured out yourself.
You enabled the "SSL -  Enable or disable SSL communication with this server. " checkbox in your real server for Vaultwarden even though the port used to connect doesn't offer SSL encryption.
https://github.com/dani-garcia/vaultwarden/wiki/Enabling-HTTPS
https://github.com/dani-garcia/vaultwarden/wiki/Private-CA-and-self-signed-certs-that-work-with-Chrome

I knew I was forgetting something  :(
I'll read up on his quides and get it all sorted.

Thank you again for your guide and help  ;D
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 09, 2021, 04:31:33 pm
Hi

@TheHellsite Thank you so much for your time and knowledge

If you have a fixed IP, does the DynDNS Configuration step need to be done? if skipped is there other settings i should put in?

If it is a must when we signup, there are 2 options :

configure your own domain
or
register under dyn.io

i have my own domain names about 10. Do i add each one to there system to get certs then duplicate the process to reverse proxy and cert the other domains?

If you use your real domain eg. www.123.com Do i need to go to my current domain registrar and change name servers to point to desec??

Thank you for the help. just want to get these vms up so i can programme again :( going to cry.
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 09, 2021, 06:29:22 pm
Hi

If you have a fixed IP, does the DynDNS Configuration step need to be done? if skipped is there other settings i should put in?

If you have a fixed IP that for sure never ever changes.
Then yes, you can safely skip setting up DynDNS on your OPNsense.
You will then only need to configure an A-Record in the DNS zone of your domains / subdomains pointing to your static IP.
You can set this up at your domain hosting provider.

Feel free to share a bit more about your current domain set up.
This will make it easier for me to help you.
- What are your domains?
- What are your subdomains?
- What is your domain hosting provider?

If it is a must when we signup, there are 2 options :

configure your own domain
or
register under dyn.io

i have my own domain names about 10. Do i add each one to there system to get certs then duplicate the process to reverse proxy and cert the other domains?

update: i used my domain name. i think that was wrong.

Seems like deSEC now also supports managing domains that are "hosted / registered" at a different hosting provider.
If I understood this correctly it allows you to manage the DNS zone of your domain at deSEC without actually transfering your domain away from you current hosting provider.

Since you are saying that you already have some domains / subdomains registered.
Something like "sub1.yourdomain1.com" ... "sub4.yourdomain5.com" and so on...

In this case you should check if your current domain hosting provider supports the DNS challenge.
And that the Let's Encrypt Plugin on OPNsense supports the DNS challenge for your hosting provider.
If not, then you have two options if you would like to use wildcard certificates...

Option 1 - Proceed setting up the managed DNS for your desired domains at deSEC. Then follow my tutorial beginning with part 2 step 3.

Option 2 - Transfer your domains to a hosting provider that supports the DNS challenge and that is also supported by the Let's Encrypt plugin.





Feel free to share more information about your domain / subdomain scenario along with the name of your hosting provider.
Of course don't expose your real domain names.
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 09, 2021, 06:45:54 pm
Wow thank you

Yes IP is fixed 100%.
- What are your domains? .com and .co.uk eg. example.com is my primary one.
- What are your subdomains? cloud.example.com dev.example.com
- What is your domain hosting provider? 123reg.co.uk

Have got to step Part 2 step 7. setting up opnsense dynamic DNS. So far have followed all steps par i registered example.com and not "anything.dedyn.io".

So before continuing i will check 123reg.co.uk options
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 09, 2021, 06:54:15 pm
For your own safety please replace your real domain name with "example", if the above is your real domain name!

I quickly checked 123reg and it seems like they do not offer an API so you can't use the DNS challenge.

Which gives you only 2 options.

1 - From now on managing the DNS zone of your 123reg domains at deSEC.
https://www.123-reg.co.uk/support/domains/how-do-i-change-the-nameservers-for-my-domain-name/
+ see image attached.

2 - Moving your domain to another registrar.
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 09, 2021, 07:10:56 pm
Thank you,

name servers updated. I have added A and MX records (set MX prefence to "10", is that right?).

In the opnsense Dynamic DNS, stuck on update URL. just type update.example.com?
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 09, 2021, 07:38:08 pm
Is dynamic dns still needed for fixed IP. You did say start from part 2 step 3. This update URL makes me think?
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 09, 2021, 08:23:33 pm
Is dynamic dns still needed for fixed IP. You did say start from part 2 step 3. This update URL makes me think?

Not needed! A record is enough.
Dynamic DNS is basically only updating the A record in your DNS Zone.
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 10, 2021, 05:24:42 pm
Hi,

So was setting up all night and all morning.

I skipped and removed dyndns plugin.
I completed the rest of the steps, except for the map as no subdomains setup yet.
I also didnt add a wild card for subdomains to the cert, when i did add it the cert failed. once removed the cert installed fine.
When i checked the local ip in browser, it works. When i type www.example.com it fails and cant ping address.

Had a hunt for what it could be, in the end decided to reboot opnsense and see if it shows errors.

Now rebooted HaProxy status is down and will not start.
I have looked in every log and there is only this:

2021-08-10T15:15:37   root[90544]   /usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy   
2021-08-10T15:15:30   root[91171]   /usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy   
2021-08-10T15:00:29   root[26043]   /usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy   
2021-08-10T14:59:58   syslog-ng[29102]   syslog-ng starting up; version='3.33.2'   
2021-08-10T14:59:56   syslog-ng[70806]   syslog-ng shutting down; version='3.33.2'   
2021-08-10T14:59:55   root[49764]   /usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy

thinking to remove HAProxy and start again?
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 10, 2021, 06:07:04 pm
No need to.
Look in your PMs.
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 01, 2021, 11:49:00 am
Hi there,
First of all, thank you very much for the tutorial. I still have two questions. I hope you are so nice to answer me :).

I want to make OPENvpn and various websites accessible via 443.

1st question:
I create a subdomain for each service. Can this subdomain then point to the same DYNDNS cname entries?
For example:
openvpn.domain.com -> CNAME -> 123.dyndns.com
seafile.domain.com -> CNAME -> 123.dyndns.com

2nd question:
How to configure HAproxy for openvpn.
Could you explain it to me as well as you did in the tutorial or add the point to the tutorial?
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 01, 2021, 04:39:55 pm
1st question:
I create a subdomain for each service. Can this subdomain then point to the same DYNDNS cname entries?
For example:
openvpn.domain.com -> CNAME -> 123.dyndns.com
seafile.domain.com -> CNAME -> 123.dyndns.com
Yes, this is exactly what the CNAME record is for! https://en.wikipedia.org/wiki/CNAME_record

If you follow my guide and use desec.io as your DynDNS provider then you can even use a wildcard CNAME record. As you can see in the attached picture.
"*.tutorial.dedyn.io" means that "a.tutorial.dedyn.io, b.tutorial.dedyn.io, c.tutorial.dedyn.io, ..." will all point to "tutorial.dedyn.io" which points to "8.8.8.8".

So if you decide to use a wildcard subdomain, then you will only need to create that single subdomain "*.domain.com".
But keep in mind that not all DNS providers / domain registrars support this!


2nd question:
How to configure HAproxy for openvpn.
Could you explain it to me as well as you did in the tutorial or add the point to the tutorial?
For this your OpenVPN server needs to run in TCP mode. (Your clients need to connect to your OpenVPN server using a TCP tunnel.)
This is because HAProxy doesn't support UDP load balancing!

If you are using OpenVPN over TCP, then the configuration is pretty straight forward.
The only difference to my example Plex configuration is that you will have to set the backend "OpenVPN_backend" to TCP mode instead of HTTP mode.
See the FAQ "Why are we doing 2-Level-SNI?" and "How can we load balance TCP traffic that we don't want to get SSL offloaded, f.e. OpenVPN over TCP?".
This should explain how to set this up.
I suggest that you first make your Seafile server working and after that proceed with your OpenVPN server.

As I am currently not using OpenVPN or any other service that requires TCP load balancing, I won't be adding this to the tutorial any time soon.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 04, 2021, 06:20:01 pm
Thanks for your answer.

I have now followed your instructions exactly before I play with a changed configuration. When I try to call up the page cloudserver.xyz.dedyn.io, I always get a 503 Service Unavailable message.

Do you have any idea what I'm doing wrong?
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 05, 2021, 10:41:02 am
Post your HAProxy config. But remove your public IP or anything that reveals your identity. But leave all local IPs in it.
There is an export option in HAProxy.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 08, 2021, 04:21:35 pm
Hi,

this is my HAproxy config:

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    option tcplog

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_6138b110159553.96461818 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6138b110159553.96461818

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6138b32401a006.77997133.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6138b15d48a964.28077676.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: SEAFILE_backend ()
backend SEAFILE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server SEAFILE_server 192.168.1.11:81 ssl verify none

Mapfile:
Code: [Select]
Name: PUBLIC_SUBDOMAINS_map
Content: cloudserver SEAFILE_backend

Do you need something else?
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sorano on September 08, 2021, 08:14:51 pm
Dropping this in here to make sure noone misses it:

https://forum.opnsense.org/index.php?topic=24668.0
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 08, 2021, 09:17:48 pm
This could be my Problem. How can i install the fix? Sorry for this dump question i did this never befoe.


Gesendet von iPhone mit Tapatalk
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sorano on September 08, 2021, 09:23:08 pm
This could be my Problem. How can i install the fix?

Lol no!! Not at all. That is a security vulnerability and not your problem.

If the information you posted is correct your problem is that you are not using FQDN in your map file.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 08, 2021, 09:28:56 pm
Oh ok.

you mean i must change the mapfile to

cloudserver.domain.com SEAFILE_backend


Gesendet von iPhone mit Tapatalk
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sorano on September 08, 2021, 09:41:58 pm
Oh ok.

you mean i must change the mapfile to

cloudserver.domain.com SEAFILE_backend


Gesendet von iPhone mit Tapatalk

Yeah
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 09, 2021, 11:49:52 am
Dropping this in here to make sure noone misses it:

https://forum.opnsense.org/index.php?topic=24668.0
Thanks!



If the information you posted is correct your problem is that you are not using FQDN in your map file.

Not necessarily, I think!
I also didn't place the whole FQDN in my mapfiles, instead I am just using the subdomains.
This works just fine. But I guess it is arguable if using the complete FQDN is the better option.
https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/syntax/

Code: [Select]
plex PLEX_backend
iot IOT_backend
...



Code: [Select]
# Backend: SEAFILE_backend ()
backend SEAFILE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server SEAFILE_server 192.168.1.11:81 ssl verify none

Your map file should still work even without using FQDNs.

Your HAProxy config also looks fine.
The only thing that makes me curious is your "SEAFILE_server" configuration!
You might wanna take a look at the "configured port" or the "SSL checkbox"!
I highly doubt that your seafile server is listening on port 81 for HTTPS traffic, which is what you have configured by ticking the SSL checkbox.
The SSL checkbox means that the server is using HTTPS/SSL on the specified port. However if the server isn't actually using HTTPS/SSL on that port then the connection will obviously not work.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 09, 2021, 04:11:09 pm
Many thanks for the help. FQDN is not necessary.

By changing to no ssl, the Seafile server now works.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 10, 2021, 09:27:30 am
Let me know if you need anymore help with your OpenVPN config.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 10, 2021, 08:09:51 pm
I need your help again.   ;D

Code: [Select]
root@OPNsense:~ # cat /usr/local/etc/haproxy.conf
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    option tcplog

    # ACTION: NOSSLservice_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/613b963c5f0851.94679524.txt)]

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_6138b110159553.96461818 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6138b110159553.96461818

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6138b32401a006.77997133.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6138b15d48a964.28077676.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: SEAFILE_backend ()
backend SEAFILE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server SEAFILE_server 192.168.30.16:80

# Backend: OPENVPN_backend ()
backend OPENVPN_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server OPENVPN_server 127.0.0.1:1194

Code: [Select]
2021-09-10T20:00:44 haproxy[11387] 192.168.1.231:51903 [10/Sep/2021:20:00:44.614] 0_SNI_frontend SSL_backend/SSL_server 1/0/4 0 -- 1/1/0/0/0 0/0
2021-09-10T20:00:44 haproxy[11387] 192.168.1.231:51903 [10/Sep/2021:20:00:44.615] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T20:00:40 haproxy[11387] 192.168.1.231:51902 [10/Sep/2021:20:00:40.526] 0_SNI_frontend SSL_backend/SSL_server 1/0/5 0 -- 1/1/0/0/0 0/0
2021-09-10T20:00:40 haproxy[11387] 192.168.1.231:51902 [10/Sep/2021:20:00:40.527] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 0_SNI_frontend SSL_backend/SSL_server 1/0/39 0 -- 1/1/0/0/0 0/0
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T19:59:26 haproxy[11387] xx.xx.xx.162:25707 [10/Sep/2021:19:59:26.004] 0_SNI_frontend SSL_backend/SSL_server 1/0/35 0 -- 1/1/0/0/0 0/0

MAP_file:
(https://i.ibb.co/wBVxxYG/map.jpg) (https://ibb.co/pbc99nV)

OPENVPN_settings:
(https://i.ibb.co/VDRxbhg/opnvpn-server-conf.jpg) (https://ibb.co/1K1MBpr)
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 12, 2021, 10:42:17 pm
Okay, first you should change the listening interface of the OpenVPN plugin from "any" to "localhost".
Because this is where HAProxy is sending the traffic to since you defined your "OpenVPN real server = 127.0.0.1".
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 13, 2021, 12:34:49 pm
Hi, I changed the interface, the problem remains the same. It looks like the VPN access is not forwarded by the SNI but by the HTTPS server, right? Unfortunately I don't know how to work around the problem.

Code: [Select]
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 0_SNI_frontend SSL_backend/SSL_server 1/0/39 0 -- 1/1/0/0/0 0/0
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T19:59:26 haproxy[11387] xx.xx.xx.162:25707 [10/Sep/2021:19:59:26.004] 0_SNI_frontend SSL_backend/SSL_server 1/0/35 0 -- 1/1/0/0/0 0/0
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 13, 2021, 06:38:48 pm
Is your OpenVPN client config correct?
- mode tcp
- port 443
- server vpn.domain.com

Some of this must be wrong.
Otherwise your SNI_frontend wouldn't pass the traffic to the configured default backend (SSL_backend) which is what it is currently doing.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 13, 2021, 07:43:52 pm
here is the extract of my configuration. I think that is all correct.

Code: [Select]
dev tun
persist-tun
persist-key
proto tcp-client
cipher AES-256-CBC
auth SHA256
client
resolv-retry infinite
remote vpn.xxxxx.dedyn.io 443 tcp
lport 0
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nullex on September 13, 2021, 10:05:21 pm
Hello,

I'm having trouble with this guide.. Very strange trouble...
I have followed the steps to a T.. except I am using cloudflare for Dynamic DNS and ACME/let's encrypt Certs. Also, I am using localhost/127.0.0.1 instead of the virtual IP option. Although I have tried it both ways with the same results...

The problem is.. weird.. So, like I said, followed the steps.. But only a few backend servers work. I have set up Plex, Sonarr, Radarr, Ombi, Cams(blueIris) and some other stuff. However, only Plex, Cams, and Radarr work. I've been fighting with it for a while, and previously only Plex and Cams were working.. not sure what I changed to make Radarr work.. but everything else just gives me Error 503 Service Unavailable.. Even though the service is up and running and is accessible through LAN.

My opnsense is fully updated, 21.7.2_1 and os-haproxy 3.5

My HAProxy config is as follows:

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug
cache opnsense-haproxy-cache
    total-max-size 512
    max-age 60

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on localhost:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_613eabd9cb19a0.51810931 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_613eabd9cb19a0.51810931

# Frontend: 1_HTTPS_frontend (Listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/613eae5151edb0.32207081.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/613eac85c00a60.86291436.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: Radarr_backend ()
backend Radarr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Radarr_server 192.168.1.111:7878

# Backend: Plex_backend ()
backend Plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Plex_server 192.168.1.159:32400

# Backend: Cams_backend ()
backend Cams_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Cams_server 192.168.1.10:81

# Backend: Ombi_backend ()
backend Ombi_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Ombi_server 192.168.1.159:5055

# Backend: Sonarr_backend ()
backend Sonarr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Sonarr_server 192.168.1.111:8989

# Backend: Tautulli_backend ()
backend Tautulli_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Tautulli_server 192.168.1.7:8181

My map File currently looks like:

Code: [Select]
sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
Ombi Ombi_backend
Tautulli Tautulli_backend

Any advice as to what I'm doing wrong? Why would some services work and others don't when they're using the same exact config?

What's also funny is that if I reconfigure the port and IP on the one of the Real Servers that works (For example, Plex_Server has IP 192.168.1.159:32400, Ombi_server has 192.168.1.159:5055, and I replace Plex_Server port to 5055 and go to Plexi.DOMAIN.com, Ombi pops up like it should. I've cloned new Real servers and Backend Pools while updating the PUBLIC_SUBDOMAINS_map  from the working Plex one, but still no go for ALL of my services.

I've also discovered that if I modify HAProxy Rules & Checks > Rules > Public_subdomains_map-rule > Default backend pool... and change it to a service that DOESN'T work with the map file.. when I hit apply I'm able to access that service on ANY rendition of my domain, as well as the root domain.com address... And if I leave the services that DO work in the map file (Plex), plex.domain.com displays plex as it should, while the rest of the domain is showing the service that doesn't work on it's own.. which further doesn't make any sense.. The map file is working for some services but not others?

Thanks in advance, I'm stumped.

Final edit:

LMFAO I think I figured out why it wasn't working... so apparently for the map file to work you have to have the first part all lowercase, cannot use any uppercase...

So when I changed my map file to:

Code: [Select]
sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
ombi Ombi_backend
tautulli Tautulli_backend


everything started working...
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: budimanjojo on September 14, 2021, 08:56:11 am
Thank you so much for this tutorial! But I have a problem and I don't know what should I do.  :'(
Maybe somebody can help me here.

I already have a Traefik reverse proxy running outside OPNSense listening on 192.168.200.244 port 80 and 443. That Traefik reverse proxying all my kubernetes cluster services including certificates handling for 2 domain names.

Now, what I want to is to have HAProxy in OPNSense to be the reverse proxy for my Traefik. For example:
- My domain names are 1stdomain.com and 2nddomain.com.
- Have a rule that: if the client go to opnsense.1stdomain.com, route it to localhost:55443 (OPNSense itself), else if the client go to *.1stdomain.com or *.2nddomain.com, route it to 192.168.200.244:443.

I don't know where to start, I tested using the tuturial but instead of SNI and HTTP frontend, I created 2 SNI frontend services but HAProxy refused to start at all.

There is an easier route for me though, that is to just create a route in Traefik to my OPNSense and be done with it. But I prefer having a firewall level reverse proxy so I can have another layer to let's say block external access like in this tutorial. Thanks before.  :)
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 01:08:53 pm
@TheHellSite

I think the problem is with the SNI frontend. Here the SSL backend is specified as the default backend. He doesn't even look at the MAP file. he forwards everything to the SSL backend. When I set the openvpn backend as default Backend for a test in the SNI frontend, openvpn work but the other things not.

do you have an idea how I can solve this?
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 02:31:50 pm
@TheHellSite

I think the problem is with the SNI frontend. Here the SSL backend is specified as the default backend. He doesn't even look at the MAP file. he forwards everything to the SSL backend. When I set the openvpn backend as default Backend for a test in the SNI frontend, openvpn work but the other things not.

do you have an idea how I can solve this?
I was just about to write you exactly this!  ;D
Your reply confirmed my guess.

Looking through the manual pages of HAProxy it seems that the "Default Backend" setting can only be overwritten by a "Use Backend" rule! Which a "Use map file" rule isn't able to.
https://www.haproxy.com/de/blog/the-four-essential-sections-of-an-haproxy-configuration/
But I can't imagine that this is the intended behaviour.

Anyways... you simply need to create a VPN_condition "host starts with vpn" and a "use backend OPENVPN_backend if VPN_condition=true" rule.
Add this rule to the SNI_frontend and set the default backend back to the SSL_backend.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 05:35:30 pm
My map File currently looks like:

Code: [Select]
sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
Ombi Ombi_backend
Tautulli Tautulli_backend

Any advice as to what I'm doing wrong? Why would some services work and others don't when they're using the same exact config?

What's also funny is that if I reconfigure the port and IP on the one of the Real Servers that works (For example, Plex_Server has IP 192.168.1.159:32400, Ombi_server has 192.168.1.159:5055, and I replace Plex_Server port to 5055 and go to Plexi.DOMAIN.com, Ombi pops up like it should. I've cloned new Real servers and Backend Pools while updating the PUBLIC_SUBDOMAINS_map  from the working Plex one, but still no go for ALL of my services.

I've also discovered that if I modify HAProxy Rules & Checks > Rules > Public_subdomains_map-rule > Default backend pool... and change it to a service that DOESN'T work with the map file.. when I hit apply I'm able to access that service on ANY rendition of my domain, as well as the root domain.com address... And if I leave the services that DO work in the map file (Plex), plex.domain.com displays plex as it should, while the rest of the domain is showing the service that doesn't work on it's own.. which further doesn't make any sense.. The map file is working for some services but not others?

Thanks in advance, I'm stumped.

Final edit:

LMFAO I think I figured out why it wasn't working... so apparently for the map file to work you have to have the first part all lowercase, cannot use any uppercase...

So when I changed my map file to:

Code: [Select]
sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
ombi Ombi_backend
tautulli Tautulli_backend


everything started working...

Glad it is working for you know.
My first guess where some misconfigured real servers (ports, ssl, ssl-verify).

BTW: Your map file is exposing your domain name! You should remove it from the forum post.

Also I did a quick scan of your domain using https://dnsdumpster.com (https://dnsdumpster.com). It lists all your subdomains since you created a single "A Record" for each of them. Consider switching to a "Wildcard A Record" in order to hide them!
If an attacker can see what services you are running it makes it easier for them to find an attack surface.

You can then still create individual a records, f.e. www.domain.tld, since the wildcard a record is resolved after all other a records have been resolved.

This is why my tutorial is using a "Wildcard A Record / Subdomain" in the form of "*.domain.tld".
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 08:23:42 pm
Still the same  :(. Any other idea :)
(https://i.ibb.co/BnKQ9sr/rule.jpg) (https://ibb.co/XYkKqJ3)
(https://i.ibb.co/nf1gxv3/condition.jpg) (https://ibb.co/VDCjfZH)
(https://i.ibb.co/37kGNLD/SNI-Frontend.jpg) (https://ibb.co/DtLsCSx)
(https://i.ibb.co/PgLz2SH/Public-Service.jpg) (https://ibb.co/bX0R4f9)
(https://i.ibb.co/FB0pX4G/logfile.jpg) (https://ibb.co/rZc932n)
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 08:39:39 pm
Please remove the default backend but leave the vpn_rule enabled.

I have another guess...
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 08:49:09 pm
does not work too. i don't think he matched the names
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 08:57:24 pm
Invert the vpn condition and name it "NoVPN_condition".
Rename the vpn rule to "NoVPN_rule" and change it to use SSL_backend.

Test it out.
1. Only the NoVPN_rule on the SNI_frontend.
2. NoVPN_rule on the SNI_frontend along with SSL_backend as default.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 09:23:55 pm
Still the Same :)

Code: [Select]
2021-09-14T21:21:05 haproxy[3256] 80.187.80.8:10670 [14/Sep/2021:21:21:05.818] 0_SNI_frontend SSL_backend/SSL_server 1/0/37 0 -- 1/1/0/0/0 0/0
2021-09-14T21:21:05 haproxy[3256] 80.187.80.8:10670 [14/Sep/2021:21:21:05.818] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-14T21:21:01 haproxy[3256] 80.187.80.8:10495 [14/Sep/2021:21:21:01.658] 0_SNI_frontend SSL_backend/SSL_server 1/0/36 0 -- 1/1/0/0/0 0/0
2021-09-14T21:21:01 haproxy[3256] 80.187.80.8:10495 [14/Sep/2021:21:21:01.658] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 09:30:49 pm
For which try?
1st or second?

If you remove the VPN_backend as default backend and place the novpn_rule in the frontend. Can you still access your seafile Server?
It should be accessible. Is it?
I want to test the sni functionality with this.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 09:41:28 pm
Your first question:
For both.

The second:
Yes i can access the Seafile server with any config. The haproxy give every request to the ssl backend.


Gesendet von iPhone mit Tapatalk
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 09:43:05 pm
It looks like that sni don’t work by OpenVPN


Gesendet von iPhone mit Tapatalk
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nullex on September 15, 2021, 12:06:38 am
My map File currently looks like:

Code: [Select]
sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
Ombi Ombi_backend
Tautulli Tautulli_backend

Any advice as to what I'm doing wrong? Why would some services work and others don't when they're using the same exact config?

What's also funny is that if I reconfigure the port and IP on the one of the Real Servers that works (For example, Plex_Server has IP 192.168.1.159:32400, Ombi_server has 192.168.1.159:5055, and I replace Plex_Server port to 5055 and go to Plexi.DOMAIN.com, Ombi pops up like it should. I've cloned new Real servers and Backend Pools while updating the PUBLIC_SUBDOMAINS_map  from the working Plex one, but still no go for ALL of my services.

I've also discovered that if I modify HAProxy Rules & Checks > Rules > Public_subdomains_map-rule > Default backend pool... and change it to a service that DOESN'T work with the map file.. when I hit apply I'm able to access that service on ANY rendition of my domain, as well as the root domain.com address... And if I leave the services that DO work in the map file (Plex), plex.domain.com displays plex as it should, while the rest of the domain is showing the service that doesn't work on it's own.. which further doesn't make any sense.. The map file is working for some services but not others?

Thanks in advance, I'm stumped.

Final edit:

LMFAO I think I figured out why it wasn't working... so apparently for the map file to work you have to have the first part all lowercase, cannot use any uppercase...

So when I changed my map file to:

Code: [Select]
sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
ombi Ombi_backend
tautulli Tautulli_backend


everything started working...

Glad it is working for you know.
My first guess where some misconfigured real servers (ports, ssl, ssl-verify).

BTW: Your map file is exposing your domain name! You should remove it from the forum post.

Also I did a quick scan of your domain using https://dnsdumpster.com (https://dnsdumpster.com). It lists all your subdomains since you created a single "A Record" for each of them. Consider switching to a "Wildcard A Record" in order to hide them!
If an attacker can see what services you are running it makes it easier for them to find an attack surface.

You can then still create individual a records, f.e. www.domain.tld, since the wildcard a record is resolved after all other a records have been resolved.

This is why my tutorial is using a "Wildcard A Record / Subdomain" in the form of "*.domain.tld".

Thanks for catching that, thought I removed all PII but apparently not lol

So I think DNSDumpster is showing those domains because I was previously using dynamic DNS on EACH of those domains and using caddy as a reverse proxy.. which all those WERE direct A records, but since following your guide I switched to wildcard domain so hopefully it should be fixed once DNSDumpster's Database is updated.. I'll keep an eye on it. Thanks again!
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Prismatic on October 06, 2021, 09:35:05 pm
Hello. First of all, many thanks for your tutorial.... working like a charm, even for an (enlightened)  newbie like me....

Im using a Synology NAS + Docker  with different services. So they are available on same LAN IP adress, but different Ports....

I would llike to setup the "access from internatl network" as on your part 6. I do understand that with my setup I cannot use the unbound split DNS option, as this doesnt work with ports...

So I'm relying on option A "NAT reflexion", but I'm unable to find it, inside the created "HAproxy rule", in Firewall/WAN section..... I do find this option in Firewall/NAT section rules, but we didnt created nothing there....

Could you please help ? and update your tutorial with a screenshot of this ?

Many thanks !


Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 06, 2021, 11:17:36 pm
Im using a Synology NAS + Docker  with different services. So they are available on same LAN IP adress, but different Ports....

I would llike to setup the "access from internatl network" as on your part 6. I do understand that with my setup I cannot use the unbound split DNS option, as this doesnt work with ports...

Of course you can use split DNS!
haproxy is handling the port scenario.

Example:
10.1.23.55:4456 = NEXTCLOUD = cloud.yourdomain.tld
10.1.23.55:4457 = PLEX = plex.yourdomain.tld

Read the Split dns part again!
You have to rewrite ALL of your 1st/2nd-level-subdomains with same IP of your OPNsense that HAProxy is listening on, f.e. the lan ip.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Prismatic on October 07, 2021, 12:55:24 am
Thanks for your answer !

Apologies, but I still don't understand something....

So my services IP & port are configured inHAproxy map file. (everything working fine, comming from WAN with certificate)..
.
this server IP for me on LANis 192.168.1.20
only using 1st level subdomains like:

audio.mydomain.tld
photo.mydomain.tld
video.mydomain.tld
etc...

My "LAN" port for opnsense on appliance is 192.168.1.1 and I did used a "virtual IP" as per your tutorial which is 192.168.50.1...

Using "hosts overrides in unbound" (not "domain") should my configuration look like this :

HOST : audio
DOMAIN: mydomain.tld
IP : 192.168.50.1 or 192.168.1.1  or 192.168.1.20 ??

tried all solutions, but I don't know if it works.... a "tracert" shows that I am always hitting my static public IP adress....

sorry for my approximative english and many thanks for your answer.
cheers !


 


Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 08, 2021, 08:46:00 am
So my services IP & port are configured inHAproxy map file. (everything working fine, comming from WAN with certificate)..
.
this server IP for me on LANis 192.168.1.20
only using 1st level subdomains like:

audio.mydomain.tld
photo.mydomain.tld
video.mydomain.tld
etc...

My "LAN" port for opnsense on appliance is 192.168.1.1 and I did used a "virtual IP" as per your tutorial which is 192.168.50.1...

Using "hosts overrides in unbound" (not "domain") should my configuration look like this :

HOST : audio
DOMAIN: mydomain.tld
IP: 192.168.1.1

IP has to be the lan ip of your opnsense.
Which I very well tell to do in my tutorial.
You should probably read it word by word again!

Also post your HAProxy config export in a code box.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 08, 2021, 05:45:47 pm
Hey all,

Is there a way to tie this in with Cloudflare?
I know ACME can use Cloudflare, but can someone add in where I need to do this for wildcards?

Also I installed the plugins and I don't see LetsEncrypt, rather I see ACME Client, and the settings are a bit different.
Any chance you can update the images?

Also I've tried to follow this to the best of my abilities.
I am getting: 503 Service Unavailable
No server is available to handle this request.

If I click on the padlock it is showing my certificate so I think its hitting right.
I'm doing things for:
homeassistant
jellyfin
radarr
sonarr

All of them say 503.

If I set a backend to maintenance I get Safari can't find the server.
So seems to be hitting it correctly.

Ok not sure what changed but now its no dice. Just server cannot be found
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 08, 2021, 11:45:41 pm
Sorry but I can't help with individual domain / dns registrars, this would simply take way to much of my time. You will have to study cloudflares documentation on your own to make it work with them.
Only difference should be the Dynamic DNS and ACME configuration, to get your certificate.
HAProxy settings are the same for every registrar.

The Let's Encrypt plugin has been renamed to ACME Client in one of the recent OPNsense updates. Apart from that nothing else has changed so all pictures are still valid. No need to change them.
I will update the text when I find the time.

Your error: Post your HAProxy export in a code box. Remove sensitive information (Public IP, Domain Name).
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 09, 2021, 10:46:19 am
Thank for taking the time to help.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_Frontend ()
frontend SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTP_Frontend ()
frontend HTTP_Frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: No_SSL_condition
    acl acl_6160768c129757.05678189 req.ssl_ver gt 0

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_6160768c129757.05678189

# Frontend: HTTPS_Frontend ()
frontend HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6160790e8358e2.93807756.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: Public_Subdomain_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61607700a205d2.15896401.txt)]

# Backend: SSL_backend (HAProxy SSL Backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: homeassistant_backend (Home Assistant Backend)
backend homeassistant_backend
    # health check: homeassistant_tcp_check
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server homeassistant 10.0.0.9:8123 check inter 2s port 8123  ssl verify none

# Backend: jellyfin_backend (Jellyfin Backend)
backend jellyfin_backend
    # health check: jellyfin_http_check
    option httpchk
    http-check send meth OPTIONS uri / ver HTTP/1.0
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server jellyfin 10.0.0.10:8090 check inter 2s port 8090  ssl verify none

The only difference I think I made was to add health checks so I can see if service is UP.
It is showing as UP so that looks good.

And my MAP looks like:
Code: [Select]
# public access subdomains
homeassistant homeassistant_backend
jellyfin jellyfin_backend

Oh one other thing to note is I have dual WAN setup.
I had this on pfSense too and HAProxy was working perfectly fine.
I just had all the external connections go through 1 WAN only which I did again here.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 09, 2021, 02:57:49 pm
Please confirm you are accessing your servers internally using:

http://10.0.0.10:8090/
http://10.0.0.9:8123/
So no https?

Even though it shouldn't matter test the access with health checks disabled.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 09, 2021, 03:17:58 pm
Please confirm you are accessing your servers internally using:

http://10.0.0.10:8090/
http://10.0.0.9:8123/
So no https?

Even though it shouldn't matter test the access with health checks disabled.

Correct if I hit those 2 IP's above internally I get the service correctly.
I did have it running with health checks disabled and same result 503 errors
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 09, 2021, 03:39:32 pm
Is opnsense the only router / firewall in your network?
To me it looks like your servers are in a totally different subnet / vlan.

Please give a little info about your local network layout.
I think your opnsense has no access to your server network!
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 09, 2021, 04:07:09 pm
Yes OPNsense is my only router/firewall.

I have dual WAN created a gateway group to double internet speed.
I created the rule on only 1 WAN gateway tho, and my a records point to the same IP.

I have 3 LANs (not VLANS) 10.0.0.1 (Main LAN) - 192.168.107.1 (IoT) - 192.168.200.1 (Guest)
My NAS Server (10.0.0.10) and OPNsense (10.0.0.1) are on 10.0.0.1/24 LAN, so no going through anything different there.

I wonder if with the dual WAN it needs a specific rule?
As with dual WAN I have to change the default gateway on the Allow LAN to any rule to the specific Multi_Gateway...
Like my LAN can no longer talk to my IOT Lan without having a specific rule to say allow...

I've just added the LAN Net to Loopback but still no dice
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 09, 2021, 05:08:43 pm
The error is not related to your dual WAN setup.  ;D
I misread something in your HAProxy config...

Code: [Select]
server jellyfin 10.0.0.10:8090 check inter 2s port 8090  ssl verify noneYou see that "ssl" word in your server config?
SSL and HTTP don't like each other!  ;)

Since your services are NOT using HTTPS but only HTTP locally... you need to UNCHECK the SSL checkbox in the HAProxy real server config for both of your servers.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 09, 2021, 05:58:49 pm
The error is not related to your dual WAN setup.  ;D
I misread something in your HAProxy config...

Code: [Select]
server jellyfin 10.0.0.10:8090 check inter 2s port 8090  ssl verify noneYou see that "ssl" word in your server config?
SSL and HTTP don't like each other!  ;)

Since your services are NOT using HTTPS but only HTTP locally... you need to UNCHECK the SSL checkbox in the HAProxy real server config for both of your servers.

Oh my days!!!
That worked! I spent all night trying to figure it out.

Thank you so very much!
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: blackhand27 on October 15, 2021, 06:06:51 am
Great Guide! Took me a bit since im using Godaddy for my domain name but figured out the cert mess and got my plex server working. Been hoping to get this setup so I can run a Synapse server at home.

I do seem to be having issues with my bitwarden site, I have the proper port set in HAProxy however I have a feeling its something with its configuration still using the internal cert vs the offload. Would anyone have some advice to see what I can look for?

figured it out, I reinstalled the server using self signed certs, for whatever reason it was still using the builtin lets encrypt certs and failing to connect.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on October 15, 2021, 07:25:02 pm
@TheHellSite

I think the problem is with the SNI frontend. Here the SSL backend is specified as the default backend. He doesn't even look at the MAP file. he forwards everything to the SSL backend. When I set the openvpn backend as default Backend for a test in the SNI frontend, openvpn work but the other things not.

do you have an idea how I can solve this?
I was just about to write you exactly this!  ;D
Your reply confirmed my guess.

Looking through the manual pages of HAProxy it seems that the "Default Backend" setting can only be overwritten by a "Use Backend" rule! Which a "Use map file" rule isn't able to.
https://www.haproxy.com/de/blog/the-four-essential-sections-of-an-haproxy-configuration/
But I can't imagine that this is the intended behaviour.

Anyways... you simply need to create a VPN_condition "host starts with vpn" and a "use backend OPENVPN_backend if VPN_condition=true" rule.
Add this rule to the SNI_frontend and set the default backend back to the SSL_backend.

Am I confused about this?

It looks like the current set up on page one will not work due to "default backend"??

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 19, 2021, 06:06:55 pm
I was just about to write you exactly this!  ;D
Your reply confirmed my guess.

Looking through the manual pages of HAProxy it seems that the "Default Backend" setting can only be overwritten by a "Use Backend" rule! Which a "Use map file" rule isn't able to.
https://www.haproxy.com/de/blog/the-four-essential-sections-of-an-haproxy-configuration/
But I can't imagine that this is the intended behaviour.

Anyways... you simply need to create a VPN_condition "host starts with vpn" and a "use backend OPENVPN_backend if VPN_condition=true" rule.
Add this rule to the SNI_frontend and set the default backend back to the SSL_backend.

Am I confused about this?

It looks like the current set up on page one will not work due to "default backend"??

I am not sure what you mean?
The setup still works as described. I also got OpenVPN over TCP working but haven't had the time yet to add it to my tutorial.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: thegriffen on October 27, 2021, 12:37:59 am
Hi everyone

First of all.. Awesome guide and even so, that you update it with new stuff as it comes along.

Iv been kinda hitting my head for a day or two now and gotta throw in the towel, and put my troubles in here.

im running a small test setup, where im gonna have a couple of web services running, but the thing is here, they are running traefik with there own LE ssl validation and so. but i cant get HAproxy to work propper.

My config looking like this:

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend ()
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:8888 name 0.0.0.0:8888
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options
    option tcplog

    # ACTION: Public_Domain_Map_Rule_2
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6178606dd3b431.48344137.txt)]
    # ACTION: Public_Domain_Map_Rule_3
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6178704b1d59f6.07557436.txt)]

# Frontend: 1_Http_Frontend ()
frontend 1_Http_Frontend
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: NoSSL_Condition
    acl acl_61781026415d99.64392811 req.ssl_ver gt 0

    # ACTION: HttpToHttps_Rule
    http-request redirect scheme https code 301 if !acl_61781026415d99.64392811

# Frontend: 1_Https_Frontend ()
frontend 1_Https_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    option httplog

    # ACTION: Public_Domain_Map_Rule_1
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/617812015049f9.00974692.txt)]

# Backend: SSL_Backend (Loopback)
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: Central_OpenSense_FW_Backend (OpenSense Firewal)
backend Central_OpenSense_FW_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Central_OpenSense_FW_SRV 10.10.5.2:55443 ssl verify none

# Backend: FRCD_Backend (domain1.com)
backend FRCD_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server DMZ_domain1_SRV 10.10.20.40:443 ssl alpn h2,http/1.1,http/1.0 verify none

# Backend: FRCD_2_Backend (domain2:8888)
backend FRCD_2_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server DMZ_domain2_SRV 10.10.20.41:8888

Best regrads
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 31, 2021, 10:44:49 am
Hi everyone

First of all.. Awesome guide and even so, that you update it with new stuff as it comes along.

Iv been kinda hitting my head for a day or two now and gotta throw in the towel, and put my troubles in here.

im running a small test setup, where im gonna have a couple of web services running, but the thing is here, they are running traefik with there own LE ssl validation and so. but i cant get HAproxy to work propper.

I don't really understand what you need haproxy for if you are already using traefik?
Can you please elaborate further what...

1. You are trying to achieve by using HAProxy?

2. What issue you are facing with HAProxy right now?

3. Explain your local network layout further!
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bringha on November 04, 2021, 10:46:42 am
Hi there,

first of all a big big thank you for this awesome and comprehensive tutorial. Very helpfull and a great contribution.

I have an additional question and I am not sure whether I suffer from a big misuderstanding.

I configured my Dyndns as suggested with dedyn.io and have now a domain.dedyn.io properly working. Your tutorial now assumes to create wildcard certificates for the *.domain.dedyn.io (in my case)

I have a main domain registered with a poster somewhere else which is domain.com. Historically I reach my dyndns based subdomains via CNAME DNS entries at my main domain provider's DNS systems, eg home.domain.com points then to home.domain.dedyn.io.

It is now possible to let the acme client generate wildcard certificates also for *.domain.com accordingly in addition/replacing the wildcard certs for *.domain.dedyn.io?

Looking forward to your reply.

Br br
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 04, 2021, 07:15:20 pm
I configured my Dyndns as suggested with dedyn.io and have now a domain.dedyn.io properly working. Your tutorial now assumes to create wildcard certificates for the *.domain.dedyn.io (in my case)

I have a main domain registered with a poster somewhere else which is domain.com. Historically I reach my dyndns based subdomains via CNAME DNS entries at my main domain provider's DNS systems, eg home.domain.com points then to home.domain.dedyn.io.

It is now possible to let the acme client generate wildcard certificates also for *.domain.com accordingly in addition/replacing the wildcard certs for *.domain.dedyn.io?

Basically there are two options to solve this

1. You replace "deSEC" as your DynDNS provider and use the DynDNS of your main domain registrar. If he supports this...
--> What is your domain registrar? (Where your real "domain.com" is registered...)

2. You transfer the DNS zone from your main "domain.com" to deSEC. Again, if your current domain registrar supports this.
This way your domain is still hosted at your current domain registrar, but deSEC will be managing all the DNS related stuff. (A records, ...)
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bringha on November 04, 2021, 07:44:54 pm
Thank you for your answer!

My domain Registrar is Ionos. Ionos has also a dyndns service and a dns management api and is contained in the opnsense dyndns list. I would like to leave the services of the main Domain there (mail, Webhosting) and have only the subdomains with my local dyndns based site.

Br br
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 04, 2021, 07:59:11 pm
IONOS works exactly like deSEC, just adopt the deSEC stuff to the IONOS settings.
get a dns API key, set up dyn dns and get your certs.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on November 05, 2021, 09:29:43 pm
Hey @TheHellSite, Do you know a way I can add Basic Auth to one of the sub domains?

I have a dashboard which just runs without login and I would really like it secured behind just basic auth if possible.
I tried creating a condition and a rule and applying it to the sub domain but it doesnt seem to work.

I also saw on the backend there is an option on the domain for basic auth checkbox, so I tried that but no dice either.

Any chance you can add a section to the guide about securing singular domains with basic auth?
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 06, 2021, 11:00:20 am
Hey @TheHellSite, Do you know a way I can add Basic Auth to one of the sub domains?

I have a dashboard which just runs without login and I would really like it secured behind just basic auth if possible.
I tried creating a condition and a rule and applying it to the sub domain but it doesnt seem to work.

I also saw on the backend there is an option on the domain for basic auth checkbox, so I tried that but no dice either.

Any chance you can add a section to the guide about securing singular domains with basic auth?

Definitely will be looking into this. You can expect this in the coming weeks, hopefully.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on November 06, 2021, 11:29:59 am
AMAZING! Thank you so much
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bringha on November 06, 2021, 10:15:27 pm
Hi there,

I tried to activate DDNS via the API with ionos as suggested but miserably failed yet.  ??? Might be somewhat unusual but I only want to activate DDNS for two subdomains, not my entire domain ‚domain.com‘ (only ‚sub.domain.com‘).

 I come up to the step that I have created my update URL, the server responds with 200 properly on the Ionos API page. This should then have activated DDNS for the requested subdomains according to the doc. However there is the no entry for these subdomain on the DNS page.

When I  apply Update URL  in the opnsense dyndns config and press ‚save and enforce update‘, my public IP is properly shown in the opnsense dyndns config page.  But still the subdomain is not shown as a DynDNS subdomain at ionos ..
as to be feared there is also no DNS with my public IP for the subdomain distributed …

Do i miss a step ?? I am looking forward to any idea.

Br br
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on November 08, 2021, 10:13:16 am
FYI I got it working the Basic Auth.

So 21.7.4 has an issue with HAProxy where changes to config are not being saved: https://forum.opnsense.org/index.php?topic=25480.0
After I applied the patch it is working.

For Basic Auth all I did was create a user under User Management
(There does seem to be some restrictions around password length or complexity, I didnt spend a load of time testing)
My 20+ Char passwords would not work until I dumbed it down a bit.

So once you have a user go to your Backend Pools, choose your desired service, scroll down to Basic Authentication -> tick the box and add the username you just created.

Boom it works.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 10, 2021, 03:33:49 pm
Hi there,

I tried to activate DDNS via the API with ionos as suggested but miserably failed yet.  ??? Might be somewhat unusual but I only want to activate DDNS for two subdomains, not my entire domain ‚domain.com‘ (only ‚sub.domain.com‘).

 I come up to the step that I have created my update URL, the server responds with 200 properly on the Ionos API page. This should then have activated DDNS for the requested subdomains according to the doc. However there is the no entry for these subdomain on the DNS page.

When I  apply Update URL  in the opnsense dyndns config and press ‚save and enforce update‘, my public IP is properly shown in the opnsense dyndns config page.  But still the subdomain is not shown as a DynDNS subdomain at ionos ..
as to be feared there is also no DNS with my public IP for the subdomain distributed …

Do i miss a step ?? I am looking forward to any idea.

Br br

Your issue is related to DynDNS I think you should fix that first and ask for help at the right place.
Maybe use google "how to set up a DynDNS subdomain at IONOS".
https://www.ionos.de/hilfe/domains/ip-adresse-konfigurieren/dynamisches-dns-ddns-einrichten-bei-company-name/

I am sorry, but since I am not using IONOS you will have to figure out their specifics on your own.
The only differences to my guide will be the DynDNS setup and the DNS challenge specifics for the ACME Client. Everything else is the same.



FYI I got it working the Basic Auth.

So 21.7.4 has an issue with HAProxy where changes to config are not being saved: https://forum.opnsense.org/index.php?topic=25480.0
After I applied the patch it is working.

For Basic Auth all I did was create a user under User Management
(There does seem to be some restrictions around password length or complexity, I didnt spend a load of time testing)
My 20+ Char passwords would not work until I dumbed it down a bit.

So once you have a user go to your Backend Pools, choose your desired service, scroll down to Basic Authentication -> tick the box and add the username you just created.

Boom it works.

Just tested it out myself. Basic Auth is so easy to set up that I am not really willing to cover it in this guide.
First create the user(s) in HAProxy. Then in the relevant backends activate basic auth and select the user(s).

The issue you are facing with your password is the way you included them in your config!
You will need to SHOULD use a password hash not the plain text password in the user management box.
Otherwise your passwords are visible in plain text in the config export.

Generate password hash using OpenSSL binary
1. (WINDOWS ONLY) Download the openssl binary, f.e. from here: http://wiki.overbyte.eu/wiki/index.php/ICS_Download#Download_OpenSSL_Binaries_.28required_for_SSL-enabled_components.29
2. (Windows) Open CMD in the directory of the binary file / (Linux) open the shell.
3. Run "openssl passwd -6" and enter your password.
4. Enter the generated password hash in the password field of the user in HAProxy Settings.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 12, 2021, 04:09:12 pm
I am having issues with modifying config which will not take effect.

as an example when attempting to modify default-dh-param from 2048 to 4096, the save and apply works but the config export is not updated.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 15, 2021, 04:41:55 pm
It was fixed after the latest update, however, I am still having connections issues... I am getting handshake failures.
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin expose-fd listeners
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend (listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_Frontend (Listening on 192.168.1.50:80)
frontend 1_HTTP_Frontend
    bind 192.168.1.50:80 name 192.168.1.50:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NOSSL_Condition
    acl acl_60f9d6d0118252.11362730 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_Rule
    http-request redirect scheme https code 301 if !acl_60f9d6d0118252.11362730

# Frontend: 1_HTTPS_Frontend (listening on 192.168.1.50:443)
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.1.50:443 name 192.168.1.50:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60f9db5421ce96.24863488.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: Public_Sub_MapRule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61698448328ff6.66158166.txt)]

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server HAP_VIP 192.168.1.50 send-proxy-v2 check-send-proxy

# Backend: TruePlex_Backend ()
backend TruePlex_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server TruePlex 192.168.1.12:32400 ssl verify none
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 16, 2021, 10:01:22 am
It was fixed after the latest update, however, I am still having connections issues... I am getting handshake failures.

From all client devices or just your smart TV(s)?
Does web browser access work?
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 16, 2021, 11:41:05 am
From all clients, nothing works.

I can connect to the plex with the local IP without an issue.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 16, 2021, 12:44:25 pm
From all clients, nothing works.

I can connect to the plex with the local IP without an issue.

1. Post a screenshot of your entire SSL Labs result. But black out any personal information from it, i.e. domain name, public IP,... You can also send me a personal message with the result link.
https://www.ssllabs.com/ssltest/


2. Go to "OPNsense --> System --> Trust --> Authorities" and post a screenshot of that page. If you want, you can black out all lines that don't have "Let's Encrypt" the name.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 16, 2021, 01:47:20 pm
There's an unknown cert and Cloudflare. I'll sent it in a bit.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 16, 2021, 02:26:15 pm
I thought you were asking about an HAProxy + Let's Encrypt issue...

Since you are using cloudflare certificates I am unable to help you. You are better off asking for help in the HAProxy forums or the cloudflare support regarding your issues.

Let me finish by giving you these informations:
1. The SSL Labs test pictures you sent me indicate that your certificate content (cn + alt name) seems to be wrong.
2. Your HAProxy HTTPS frontend settings do not match the ones I provide in my guide.


You could however just follow my tutorial step by step and would end up with a working setup. ;)
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on November 17, 2021, 03:04:01 pm
Thanks TheHellSite for the time taken to share the setup and help people.
This is my first foray into HA because I wanted to open traffic to a webserver on my lan for the first time. First I was going to just forward the port on OPN but then I saw your post and realised we can get so much more functionality from HA and your setup suggestion.

I have setup my webserver (nginix) to listen on custom port say 8082, and there is where I got the letsencrypt & dedyn.io setup first, as a kid of test before moving to OPN.
I've now done the cert as per your guide, essentially moved cert infrastrucure from the real server to OPN. All is good.

I've done the "pre-requisites" in my mind. I'm getting ready to string it together and then my question if I may.

Part 4, step 1. It says to change OPN Admin interface to another port(s) for http and https, usual 80, 443 because HA would expect traffic there.
Is it obligatory or could I keep my real server on 8082 and setup HA to exect that and keep OPN as is on 80 & 443?
I expect to use https with a self signed cert on the real server -instead of the current cert now in OPN-, or maybe only http on that port at the beginning.
Thanks.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 17, 2021, 07:16:00 pm
I thought you were asking about an HAProxy + Let's Encrypt issue...

Since you are using cloudflare certificates I am unable to help you. You are better off asking for help in the HAProxy forums or the cloudflare support regarding your issues.

Let me finish by giving you these informations:
1. The SSL Labs test pictures you sent me indicate that your certificate content (cn + alt name) seems to be wrong.
2. Your HAProxy HTTPS frontend settings do not match the ones I provide in my guide.


You could however just follow my tutorial step by step and would end up with a working setup. ;)

I am not sure what I am missing... I went over the HA HTTPS Frontend a dozen times and I am not seeing what's not matching... :(
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 18, 2021, 11:08:11 am
Part 4, step 1. It says to change OPN Admin interface to another port(s) for http and https, usual 80, 443 because HA would expect traffic there.
Is it obligatory or could I keep my real server on 8082 and setup HA to exect that and keep OPN as is on 80 & 443?
I expect to use https with a self signed cert on the real server -instead of the current cert now in OPN-, or maybe only http on that port at the beginning.
Thanks.

You can run your real servers (f.e. nginx web server) on whatever port you like (80, 443, 50443, ......) that doesn't matter at all.
The reason why you have to change the OPNsense web server port is quite simple.

OPNsense is obviously running a built in webserver by default (LAN INTERFACE / PORT 443), otherwise you wouldn't be able to access the web interface at all.
HAProxy is listening on ALL INTERFACES by default.

Web Server = Interface = IP:Port
HAProxy = ALL = 0.0.0.0:0 or 0.0.0.0:80+443
OPNsense = LAN = 192.168.1.1:443

This is the conflict! You can't have two or more services listening on the same "Interface + IP:Port". How would they know which traffic belongs to whom? As soon as one of them grabs the traffic the other one will never get it.

TL;DR:
1. Yes, you can run your real servers on any port (f.e. 8082).
2. No you can't change the OPNsense back to port 443 because you wouldn't be able to reach the OPNsense web interface anymore and or HAProxy will refuse to start.
3. I strongly advise you to also run your real server(s) with a self-signed SSL certificate to increase security. It is however not necessary.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 18, 2021, 11:20:00 am
I am not sure what I am missing... I went over the HA HTTPS Frontend a dozen times and I am not seeing what's not matching... :(

As I already said, it has something to do with your cloudflare domain config!
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on November 18, 2021, 12:46:35 pm

Web Server = Interface = IP:Port
HAProxy = ALL = 0.0.0.0:0 or 0.0.0.0:80+443
OPNsense = LAN = 192.168.1.1:443

This is the conflict! You can't have two or more services listening on the same "Interface + IP:Port". How would they know which traffic belongs to whom? As soon as one of them grabs the traffic the other one will never get it.

TL;DR:
1. Yes, you can run your real servers on any port (f.e. 8082).
2. No you can't change the OPNsense back to port 443 because you wouldn't be able to reach the OPNsense web interface anymore and or HAProxy will refuse to start.
3. I strongly advise you to also run your real server(s) with a self-signed SSL certificate to increase security. It is however not necessary.
Thank you for taking the time to answer. The problem with conflicting ip+ports is one I understand but I failed to articulate my question to explain what I meant.
I have been following the guide step by step, back and forth to understand each part. I made a little diagram for me to understand it better. I hope you don't mind if I spoil your thread with it. It just works for me to understand and better ask a question.
(https://i.ibb.co/WNXCxbb/HAP-understanding.png)
Essentially my question was if I could bind the SNI front end internally to a custom port instead of the usual 80,443. I probably should think again if what I'm asking is a sane question.
I've used this diagram and read&re-read your guide and I think I have a better grasp on it.
The nginx in my picture is me trying to figure out where/how I need to do the setup for my use. I hope I have not misrepresented it.
Let me come back with a better question when I've tested things. Much obliged.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 18, 2021, 04:00:00 pm
(https://i.ibb.co/WNXCxbb/HAP-understanding.png)
Essentially my question was if I could bind the SNI front end internally to a custom port instead of the usual 80,443.

First off all: B in your picture. The HTTP frontend shouldn't have the SSL_backend as the backend!!! Instead you only place the HTTPtoHTTPS redirect rule on it and if you need it, the HTTP services that don't require SSL.
You are also missing the PLEX_backend containing the PLEX_server.


Short Answer: No, because then you would have to access your services using that custom port in addition to having a subdomain for them. Which is pretty pointless, imho.

Example with default http/https port = http(s)://plex.yourdomain.tld/
Example with custom Port = 1234 = http(s)://plex.yourdomain.tld:1234/


Long Answer: It might be possible if you create a port forward in your OPNsense that forwards any WAN traffic that hits on TCP 80/443 to 127.0.0.1:1234.



My question to you: Why do you want't to do this in the first place? It makes this setup even more complex, for no reason...
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on November 18, 2021, 08:26:40 pm
Thank you for the clarifications. They help. I'm revising it now.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 11:05:05 am
Hello,

when I started implementing HAProxy in my network I couldn't find any complete and well written guide out there. I had to puzzle everything together from various websites.
So I thought I would save many of you a lot of time and provide my ultimate HAProxy on OPNsense guide.  :)

This tutorial will show you how to configure HAProxy as a reverse proxy on OPNsense using wildcard certificates from Let's Encrypt.
It is going to be a step-by-step guide with images on how to set things up while also explaining why we set things up in a certain way.
I will try to make this as complete and detailed as possible.
If you think that there is anything wrong or missing, feel free to tell me about it and I will consider changing it.

If this guide was helpful to you then please leave me a thanks down below as it took me several days to write this down.

Kind Regards
TheHellSite

This guide was a life-saver. And while implementing, I finally had the chance to understand a little more how HAProxy works. Thank you so much for writing it.

Everything worked as documented, and in a couple of hours I was up and running, the only problem was getting 100% on the ssl quality test. The Cipher Strength was always 90%, so I got A+, with a 90% overall score.

After a couple of hours of tinkering with the ciphers, I figured out that the test doesn't like 128bit ciphers. So I removed those from the Cipher List (TLSv1.2) and Cipher Suites (TLSv1.3) of the HTTPS_frontend, and I finally got the A+ 100% score.

In case anyone faces the same issue, these are the cipher settings for HTTPS_frontend:

Quote
Cipher List: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305

Cipher Suites: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

Thanks again,

Alessandro
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 23, 2021, 11:32:25 am
After a couple of hours of tinkering with the ciphers, I figured out that the test doesn't like 128bit ciphers. So I removed those from the Cipher List (TLSv1.2) and Cipher Suites (TLSv1.3) of the HTTPS_frontend, and I finally got the A+ 100% score.

In case anyone faces the same issue, these are the cipher settings for HTTPS_frontend:

Quote
Cipher List: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305

Cipher Suites: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

Thanks again,

Alessandro

Happy to hear it is working for you.
I think you missed the part of my tutorial where I am giving needed ciphers and cipher suites to get an 100% A+ rating.

Quote
My Tutorial
Cipher List: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384
Cipher Suites: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

yours
Cipher List: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305
Cipher Suites: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

The only difference between my guide and your list is the "DHE-RSA-AES256-GCM-SHA384" cipher.
Both will be scoring 100 % A+ while mine offer even more client compatibility.

Seems like I forgot to mention it again during tutorial at the part where the HTTPS_frontend is created. I now put a reference in there pointing to the beginning of my post where I provide the current best cipher list and cipher suits.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 11:54:24 am
Happy to hear it is working for you.
I think you missed the part of my tutorial where I am giving needed ciphers and cipher suites to get an 100% A+ rating.

Indeed I missed it, even though I followed the guide with great attention. Problem is that it was not in the tutorial, but at the beginning of the post. Sorry about that, I paid attention mainly to the tutorial. Would have saved a lot of time tinkering with the ciphers. :)

Quote
The only difference between my guide and your list is the "DHE-RSA-AES256-GCM-SHA384" cipher.
Both will be scoring 100 % A+ while mine offer even more client compatibility.

Yes I noticed that single difference. I started from the default ciphers of HAProxy, and that one wasn't in there. I simply removed the AES128 ones from the defaults. The link you put to that  Mozilla Config is fantastic, never even thought something like that existed. Thank you again.

Quote
Seems like I forgot to mention it again during tutorial at the part where the HTTPS_frontend is created. I now put a reference in there pointing to the beginning of my post where I provide the current best cipher list and cipher suits.

That's good, because I missed it completely, would've saved me quite some time. My bad. :)

One thing I wanted to ask you: I followed your naming conventions and I noticed you had the 1_ prefix both for the HTTP and HTTPS frontend. I renamed the HTTPS to 2_HTTPS_frontend, don't know if it was intentional or not, but I interpreted it as a progressive number so that one was a 2.


Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 23, 2021, 11:57:42 am
One thing I wanted to ask you: I followed your naming conventions and I noticed you had the 1_ prefix both for the HTTP and HTTPS frontend. I renamed the HTTPS to 2_HTTPS_frontend, don't know if it was intentional or not, but I interpreted it as a progressive number so that one was a 2.

I did that on purpose to express the "level" of reverse proxying.

Level 1 - SNI traffic
Level 2 - HTTP + HTTPS traffic

However you can name it as you like. It doesn't matter in terms of functionality.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 12:01:57 pm
I did that on purpose to express the "level" of reverse proxying.

Level 1 - SNI traffic
Level 2 - HTTP + HTTPS traffic

However you can name it as you like. It doesn't matter in terms of functionality.

Thanks for the explanation.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 12:30:11 pm
Seems like I forgot to mention it again during tutorial at the part where the HTTPS_frontend is created. I now put a reference in there pointing to the beginning of my post where I provide the current best cipher list and cipher suits.

I moved the frontend cipher settings and bind options to the Global Parameters -> SSL Default Settings section, emptied the relevant fields in HTTPS_frontend, it works fine and I think it's simpler:

(https://i.imgur.com/6sqazDa.png)

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 23, 2021, 12:43:27 pm
Arguable! ;)
If you have many HTTPS frontends (on different ports) that might need different SSL settings then my way is better.

Otherwise it doesn't really matter where you put the settings. Just note that the SSL default settings get overwritten once you set anything in the associated boxes on the HTTPS frontends.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 12:49:39 pm
Arguable! ;)
If you have many HTTPS frontends (on different ports) that might need different SSL settings then my way is better.

Otherwise it doesn't really matter where you put the settings. Just note that the SSL default settings get overwritten once you set anything in the associated boxes on the HTTPS frontends.

Your second point is the answer to your first observation: Global settings are a "global default". If you need a custom config for a frontend, you can customize its config and it will supersede the global one. ;)

That's why I like it, it's the classic enterprise approach to organizing things, more intuitive for me (global...and custom as an exception).

But like you said...it's all very subjective obviously. Individual preferences.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: dima1002 on November 28, 2021, 09:01:48 am
Hello,

what is wrong here?

In Firefox I get the following error: PR_END_OF_FILE_ERROR

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log audit debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): LetsEncrypt_443 ()

# Frontend (DISABLED): LetsEncrypt_80 ()

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_61a24897421141.86617043 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_61a24897421141.86617043

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61a24a78aa9cc4.11915455.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61a249350142e3.01879320.txt)]

# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Backend: Mail ()
backend Mail
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Main 192.168.111.2:443 ssl verify none

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend: Nextcloud ()
backend Nextcloud
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Nextcloud 192.168.111.3:443 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: Bitwarden_Backend ()
backend Bitwarden_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Vaultwarden80 192.168.111.77:80
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 28, 2021, 11:16:09 am
Hello,

what is wrong here?

In Firefox I get the following error: PR_END_OF_FILE_ERROR

Are you only getting this error when using Firefox?
If so, which version of Firefox are you running?
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: SOUK on December 04, 2021, 03:56:42 pm
Delete
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: evathesalmon on December 12, 2021, 11:41:33 pm

In Firefox I get the following error: PR_END_OF_FILE_ERROR


Hello,

I have same issue in firefox and chrome. In logs it is said "Received something which does not look like a PROXY protocol header"

(https://i.imgur.com/sjyyqey.png)

I thought at first that it is a proxy problem and double checked your 20210613 update but my bind option pass-through set as accept-proxy

(https://i.imgur.com/tkMNypC.png)

(https://i.imgur.com/HrWTwSa.png)

My current HAProxy version is 3.7 and opnsense is 21.7.6
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on December 14, 2021, 09:27:56 am
In Part 6, NAT Reflection: it applies to port forwarding rules, but in the guide you switched to a simple filter rule.

So there's only one option remaining: split DNS.

HAProxy has been rock solid, thanks again for your guide. I'm having a hard time only for Uptime Kuma, it uses websockets, and it's the only service that doesn't work behind HAProxy. The dev published a guide for the configuration behind several reverse proxies, unfortunately the only one missing is HAProxy: https://github.com/louislam/uptime-kuma/wiki/Reverse-Proxy#nginx

I'm sure there's a way to make it work but I can't find it...
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 14, 2021, 10:35:36 am

In Firefox I get the following error: PR_END_OF_FILE_ERROR


Hello,

I have same issue in firefox and chrome. In logs it is said "Received something which does not look like a PROXY protocol header"

I thought at first that it is a proxy problem and double checked your 20210613 update but my bind option pass-through set as accept-proxy

My current HAProxy version is 3.7 and opnsense is 21.7.6

I am running the setup with the latest OPNsense updates just fine, using Firefox (mainly) but Chrome or Edge work fine either.
It seems you did misconfigure something!

Post your haproxy export in a code box. (redact any sensitive information, but leave in the local IPs!)
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 14, 2021, 11:05:13 am
In Part 6, NAT Reflection: it applies to port forwarding rules, but in the guide you switched to a simple filter rule.

So there's only one option remaining: split DNS.

Thank you for pointing this out! I changed the guide some time ago and forgot to update that part.
Just keep in mind that there is nothing wrong with Split DNS, it is even the preferred way of doing it!

HAProxy has been rock solid, thanks again for your guide. I'm having a hard time only for Uptime Kuma, it uses websockets, and it's the only service that doesn't work behind HAProxy. The dev published a guide for the configuration behind several reverse proxies, unfortunately the only one missing is HAProxy: https://github.com/louislam/uptime-kuma/wiki/Reverse-Proxy#nginx

I'm sure there's a way to make it work but I can't find it...

Isn't that just a simple web service displaying uptime of servers?
How are you accessing it on your local network? f.e. http://192.168.2.55:3001/ or https://192.168.2.55:3001/
If it is http then you will need to DISABLE SSL in the real server settings for uptime kuma.
If it is https then you will need to ENABLE SSL but DISABLE SSL verification in the real server settings for uptime kuma.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on December 14, 2021, 11:16:25 am
Thank you for pointing this out! I changed the guide some time ago and forgot to update that part.
Just keep in mind that there is nothing wrong with Split DNS, it is even the preferred way of doing it!

I know, it's my preferred way too. I love DNS. Was just pointing out that there's only one option. :)

Quote
Isn't that just a simple web service displaying uptime of servers?
How are you accessing it on your local network? f.e. http://192.168.2.55:3001/ or https://192.168.2.55:3001/
If it is http then you will need to DISABLE SSL in the real server settings for uptime kuma.
If it is https then you will need to ENABLE SSL but DISABLE SSL verification in the real server settings for uptime kuma.

I've already reverse-proxied a lot of services, I know how that works. The problem is how Uptime Kuma works: it uses ws:// (websockets) connections in addition to HTTP, so you connect in http first to auth, then it starts communicating through WS, through a sort of tunnel. If you check that link I provided, you will see that for many proxies there's some custom configs to support that. The only proxy that does one-line config magic is caddy...it's tempting me a lot...everybody told me that caddy is the simplest one and it simply works, without doing any hard config work. But I already have HAProxy in place, and would like to stick to it.

UPDATE: I found this article https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/#advanced-configuration

Now I have to understand where to put those things in the UI on OPNsense. Is there a way to edit the config files directly?
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 14, 2021, 11:36:16 am
I've already reverse-proxied a lot of services, I know how that works. The problem is how Uptime Kuma works: it uses ws:// (websockets) connections in addition to HTTP, so you connect in http first to auth, then it starts communicating through WS, through a sort of tunnel. If you check that link I provided, you will see that for many proxies there's some custom configs to support that. The only proxy that does one-line config magic is caddy...it's tempting me a lot...everybody told me that caddy is the simplest one and it simply works, without doing any hard config work. But I already have HAProxy in place, and would like to stick to it.

Did you try something like this?
https://stackoverflow.com/a/22735431/17193869
or this
https://discourse.haproxy.org/t/using-reverse-proxy-with-secured-web-sockets-wss/2917

Otherwise your best bet is to ask in the haproxy discourse forum including your config export.
Sadly, I can't help everyone to get specific services running.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on December 14, 2021, 01:14:45 pm
Did you try something like this?
https://stackoverflow.com/a/22735431/17193869
or this
https://discourse.haproxy.org/t/using-reverse-proxy-with-secured-web-sockets-wss/2917

I found this article from HAProxy guys:
https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/#simple-configuration

That's why I wanted to know if I can configure it via shell, working on the files directly. Do you know if that's possible?

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 14, 2021, 01:36:46 pm
I found this article from HAProxy guys:
https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/#simple-configuration

That's why I wanted to know if I can configure it via shell, working on the files directly. Do you know if that's possible?

Oh very well, of course there is!
You just need to place these settings (one per line) in the advanced settings of the frontend/backend.

Frontends --> yourfrontend --> top left, enable advanced mode --> Advanced settings --> Option pass-through

As far as I am aware settings these options on the frontend will apply them to ALL services that are going through it.
But you should also be able to set them on the corresponding backend so that it will only apply to the specific service.

Code: [Select]
## routing based on websocket protocol header
  acl hdr_connection_upgrade hdr(Connection)  -i upgrade
  acl hdr_upgrade_websocket  hdr(Upgrade)     -i websocket
  use_backend bk_ws if hdr_connection_upgrade hdr_upgrade_websocket
  default_backend bk_web

This however is also just a combination of acl=condition and rule=use_backend.
It is basically saying: IF HTTP_header=Connection+Upgrade THEN USE_BACKEND AKUMA_ws_backend
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on December 14, 2021, 02:46:51 pm
As far as I am aware settings these options on the frontend will apply them to ALL services that are going through it. But you should also be able to set them on the corresponding backend so that it will only apply to the specific service.

Not working, I don't think it's a config issue...

Seems like there were issues with websockets with older versions, newest v2.5 fixed several things regarding websockets support: https://www.haproxy.com/blog/announcing-haproxy-2-5/

Latest plugin has HAProxy v2.2.18, will have to wait for the upgrade...unless I can upgrade it manually...

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: evathesalmon on December 15, 2021, 12:07:26 am

Post your haproxy export in a code box. (redact any sensitive information, but leave in the local IPs!)


Thank you for reply.

I only have map for internal network because I don't pass-through external traffic. And I also set host binding via unbound overrides all to 192.168.64.1.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 127.0.0.1:80, 127.0.0.1:443)
frontend 0_SNI_frontend
    bind 127.0.0.1:443 name 127.0.0.1:443
    bind 127.0.0.1:80 name 127.0.0.1:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_61a3f4fdabe975.79392880 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_61a3f4fdabe975.79392880

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61a3f8200d98a7.85727127.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61a3f5af2446f3.11405534.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: unraid_backend ()
backend unraid_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server unraid_server 192.168.0.20:443 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: opnsense_backend ()
backend opnsense_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opnsense_server 192.168.0.1:1443 ssl verify none


(https://i.imgur.com/gD1BC97.png)
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 15, 2021, 01:02:57 am
Thank you for reply.

I only have map for internal network because I don't pass-through external traffic. And I also set host binding via unbound overrides all to 192.168.64.1.

Check the cipher list and cipher suites on your HTTPS frontend! They don't match the ones I provide in my first post!

That one thing wonders me and might be your issue... Why did you set the SNI_frontend to listen on the localhost address? This makes no sense at all!
You are allowing inbound traffic on your WAN address 80+443 with our firewall rule but your SNI_frontend is only listening on the localhost address. This way it will never catch any traffic at all!

I never said to do so in my guide! Apart from the SSL_server address which can safely use the localhost address.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: evathesalmon on December 15, 2021, 09:11:15 pm

You are allowing inbound traffic on your WAN address 80+443 with our firewall rule but your SNI_frontend is only listening on the localhost address. This way it will never catch any traffic at all!



UPD: nevermind, it was my local cache and unbound cache. After purging everything it works just fine without any specific binding. Thank you for your help. 

I have amended my config and compared again with the guide but I must be missing something.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_61a3f4fdabe975.79392880 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_61a3f4fdabe975.79392880

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61a3f8200d98a7.85727127.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNET_condition
    acl acl_61a3f9b4ed7092.44798843 src 192.168.0.0/24

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61a3f5af2446f3.11405534.txt)] if acl_61a3f9b4ed7092.44798843

# Backend: unraid_backend ()
backend unraid_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server unraid_server 192.168.0.20:443 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: opnsense_backend ()
backend opnsense_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opnsense_server 192.168.0.1:1443 ssl verify none

# Backend: qbittorrent_backend ()
backend qbittorrent_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server qbittorrent_server 192.168.0.20:8080

The setup is working only if I bind my host->ip in unbound overrides the way they are physically, in example unraid is binded to 192.168.0.20, opnsense is binded to 192.168.0.1 and other (see the pic). But if the redirects are set to 192.168.54.1 just like in the guide it just unable to connect.

In other words seems like 0_SNI_frontend doesn't listen to all addresses even if it is configured so.

Also a question: should I set any WAN rules even if I only routing the LAN traffic, cause now I didn't set any.

(https://i.imgur.com/by1yDJ0.png)
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: boredpanda on January 03, 2022, 01:22:51 pm
Hello,

when I started implementing HAProxy in my network I couldn't find any complete and well written guide out there. I had to puzzle everything together from various websites.
So I thought I would save many of you a lot of time and provide my ultimate HAProxy on OPNsense guide.  :)

Hello! And thank you very much for your well-written guide. It made my switch from pfSense to OPNsense far smoother! HAProxy in pfSense looks quite different from HAProxy in OPNsense.

I self-host a bunch of services on a local server, and all the services are in dockers, meaning they all have the same IP but different ports. Most of the services worked just fine following your guide, but I'm noticing a few odd things. For instance, on my Firefly III installation, I get the error in the attachment and I'm unable to log in. If I visit the IP:PORT address, it works fine. Any suggestions on changes I could maybe make to my configuration?

None of my services are publicly exposed and I'm using a FQDN just to not having to remember port numbers.

My config is below. Thanks! :)

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_61d029838380d8.68540995 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_61d029838380d8.68540995

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61d02d156c0846.98881851.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    # ACL: EXTERNAL_conditions
    acl acl_61d066f2cc9639.62892989 src 193.138.218.219
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_61d05461151001.71548589 src 192.168.1.0/24 192.168.70.0/24

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61d053aef188f4.27343600.txt)] if acl_61d066f2cc9639.62892989 || acl_61d05461151001.71548589
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61d029eb5a9da6.54806678.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: nextcloud_backend ()
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nextcloud_server 192.168.1.12:444 ssl verify none

# Backend: miniflux_backend ()
backend miniflux_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server miniflux_server 192.168.1.12:5600

# Backend: joplin_backend ()
backend joplin_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server joplin_server 192.168.1.12:22300

# Backend: calibre_backend ()
backend calibre_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server calibre_server 192.168.1.12:8083

# Backend: emby_backend ()
backend emby_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server emby_server 192.168.1.12:8096

# Backend: grocy_backend ()
backend grocy_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server grocy_server 192.168.1.12:9283

# Backend: hydra_backend ()
backend hydra_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hydra_server 192.168.1.12:5076

# Backend: piwigo_backend ()
backend piwigo_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server piwigo_server 192.168.1.12:8099

# Backend: collabora_backend ()
backend collabora_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server collabora_server 192.168.1.12:9980

# Backend: freshrss_backend ()
backend freshrss_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server freshrss_server 192.168.1.12:8066

# Backend: wallabag_backend ()
backend wallabag_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server wallabag_server 192.168.1.12:6500

# Backend: wikijs_backend ()
backend wikijs_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server wikijs_server 192.168.1.12:3000

# Backend: heimdall_backend ()
backend heimdall_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server heimdall_server 192.168.1.12:8538

# Backend: monica_backend ()
backend monica_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server monica_server 192.168.1.12:8956

# Backend: firefly_backend ()
backend firefly_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server firefly_server 192.168.1.12:8088

# Backend: paperless_1_backend ()
backend paperless_1_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server paperless_1_server 192.168.1.12:8016

# Backend: paperless_2_backend ()
backend paperless_2_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server paperless_2_server 192.168.1.12:8006
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 04, 2022, 01:16:56 am
First of all please use the correct cipher list and suites, see the beginning of my OP.
You are still using AES128 ciphers indicating you didn't read my tutorial correctly.

Next thing would be to clear your browser cache.
Is it only firefly that is not working or are others also affected?

Are you using a trusted lets encrypt cert or a selfsigned one?
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: boredpanda on January 05, 2022, 06:04:45 pm
First of all please use the correct cipher list and suites, see the beginning of my OP.
You are still using AES128 ciphers indicating you didn't read my tutorial correctly.

Next thing would be to clear your browser cache.
Is it only firefly that is not working or are others also affected?

Are you using a trusted lets encrypt cert or a selfsigned one?

That's fair enough! I have updated my configuration with the correct cipher list and suites. My browser cache has been cleared and I've tried multiple browsers on different computers. The issue still remains. I am using a trusted let's encrypt cert (wildcard domain managed by ACME on OPNsense).

Two of my services aren't working as expected at the moment, and that's Firefly III and Grocy. Everything else is working.

For Firefly, I came across this thread (https://github.com/firefly-iii/firefly-iii/discussions/5118) on Github discussing my exact issue. It was apparently fixed in this comment (https://github.com/firefly-iii/firefly-iii/discussions/5118#discussioncomment-1398790). My Docker env TRUSTED_PROXIES is set to ** already. Do I need to edit 1_HTTPS_frontend or 1_HTTP_frontend? I see that both of those have the option X-Forwarded-For header enabled in my HAProxy.

For Grocy, I'm having the issue described here (https://github.com/linuxserver/docker-grocy/issues/18), on Github. Someone using nginx mentioned they solved the issue by adding proxy_set_header X-Forwarded-Proto https; to their nginx config.

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 05, 2022, 06:34:51 pm
Two of my services aren't working as expected at the moment, and that's Firefly III and Grocy. Everything else is working.

For Firefly, I came across this thread (https://github.com/firefly-iii/firefly-iii/discussions/5118) on Github discussing my exact issue. It was apparently fixed in this comment (https://github.com/firefly-iii/firefly-iii/discussions/5118#discussioncomment-1398790). My Docker env TRUSTED_PROXIES is set to ** already. Do I need to edit 1_HTTPS_frontend or 1_HTTP_frontend? I see that both of those have the option X-Forwarded-For header enabled in my HAProxy.

For Grocy, I'm having the issue described here (https://github.com/linuxserver/docker-grocy/issues/18), on Github. Someone using nginx mentioned they solved the issue by adding proxy_set_header X-Forwarded-Proto https; to their nginx config.

You should (but don't have to) leave the x-forwarded-for header enabled. It is very useful since the real servers (firefly, grocy, ...) will get to know the original IP of the client trying to access it.

The links you posted both imply that your issues COULD be resolved by adding the following to your "HTTPS_frontend".

HAProxy --> Virtual Services --> Public Services --> 1_HTTPS_frontend --> Edit --> Enable "advanced mode" (top left corner) --> Scroll down to "Advanced settings" --> Option pass-through --> insert the below code --> Save --> Apply

Code: [Select]
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Real-IP %[src]

You might not need both lines so play around until you find the necessary line(s) and please post the final solution!

If both of your services are working now then I suggest to remove that setting from your HTTPS_frontend and set it on the firefly and grocy backends instead!

HAProxy --> Virtual Services --> Public Services --> firefly/grocy backend --> Edit --> Enable "advanced mode" (top left corner) --> Scroll down to "Tuning options" --> Option pass-through --> insert the above code --> Save --> Apply
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: boredpanda on January 05, 2022, 07:00:36 pm
The links you posted both imply that your issues COULD be resolved by adding the following to your "HTTPS_frontend".

HAProxy --> Virtual Services --> Public Services --> 1_HTTPS_frontend --> Edit --> Enable "advanced mode" (top left corner) --> Scroll down to "Advanced settings" --> Option pass-through --> insert the below code --> Save --> Apply

Code: [Select]
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Real-IP %[src]

You might not need both lines so play around until you find the necessary line(s) and please post the final solution!

The second line didn't work, but http-request set-header X-Forwarded-Proto https if { ssl_fc } solved the issue with both services, so I have added the line to both of the back ends. Thank you so much! ;)
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on January 10, 2022, 07:24:43 pm
I wonder if I could ask this question here. I've followed the instruction to the letter, all is good.
I have an additional requirement however, to proxy TCP mode to the real server after TLS offloading at HA.
Say the real server is listening on port 8053 and we can live without tls between HA and it to begin with, is in the internal LAN. The public port for the service however is different, say 853. So HA needs to receive in one and sent to another.
I understand we can't mix tcp and http modes and the tutorial explains it in the relevant part.

My current thinking is that since 0_SNI_frontend and SSL_backend are already in TCP mode, the first part is adding the port, ending with 0.0.0.0:80, 0.0.0.0:443: 0.0.0.0:851
- Since the TLS is being terminated at the 1_HTTPS_frontend but is not TCP, that is not the right place to go next.
I thought then I needed to create a new frontend in TCP mode. I duplicated 1_HTTPS_frontend and changed name to 1_TCP_frontend. Set it to listen on the vip:851. I left the map rule on it and disabled SSL offload.

Testing gives me error:
2022-01-10T17:11:39   haproxy[81388] externalip:port [10/Jan/2022:17:11:39.859] 1_TCP_frontend 1_TCP_frontend/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0   
2022-01-10T17:11:39   haproxy[81388] Connect from externalip:port to wanip:851 (0_SNI_frontend/TCP)


Makes me think I'm not too far.
My question to you knowledgeable people is if the setup of this tutorial can be expanded to do this, or I'm trying to do something totally that'll never work, and some pointers to what I need to do would be wonderful please.
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 10, 2022, 10:10:57 pm
You have to create a condition and rule for that service. And then place that rule on the 0_SNI_frontend.

However you CAN NOT do url conditions, f.e. service.domain.com ....


OR

You create a new TCP frontend that listens on that port.
Then create server + backend for the service and set this backend on the new tcp frontend as default backend.

OR

you simply use a port forwarding.....
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on January 11, 2022, 12:07:21 am
Option 2 was what I was hoping was possible. It is working and what I found was that I needed to add the port to the 0_SNI_frontend too, which I wasn't sure about. I had it from earlier attempts but once working, my new proxy would not work without it.
So in summary, added the port to 0_SNI_frontend; added the new default backend to the new front end I had created (that I had missing), as well as has the map file rule. I haven't tested without this map yet, that's next.
I'm going to read and re-read the config that is working against the documentation to understand how. I can't follow the new path at present due to my lack of knowledge on the HA options used.
Thank you TheHellSite.
Looks like this and allows me to put my subdomain in my android phone to get DNS over TLS on my own infra that filters domains with Adguard before going out on TLS to the open internet. That was the goal.
For anyone wondering is here:
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    1
    hard-stop-after             60s
    maxconn                     10
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 10
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening on 0.0.0.0:80, 0.0.0.0:443, 0.0.0.0:853)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:853 name 0.0.0.0:853
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (listening on 192.168.5.100:80 i.e. http only)
frontend 1_HTTP_frontend
    bind 192.168.5.100:80 name 192.168.5.100:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_619439805021f2.97978352 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_619439805021f2.97978352

# Frontend: 1_HTTPS_frontend (Listening on 192.168.5.100:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.5.100:443 name 192.168.5.100:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61952b9d47d700.25962675.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Frontend: 1_TCP_frontend (Listening on 192.168.5.100:853)
frontend 1_TCP_frontend
    bind 192.168.5.100:853 name 192.168.5.100:853 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61dc51606078d9.11258474.certlist
    mode tcp
    default_backend nginx_backend-tcp
    # tuning options
    timeout client 15m

    # logging options
    option tcplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.5.100 send-proxy-v2 check-send-proxy

# Backend: nginx_backend-tcp ()
backend nginx_backend-tcp
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server nginx_1 192.168.5.152:8053
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Tattoofreak on January 24, 2022, 02:18:58 pm
Thank you for this tutorial.
Unfortunately my requirements are a bit different.
Basically, what I want to do, is:

I want to set up HAProxy just for routing traffic based on URLs (https://xyz.domain.com goes to server 1 and https://abc.domain.com goes to server 2, etc...).
All SSL stuff for the destination web servers is being handled by a separate Linux certificate server and the web servers themselfes, independent from OPNsense/HAProxy. HAProxy is really only needed for routing traffic based on URLs, nothing more, nothing less.

The websites are all working when I use NAT rules, but I was not able yet to make it run with HAProxy in between.

For testing it with just one webserver, I have configured the following:

OPNsense settings:

System --> Settings --> Administration:
TCP port: 4443

Firewall --> Rules --> WAN:
Protocol: IPv4/TCP | Source: any | Port: Any | Destination: This Firewall | Port: 80 (HTTP) | Rule: Pass
Protocol: IPv4/TCP | Source: any | Port: Any | Destination: This Firewall | Port: 443 (HTTPS) | Rule: Pass

HAProxy settings (everthing not mentioned I left on default settings):

Services --> HAProxy --> Settings --> Settings:
Service:
Enable HAProxy: Checked

Services --> HAProxy --> Settings --> Real Servers:
Real Servers:
Name or Prefix: web01_http | FQDN/IP: 192.168.1.10 | Port: 80
Name or Prefix: web01_https | FQDN/IP: 192.168.1.10 | Port: 443

Services --> HAProxy --> Settings --> Virtual Services:
Backend Pools:
Name: web01_backendpool01 | Mode: TCP (Layer4) | Servers: web01_http, web01_https

Services --> HAProxy --> Settings --> Rules & Checks:
Conditions:
Name: myWebsite01_condition01 | Condition type: Host starts with | Host prefix: <mysubdomain>.<mydomain>.com

Services --> HAProxy --> Settings --> Rules & Checks:
Rules:
Name: myWebsite01_rule01 | Select conditions: myWebsite01_condition01 | Execute function: Use specific Backend Pool | Use backend pool: web01_backendpool01

Services --> HAProxy --> Settings --> Virtual Services:
Public Services:
Name: web01_publicService01 | Listen Addresses: <myPublicIP>:80, <myPublicIP>:443 | Type: TCP | Rules: myWebsite01_rule01


Any help would be very appreciated, thank you in advance!
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Tattoofreak on January 27, 2022, 01:26:31 pm
Anyone?

I'm wondering if it is even possible to catch URL based packets when Public Services Type is set to TCP? Would this work?
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 29, 2022, 09:37:27 am
You have to set your backends and frontends to HTTP Mode.
Also disable SSL offloading on the frontends.

But I can't guarantee for sure that it will work.
TCP Mode will never (with a few exceptions) work because there is no header in the packets that would tell HAProxy which service to send the traffic to.

HTTP Mode could work, but you might need to create some "http header contains..." conditions.
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: afall on January 31, 2022, 05:42:25 pm
First of all thank you for the Guide it was great and helped me out a lot.

I just have a question as I have to use a vpn adapter call for work called Zscaler and it try to make a tunnel using port 443. How do i tell HAproxy as a default to only route only thing in the map file and dont intercept anything else. So the vpn client can create a tunnel. 
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: vladnik on February 01, 2022, 04:16:24 pm
Hi @TheHellSite,

just wanted to say thank you for this excellent guide, worked like a charm and thaught me much about how HAProxy works. Thank you for a job well done!  :)

Cheers!
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 03, 2022, 01:37:56 pm
I have two Server with several vhosts behind a OPNsense Router/Firewall

Is this also possible to have two server which need certs? Or only one?

When if it’s possible, how I have to do it?
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Tattoofreak on February 10, 2022, 03:40:27 pm
@TheHellSite
Maybe you would also like on how to enable Websockets on your frontend(s) if your incoming clients are looking for such one(s). Websockets are basically used for example for streaming services over web. I was looking for so long on how to resolve my problem for making HAProxy work with Synology's DS Cam Android app which tries to connect from remote to the Synology Surveillance Station NAS behind HAProxy and I finally found out. You have to insert the following on your frontend (where you have to replace <myBackend(Pool)> with your according backend, of course):

Code: [Select]
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend <myBackend(Pool)> if is_websocket

Here's the link to my original and solved issue:
 https://www.synoforum.com/threads/connecting-synology-ds-cam-android-app-to-synology-surveillance-station-through-opnsense-haproxy-plugin.7969/ (https://www.synoforum.com/threads/connecting-synology-ds-cam-android-app-to-synology-surveillance-station-through-opnsense-haproxy-plugin.7969/)
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 12, 2022, 09:13:03 pm
I got this error
Quote
[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 02:38:21 pm
First of all thank you for the Guide it was great and helped me out a lot.

I just have a question as I have to use a vpn adapter call for work called Zscaler and it try to make a tunnel using port 443. How do i tell HAproxy as a default to only route only thing in the map file and dont intercept anything else. So the vpn client can create a tunnel.

Since your provided little to no info of the setup I am not really able to help you.
Also my guide is just to show beginners how things are done.

However if I understood your issue correctly, you can try the below. If that doesn't work you are better of asking in the HAProxy forums!

Add the following in the "Option pass-through" field of your 0_SNI_frontend.
Don't forget to create the backend along with the server for Zscaler. My example uses OpenVPN.
Code: [Select]
tcp-request inspect-delay 5s
use_backend OPENVPN_backend if req_ssl_hello_type 1
tcp-request content accept if !{ req_ssl_hello_type 1 }
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 02:46:13 pm
I have two Server with several vhosts behind a OPNsense Router/Firewall

Is this also possible to have two server which need certs? Or only one?

When if it’s possible, how I have to do it?

I got this error
Quote
[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

Never had this issue!

1. Go through my guide again. If you follow it step by step, you WILL HAVE a working setup. Otherwise you did something wrong.

2. Where is your config export? Nobody will be able to help you without a HAProxy config export!

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 15, 2022, 03:01:33 pm
I have following configuration:

NAS (192.168.1.118) → Apache → Nextcloud/DAV
/
ROUTER/Firewall/HAproxy (192.168.1.1)
\
Server (192.168.1.100) → Apache with severals vhost

I want to bring up one vhost as test with following configuration

Quote
<VirtualHost *:80>
     ServerAdmin mail@xxx.com
     ServerName xxx.ch
     ServerAlias www.xxxx.ch
     DocumentRoot /usr/share/webapps/blog/
     DirectoryIndex index.php
     RemoteIPProxyProtocol On
     <Directory /usr/share/webapps/blog>
        Options +Indexes +FollowSymLinks +MultiViews
        AllowOverride All
        Order allow,deny
        allow from all

    <FilesMatch \.php$>
      # For Apache version 2.4.10 and above, use SetHandler to run PHP as a fastCGI process server
      SetHandler "proxy:unix:/run/php-fpm/php-fpm.sock|fcgi://localhost"
    </FilesMatch>
    <Files "*.php">
      MultiviewsMatch Any
    </Files>
    </Directory>
     ErrorLog /var/log/httpd/blog_error.log
     CustomLog /var/log/httpd/blog_access.log combined
</VirtualHost>
My HAproxy.conf file looks like this

cat haproxy.conf

Quote
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: O_SNI_fronted (Listening 0.0.0.0:80 0.0.0.0:443)
frontend O_SNI_fronted
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NO_SSL_Rule
    acl acl_620808a860e296.91534155 req.ssl_ver gt 0

    # ACTION: HTTP_TO_HTTPS_RULE
    http-request redirect scheme https code 301 if !acl_620808a860e296.91534155

# Frontend: 1_HTTPS_frontend (Listening 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6208140971a7a3.08696099.certlist
    mode http
    option http-keep-alive
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_MAP_RULE
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/620809e036a6d1.87483247.txt)]

# Backend: SSL_backend (SSL  backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: 5erver_backend (Server backend)
backend 5erver_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80
# ERROR: server data not found (0b989d9b-eb50-4dff-8a2f-6bc56245fd74)

# Backend: NAS_backend (NAS backend)
backend NAS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nas_Server_80 192.168.1.118:80
# ERROR: server data not found (36c63574-bd94-43f7-836e-cd78c8edc6c0)
My map files looks like this
Quote
#public subdomains mapping
flood 5erver_backend
frank 5erver_backend
www 5erver_backend
torrent 5erver_backend
grafana 5erver_backend
nas 5erver_backend
kvm 5erver_backend
monitoring 5erver_backend
speedtest 5erver_backend
sync 5erver_backend
tracker 5erver_backend
cloud NAS_backend
dav NAS_backend
How I can fix this error
503 error?

Thanks for reply
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 03:25:08 pm
FIRST: You should remove your personal info from your post.

SECOND: Another issue from not properly reading my guide.

Your solution is in Part 5 - Step 6.

Quote
Now we create the backend that belongs to an actual service. You will need one backend for each service.
If you have multiple servers serving the exact same content than you will want to add all servers into a single backend so HAProxy can actually balance the load between the servers.

YOU NEED: ... one backend for each service.

YOU DID: ... one backend for each server hosting individual services.

Just think about it... How should HAProxy even be able to talk to one of your services when you are only pointing him to the IP:Port of the server virtually hosting the service!? This makes no sense...
It is like telling someone "Meet me in New York in a bar." without telling him which bar.
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 15, 2022, 04:17:31 pm
FIRST: You should remove your personal info from your post.

SECOND: Another issue from not properly reading my guide.

Your solution is in Part 5 - Step 6.

Quote
Now we create the backend that belongs to an actual service. You will need one backend for each service.
If you have multiple servers serving the exact same content than you will want to add all servers into a single backend so HAProxy can actually balance the load between the servers.

YOU NEED: ... one backend for each service.

YOU DID: ... one backend for each server hosting individual services.

Just think about it... How should HAProxy even be able to talk to one of your services when you are only pointing him to the IP:Port of the server virtually hosting the service!? This makes no sense...
It is like telling someone "Meet me in New York in a bar." without telling him which bar.

Ok, thanks for finding the error.

English isn't my mother language so can you little bit clarify the steps.

So I have to copy the backend and adjust it for every virtual host of Apache even if they are on the same port?

I have one apache on both SERVER(192.168.1.100 and 192.168.1.118) on port 80 an only one webserver with several vhosts. I thought the map file will do the split of the to the vhosts...

For me is Apache one service on which is on both SERVER and so It's has one backend each for this two server?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 15, 2022, 04:29:47 pm
(https://abload.de/img/haproxydgj5s.png)

So I have to do this for every single vhost?

So it's looking so?

(https://abload.de/img/haproxy_backendlgk4r.png)

So the maps file is also wrong I have to change nas_backend to nas_server_80 and 5erver_backend to 5erver_server_80 ?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 05:37:21 pm
This is not a question of your mother language not being english.  ;D
This is a question of you not understanding the layout of HAProxy.

It is really hard for me to help here since you don't seem to understand the basics of a reverse proxy.
A reverse proxy is not a web server to host your sites! It is meant to route traffic to a web server.

Rule 1: You need at least ONE server for each backend.
With server I mean server created in HAProxy. A server in HAProxy represents the service (f.e. nextcloud, plex, ...) running on one of your physical servers.

Rule 2: Each backend will then present your individual services to your frontend.

HTTPS Frontend --> Mapfile --> PLEX_backend --> PLEX_server --> 192.168.2.30:32400
HTTPS Frontend --> Mapfile --> BLOG_backend --> BLOG_server --> 192.168.2.30:80
HTTPS Frontend --> Mapfile --> CLOUD_backend --> CLOUD_server --> 192.168.2.40:43569

Rule 3: A mapfile is simply mapping an access URL, f.e. www.yourdomain.com, against a backend.


To be fair it seems I misunderstood your setup/goal. But since you still didn't explain exactly what you are trying to achieve I am assuming you are trying to something like this.

Apache is your webserver. (www.yourdomain.com)
These vhost seem to provide subdirectories or individual websites. (f.e. service.yourdomain.com/subdir/)

Code: [Select]
#public subdomains mapping
flood 5erver_backend
frank 5erver_backend
www 5erver_backend
torrent 5erver_backend
grafana 5erver_backend
nas 5erver_backend
kvm 5erver_backend
monitoring 5erver_backend
speedtest 5erver_backend
sync 5erver_backend
tracker 5erver_backend
cloud NAS_backend
dav NAS_backend

Let me further explain to you the map file. I mean actually my first post does it, but you don't seem to understand or didn't read...

Code: [Select]
nas 5erver_backendnas.yourdomain.com --> NAS_backend

Code: [Select]
www 5erver_backendwww.yourdomain.com --> 5erver_backend

Code: [Select]
grafana 5erver_backendgrafana.yourdomain.com --> 5erver_backend
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 15, 2022, 06:00:06 pm
Thanks for your help. I appreciated it!

Quote
cat haproxy.conf
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: O_SNI_fronted (Listening MYIP:80 MYIP:443)
frontend O_SNI_fronted
    bind MY IP:80 name MYIP:80
    bind MY IP:443 name MY IP:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening 192.168.1.1:80)
frontend 1_HTTP_frontend
    bind 192.168.1.1:80 name 192.168.1.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NO_SSL_Rule
    acl acl_620808a860e296.91534155 req.ssl_ver gt 0

    # ACTION: HTTP_TO_HTTPS_RULE
    http-request redirect scheme https code 301 if !acl_620808a860e296.91534155

# Frontend: 1_HTTPS_frontend (Listening 192.168.1.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.1.1:443 name 192.168.1.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6208140971a7a3.08696099.certlist
    mode http
    option http-keep-alive
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_MAP_RULE
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/620809e036a6d1.87483247.txt)]

# Backend: SSL_backend (SSL  backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 192.168.1.1 send-proxy-v2 check-send-proxy

# Backend: blog_server_backend (Server backend blog)
backend blog_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: cloud_nas_backend (cloud  backend NAS)
backend cloud_nas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nas_Server_80 192.168.1.118:80 send-proxy-v2 check-send-proxy

# Backend: dav_nas_backend (dav  backend NAS)
backend dav_nas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nas_Server_80 192.168.1.118:80 send-proxy-v2 check-send-proxy

# Backend: frank_server_backend (Server backend frank)
backend frank_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: flood_server_backend (Server backend flood)
backend flood_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: sync_server_backend (Server backend sync)
backend sync_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: monitoring_server_backend (Server backend monitoring)
backend monitoring_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: kvm_server_backend (Server backend kvm)
backend kvm_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: nas_server_backend (Server backend nas)
backend nas_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: tracker_server_backend (Server backend tracker)
backend tracker_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

and my mapfile

Quote
#public subdomains mapping
flood flood_server_backend
frank frank_server_backend
grafana grafana_server_backend
nas  nas_server_backend
kvm kvm_server_backend
monitoring monitoring_server_backend
sync sync_server_backend
tracker tracker_server_backend
cloud cloud_nas_backend
dav dav_nas_backend

You see anymore errors?

EDIT1: Do you recommend to change to nginx who can handle as web and reverse proxy
EDIT2: After studying your latest post I got it.
EDIT3: Why can’t Haproxy multiple services on same port?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 06:27:48 pm
You still don't understand.  ???
Your first mapfile was correct.

Just do this and tell me if it works.

1. create a mapfile with the below content.

Code: [Select]
#public subdomains mapping
flood WEBSERVER_backend
frank WEBSERVER_backend
www WEBSERVER_backend
torrent WEBSERVER_backend
grafana WEBSERVER_backend
nas WEBSERVER_backend
kvm WEBSERVER_backend
monitoring WEBSERVER_backend
speedtest WEBSERVER_backend
sync WEBSERVER_backend
tracker WEBSERVER_backend
cloud NAS_backend
dav NAS_backend

2. create the following backends and servers
Code: [Select]
WEBSERVER_backend --> contains --> WEBSERVER_server

WEBSERVER_server=192.168.1.100:80


NAS_backend --> contains --> NAS_server

NAS_server=192.168.1.118:80

3. Try to access your webserver and nas.

Code: [Select]
# WEBSERVER_backend
flood.yourdomain.com
frank.yourdomain.com
www.yourdomain.com
torrent.yourdomain.com
grafana.yourdomain.com
nas.yourdomain.com
kvm.yourdomain.com
monitoring.yourdomain.com
speedtest.yourdomain.com
sync.yourdomain.com
tracker.yourdomain.com


# NAS_backend
cloud.yourdomain.com
dav.yourdomain.com



If it still doesn't work then I can only offer you paid support. I hope you understand. It is my free time and I can only help for free up to a certain point.
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opns_neuling on February 19, 2022, 01:17:00 pm
I got this error
Quote
[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

https://forum.opnsense.org/index.php?topic=27065.msg131206#msg131206

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 20, 2022, 05:25:18 pm
I have just tried TCP mode with map file, there is a few more steps to achieve the goal instead of placing the map rule directly to 0_SNI
(I checked the package and found SNI inside, however, haproxy doesn't recognize it in TCP mode, that's why we need to force it to recognize SNI)

1. Create a "Condition" to request client hello
Name: SSL_Hello
Condition type: Custom condition (option pass-through)
Option pass-through: req_ssl_hello_type 1
(https://i.postimg.cc/rD89fwvk/cond-hello.jpg) (https://postimg.cc/rD89fwvk)

2. Create a "Rule" to wait accept SSL hello
Name: tcp_request_content_accept_ssl
Select conditions: SSL_Hello
Execute function: tcp-request-content-accept
(https://i.postimg.cc/mcnNZNhL/rule-ssl.jpg) (https://postimg.cc/mcnNZNhL)

3. Create a "Rule" to wait for inspect
Name: tcp_request_inspect_delay
Optional condition: none
Execute function: tcp-request-inspect-delay
TCP inspection delay: 5s
(https://i.postimg.cc/gX6yxX6v/rule-delay.jpg) (https://postimg.cc/gX6yxX6v)

4. Place the Rule to 0_SNI_frontend in following order
tcp_request_inspect_delay
tcp_request_content_accept_ssl
map
(https://i.postimg.cc/sBx4R0jH/rule-order.jpg) (https://postimg.cc/sBx4R0jH)
(hmdir_ru is my map rule)


Update according to findings in #183 (https://forum.opnsense.org/index.php?topic=23339.msg131582#msg131582)

5. Change the no_SSL condition to Traffic is SSL (locally deciphered)
(https://i.postimg.cc/Cng6Mdtn/nossl.jpg) (https://postimg.cc/Cng6Mdtn)

*Remark
It is advised to use another map file for 1_HTTPS_frontend if necessary
If you really don't want to create another map file, use "SNI TLS extension matches (locally deciphered)" instead
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 20, 2022, 06:04:58 pm
I got this error
Quote
[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

https://forum.opnsense.org/index.php?topic=27065.msg131206#msg131206
Have you Disable web GUI redirect rule in Part 4-1?
If you haven't disable it, your web GUI will keep listening to port 80.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 20, 2022, 07:31:40 pm
I have http webui with port 4444 impossible that listen on 80
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 20, 2022, 09:03:34 pm
I have http webui with port 4444 impossible that listen on 80

(https://i.postimg.cc/YGBYddZD/Screenshot-20220221-035236-01.jpg) (https://postimg.cc/YGBYddZD)
That's what I get when I try to add another frontend with 192.168.6.1:80, which already exists in my running system.
You must have another process running which listening to port 80 when you get this error

That's why I asked rather you disabled the GUI redirect.
As this redirect process is listening to port 80 of your router IP, and help you redirect any port 80 to the HTTPS port you typed.
If you already disabled, you need to figure out what other plugin you have installed and using port 80.

Guess: GUI redirect, nginx, firewall port forward
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 21, 2022, 08:41:30 am
Just take a look at the config file, I feel strange for some part of it
1. You don't need to type WAN IP in 0_SNI_frontend
instead, it should be 0.0.0.0:80 and 0.0.0.0:443
0.0.0.0 means any IP that points to your router.

2. What is your router IP?
If your router is 192.168.1.1, then 1_HTTP_frontend and 1_HTTPS_frontend will obviously conflict with 0_SNI_fronted
Since 0_SNI_fronted is already listening to 80 and 443 port of your router, you won't able to listen it with 192.168.1.1
Please follow Part 4-2 to create Virtual IP, and set 1_HTTP_frontend and 1_HTTPS_frontend to the virtual IP

If you don't want to create any Virtual IP, please remove 0_SNI_frontend
set 1_HTTP_frontend with 0.0.0.0:80 and 1_HTTPS_frontend 0.0.0.0:443 instead
Since all of your servers are running in http mode, it should work for having no SNI frontend
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 21, 2022, 01:13:24 pm
I have just tried TCP mode with map file, there is a few more steps to achieve the goal instead of placing the map rule directly to 0_SNI
(I checked the package and found SNI inside, however, haproxy doesn't recognize it in TCP mode, that's why we need to force it to recognize SNI)

As I said in the tutorial I am not running any services that would require me to load balance / reverse proxy plain TCP traffic. This is why I never tested it and only could provide theoretical assumptions on how it could work.
So thank you very much for providing a good guide on this.

If you don't mind I will add this to the tutorial (credits to your post) when I find the time!

Just a quick question: Which service are you running in tcp mode? Does the service send the SNI header?
Or does you solution work with any TCP based service? Becuase as far as I am aware the service needs to add the SNI header otherwise the access URL is not beeing sent to HAProxy.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 21, 2022, 01:21:52 pm
Just take a look at the config file, I feel strange for some part of it
1. You don't need to type WAN IP in 0_SNI_frontend
instead, it should be 0.0.0.0:80 and 0.0.0.0:443
0.0.0.0 means any IP that points to your router.

2. What is your router IP?
If your router is 192.168.1.1, then 1_HTTP_frontend and 1_HTTPS_frontend will obviously conflict with 0_SNI_fronted
Since 0_SNI_fronted is already listening to 80 and 443 port of your router, you won't able to listen it with 192.168.1.1
Please follow Part 4-2 to create Virtual IP, and set 1_HTTP_frontend and 1_HTTPS_frontend to the virtual IP

If you don't want to create any Virtual IP, please remove 0_SNI_frontend
set 1_HTTP_frontend with 0.0.0.0:80 and 1_HTTPS_frontend 0.0.0.0:443 instead
Since all of your servers are running in http mode, it should work for having no SNI frontend

I am already on him with the fix.
Also explained to him that the HTTP(s)_frontend IPs could cause issues.
His Apache is also misconfigured badly.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 21, 2022, 03:27:41 pm
Sure, just add it to your tutorial if you like.
I have 2 TCP servers running. OpenVPN and v2ray
(both of them have SNI header with it)

I'm sure not all of the TCP services can use haproxy, for example minecraft server without additional tools.
(One of the ways is to add one more rule to redirect other SSL connections to SSL_backend, and set minecraft server as default backend of 0_SNI, as no conditions or rules in haproxy can catch connections that doesn't have SNI header).
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 21, 2022, 03:29:31 pm
Sure, just add it to your tutorial if you like.
I have 2 TCP servers running. OpenVPN and v2ray
(both of them have SNI header with it)

I'm sure not all of the TCP services can use haproxy, for example minecraft server without additional tools.
(One of the ways is to add one more rule to redirect other SSL connections to SSL_backend, and set minecraft server as default backend of 0_SNI, as no conditions or rules in haproxy can catch connections that doesn't have SNI header).

Thank you for confirming!
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 22, 2022, 08:30:30 am
Just checked the L6 warning a bit.
This is the code from haproxy github (https://github.com/haproxy/haproxy/blob/master/src/cfgparse.c)
Code: [Select]
int warnif_tcp_http_cond(const struct proxy *px, const struct acl_cond *cond)
{
if (!cond || px->mode != PR_MODE_HTTP)
return 0;

if (cond->use & (SMP_USE_L6REQ|SMP_USE_L6RES)) {
ha_warning("Proxy '%s': L6 sample fetches ignored on HTTP proxies (declared at %s:%d).\n",
   px->id, cond->file, cond->line);
return ERR_WARN;
}
return 0;
}

The condition: req.ssl_ver (Traffic is SSL....)
This condition might get error code, as http doesn't have tls packages.
Although negating this condition will still redirect packages to 443, haproxy will still run.
But we can simply use "Traffic is HTTP" here.
(Then, I don't know why I get such error when I create 2 port 80 frontend) ???
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 22, 2022, 09:29:18 am
Please repost your current config.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 22, 2022, 11:19:30 am
You mean my working config?

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if xray)
frontend tcp_front
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_6212326a7c07e4.28981163 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_6212326a7c07e4.28981163
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6214a3ae639096.17472719.txt)]

# Frontend: http_front (Listen to VIP:80 and redirect to 443)
frontend http_front
    bind 192.168.6.1:80 name 192.168.6.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: http
    acl acl_62123bbee27260.60165685 ssl_fc

    # ACTION: http_to_https
    http-request redirect scheme https code 301 if !acl_62123bbee27260.60165685

# Frontend: ssl_front (Listen to VIP 443, SSL offload cert)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.6.1:443 name 192.168.6.1:443 accept-proxy ssl no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621244f0de5919.36753000.certlist
    mode http
    option http-keep-alive
    default_backend hkbn_back
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: local
    acl acl_62123a1cebe813.09309501 src 192.168.3.0/24 192.168.5.0/24

    # ACTION: local_ru
    use_backend opn_back if acl_62123a1cebe813.09309501

# Backend: hmdir_back (Backend of v2ray)
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.3.3:443

# Backend: hkbn_back (Backend of HKBN)
backend hkbn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hkbn 192.168.4.2:443 ssl verify none

# Backend: SSL_back (Backend to redirect SSL servers)
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server ssl 192.168.6.1 send-proxy-v2 check-send-proxy

# Backend: opn_back (Backend of opnsense with SSL)
backend opn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opn 192.168.3.1:8443 ssl verify none

# Backend: open_back (Backend of OpenVPN)
backend open_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server open 192.168.3.1:10443

BTW, I know why I will get such warning after reading the code.
Since my TCP layer already received TLS packages and inspected it already.
For HTTP and HTTPS frontend, any more "request" for SSL/TLS will probably get such warning.

Thus, for anyone using map file in 0_SNI_frontend.
They not only need to create "tcp_request_inspect_delay", "tcp_request_content_accept_ssl" and "SSL_Hello" condition
but also need to change the non-HTTP traffic condition to either
Negate of Traffic is SSL (locally deciphered) (ssl_fc), which will not request another TLS package
or
Traffic is HTTP (req.proto_http)
but not
Traffic is SSL (TCP request content inspection) (req.ssl_ver)

If Someone already used map file in 0_SNI_frontend
And they have multiple HTTP server need to offload in HTTPS_frontend, they should either use another map file
or
SNI TLS extension matches (locally deciphered) (ssl_fc_sni)
but not
SNI TLS extension ... (TCP request content inspection) (req.ssl_sni)

*For Morta's case, this might also due to multiple inspection of TLS or another port 80 listener
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 22, 2022, 06:28:24 pm
I have looked wir netsocket but no other app is on port 80 and is working so far

I have also problem that the cert redirect automatically from www.xx.ch to xx.ch

When I insert xx.ch in the map file all in existing sub domains goes to xx.ch as examples buff.xx.ch goes to xx.ch

How can I map at least www.xx.ch to a service without redirect or better how can I map xx.ch and www.xx.ch without redirecting all subdomain to xx.ch vhost
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 22, 2022, 07:14:45 pm
Just set your Webserver as default backend on the https frontend.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 22, 2022, 07:32:33 pm
I have two Webserver. So I can took only one.
Which one? WEBSERVER_backend or NAS_backend?


Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 22, 2022, 08:35:47 pm
Obviously your Webserver and not your NAS.......  ::)

Also you just exposed your Domain again  :-\

Please don't get me wrong but I really hope you know what you are doing by self-hosting and exposing any services from your private internet connection!
There are many risks involved and you don't seem to me like someone that is aware of everything he is doing and why it needs to be done.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 23, 2022, 08:07:18 pm
Would be possible to adjust the tutorial with ipv6 support or a hint how to add ipv6 support to a existing configuration?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 24, 2022, 01:22:48 am
Would be possible to adjust the tutorial with ipv6 support or a hint how to add ipv6 support to a existing configuration?

For the frontend that using 0.0.0.0:80, "add" [::]:80
For the frontend that using 0.0.0.0:443, "add" [::]:443

In firewall rule, add one more rule to "allow" "ipv6" "TCP" to "this firewall" in "port 80 and 443"

Note: your DDNS need to update ipv6 too
ddclient 3.9.1 should be able to update ipv6 . However, I don't know rather OPNsense had implemented to update ipv6 automatically.

P.S. dyndns plugin is too old, it doesn't support ipv6
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: figadore on February 25, 2022, 07:27:07 am
When creating the virtual IP, there's a new option not included in the picture. Is there a correct setting for "Allow service binding"? Does it make a difference?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 27, 2022, 11:30:42 am
I mean it is self-explanatory.

To answer your question: You will want this enabled.
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opns_neuling on March 13, 2022, 09:42:06 pm
I got this error
Quote
[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

on Step:
Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
Here we will only create a "NoSSL_condition", which is necessary in order to identify non-HTTP traffic.


Please replace "req.ssl_ver gt 0" with "ssl_fc" also, select "custom" and on custom field enter "ssl_fc"

https://redmine.pfsense.org/issues/9261


Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 14, 2022, 01:23:40 pm
Can you make a screenshot? I don’t understand you instructions

I have no req.ssl_ver 0 in my configuration!

So?

(https://abload.de/img/auswahl_024oaj7m.png)
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 14, 2022, 08:22:32 pm
Mentioned in #183
Traffic is SSL (locally deciphered) is ssl_fc

And the condition need to be replaced is Part 5.7.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 15, 2022, 06:55:39 pm
Thanks for detailed instructions, I've follow step by step to make a web hosting running nginx with https support.

My HAProxy Config:

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_622eebaf197419.36314953 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_622eebaf197419.36314953

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/622eef9a9d7268.16491040.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Backend: backend_pool_abc ()
backend backend_pool_meet_huuich_vn
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server PLEX_server 192.168.82.11:32400 ssl verify none

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

My Map File content:

Code: [Select]
# public access subdomains
plex PLEX_backend

My nginx website config

Code: [Select]
server {
        listen 32400;
        listen [::]:32400;

        root /var/www/mywebsite.com/html;
        index index.html index.htm index.nginx-debian.html;

        server_name mywebsite.com;

        location / {
                try_files $uri $uri/ =404;
        }
}

I can go my website on internal http://192.168.82.11:32400 is ok but when I access http://mywebsite.com browser go to https://mywebsite.com and show error

Code: [Select]
503 Service Unavailable
No server is available to handle this request.

How can I fix this and show my website https://mywebsite.com ok? Thanks!
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 15, 2022, 07:15:48 pm
I can go my website on internal http://192.168.82.11:32400 is ok but when I access http://mywebsite.com browser go to https://mywebsite.com and show error

Code: [Select]
503 Service Unavailable
No server is available to handle this request.

How can I fix this and show my website https://mywebsite.com ok? Thanks!


First.
The entry "plex PLEX_backend" in the mapfile means that you will have to access it using the "plex" subdomain. --> In your case "plex.mywebsite.com"!

Alternatively just set the PLEX_backend as default backend on your HTTPS_frontend.

Second.
http will always get redirected to https. This is intended and you will most probably want this! This is configured using the HTTPtoHTTPS_rule and NoSSL_condition.

Third.
Apart from the above your config looks good. (just took a very short look at it)

Fourth.
You might have to disable the SSL checkbox in the PLEX_server settings.
But only if you are REALLY accessing it locally using http://IP:32400 and the service is NOT redirecting you to HTTPS. But I highly doubt this since Plex is running on a self-signed SSL cert by default...
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 15, 2022, 11:00:02 pm
First.
The entry "plex PLEX_backend" in the mapfile means that you will have to access it using the "plex" subdomain. --> In your case "plex.mywebsite.com"!

Alternatively just set the PLEX_backend as default backend on your HTTPS_frontend.

Second.
http will always get redirected to https. This is intended and you will most probably want this! This is configured using the HTTPtoHTTPS_rule and NoSSL_condition.

Third.
Apart from the above your config looks good. (just took a very short look at it)

Fourth.
You might have to disable the SSL checkbox in the PLEX_server settings.
But only if you are REALLY accessing it locally using http://IP:32400 and the service is NOT redirecting you to HTTPS. But I highly doubt this since Plex is running on a self-signed SSL cert by default...

You are right, mysubdomain.mywebsite.com, and follow your guide I've edit my Map File to "mysubdomain PLEX_backend" and "disable the SSL checkbox in the PLEX_server settings" and voila my subdomain website run https:// ok with A+ score, Thank you so much for quick and detailed reply!

P/s: I've follow your guide with cloudflare instead. (almost every settings are the same like your guide)
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 16, 2022, 02:47:59 am
I want to set up HAProxy just for routing traffic based on URLs (https://xyz.domain.com goes to server 1 and https://abc.domain.com goes to server 2, etc...).
All SSL stuff for the destination web servers is being handled by a separate Linux certificate server and the web servers themselfes, independent from OPNsense/HAProxy. HAProxy is really only needed for routing traffic based on URLs, nothing more, nothing less.I want to set up HAProxy just for routing traffic based on URLs (https://xyz.domain.com goes to server 1 and https://abc.domain.com goes to server 2, etc...).
All SSL stuff for the destination web servers is being handled by a separate Linux certificate server and the web servers themselfes, independent from OPNsense/HAProxy. HAProxy is really only needed for routing traffic based on URLs, nothing more, nothing less.
Did you find solution for your purpose? I wanna do same like you.
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 16, 2022, 03:52:23 am
You have to set your backends and frontends to HTTP Mode.
Also disable SSL offloading on the frontends.

But I can't guarantee for sure that it will work.
TCP Mode will never (with a few exceptions) work because there is no header in the packets that would tell HAProxy which service to send the traffic to.

HTTP Mode could work, but you might need to create some "http header contains..." conditions.
I can do that using SSL passthrough follow step by step this guide (https://forum.opnsense.org/index.php?topic=18538.msg84958#msg84958) from @alh

Every works ok except my Nginx log can not receive real ip from client, it's only show ip  local of my OPNSense server in file /var/log/nginx/access.log

Do you think can mix SSL passthrough with your guide (edit 0_SNI_frontend with rules from SSL passthrough and Default Backend Pool) or any solution to receive real ip from client with SSL passthrough? Thanks!

Code: [Select]
This should work for any TCP-based SSL/TLS encrypted service in passthrough (HAProxy: TCP) mode... It does NOT work for STARTTLS!

In this example I use TCP port 443.

HAProxy plugin: Create "Real Server" (enter name, IP/FQDN and port number if different from 443, the rest can be left at default)
HAProxy plugin: Create "Backend Pool" (enter name, set mode to TCP and select the real server from step 1)
HAProxy plugin: Create "Condition" (enter name ["traffic_ssl"], condition type is "custom condition (option pass-through)" with value "req_ssl_hello_type 1")
HAProxy plugin: Create "condition" (enter name ["myservice_sni"], condition type is "SNI TLS extension matches (TCP request content inspection)" with value "myservice.example.com" or whatever your FQDN is)
HAProxy plugin: Create "Rule" (enter name ["request_inspect_delay"], select no condition, function is "tcp-request inspect delay" with value "5s" or whatever suits you)
HAProxy plugin: Create "Rule" (enter name ["request_content_accept_ssl"], select condition of 3 ["traffic_ssl"], function is "tcp-request content accept")
HAProxy plugin: Create "Rule" (enter name ["myservice_sni"], select condition of 4 ["myservice_sni"], function is "Use specific backend pool" with your pool from 2)
HAProxy plugin: Create "Public service" (enter name ["https_passthrough"], choose a listen address [":443" for all], type is "TCP" and select the 3 rules created earlier)
HAProxy plugin: Enable plugin or test/apply
Firewall: allow incoming traffic to WAN (address) or whatever for TCP port 443.

That works at least for me. If you have double NAT you would need to disable port randomization for the proxied port...

Does that help you?
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 16, 2022, 11:51:30 am
If all SSL certificate are handled by webserver themselves.
Follow #176 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354), step 1-4 will allow 0_SNI_FRONTEND to recognize TLS package and redirect under TCP mode

Note: All backend redirect from 0_SNI_FRONTEND should be in TCP mode
HAProxy only work with server that using TLS package. For those server that doesn't contain TLS package (for example, game servers), HAProxy won't work. Although you can set default backend server to game server, but there's only one default backend server.

BTW, for backend server getting source IP, enabling X-Forwarded-For header for all frontend should work
However, this one only apply to HTTP.
reference (https://www.haproxy.com/fr/blog/preserve-source-ip-address-despite-reverse-proxies/)
 reference 2 (https://www.haproxy.com/blog/layer-7-load-balancing-transparent-proxy-mode/)
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 16, 2022, 09:20:26 pm
If all SSL certificate are handled by webserver themselves.
Follow #176 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354), step 1-4 will allow 0_SNI_FRONTEND to recognize TLS package and redirect under TCP mode

Note: All backend redirect from 0_SNI_FRONTEND should be in TCP mode
HAProxy only work with server that using TLS package. For those server that doesn't contain TLS package (for example, game servers), HAProxy won't work. Although you can set default backend server to game server, but there's only one default backend server.

BTW, for backend server getting source IP, enabling X-Forwarded-For header for all frontend should work
However, this one only apply to HTTP.
reference (https://www.haproxy.com/fr/blog/preserve-source-ip-address-despite-reverse-proxies/)
 reference 2 (https://www.haproxy.com/blog/layer-7-load-balancing-transparent-proxy-mode/)
I've follow your guide but though my backend server ctl_backend has code
Code: [Select]
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Real-IP %[src]

But it can not get source ip, here is my full config, could you show me how to get source ip? Or I have to have 2 OPNSense, one for SSL passthrough and one use Let's Encrypt Wildcard Certificates like this topic guide? Could 1  OPNSense sever run both nginx reverse proxy and HAProxy?

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    # tuning options
    timeout client 30s

    # logging options
    # ACL: traffic_ssl
    acl acl_601a842f14cee3.17646593 req_ssl_hello_type 1

    # ACTION: request_content_accept_ssl
    tcp-request content accept if acl_601a842f14cee3.17646593
    # ACTION: request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Frontend (DISABLED): 1_HTTP_frontend (Listening on 192.168.64.1:80)

# Frontend (DISABLED): 1_HTTPS_frontend (Listening on 192.168.64.1:443)

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Real-IP %[src]
    http-reuse safe
    server PLEX_server 192.168.82.11:32400

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend (DISABLED): SSL_backend ()

# Backend: ctl_backend ()
backend ctl_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Real-IP %[src]
    http-reuse safe
    server ctl_server 192.168.82.11:32401
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 17, 2022, 04:15:01 am
Notice that you are using TCP mode frontend only.
X-forwarded-for will not simply work under any case that using TCP mode, as X-forwarded-for will only work under HTTP mode and need to enable in all layers (TCP mode will not forward any header)

For TCP mode, you need to set all backend that wants to get real client IP and redirected from this frontend to use Proxy Protocol (either version 1 or 2, while verion 1 sending ASCII data and version 2 sending binary)
You will find "send-proxy" under that backend, which means it will send another package via proxy protocol to your real server.

Your real server need to modify config to read proxy protocol package too.
 Nginx guide to set it accept, reading and logging proxy protocol address (https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/)
(Not 2 lines, tones should be modified whenever there is a layer of TCP mode)

P.S. Theoretically, setting like OP having (SNI frontend->send-proxy backend->HTTPS frontend with X-forwarded-for) should also forward proxy protocol package to real server too.

Example: my v2ray server is living in TCP mode (although I don't care about the IP log)
I will get router IP in log if I don't user proxy protocol.
I cannot reach v2ray server if I use proxy protocol in haproxy but not changing any config of v2ray server (probably rejected by the server itself)
I will get correct client IP if I use proxy protocol in haproxy and changing the config of v2ray to accept proxy protocol

My HAProxy config is as follow:
1. open_back is OPENVPN live in TCP mode, and will not get any real client IP (Always 192.168.3.1)
2. hmdir_back is v2ray (another proxy server) live in TCP mode, and will get real client IP (I need to modify v2ray config to make it read and log proxyprotocol
3. opn_back is opnsense page live in HTTP mode, will not get real client IP as I don't care.
4. unifi_back is unifi controller page live in HTTP mode, I don't care rather it get real IP too
5. hkbn_back is ISP router page, which has no access right to my LAN, just a dummy to fake ISP that I'm using their device, all unwanted access will reditect to this page (Their device always open 443 to public access)
5. opn_back and unifi_back can only be accessed with source IP as 192.168.3.0/24 and 192.168.5.0/24 and I have tested no one can access to these pages except my LAN clients. Thus, real client IP should be passed to SSL_frontend successfully too.
Just need to add backend with http-request set-header X-Real-IP %[src] (Although there will be warning, you can do it in frontend, as HAProxy config manual (http://cbonte.github.io/haproxy-dconv/2.0/configuration.html)mentioned both ways will work) and config webserver read and log X-Real-IP
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if v2ray)
frontend tcp_front
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_6212326a7c07e4.28981163 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_6212326a7c07e4.28981163
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6214a3ae639096.17472719.txt)]

# Frontend: http_front (Listen to VIP:80 and redirect to 443)
frontend http_front
    bind 192.168.6.1:80 name 192.168.6.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: http
    acl acl_62123bbee27260.60165685 ssl_fc

    # ACTION: http_to_https
    http-request redirect scheme https code 301 if !acl_62123bbee27260.60165685

# Frontend: ssl_front (Listen to VIP 443, SSL offload cert)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.6.1:443 name 192.168.6.1:443 accept-proxy ssl no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621244f0de5919.36753000.certlist
    mode http
    option http-keep-alive
    default_backend hkbn_back
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: local
    acl acl_62123a1cebe813.09309501 src 192.168.3.0/24 192.168.5.0/24

    # ACTION: local_map_ru
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623230d7bffd04.94702836.txt)] if acl_62123a1cebe813.09309501

# Backend: hmdir_back (Backend of v2ray)
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.3.3:443 send-proxy-v2 check-send-proxy

# Backend: hkbn_back (Backend of HKBN)
backend hkbn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hkbn 192.168.4.2:443 ssl verify none

# Backend: SSL_back (Backend to redirect SSL servers)
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server ssl 192.168.6.1 send-proxy-v2 check-send-proxy

# Backend: opn_back (Backend of opnsense with SSL)
backend opn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opn 192.168.3.1:8443 ssl verify none

# Backend: unifi_back (Backend of unifi with SSL)
backend unifi_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server unifi 192.168.3.4:8443 ssl verify none

# Backend: open_back (Backend of OpenVPN)
backend open_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server open 192.168.3.1:10443
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 17, 2022, 09:38:06 am
@Bunch Thank you for detailed guide and attached config. I've tried to simulate like your server but now my server not working

This is my config

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if v2ray))
frontend tcp_front
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: traffic_ssl
    acl acl_601a842f14cee3.17646593 req_ssl_hello_type 1

    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_601a842f14cee3.17646593
    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Frontend: http_front (Listening on 192.168.64.1:80)
frontend http_front
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_622eebaf197419.36314953 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_622eebaf197419.36314953

# Frontend: ssl_front (Listening on 192.168.64.1:443)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/622eef9a9d7268.16491040.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Backend: hmdir_back ()
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.82.11:32401 send-proxy-v2 check-send-proxy

# Backend (DISABLED): PLEX_backend ()

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend: SSL_back ()
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend (DISABLED): ctl_backend ()

My hmdir_ru is link to PUBLIC_SUBDOMAINS_map map file with content:
Code: [Select]
# public access subdomains
c hmdir_back

Could you share content of map file in local_map_ru in this config

Code: [Select]
# ACTION: local_map_ru
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623230d7bffd04.94702836.txt)] if acl_62123a1cebe813.09309501

And ACL http in this config

Code: [Select]
# logging options
    # ACL: http
    acl acl_62123bbee27260.60165685 ssl_fc

Thank you!
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 17, 2022, 10:07:30 am
My local_map_rule is:
Test type= if
Selection condition= local
Execute function= Map domains to backend pools using a map file
Map_file= local_map

Condition local:
Condition type= Source IP matches specified IP
Source IP=192.168.3.0/24 192.168.5.0/24

Local_map:
Code: [Select]
opn opn_back
unifi unifi_back

For ssl_fc, it is mentioned in #174 Step 5 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354)
And #183 (https://forum.opnsense.org/index.php?topic=23339.msg131582#msg131582)

For testing why it won't work in your case,
You can try not using Proxy Protocol first. If you can access your backend, that means your webserver config is not accepting proxy protocol (It won't work both way at the same time)
You need to use Proxy Protocol and modify webserver config ( for example, /etc/nginx/sites-enabled/default) at the same time

Similar test case mentioned in previous post
Quote
Example: my v2ray server is living in TCP mode (although I don't care about the IP log)
I will get router IP in log if I don't user proxy protocol.
I cannot reach v2ray server if I use proxy protocol in haproxy but not changing any config of v2ray server (probably rejected by the server itself)
I will get correct client IP if I use proxy protocol in haproxy and changing the config of v2ray to accept proxy protocol
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 17, 2022, 10:15:45 am

For testing why it won't work in your case,
You can try not using Proxy Protocol first. If you can access your backend, that means your webserver config is not accepting proxy protocol (It won't work both way at the same time)
You need to use Proxy Protocol and modify webserver config ( for example, /etc/nginx/sites-enabled/default) at the same time

Similar test case mentioned in previous post
Quote
Example: my v2ray server is living in TCP mode (although I don't care about the IP log)
I will get router IP in log if I don't user proxy protocol.
I cannot reach v2ray server if I use proxy protocol in haproxy but not changing any config of v2ray server (probably rejected by the server itself)
I will get correct client IP if I use proxy protocol in haproxy and changing the config of v2ray to accept proxy protocol

My /etc/nginx/sites-enabled is

Code: [Select]
server {
listen 32401;
listen [::]:32401;

server_name c.mywebsite.com;

root /var/www/html;

location / {
try_files $uri $uri/ /yourls-loader.php$is_args$args;
}

location ~ \.php$ {
include fastcgi.conf;

fastcgi_index index.php;

fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}

How could use Proxy Protocol and modify webserver config with my HAProxy config above?
Sorry for basic question. Thank you for detailed explaination.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 17, 2022, 11:04:16 am
That's not basic question IMO.  :D
The modification method is mentioned in Nginx guide (https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/)

Should be modified as below (haven't tested it)
Code: [Select]
server {
listen 32401 proxy_protocol;
listen [::]:32401 proxy_protocol;

server_name c.mywebsite.com;

root /var/www/html;

location / {
try_files $uri $uri/ /yourls-loader.php$is_args$args;
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr
}

location ~ \.php$ {
include fastcgi.conf;

fastcgi_index index.php;

fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}

For the log config, I forgot rather it is /etc/nginx/nginx.conf by default
You need to change the log format inside http{} to something like
Code: [Select]
log_format combined '$proxy_protocol_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent"';
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 18, 2022, 10:05:44 am
The modification method is mentioned in Nginx guide (https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/)

Should be modified as below (haven't tested it)

Your code "haven't tested but working for me with http website, but when I change to https with certbot

Code: [Select]
certbot --nginx -d c.mywebsite.com
My https website can not access though I've change my server port to 443, this is my nginx website config

Code: [Select]
server {
listen 32401 proxy_protocol;
listen [::]:32401 proxy_protocol;

listen [::]:443 ssl ipv6only=on proxy_protocol; # managed by Certbot
    listen 443 ssl proxy_protocol; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/c.mywebsite.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/c.mywebsite.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

server_name c.mywebsite.com;

root /var/www/html;

location / {
try_files $uri $uri/ /yourls-loader.php$is_args$args;
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr;
}

location ~ \.php$ {
include fastcgi.conf;

fastcgi_index index.php;

fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}
}

and my current HAProxy configure

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if v2ray))
frontend tcp_front
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: traffic_ssl
    acl acl_601a842f14cee3.17646593 req_ssl_hello_type 1

    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_601a842f14cee3.17646593
    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Frontend: http_front (Listening on 192.168.64.1:80)
frontend http_front
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_622eebaf197419.36314953 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_622eebaf197419.36314953

# Frontend: ssl_front (Listening on 192.168.64.1:443)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/622eef9a9d7268.16491040.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Backend: hmdir_back ()
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.82.11:443 ssl verify none send-proxy-v2 check-send-proxy

# Backend (DISABLED): PLEX_backend ()

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend: SSL_back ()
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend (DISABLED): ctl_backend ()

Could you take a look and guide me how to run https on my website?

For the log config, I forgot rather it is /etc/nginx/nginx.conf by default
You need to change the log format inside http{} to something like
Code: [Select]
log_format combined '$proxy_protocol_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent"';
Fire I use above code has a error, so I google and found a solution for working log, I post here for anyone need

Code: [Select]
log_format my_log '$proxy_protocol_addr - $remote_user [$time_local] '
                       '"$request" $status $body_bytes_sent '
                       '"$http_referer" "$http_user_agent"';

access_log /var/log/nginx/access.log my_log;
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 18, 2022, 12:59:38 pm
Better check /etc/letsencrypt/options-ssl-nginx.conf
Check rather the SSL protocol is too old (is it having something like TLSv1 TLSv1.1)
If the settings are too old, try to change those settings in that file according to this link (https://ssl-config.mozilla.org/#server=nginx&version=1.17.7&config=intermediate&openssl=1.1.1d&guideline=5.6)

Guess below lines need to be modified if case exists.
Code: [Select]
ssl_session_timeout 1d;
    ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
    ssl_session_tickets off;

    # intermediate configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    # HSTS (ngx_http_headers_module is required) (63072000 seconds)
    add_header Strict-Transport-Security "max-age=63072000" always;

    # OCSP stapling
    ssl_stapling on;
    ssl_stapling_verify on;

For real server in HAProxy, I have tried turning SSL on or off doesn't affect my servers under TCP mode, don't know rather it is the same in your case, please try it too.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on March 18, 2022, 01:06:28 pm
I'm probably out of place saying this, as is not my thread, but should't this discussion go to another thread and leave this one for it's original purpose?
It has branched off now to "how can I enable TLS on my website", from "how can I log the client ip not the proxy ip on the backend webserver" and "how do I use proxy_protocol".
What do you think?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 18, 2022, 01:13:51 pm
Yes, I think it is out of HAProxy scope too.
When send-proxy or X-forwarded-for part is done in HAProxy, nothing can be done in HAProxy afterwards.
All other settings are related to Nginx or some other server settings.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 18, 2022, 02:14:35 pm
@cookiemonster Thank for your comment about back to original purpose, I'll create a new topic relate my questions. Thanks.

@Bunch: I'm very thankful to you for your help. Best Regards!
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: The_Dave on March 18, 2022, 03:10:11 pm
I have a very serious problem with this haproxy config since I updated to 22.1.3. Suddenly haproxy didn't start anymore. On further investigation trying to start haproxy through the commandline showed that suddenly the ipadresses for the frontend cannot be bound anymore:

Code: [Select]
root@OPNsense:/home/David # /usr/local/etc/rc.d/haproxy start
Starting haproxy.
[ALERT]    (21351) : Starting frontend 1_HTTP_frontend: cannot bind socket (Can't assign requested address) [192.168.64.1:80]
[ALERT]    (21351) : Starting frontend 1_HTTPS_frontend: cannot bind socket (Can't assign requested address) [192.168.64.1:443]
[ALERT]    (21351) : [/usr/local/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.
/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy

Any help on how to fix this would be really appreciated
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 18, 2022, 06:12:04 pm
It seems that it is the same issue as This thread (https://forum.opnsense.org/index.php?topic=27547.msg133659#msg133659)
I have the same issue after update and reboot.
For temporary fix, edit the VIP, save without any changes, then apply.
You will able to start HAProxy again.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: The_Dave on March 18, 2022, 08:52:27 pm
For temporary fix, edit the VIP, save without any changes, then apply.

Thank you this actually works for now, I hope there will be a proper solution or fix for this soon though
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: balrog on March 20, 2022, 10:49:35 am
I had the problem after updating to OPNsense 22.1.3 that the HAProxy service did not start anymore. I was able to solve the problem by editing the Virtual IP and saving it again without adjusting it. After that the service could be started again.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 20, 2022, 02:13:15 pm
Code: [Select]
2022/03/20 15:00:53 [error] 1124599#1124599: *22208 upstream prematurely closed connection while reading response header from upstream, client: 192.168.1.1, server: sync.xxx.ch, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/run/uwsgi/mozilla-firefox-sync-server.sock:", host: "sync.xxx.ch"

I got a 502 Gateway error on this site with nginx.

Where is the option of keep-alive for the backends? Or anybody knows how to fix this error?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 22, 2022, 08:19:47 am
Check and change Services: HAProxy: Settings: Default Parameters
Check Tuning Options of your frontends rather you overwrite the setting too.
If it doesn't fix the issue, then it is Nginx setting problem
Check this link (https://stackoverflow.com/questions/36488688/nginx-upstream-prematurely-closed-connection-while-reading-response-header-from) too for Nginx setting
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 22, 2022, 09:37:57 am
Thanks for reply bunch!

The link is for the settings with a reverse proxy with nginx.

I will look at the tunables of Haproxy.

I found this link

https://serveanswer.com/questions/upstream-prematurely-closed-connection-while-reading-upstream-large-files (https://serveanswer.com/questions/upstream-prematurely-closed-connection-while-reading-upstream-large-files)

Also possible that is a connection error to the SQLite database.

I will give a try at home.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 22, 2022, 02:30:22 pm
I try to add full IPv6 support to my Haproxy configuration.
Now I have a problem with IPv6 localhost or loopback address

I did a real server ssl_server_ipv6 with IPv6 ::1

(https://abload.de/img/e083e5c0-101f-494d-984zjpo.png)
(https://abload.de/img/52010dc8-40b7-4474-abm6jvq.png)

I got following error when I add ssl_server_ipv6 to SSL_backend
(https://abload.de/img/ea69a72d-3153-4847-bb8fjuf.png)

How I can fix it or do a workaround?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 22, 2022, 03:41:30 pm
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 1000s
    timeout connect 1000s
    timeout server 1000s
    retries 3
    default-server init-addr libc,last
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend (Listening o)
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind :::80 name :::80
    bind :::443 name :::443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 1000s

    # logging options

# Frontend: HTTP_frontend (Listening 127.0.0.1:80)
frontend HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    bind [::1]:80 name [::1]:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 1000s

    # logging options
    # ACL: NoSSL_condition
    acl acl_621d0b77c74989.24704837 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_621d0b77c74989.24704837

# Frontend: HTTPS_frontend (Listinging on 127.0.0.1:443)
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621d11c7cad951.61400293.certlist
    bind [::1]:443 name [::1]:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621d11c7cad951.61400293.certlist
    mode http
    option http-keep-alive
    default_backend WEBSERVER_backend
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/621d0c7054ddb7.46420139.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: WEBSERVER_backend ()
backend WEBSERVER_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server WEBSERVER_server 192.168.1.100:80 send-proxy-v2 check-send-proxy
    server WEBSERVER_server_ipv6 2a02:XXX:XXX::2000:80 send-proxy-v2 check-send-proxy

# Backend: NAS_backend ()
backend NAS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server NAS_server 192.168.1.118:80
    server NAS_server_ipv6 2a02:XXX:XXX::1000:80

# Backend: WEBSERVER_SSL_backend ()
backend WEBSERVER_SSL_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server WEBSERVER_server_ssl 192.168.1.100:443
    server WEBSERVER_server_ssl_ipv6 2a02:XXX:XXX::2000:443

It's this a correct and possible configuration?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 22, 2022, 04:01:34 pm
I don't think you need to create another ipv6 real server, as long as it is the same sever in ipv4
You only need to add :::443 and :::80 to frontend listener (in frontend, [::]:80 is the same as :::80, in case you confused with the syntax)
That will be ipv6 to 4 setup.

If you add 2 real server to the same backend, you are load balancing them.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 22, 2022, 05:35:53 pm
I don't think you need to create another ipv6 real server, as long as it is the same sever in ipv4
You only need to add :::443 and :::80 to frontend listener (in frontend, [::]:80 is the same as :::80, in case you confused with the syntax)
That will be ipv6 to 4 setup.

If you add 2 real server to the same backend, you are load balancing them.

Thanks for the Input.
Real IPv6 or Dual Stack support of HAproxy would be nice but so I have a fallback opportunity if IPv4 or IPv6 are out of service by me or the ISP! 
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: os_admin on March 27, 2022, 08:59:47 pm
I have a very serious problem with this haproxy config since I updated to 22.1.3. Suddenly haproxy didn't start anymore. On further investigation trying to start haproxy through the commandline showed that suddenly the ipadresses for the frontend cannot be bound anymore:


This behaviour seems to be gone since 22.1.4_1. No guarantee.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: os_admin on March 27, 2022, 09:43:26 pm
Just tested it out myself. Basic Auth is so easy to set up that I am not really willing to cover it in this guide.
First create the user(s) in HAProxy. Then in the relevant backends activate basic auth and select the user(s).

Thanks for this tutorial. It saved my ass. I learned a lot about OPNsense and HAProxy. At last I enabled basic auth. on one of my backends. Anything was fine before, but after activating it I can't no longer login into the service web frontend itself. If I access the frontend browser asked for the basic auth, after that I see the login screen of the service, but after put in the service credentials the FE refresh and shows the login screen again.
If I disable basic auth, the service FE works as expected.

What do I miss? Maybe someone out there has a hint...
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 31, 2022, 06:58:19 am
Strange,  have just tested one of my backend
Create user->Enable Basic Auth and select Allowed users in backend
It works as expected.

Edit: Just tested a bit deeper, for some pages like unifi controller, it will always redirect to wrong page
For some pages like opnsense web UI, in chrome(PC), it will keep prompting for auth, but in firefox(PC & mobile), everything works normally

BTW, nothing more can be done in haproxy too, as some site in some browser works normally. Thus, the problem is due to webserver and browser.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 01, 2022, 05:58:07 pm
I'm probably out of place saying this, as is not my thread, but should't this discussion go to another thread and leave this one for it's original purpose?
It has branched off now to "how can I enable TLS on my website", from "how can I log the client ip not the proxy ip on the backend webserver" and "how do I use proxy_protocol".
What do you think?

Absolutely true!  ;D

I don't want to sound like an asshole here, but this tutorial was intended to get the basics working for new users.
This is also why I stopped answering questions about issues like "my service_abc has the requirement_xyz how work????".

If there are any questions in that regard then people should consider posting them...
here: https://forum.opnsense.org/index.php?board=28.0
or here: https://discourse.haproxy.org/
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 01, 2022, 05:58:31 pm
It seems that it is the same issue as This thread (https://forum.opnsense.org/index.php?topic=27547.msg133659#msg133659)
I have the same issue after update and reboot.
For temporary fix, edit the VIP, save without any changes, then apply.
You will able to start HAProxy again.

Thank you for posting the workaround!
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 11, 2022, 09:23:22 pm
Just a quick notifcation for everyone following the thread.
It seems like Let's Encrypt changed something regarding wildcard certificates.

I updated the picture in Part 3 - Step 6 to reflect the changes necessary in order to obtain a certificate.

You will have to remove the alt name "*.yourdomain.tld" and change the common name to "*.yourdomain.tld".
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: ElliotAlderson on April 12, 2022, 03:54:12 am
Code: [Select]
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind website.com:443 name website.com:443
    bind website.com:80 name website.com:80
    mode tcp
    default_backend SSL_backend
    timeout client 30s

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 10.10.10.1:80 name 10.10.10.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 30s

    # ACL: NoSSL_cond
    acl acl_62548efaf067e6.21908045 req.ssl_ver gt 0
    # ACTION: HTTPupgrade_rule
    http-request redirect scheme https if !acl_62548efaf067e6.21908045

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 10.10.10.1:443 name 10.10.10.1:443 accept-proxy ssl ssl-min-ver TLSv1.3 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62549082216928.65241361.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 30s

    # ACTION: PUBLIC_SUBDOMAINS_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62548f2d97ef05.80304462.txt)]

# Backend: club_backend ()
backend club_backend
website.com    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server club_host 10.0.0.94:3000 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
website.com    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    timeout connect 30s
    timeout server 30s
    server SSL_host 10.10.10.1 send-proxy-v2 check-send-proxy

I switched over from pfSense to OPNSense months ago and I had to set my side projects to the side because I simply could not replicate my HAProxy setup from before. I'm thankful for this tutorial since it's seems like the closest to what I used to have.

I'm extremely lost here. I have gone through this tutorial many times to double check my steps, I have tried changing things on my own.
At the moment, the HTTP->HTTPS redirect doesn't seem to work at all (empty response, no redirected) and the https site gives a blank response as well.
I have checked it with tcpdump directly on the OPNSense shell and could see that packets do get exchanged between my host and the virtual IP.  (TCP handshake, TLSv1 Client Hello, End connection )
The site itself is definitely working correctly internally at the host specified in the config file.

In the config I posted, I used website.com:443 in the SNI frontend.
I have tried it with 0.0.0.0:443 and my public IP with no success.

10.0.0.94 is in my LAN.
10.10.10.1 is the virtual interface I created.

My wildcard certificate seems to be working correctly.

I would really appreciate some help  :'(


By the way, what would the process be for getting another domain and wildcard cert to work added to this setup?
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 12, 2022, 07:06:19 pm
You are missing "code 301" in HTTPupgrade_rule
(Part 5-9)

Explanation of the code can be found here (https://www.haproxy.com/blog/redirect-http-to-https-with-haproxy/)
Quote
This technique will only work when using mode http because it redirects at the HTTP layer using a 302 Found HTTP response status, which is known as a temporary redirect. Once you’re fully committed to using HTTPS and have tested it thoroughly on your website, you may wish to instruct the browser to cache the redirect, which will save one round trip between the browser and HAProxy, speeding up page load times. Set the code parameter to 301 to send a 301 Moved Permanently status back, which browsers can cache:

Although it should work with 302, just give it a try

BTW, I don't know why there is website.com in the code of both backend. There shouldn't be something like this.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: ElliotAlderson on April 12, 2022, 09:01:22 pm
Oh, yeah. I actually did have the code 301 there originally. It's still the same effect with that though. That's just something I forgot to change back when I was trying new things. (Saw an older forum post that didn't use the "code 301" part.)

Regarding about the website.com,
In the config I posted, I used website.com:443 in the SNI frontend.
I have tried it with 0.0.0.0:443 and my public IP as well with no success.

I just can't get it to work so I'm trying different things.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 13, 2022, 02:16:04 am
I mean
Quote
# Backend: SSL_backend ()
backend SSL_backend
website.com    mode tcp
And
Quote
# Backend: club_backend ()
backend club_backend
website.com    mode http

Usually the wouldn't have SNI or domain names hard code in backends
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 13, 2022, 10:36:48 am
This week i have moved away from pfSense, I had acme, cloudflare & HAProxy working prior to the switch. Installed opnsense while slowly getting my services back online I came across this well written tutorial which seems more in-depth than my old setup but run into issues while accessing the hosted web service, it is failing to load with a 522 error, the connection if timing out before a response I think?

I have not got any further in the guide than part 5, step 10. Accessing from outside of my network as this is not possible so far.

I have a static WAN IP.. in cloudflare a have [A record *.example.com > Static IP]

I have double checked all the settings in this tutorial and after some googling i came across a reddit post, suggesting they fixed the 522 error in opnsense because HAProxy wasn't listening on port 80 during the HTTPtoHTTPS redirect. Is there a way I can diagnose this issue and trace the route somehow.

Lastly

ACME do not show any error in the log files. 
Code: [Select]
2022-04-13T18:53:42 php AcmeClient: running automation (configd): Restart HAProxy
2022-04-13T18:53:42 php AcmeClient: running automations for certificate: *.example.com
2022-04-13T18:53:42 opnsense AcmeClient: updated ACME X.509 certificate: *.example.com
2022-04-13T18:53:42 opnsense AcmeClient: successfully issued/renewed certificate: *.example.com
2022-04-13T18:51:27 opnsense AcmeClient: using challenge type: CloudFlare_DNS-01
2022-04-13T18:51:27 opnsense AcmeClient: account is registered: example.com
2022-04-13T18:51:27 opnsense AcmeClient: using CA: letsencrypt
2022-04-13T18:51:27 opnsense AcmeClient: issue certificate: *.example.com

HAProxy has no errors in the log file either

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62565b172acae6.05588153 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62565b172acae6.05588153

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62565eb5d0ff12.02152772.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62565c00b116b3.27816426.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: MineOS_backend ()
backend MineOS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server MineOS_server 192.168.1.103:8443 ssl verify none
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: hopey on April 13, 2022, 03:34:46 pm
@TheHellSite: Thanks for your tutorial, it helped me to understand things a bit better!

I just had this problem: https://forum.opnsense.org/index.php?topic=27903.new#new. That's why I was reading your thread.

 What was strange to me, that you need to change the port of your opensense web interface! That's the point were I thought there might be a better solution 😬 I was reading and found this tutorial: https://schulnetzkonzept.de/opnsense. This guy just used a virtual IP, like you did. But instead of pointing to the lookup adress he just introduced a new adress and let haproxy listen on. And so did I - and it worked like charm! (detailes in the upper linked problem)

Maybe you wanna add it to your tutorial too?!

Have a nice day!
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: pottproll on April 13, 2022, 04:47:13 pm
Thanks for the tutorial! Its working great so far! I have one question: in my ACME Client log it says after renewal: php[2613]   AcmeClient: automation not supported: restart_haproxy
Is this just me or maybe because i didnt check HA-Proxy Integration?

Edit: never mind, worked perfectly the next renewal.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 14, 2022, 04:22:13 am
I usually trace the session in Services: HAProxy: Statistics, Counters
When saving HAProxy setting, it will reset the session stat.
So, we can check the session stop at which server if we try to access immediately after a reset.
For example, I expect a session goes through
1. Frontend: TCP_front
2. Frontend: SSL_front
3. Backend: opn_back (usually don't have problem if you type correctly)
4. Server: opn

If all of them have session counts, but you still failed to access the website.
That means, it should be problem between browser and server, but you can access to the page when you type the server IP and port directly. Then, there might be issue about your ciphers, certs, OCSP settings, etc. You might find handshake error in server log too.

If it stop at 2. That means, your haproxy is not recognizing SNI correctly. Check your map file, or you can try to create condition with "SNI TLS extension matches (locally deciphered)" and your full SNI (the.domain.com), then create a rule to "Use specified Backend Pool" when condition matches.
Removing all rules and set default backend to test server first can also be a choice. (At least you will know rather it fails only in SNI part or more parts suffers)

If it stop at 1. There might be issue with the VIP again. (The bug similar to this (https://forum.opnsense.org/index.php?topic=27547.msg133659#msg133659))

If no session count. The listener is not working. TCP_frontend have wrong Listen Addresses. If you try it from WAN, check your firewall rules too, or maybe DNS record issue.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 14, 2022, 05:03:24 am
Okay i can now access my webservices but in doing so a missed out the Virtual IP step.
After thinking about my issue some more I am listening HTTP & HTTPS traffic on 192.168.64.1 which I think is when the time out happens.

So I;
Service > HAProxy > Settings > Real Servers > SSL_server: changed FQDN or IP, from 192.168.64.1 to 192.168.1.1

Service > HAProxy > Settings > Virtual Services > 1_HTTPS_frontend: changed Listen Addresses, from 192.168.64.1:443 to 192.168.1.1:443

Service > HAProxy > Settings > Virtual Services > 1_HTTP_frontend: changed Listen Addresses, from 192.168.64.1:80 to 192.168.1.1:80

Now it is all working, What did I do wrong in setting up the Virtual IP I wonder.

0_SNI_frontend > Listen Addresses:0.0.0.0:80, 0.0.0.0:443
should this need to be the Virtual IP as opnsense runs on 192.168.1.1

^^fyi thankyou for the tips on tracing
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 14, 2022, 05:42:11 am
Quote
0_SNI_frontend > Listen Addresses:0.0.0.0:80, 0.0.0.0:443
should this need to be the Virtual IP as opnsense runs on 192.168.1.1

^^fyi thankyou for the tips on tracing

For this question, lets clear the package path that OP wants to do first.
Assume the following IP config:
WAN IP: 1.2.3.4
Firewall IP: 192.168.1.1
VIP: 192.168.64.1
Server IP:port and SNI: 192.168.1.2:80, the.website.com

A browser try to access https://the.website.com from internet.
It asked system to resolve from DNS server: the.website.com. DNS server replies it is 1.2.3.4
Browser try to access 1.2.3.4 with port 443, sending TLS package with SNI=the.website.com

Since haproxy SNI_frontend is listening to 0.0.0.0:443, that means it is listening to port 443 that all IP can represent the firewall. In this case, it is 1.2.3.4:443 and 192.168.1.1:443

SNI_frontend catches the TLS package in 1.2.3.4:443 and passes to SSL_backend(VIP) without changing port

Since SSL_frontend is listening to 192.168.64.1:443, it takes the TLS package and knows that it try to access 192.168.1.2:80. SSL_frontend communicate to the browser, exchanging the SSL cert and keys according to ciphers.

Session to webserver_backend start, SSL_frontend redirect remaining packages to webserver_backend.

I don't know why your VIP won't work, maybe missing opnsense update, misconfig of VIP or another new bug. (You can try to ping the VIP in LAN to check rather it reply. If the VIP is working normally, it should reply)
And SNI frontend wont use VIP unless you use hopey's method (He still need to add NAT rule to redirect packages to VIP as I mentioned in another thread)
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 14, 2022, 07:21:31 am
You guys are awesome, really appreciate you explaining the process. It is easy to follow along to a guide but to understand what is happening makes it that much easier down the line.

After pinging the VIP 192.168.64.1, it was timing out.

Checked the setting and all is correct to the tutorial.
Decided to changed the submask from 32 to 24.
Then I was able to ping the VIP and access my web services.
Reset the submask back to 32 and i am still able to ping the VIP and web services working with HTTP & HTTPS listening on the VIP. Very strange but it seems resolved.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 14, 2022, 07:43:34 am
The issue is really similar to this one (https://forum.opnsense.org/index.php?topic=27547.msg133659)
I guess the issue should be solved in 22.1.4.
I run the patch in 22.1.3, so I don't know rather it is really fixed. But I don't have such problem anymore (currently 22.1.5)

If your are in 22.1.4 or 22.1.5, having similar issue. But general log don't have something like
Code: [Select]
/firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '192.168.64.1' -alias' failed to executePlease create another thread under 22.1 Production Series (https://forum.opnsense.org/index.php?board=41.0)

If you have exactly the same log and you are in 22.1.4 or 22.1.5. Please try to reply to the thread (https://forum.opnsense.org/index.php?topic=27547.msg133659)
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 14, 2022, 08:05:12 am
I will make my way over to the other thread you linked then as it is a similar issue.

I am on 22.1.5 and after checking the general log I also have the below
Code: [Select]
2022-04-14T16:42:58 Error opnsense /firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '192.168.64.1' -alias' failed to execute


--EDIT--
After reading the linked thread, a patch was applied in 22.1.4 but you also need to untick "Allow service binding".
I can confirm this has fixed the issue. Thanks again for point me in the right direction.

---EDIT 2---
I am now interested in the last part of the tutorial, I did not have anything like this setup on my previous pfsense setup so it will be a great addition. It would make for remembering local ip/ ports easier when navigating to locally hosted services.
I have followed the steps to make these subdomains accessible only from my local network but getting a "503 service unavailable".

I think its not reading the local map file correctly. I followed the previious steps for tracing and can see the counters go up when I try to access local.website.com but nothing is being passed to the prism_backend.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62565b172acae6.05588153 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62565b172acae6.05588153

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62565eb5d0ff12.02152772.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6257dfacde7e16.43417850 src_is_local

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6257d684d34507.32920094.txt)] if acl_6257dfacde7e16.43417850
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62565c00b116b3.27816426.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: MineOS_backend ()
backend MineOS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server MineOS_server 192.168.1.103:8443 ssl verify none

# Backend: Prism_backend ()
backend Prism_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Prism_server 192.168.1.103:2342

PUBLIC_SUBDOMAINS_map
Code: [Select]
# public access subdomains
mineos MineOS_backend
LOCAL_SUBDOMAINS_map
Code: [Select]
# local access subdomains
prism Prism_backend

# public access subdomains
mineos MineOS_backend

just to confirm;
mineos.website.com > works locally and externally
prism.website.com > 503 error locally and externally
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 14, 2022, 03:41:11 pm
Quote
mineos.website.com > works locally and externally
prism.website.com > 503 error locally and externally
Sorry, I haven't read the error 503.

Try not to use "src_is_local"
I remember that there is issue with this function, especially under proxy protocol. (It's hyproxy issue since 1.7?)
Try to use "Source IP matches a specific IP" instead
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 15, 2022, 12:22:50 am
Try to use "Source IP matches a specific IP" instead

I actually set this first time around which gave the same output. So I then decided to try a broader range with "IP is local".

I tried again "Source IP matches a specific IP" of the specific subnet again with no luck. have even tried isolating it to the IP of the client I am currently using.

the counter still is not being passed on the final stage of the route. After checking the HAProxy log file, as I navigate to local.website.com i am receiving a external request from cloudflare. I suspect the issue lies within the browser redirecting the request out or HAproxy not grabbing it locally before it goes out??
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 17, 2022, 01:03:33 pm
Strange, there shouldn't be something related to cloudflare
Maybe try to create condition: "SNI TLS extension matches (locally deciphered)" with your full sni, prism.website.com
Create another rule that
Use specified Backend Pool: Prism_backend
When
Source IP matches a specific IP...
And
SNI TLS extension matches (locally deciphered)

Use this rule to replace the one with local map file first
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bothson on April 19, 2022, 05:28:18 pm
Hello and thank you for this tutorial.
It helped me alot.

My services are available from the outside.
But from inside they are not accessible.

In the logs i see:
Code: [Select]
Informational haproxy 10.10.10.206:63264 [19/Apr/2022:17:26:27.483] 1_HTTPS_frontend/10.12.0.1:443: Received something which does not look like a PROXY protocol header
I already checked the ciphers. They seem to be ok.
Does anybody know where i can search for the problem?

My config:
Code: [Select]
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 10.12.0.1:80 name 10.12.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_60d1a0c1b278f7.63252237 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_60d1a0c1b278f7.63252237

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 10.12.0.1:443 name 10.12.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6256591773a972.14047672.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/625655d89e4274.43878203.txt)]

# Backend: bitwarden_backend ()
backend bitwarden_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server bitwarden_host 10.10.10.11:8080

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 10.12.0.1 send-proxy-v2 check-send-proxy

Best
Mathias
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 19, 2022, 06:26:15 pm
My services are available from the outside.
But from inside they are not accessible.

Well, there you got the point of error.
You probably configured the wrong IP in your DNS overwrites.

What is your OPNsense LAN IP, what is the DNS Overwrite IP and what is the virtual IP of the "HAProxy SSL Server"?
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bothson on April 19, 2022, 08:50:05 pm
Quote
Well, there you got the point of error.
You probably configured the wrong IP in your DNS overwrites.

What is your OPNsense LAN IP, what is the DNS Overwrite IP and what is the virtual IP of the "HAProxy SSL Server"?

My LAN IP is 10.10.10.1/24
My DNS Override points to    10.12.0.1
Which is also my Virtual IP (Loopback).

I do not get the point.

Edit: Ok, changed the Override IP to my LAN IP (10.10.10.1). Now it works.
But to be honest, i do not understand why.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 19, 2022, 08:59:40 pm
Edit: Ok, changed the Override IP to my LAN IP (10.10.10.1). Now it works.
But to be honest, i do not understand why.

You error explains why!

Code: [Select]
Informational haproxy 10.10.10.206:63264 [19/Apr/2022:17:26:27.483] 1_HTTPS_frontend/10.12.0.1:443: Received something which does not look like a PROXY protocol header
The HTTPS_frontend expects that all data sent to it has the "proxy protocol header".
Since you pointed your internal requests directly to your HTTPS_frontend (HAProxy_VIP) instead of your SNI_frontend (any of the real local IPs of your OPNsense) the data didn't get the PROXY protocol header attached by the SSL_backend.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bothson on April 20, 2022, 11:31:06 am
Edit: Ok, changed the Override IP to my LAN IP (10.10.10.1). Now it works.
But to be honest, i do not understand why.

You error explains why!

Code: [Select]
Informational haproxy 10.10.10.206:63264 [19/Apr/2022:17:26:27.483] 1_HTTPS_frontend/10.12.0.1:443: Received something which does not look like a PROXY protocol header
The HTTPS_frontend expects that all data sent to it has the "proxy protocol header".
Since you pointed your internal requests directly to your HTTPS_frontend (HAProxy_VIP) instead of your SNI_frontend (any of the real local IPs of your OPNsense) the data didn't get the PROXY protocol header attached by the SSL_backend.

Ah ok, i get the point.
Thank you very much for the explanation.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 24, 2022, 05:45:11 am
I am not sure if this is the correct way to achieve multiple domains pointing to different backends but it seems to be working for me.
At first I ran into a issue were all domains could access the same subdomain, this is when I realized I just needed some extra conditions.

Here are the steps to achieve; service.example.com & service1.example1.com

Services --> ACME Client --> Certificates
Add the certificate for your extra domains and forcefully issue your certificate

Services --> HAProxy --> Settings --> Advanced --> Map Files
Here we will create a new map file for each domain "PUBLIC_SUBDOMAINS_map-example" & "PUBLIC_SUBDOMAINS_map-example1"

Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
Add a new condition for each domain that you have added.
Name = "example1_condition"
Description = "Traffic matches example1.com"
Condition type = "host contains"
Host Contains = "example1.com"

Services --> HAProxy --> Settings --> Rules & Checks --> Rules
Add a map rule for each domain while also selecting our newly created "example1_condition"
Name = "PUBLIC_SUBDOMAINS_map-rule-example1"
Select conditions = "example1_condition"
Map file = "PUBLIC_SUBDOMAINS_map-example1"

Services --> HAProxy --> Settings --> Virtual Services --> Public Services
Finally we edit our "1_HTTPS_frontend"
Add all extra domains in the "Certificates" input.
Scroll down and add each map-rule-example1 in the "Select Rules" input
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 24, 2022, 11:51:04 am
I am not sure if this is the correct way to achieve multiple domains pointing to different backends but it seems to be working for me.
At first I ran into a issue were all domains could access the same subdomain, this is when I realized I just needed some extra conditions.

Here are the steps to achieve; service.example.com & service1.example1.com

Services --> ACME Client --> Certificates
Add the certificate for your extra domains and forcefully issue your certificate

Services --> HAProxy --> Settings --> Virtual Services --> Public Services
Finally we edit our "1_HTTPS_frontend"
Add all extra domains in the "Certificates" input.
Just the steps above are necessary and the following step.

Then edit the "PUBLIC_SUBDOMAINS_map" and change the subdomain entries to full FQDNs.

Code: [Select]
old
===
nas NAS_backend

new
===
nas.domain1.com NAS_1_backend
nas.domain2.com NAS_2_backend

Of course you could also create dedicated map-files and rules for each domain, but in terms of functionality it is not necessary.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 25, 2022, 04:45:20 am
Although it has the same outcome, The steps you provided using only 1 map file is a lot cleaner and easier to follow.

Thanks again for pointing this out.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: christian_domes on April 29, 2022, 09:19:57 am
@theHellSite
Hello
When I overide the dns Server will the haproxy still be used or not?
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on May 11, 2022, 05:11:28 am
@theHellSite
Hello
When I overide the dns Server will the haproxy still be used or not?

It depends on how you override the dns record.
Assume that you set your SNI frontend with 0.0.0.0:80 and 0.0.0.0:443

For example, you added a DNS record in Cloudflare "abc.domain.com" pointing to your WAN IP, and your tested it and found HAProxy working both locally and externally.
Then you removed the DNS record from Cloudflare, and add one in unbounded "abc.domain.com" pointing to your OpnSense IP (either LAN or WAN, doesn't metter)

Then your HAProxy should work locally, but failed externally.
*If you want to override DNS record in unbound, always point to SNI frontend. If you set it to either HTTP frontend or HTTPS frontend, it will fail.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 04, 2022, 03:42:34 pm
I just updated the tutorial with a very important change to the DynDNS part. It therefore only affects users with a dynamic WAN IP.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: schnerring on June 10, 2022, 12:57:32 am
I've been following this excellent guide to a tee, but I get the following warning:

Quote
[WARNING] (51339) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:69).
Warnings were found.
Configuration file is valid

This is the affected part of the HAProxy config (the last line is 69):

Code: [Select]
# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_629f48c6073c95.86527303 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_629f48c6073c95.86527303

For reference, other people also encountered this issue:


The issue seems to be the NoSSL_condition. The suggested solution is to replace the condition type from Traffic is SSL (TCP request content inspection)  to Traffic is SSL (locally deciphered):

Code: [Select]
     # logging options
     # ACL: NoSSL_condition
-    acl acl_629f48c6073c95.86527303 req.ssl_ver gt 0
+    acl acl_629f48c6073c95.86527303 ssl_fc
 
     # ACTION: HTTPtoHTTPS_rule
     http-request redirect scheme https code 301 if !acl_629f48c6073c95.86527303

When doing so, the warning is gone. However, this is the first time I'm using HAProxy and I don't really know what I'm doing, so I wanted to check-in with you guys to ensure my solution is correct?

If so, I'd appreciate if you updated the screenshot in step 5.7 @TheHellSite
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 10, 2022, 02:14:12 pm
The issue seems to be the NoSSL_condition. The suggested solution is to replace the condition type from Traffic is SSL (TCP request content inspection)  to Traffic is SSL (locally deciphered):

Code: [Select]
     # logging options
     # ACL: NoSSL_condition
-    acl acl_629f48c6073c95.86527303 req.ssl_ver gt 0
+    acl acl_629f48c6073c95.86527303 ssl_fc
 
     # ACTION: HTTPtoHTTPS_rule
     http-request redirect scheme https code 301 if !acl_629f48c6073c95.86527303

When doing so, the warning is gone. However, this is the first time I'm using HAProxy and I don't really know what I'm doing, so I wanted to check-in with you guys to ensure my solution is correct?

If so, I'd appreciate if you updated the screenshot in step 5.7 @TheHellSite

Thanks for the detailed explanation!
I already had this on my to do list, but then couldn't find the time for it and in the end simply forgot about it.

The HAProxy docs are also stating to use the "ssl_fc" condition.
https://www.haproxy.com/blog/redirect-http-to-https-with-haproxy/

I will change it in the next couple of days!
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 11, 2022, 03:20:57 pm
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: PaulePils on June 14, 2022, 11:41:40 am
Hello,

I just recently upgraded my home network with an opnsense and want to reconfigure some stuff in the same run.
Luckily I found this tutorial which was really easy to follow through, especally because of the screenshots :-)
Sadly I must have done something wrong but I can't put my finger on it.
When I try to access from internal LAN (IP address or Name), I get the following error:
From external (via mobile data):
503 Service Unavailable

If I enter http://IP_ADRESS then it opens the Nextcloud Login page but after entering the login data, it switches to https:// and the error appears.

HAProxy config
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62a76f360f0732.68695084 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62a76f360f0732.68695084

# Frontend: 2_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 2_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets no-tlsv12 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62a772caaae308.49400660.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62a76fbf29df39.71162057.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Nextcloud_backend ()
backend Nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Nextcloud 192.168.10.40:443 ssl verify none


Map files
Code: [Select]
#public access subdomains
nextcloud NEXTCLOUD_backend

Log files
Code: [Select]
Informational haproxy 192.168.10.101:4054 [14/Jun/2022:11:11:05.082] 2_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
My Network:
ISP --> Modem --> OPNSense --> Proxmox --> Server 1 Nextcloud

Maybe you can push me in the right direction :-)
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on June 15, 2022, 05:01:20 am
Hello,

I just recently upgraded my home network with an opnsense and want to reconfigure some stuff in the same run.
Luckily I found this tutorial which was really easy to follow through, especally because of the screenshots :-)
Sadly I must have done something wrong but I can't put my finger on it.
When I try to access from internal LAN (IP address or Name), I get the following error:
  • Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
  • Edge:ERR_SSL_PROTOCOL_ERROR
From external (via mobile data):
503 Service Unavailable

If I enter http://IP_ADRESS then it opens the Nextcloud Login page but after entering the login data, it switches to https:// and the error appears.

HAProxy config
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62a76f360f0732.68695084 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62a76f360f0732.68695084

# Frontend: 2_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 2_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets no-tlsv12 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62a772caaae308.49400660.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62a76fbf29df39.71162057.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Nextcloud_backend ()
backend Nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Nextcloud 192.168.10.40:443 ssl verify none


Map files
Code: [Select]
#public access subdomains
nextcloud NEXTCLOUD_backend

Log files
Code: [Select]
Informational haproxy 192.168.10.101:4054 [14/Jun/2022:11:11:05.082] 2_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
My Network:
ISP --> Modem --> OPNSense --> Proxmox --> Server 1 Nextcloud

Maybe you can push me in the right direction :-)

Since you are forcing HAProxy to use TLS 1.3. (As you use no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets no-tlsv12)
There might be issue taking out TLS_AES_128_GCM_SHA256.

According to RFC 8446, S9.1 (https://www.rfc-editor.org/rfc/rfc8446#section-9.1)
Quote
A TLS-compliant application MUST implement the TLS_AES_128_GCM_SHA256 [GCM] cipher suite and SHOULD implement the TLS_AES_256_GCM_SHA384 [GCM] and TLS_CHACHA20_POLY1305_SHA256 [RFC8439] cipher suites (see Appendix B.4).

Although it shouldn't have any effect by implementing a Cipher List, but Cipher List should be removed under TLS 1.3.



From external (via mobile data):
503 Service Unavailable

one of the reasons can be: it cannot read the map file, so it doesn't redirect to target backend.
Some device is hard to load map files during boot time. (One of my setup under Esxi is having such issue)
Apply the HAProxy one more time after all services finished loading can make it read the map file again. Or you can implement condition and rule to ignore such issue
example:
Condition name: nextcloud_cond, condition type: host contains, Host Contains: nextcloud
Rule name: nextcloud_ru, Test type: IF, Select conditions: nextcloud_cond, Execute function: Use specified backend Pool, Use backend pool: NEXTCLOUD_backend
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: PaulePils on June 15, 2022, 11:26:53 am
Thanks for your input. I changed the cipher accordingly to https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6 (https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6) and also applied your rule but still no success.
I am just curios because in the main post the value TLS_AES_128_GCM_SHA256 is also left out  ???

Out of curiosity I tried do enter the only the main domain DOMAIN.dedyn.io and not SUBDOMAIN.DOMAIN.dedyn.io. Then I get a different error
Quote
Firefox vertraut dieser Website nicht, weil das von der Website verwendete Zertifikat nicht für DOMAIN.dedyn.io gilt. Das Zertifikat ist nur gültig für *.DOMAIN.dedyn.io.
SSL_ERROR_BAD_CERT_DOMAIN
I can accept the risk but this is something I need to do on all devices...

Could the problem be that I already had DynDNS by another provider and LE certificate? If yes how can I revoke it?
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 15, 2022, 03:15:28 pm
When I try to access from internal LAN (IP address or Name), I get the following error:
  • Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
  • Edge:ERR_SSL_PROTOCOL_ERROR

This error usually means that you tried to access a service using HTTPS that only supports HTTP.

Quote
internal LAN (IP address or Name)
What do you mean by that? You should use the FQDN and not the local hostname / IP in order to use the reverse proxy (HAProxy).

If I enter http://IP_ADRESS then it opens the Nextcloud Login page but after entering the login data, it switches to https:// and the error appears.
Again, which IP are you entering? The nextcloud local IP or your public IP?
Your public IP should ALWAYS forward HTTP to HTTPS and not display any webpages via HTTP whatsoever! (HTTPtoHTTPS_rule)

My Network:
ISP --> Modem --> OPNSense --> Proxmox --> Server 1 Nextcloud
Is the firewall in your modem disabled or is it still doing NAT?

Thanks for your input. I changed the cipher accordingly to https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6 (https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6) and also applied your rule but still no success.
I am just curios because in the main post the value TLS_AES_128_GCM_SHA256 is also left out  ???

You won't need any 128 bit ciphers unless you are using very very old devices to access your services which I highly doubt since you only want to enable TLS v1.3 anyway.
So you can safely use the cipher suites in my first post which are identical to the ones from the Mozilla SSL configurator but have the "insecure / weak" 128 bit ciphers removed.

Out of curiosity I tried do enter the only the main domain DOMAIN.dedyn.io and not SUBDOMAIN.DOMAIN.dedyn.io. Then I get a different error
Quote
Firefox vertraut dieser Website nicht, weil das von der Website verwendete Zertifikat nicht für DOMAIN.dedyn.io gilt. Das Zertifikat ist nur gültig für *.DOMAIN.dedyn.io.
SSL_ERROR_BAD_CERT_DOMAIN
I can accept the risk but this is something I need to do on all devices...

Could the problem be that I already had DynDNS by another provider and LE certificate? If yes how can I revoke it?

Well, the error is pretty much self-explanatory isn't it?
In my tutorial the wildcard certificate is only valid for the 2nd-level subdomains "*.DOMAIN.dedyn.io" but not for the 1st-level subdomain "DOMAIN.dedyn.io" itself.

If you want the certificate to also cover for "domain.dedyn.io" then you will have to change the certificate in the ACME client to match that. See Part 3 - Step 6.
You will however only need this if you are serving a website in the domain root without "www" infront of it.

Code: [Select]
# currently configured
Common Name = *.DOMAIN.dedyn.io

# what you MIGHT want
Common Name = DOMAIN.dedyn.io
Alt Names = *.DOMAIN.dedyn.io

This will cover the 1st-level subdomain including all 2nd-level subdomains.
Don't forget to reissue the certificate.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 15, 2022, 03:25:18 pm
Since you are forcing HAProxy to use TLS 1.3. (As you use no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets no-tlsv12)
There might be issue taking out TLS_AES_128_GCM_SHA256.

According to RFC 8446, S9.1 (https://www.rfc-editor.org/rfc/rfc8446#section-9.1)
Quote
A TLS-compliant application MUST implement the TLS_AES_128_GCM_SHA256 [GCM] cipher suite and SHOULD implement the TLS_AES_256_GCM_SHA384 [GCM] and TLS_CHACHA20_POLY1305_SHA256 [RFC8439] cipher suites (see Appendix B.4).

Well, in order to get an 100% A+ rating at SSL Labs it is mandatory to remove any ciphers below or equal to 128 bit. This is why I haven't included them in my cipher list and cipher suites.
So unless you have really old devices with outdated browsers I would suggest to only use ciphers with a strength of 256 bit or above.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: PaulePils on June 15, 2022, 05:28:26 pm
When I try to access from internal LAN (IP address or Name), I get the following error:
  • Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
  • Edge:ERR_SSL_PROTOCOL_ERROR

This error usually means that you tried to access a service using HTTPS that only supports HTTP.
Does this mean I need an additional vhost config for the Nextcloud? One for :80 and one for :443? But the certificate lies on a different machine (opnsense). How can I point it there? Or am I missing something?
By deleting the "overwirte" codes in the nextcloud config I at least got access in my internal LAN. I assume the HAProxy config is correct and I need to make corrections in the nextcloud config.

Quote
Quote
internal LAN (IP address or Name)
What do you mean by that? You should use the FQDN and not the local hostname / IP in order to use the reverse proxy (HAProxy).
It was the IP address of the Nextcloud machine. But if it doesn't work that way that is new for me but thanks for pointing it out.

Quote
If I enter http://IP_ADRESS then it opens the Nextcloud Login page but after entering the login data, it switches to https:// and the error appears.
Again, which IP are you entering? The nextcloud local IP or your public IP?
Your public IP should ALWAYS forward HTTP to HTTPS and not display any webpages via HTTP whatsoever! (HTTPtoHTTPS_rule)
I entered the local IP address of the nextcloud machine. But this also happens when I enter the FQDN.

Quote
My Network:
ISP --> Modem --> OPNSense --> Proxmox --> Server 1 Nextcloud
Is the firewall in your modem disabled or is it still doing NAT?
It is dumb modem (TC-4400) therefor it should not have NAT. It only provides the connection to my ISP (Vodafone Cable).

Quote
Thanks for your input. I changed the cipher accordingly to https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6 (https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6) and also applied your rule but still no success.
I am just curios because in the main post the value TLS_AES_128_GCM_SHA256 is also left out  ???
Quote
You won't need any 128 bit ciphers unless you are using very very old devices to access your services which I highly doubt since you only want to enable TLS v1.3 anyway.
So you can safely use the cipher suites in my first post which are identical to the ones from the Mozilla SSL configurator but have the "insecure / weak" 128 bit ciphers removed.
OK
Quote
Out of curiosity I tried do enter the only the main domain DOMAIN.dedyn.io and not SUBDOMAIN.DOMAIN.dedyn.io. Then I get a different error
Quote
Firefox vertraut dieser Website nicht, weil das von der Website verwendete Zertifikat nicht für DOMAIN.dedyn.io gilt. Das Zertifikat ist nur gültig für *.DOMAIN.dedyn.io.
SSL_ERROR_BAD_CERT_DOMAIN
I can accept the risk but this is something I need to do on all devices...

Could the problem be that I already had DynDNS by another provider and LE certificate? If yes how can I revoke it?

Well, the error is pretty much self-explanatory isn't it?
In my tutorial the wildcard certificate is only valid for the 2nd-level subdomains "*.DOMAIN.dedyn.io" but not for the 1st-level subdomain "DOMAIN.dedyn.io" itself.

If you want the certificate to also cover for "domain.dedyn.io" then you will have to change the certificate in the ACME client to match that. See Part 3 - Step 6.
You will however only need this if you are serving a website in the domain root without "www" infront of it.

Code: [Select]
# currently configured
Common Name = *.DOMAIN.dedyn.io

# what you MIGHT want
Common Name = DOMAIN.dedyn.io
Alt Names = *.DOMAIN.dedyn.io

This will cover the 1st-level subdomain including all 2nd-level subdomains.
Don't forget to reissue the certificate.
I don't need the coverage of "domain.dedyn.io". So this should be fine.[/quote]
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 15, 2022, 10:23:09 pm
When I try to access from internal LAN (IP address or Name), I get the following error:
  • Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
  • Edge:ERR_SSL_PROTOCOL_ERROR

This error usually means that you tried to access a service using HTTPS that only supports HTTP.
Does this mean I need an additional vhost config for the Nextcloud? One for :80 and one for :443? But the certificate lies on a different machine (opnsense). How can I point it there? Or am I missing something?
By deleting the "overwirte" codes in the nextcloud config I at least got access in my internal LAN. I assume the HAProxy config is correct and I need to make corrections in the nextcloud config.


Code: [Select]
    server Nextcloud 192.168.10.40:443 ssl verify noneThis line in your HAProxy config indicates that you are accessing your nextcloud webinterface from your local network using "https://192.168.10.40".
If the above is not true and you are accessing it using "http://192.168.10.40" instead then you should have figured your error after reading this sentence.

IF you can already access nextcloud from within your local using its Local_IP:Port then you won't have to change any settings at nextcloud.

Also... Please do yourself a favour and read about what a reverse proxy is, what it does and how it works.
No offense but I think you don't really know what you are doing here.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brynjolm on June 17, 2022, 09:07:30 pm
Hi! First off, i want to thank you for the detailed guide you posted. I am new to HAProxy and have some questions regarding some configs, do you still reply to this post? or should i ask or open another thread somewhere? Thanks!
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 17, 2022, 10:03:40 pm
Hi! First off, i want to thank you for the detailed guide you posted. I am new to HAProxy and have some questions regarding some configs, do you still reply to this post? or should i ask or open another thread somewhere? Thanks!

As long as the question is related to THIS tutorial then feel free to ask!

Otherwise please ask here: https://forum.opnsense.org/index.php?board=28.0
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brynjolm on June 17, 2022, 11:39:15 pm
Thanks! So, i set up HAP as per the guide, and am wondering if its a normal consequence to have 503 error on my truenas webui. Internally i can acces it no problems, outside of the home network i am unable to. Sidenote, i do not plan to expose the web ui or planned to. I was just using it as a sort of test run to see if it will resolve to my domain.truenas.com. I also did expose plex under port 32400. But seem to encounter some TLS handshake errors. It would be nice if you could take a look at my config and point me in the right direction. I did only a A and AAAA record for wildcard hostname i.e *.mydomain.com. I dont know if thats enough.
My general goal is to only expose plex and some other services like nextcloud, but thats for another section of the forums. Also my network is kinda flat. All services run through a single VLAN.
I created two Servers and two backends. One for Truenas and plex respectively.
i have been trying to make a stable connection to plex but it just seems to either time out or lose connection after a while.

Heres and output of my config file, i have some disabled stuff in the config file since i turn on and off stuff for testing. Also renamed some stuff, truenas is just the Physical server with the IP. (myloopbackip) is the virutal address.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

#
# NOTE: HAProxy is currently DISABLED
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr libc,last
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend (Listening on ip:80 / ip:443)
frontend SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind :::80 name :::80
    bind :::443 name :::443
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTP_frontend (Listening on 127.10.20.5)
frontend HTTP_frontend
    bind (myloopbackip):80 name (myloopbackip):80 accept-proxy
    bind :::80 name :::80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_62aa8dcf894a87.42381056 ssl_fc

    # ACTION: HTTPtoHTTPS
    http-request redirect scheme https code 301 if !acl_62aa8dcf894a87.42381056

# Frontend: HTTPS_frontend (Listening on (myloopbackip))
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind (myloopbackip):443 name (myloopbackip):443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62aad04a028639.71957640.certlist
    bind :::443 name :::443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62aad04a028639.71957640.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: Public_subdomains_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62aa8e31993357.88056717.txt)]
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server (myloopbackip) send-proxy-v2 check-send-proxy

# Backend (DISABLED): router_Backend (router Backend)

# Backend: truenas_Backend (truenas Backend)
backend truenas_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server truenas truenasip:443 ssl sni str(truenas) verify none send-proxy-v2 check-send-proxy

# Backend: plex_backend (plex Backend)
backend plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Plex truenasip:32400 send-proxy-v2 check-send-proxy


This is my map file:

Code: [Select]
# public access domains
truenas truenas_backend
plex plex_backend

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: droren on June 18, 2022, 06:17:16 am
I've followed the article and able to setup few subdomain to internal machines in my network.

I have dumb questions... when I created new real server and backend server for one my Synology packages, I initially used HTTPS port and received 400 bad request error The plain HTTP request was sent to HTTPS port

When I changed the port in real server settings to HTTP request, everything worked fine and I am able to access the internal server with SSL certificate.

Why?
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 21, 2022, 10:49:24 am
I've followed the article and able to setup few subdomain to internal machines in my network.

I have dumb questions... when I created new real server and backend server for one my Synology packages, I initially used HTTPS port and received 400 bad request error The plain HTTP request was sent to HTTPS port

When I changed the port in real server settings to HTTP request, everything worked fine and I am able to access the internal server with SSL certificate.

Why?

FAQ - Do I need to enable "SSL" in the Real Server configuration of a service?

Thanks for NOT reading my first post properly.  :(
I explained everything there very well.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 21, 2022, 10:58:08 am
Heres and output of my config file, i have some disabled stuff in the config file since i turn on and off stuff for testing. Also renamed some stuff, truenas is just the Physical server with the IP. (myloopbackip) is the virutal address.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

#
# NOTE: HAProxy is currently DISABLED
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr libc,last
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend (Listening on ip:80 / ip:443)
frontend SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind :::80 name :::80
    bind :::443 name :::443
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTP_frontend (Listening on 127.10.20.5)
frontend HTTP_frontend
    bind (myloopbackip):80 name (myloopbackip):80 accept-proxy
    bind :::80 name :::80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_62aa8dcf894a87.42381056 ssl_fc

    # ACTION: HTTPtoHTTPS
    http-request redirect scheme https code 301 if !acl_62aa8dcf894a87.42381056

# Frontend: HTTPS_frontend (Listening on (myloopbackip))
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind (myloopbackip):443 name (myloopbackip):443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62aad04a028639.71957640.certlist
    bind :::443 name :::443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62aad04a028639.71957640.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: Public_subdomains_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62aa8e31993357.88056717.txt)]
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server (myloopbackip) send-proxy-v2 check-send-proxy

# Backend (DISABLED): router_Backend (router Backend)

# Backend: truenas_Backend (truenas Backend)
backend truenas_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server truenas truenasip:443 ssl sni str(truenas) verify none send-proxy-v2 check-send-proxy

# Backend: plex_backend (plex Backend)
backend plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Plex truenasip:32400 send-proxy-v2 check-send-proxy


This is my map file:

Code: [Select]
# public access domains
truenas truenas_backend
plex plex_backend

1. The map file is case sensitive. Fix it.

2. Remove the "send-proxy-v2 check-send-proxy" directives from the backends of your actual services. These two options are only necessary on the "SSL_backend".

3. Why do you have the sni setting configured in your truenas real server? that shouldn't be necessary at all. Also remove the "send-proxy-v2 check-send-proxy" directives from the backend (see 1.).
Code: [Select]
server truenas truenasip:443 ssl sni str(truenas) verify none send-proxy-v2 check-send-proxy
4. Your "HTTP_frontend" and "HTTPS_frontend" should ONLY be listening to your SSL_server IP address. Not to "0.0.0.0:0" or "::::0". Just think about it and take a look at my diagram in the first post... You should quickly figure that this doesn't make any sense.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brynjolm on June 25, 2022, 12:16:30 am
Quote
1. The map file is case sensitive. Fix it.

2. Remove the "send-proxy-v2 check-send-proxy" directives from the backends of your actual services. These two options are only necessary on the "SSL_backend".
4. Your "HTTP_frontend" and "HTTPS_frontend" should ONLY be listening to your SSL_server IP address. Not to "0.0.0.0:0" or "::::0". Just think about it and take a look at my diagram in the first post... You should quickly figure that this doesn't make any sense.


Thanks! That solved a lot of the problems i had.

I also had to edit some settings on my cloudflare setup in order for it to work. I now have full access to everything aside from nextcloud and some other stuff. while still being proxied behind cloudflare. I dont know if it will be useful for other cloudflare users. But for me at least since i wanted to go cloudflare for everything. I ended up also using the cloudflare certs and uploading it to HAP while still being onf Full(Strict) mode. i dont know if its and intended sideffect but i didnt have to use split dns for this. I could just call up everything by subdomain.domain.com locally.
Also with regards to the map file for the local subdomanins. The format should look like this?
Quote
local loca_backend

plex plex_backend

Should there be a space in between?
Last questions. Is it possible to use the generated Cert for both truenas and opnsense instead of the self signed one? would it break the config? Also is there a specifi HAP sub, or should i just post in the General section?
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 27, 2022, 04:08:58 pm
I could just call up everything by subdomain.domain.com locally.
This is because your domains resolve to cloudflares IP and not your own public IP.

Also with regards to the map file for the local subdomanins. The format should look like this?
Quote
local loca_backend

plex plex_backend

Should there be a space in between?
https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/syntax/


Last questions. Is it possible to use the generated Cert for both truenas and opnsense instead of the self signed one? would it break the config? Also is there a specifi HAP sub, or should i just post in the General section?
Shouldn't break anything but isn't neccessary.
https://discourse.haproxy.org/
or
https://forum.opnsense.org/index.php?board=28.0
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: xxavarlonxx on June 29, 2022, 01:04:47 pm
Thnak you very much. Thats the best guide about HAProxy - Lets Encrypt I've ever found. Everything works and its not a mickey mouse setup, its a good setup for production..