OPNsense Forum

English Forums => Tutorials and FAQs => Topic started by: TheHellSite on May 31, 2021, 01:06:11 pm

Title: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 31, 2021, 01:06:11 pm
Hello,

when I started implementing HAProxy in my network I couldn't find any complete and well written guide out there. I had to puzzle everything together from various websites.
So I thought I would save many of you a lot of time and provide my ultimate HAProxy on OPNsense guide.  :)

This tutorial will show you how to configure HAProxy as a reverse proxy on OPNsense using wildcard certificates from Let's Encrypt.
It is going to be a step-by-step guide with images on how to set things up while also explaining why we set things up in a certain way.
I will try to make this as complete and detailed as possible.
If you think that there is anything wrong or missing, feel free to tell me about it and I will consider changing it.

If this guide was helpful to you then please leave me a thanks down below as it took me several days to write this down.

Kind Regards
TheHellSite



This configuration is tested to be working on OPNsense 22.7.x (OpenSSL flavour) with latest updates as of 20220808.



How to ask for help.

From now on I will no longer provide support to people asking for help without giving any additional information about their setup.

1. ALWAYS include the HAProxy Config Export.
2. ALWAYS include relevant HAProxy errors and/or log entries.
3. ALWAYS include details about your setup, your goal, the service, ... anything relevant to the issue.




Changelog



Current Ciphers and Cipher Suites for a 100% A+ rating at SSLLabs
Last updated on 20220615 using Mozilla SSL Configuration Generator.
https://ssl-config.mozilla.org/#server=haproxy&version=2.4.17&config=intermediate&openssl=1.1.1o&guideline=5.6

All ciphers with a strength of 128 bit or below have been removed in order to get a 100% A+ rating at SSL Labs.

Code: [Select]
Cipher List
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384

Cipher Suites
TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

What will the end result look like?
We will have a wildcard SSL certificate from Let's Encrypt that gets an A+ rating with 100% score at all points from SSLLabs.
https://www.ssllabs.com/ssltest/

We will also have two levels of load balancing our services.

Level 0 - SSL Offloading disabled
WWW --> WAN interface --> OPNsense --> HAProxy SNI Frontend --> internal servers / services

Level 1 - SSL Offloading enabled
WWW --> WAN interface --> OPNsense --> HAProxy SNI Frontend --> HAProxy SSL Frontend --> internal servers / services

Visual Schematic
(Idea: @cookiemonster (https://forum.opnsense.org/index.php?topic=23339.msg123295#msg123295))
(https://i.postimg.cc/MM2dj6zQ/P000-001-Schematic.png) (https://postimg.cc/MM2dj6zQ)


What are we going to do?

FAQ



The Configuration

Part 1 - Plugin Installation




Part 2 - DynDNS Configuration




Part 3 - Let's Encrypt (ACME Client)




Part 4 - System Preparation




Part 5 - HAProxy Configuration




Part 6 - Access from internal networks
If you try to access your URL "your_service.your_subdomain.dedyn.io" from a device in your internal network, it should fail.
There are two ways of fixing this. I will cover both options but keep in mind that Split DNS (Option A) is the suggested way of doing it.
NAT Reflection (Option B) is an inferior solution since you lose the ability to track originating source IP in HAProxy when going through NAT. (@sorano (https://forum.opnsense.org/index.php?topic=23339.msg111439#msg111439))

Option A - Split DNS (https://docs.opnsense.org/manual/unbound.html#overrides)
Option B - NAT Reflection (https://docs.opnsense.org/manual/nat.html)

Option A - Split DNS (DNS Overrides)
Since you are using OPNsense you are probably also using the Unbound DNS plugin as your local DNS server.
Because of that you can easily set up DNS overrides.

Option B - NAT Reflection
Please note that NAT Reflection is only applicable on port forwarding rules so you will have to change the "Allow HAProxy" rule to a port forwarding rule with the localhost (or some random virtual IP on the localhost) as target.




Part 7 - Advanced Configuration: local-access-only subdomains
Imagine you have a service that you would like to access / protect using your brand new reverse proxy without making it available on the internet?
Well, HAProxy has got you covered!

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: browne on June 01, 2021, 10:35:23 am
Thank you very much!
This helped me switching from regular certificates to wildcard certificates!
I now also do score 100% A+ in the SSL test.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: skittlebrau on June 01, 2021, 04:23:25 pm
Thank you so much for this guide! I was completely lost in the new UI layout before.

Do you mind showing in Step 5 an example of how you've configured 'PLEX_backend'? I noticed you referred to it, but there wasn't a screenshot for it.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 01, 2021, 06:52:28 pm
The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different. :)

--> I will add it to the tutorial.

Reason: https://forum.opnsense.org/index.php?topic=23339.msg111143#msg111143


It is also almost everytime the same procedure to configure / add a new service in HAProxy.
--> I will add this to the FAQ.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: ejball02 on June 01, 2021, 08:55:28 pm
I'm using a self-signed cert for HTTPS inspection for content filtering. I've got OPT2 configured as a guest network on my Protectli, and content filtering, using shallalist works great. Only downside, is that when I try to access am HTTPS site, Firefox/Chrome always give a warning page: "Your connection is not private" "ERR_CERT_AUTHORITY_INVALID". After much Googling, I came across an old post, that said Let's Encrypt can give public certs which would get rid of the message.

I added the LE plugin but couldn't figure out, how to create a cert for use on the Foward Proxy "CA to use" field, required for SSL inspection. Looking through this walkthrough, I'm wondering if there is something here that can help achieve creating a cert for content filtering. Anyone have any experience with this?
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 01, 2021, 11:36:13 pm
I'm wondering if there is something here that can help achieve creating a cert for content filtering. Anyone have any experience with this?
No offense, but this is meant to be a tutorial related to HAProxy in a reverse proxy setup.  :)
As this is the tutorial sub-forum it is not really meant to ask for help and especially not regarding different topics.
You can of course follow this guide to the part where you get your Let's Encrypt certificate, but from there on you will be on your own.

I think you should probably open up your own thread in the help forums and ask for help there.

https://forum.opnsense.org/index.php?board=27.0
or here
https://forum.opnsense.org/index.php?board=28.0
or here
https://forum.opnsense.org/index.php?board=1.0
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: skittlebrau on June 02, 2021, 02:24:48 am
The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different. :)

I assumed that it would be the case, so thanks for confirming  ;D

One part that's tripping me up is in my plex_backend pool and other ones I set for my internal services like Resilio Sync, if I set the mode to TCP (Layer 4) then I get a syntax error below. Switching the mode to HTTP for those two backends resolves the syntax error, but produces blank pages. Stats area in HAProxy shows the services as being UP.

[NOTICE] 152/082107 (17536) : haproxy version is 2.2.14-a07ac36
[ALERT] 152/082107 (17536) : http frontend '1_HTTPS_frontend' (/usr/local/etc/haproxy.conf.staging:70) tries to use incompatible tcp backend 'sync1_backend' (/usr/local/etc/haproxy.conf.staging:117) in a 'use_backend' rule (see 'mode').
[ALERT] 152/082107 (17536) : http frontend '1_HTTPS_frontend' (/usr/local/etc/haproxy.conf.staging:70) tries to use incompatible tcp backend 'plex_backend' (/usr/local/etc/haproxy.conf.staging:104) in a 'use_backend' rule (see 'mode').
[ALERT] 152/082107 (17536) : Fatal errors found in configuration.

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 02, 2021, 08:08:18 am
The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different. :)

I assumed that it would be the case, so thanks for confirming  ;D

One part that's tripping me up is in my plex_backend pool and other ones I set for my internal services like Resilio Sync, if I set the mode to TCP (Layer 4) then I get a syntax error below. Switching the mode to HTTP [...]
[/font]

Now I remember correctly.
The SSL_backend is different from most other backends since it is NOT running in HTTP mode.
But HTTP mode is usually the one you want for a web based service.
I changed that part in the tutorial. Please take a look at the backend creation.

My bad! I wrote the tutorial from brain memory. So I guess you are my beta tester.  ;D
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: skittlebrau on June 02, 2021, 08:54:51 am
All good, you've been so helpful.

I checked through everything again and realised that when setting up the NAT port forward, I forgot to enter the dropdown menu at the end to add the associated filter rule. I was too used to pfSense automatically selecting that by default, so no wonder it wasn't working despite changing from TCP to HTTP mode for the backend services!

For posterity sake, I've saved your guide for my own reference and archived it to PDF as well.

If there's a particular charity you support, send me a private message and I'll happily donate to one in your name, along with a donation to the OPNsense project.  :D
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: michaelgo on June 02, 2021, 03:19:36 pm
Hi,
thank you for an amazing guide.

in part 5.5 the picture is broken, can you please re-post (it's working now)
also, 5.7 "plex condition" is it a rule? (5.7 and 5.8 screenshots are vice versa)

also, i don't have "Backends" only "Backend pools" there
Services --> HAProxy --> Settings --> Virtual Services --> Backends
is it what you mean?
thanx
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 03, 2021, 01:01:13 am
Yep, those pictures are in the wrong order and yes, I meant Backend Pools.
I will change this.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 03, 2021, 01:24:05 pm
If there's a particular charity you support, send me a private message and I'll happily donate to one in your name, along with a donation to the OPNsense project.  :D
A donation to OPNsense would be happily welcome I guess. :)
I myself don't want any money for this. We are all more or less using OPNsense free of charge so the least anyone can do is help the community or donate to the project.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: sorano on June 05, 2021, 03:00:21 pm
@TheHellSite Great guide! I'm sure it will help alot of people trying to get this kind of setup up and running.

I'm running a similar setup and have some suggestions for improvement to your guide.

1. You dont need to use virtual IP's.
If you bind the HAProxy frontends to 0.0.0.0:80 & 0.0.0.0:443 it will bind to you WAN interface (even if it's dynamic). And when you do there is no need for NAT forwarding to the virtual IP's so a simple firewall rule for 80/443 on the WAN interface is enough.
Then you can bind the SSL terminating frontend to 127.0.0.1:[port] and use that IP for your SSL terminating "real server".
With the added bonus of that it performs better in a CARP setup.

2. Use map files {Advanced --> Map files}
Using map files to map domains to backends will keep your config rules alot less cluttered, especially when you have many subdomains to match. With map files 1 rule is enough to map all of your domains.

Example for map file:
plex.mydomain.tld Backend_plex

Example for rule:
(Execute function: Map domains to backend pools using a map file
Map file: [name of your map file]

Then apply the map file rule to your SSL terminating frontend.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 07, 2021, 11:29:22 am
1. You dont need to use virtual IP's.
2. Use map files {Advanced --> Map files}

1. You dont need to use virtual IP's.
I totally get your point! This makes indeed sense but I think only if you have a static WAN IP.
As it would break the access from internal networks to the external URLs "service.subdomain.mydomain.tld" if one enabled that access using DNS rewrite rules. I am not aware of a way to rewrite DNS entries in Unbound to the WAN interface address.

With NAT reflection your way of setting this up can of course work.


2. Use map files {Advanced --> Map files}
I haven't used those yet but looks very promising!
This really makes sense in a big environment with lots of subdomains.
Thank you for pointing this out! I will add it to the FAQ.  :)
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: sorano on June 07, 2021, 02:21:02 pm
1. You dont need to use virtual IP's.
I totally get your point! This makes indeed sense but I think only if you have a static WAN IP.

Well, I only have dynamic IP's for my WAN interfaces. MultiWAN consisting of a fiber primary with LTE failover, two OPNsense hosts running CARP on all interfaces except for WAN (since I cannot get proper stateful failover with dynamic WAN IP's).

As it would break the access from internal networks to the external URLs "service.subdomain.mydomain.tld" if one enabled that access using DNS rewrite rules. I am not aware of a way to rewrite DNS entries in Unbound to the WAN interface address.

With NAT reflection your way of setting this up can of course work.

The way I'm doing access from internal networks is with Split DNS (DNS override as you call it).
In my opinion NAT reflection is an inferior solution since you lose the ability to track originating source IP in HAProxy when going through NAT.

Since HAProxy is already listening on 0.0.0.0 (all available IPv4 interfaces) I resolve the Split DNS to the internal IP of my DMZ CARP IP (but any internal IPv4 interface will do as long as you allow 80/443).

I also have certain domains I don't want reachable from the Internet so I use two map file rules, one for internal domains along with a condition that checks that source is RFC1918.

And one for external domains where I also require additional authentication.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 09, 2021, 12:03:57 pm
Since HAProxy is already listening on 0.0.0.0 (all available IPv4 interfaces) I resolve the Split DNS to the internal IP of my DMZ CARP IP (but any internal IPv4 interface will do as long as you allow 80/443).
So this means you are actually also using sort of a virtual IP.  :D
Okay so you say the easier way is like this:
- Firewall rule on WAN allowing 443+80
- SNI_frontend listening on 0.0.0.0:80+443 (Which covers all interfaces)
- SSL_frontend listening on 127.0.0.1:443
- Split DNS pointing to f.e. the LAN IP of OPNsense.

You are right, with that you could skip the virtual IPs.
I would however still prefer to use a virtual IP for the SSL_frontend as I don't want to hijack the localhost for it. I just prefer to keep things seperated.

I also have certain domains I don't want reachable from the Internet so I use two map file rules, one for internal domains along with a condition that checks that source is RFC1918.
For this I guess you have to use 0.0.0.0 on the SNI_frontend otherwise you would need another NAT rule forwarding 443-LAN traffic to the virtual IP.

EDIT 20210611:
Implemented @sorano (https://forum.opnsense.org/index.php?topic=23339.msg111339#msg111339)'s enhancements  :)
Title: Re: Tutorial: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ SSLLabs Rating
Post by: TheHellSite on June 13, 2021, 04:13:00 pm
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: pintu1228 on July 08, 2021, 04:49:36 pm
This is exactly what I was looking for, have had trouble coming from pfsense to opnsense to setup haproxy/let's encrypt.  I have cloudflare setup to use DNS.

I am using google domain, how do I go about setting up the 1st part (Dynamic DNS), do I need to create 3 custom records:

domain.com (A type)
*.domain.com (A type)
www.domain.com (CNAME)

And also I created separate dynamicDNS for plex.domain.com to use for part 7 (configure Dynamic DNS on opnsense).

Thanks
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 09, 2021, 12:28:11 pm
I am using google domain, how do I go about setting up the 1st part (Dynamic DNS), do I need to create 3 custom records:

domain.com (A type)
*.domain.com (A type)
www.domain.com (CNAME)

It all depends on what you would like to achieve!
You can set your DynDNS to update the IP of "domain.com" and then have a CNAME record in the form of "subdomain.domain.com" pointing to the IP of "domain.com".

However since you might also want to host a website on "domain.com / www.domain.com", you will probably want a subdomain for your domain first!
I myself always create individual 1st-level-subdomains for each physical location that is sitting in a different building / city / country having there own public IP.

f.e. location.domain.tld = earth.thehellsite.com / moon.thehellsite.com / mars.thehellsite.com and so on

And then (as I already explained in my tutorial) a wildcard 2nd-level-subdomain.
Which means that any string in the 2nd-level-subdomain is pointing to my 1st-level-subdomain.

f.e. any_string.location.domain.tld = *.earth.thehellsite.com = abcxyz.earth.thehellsite.com / 123.earth.thehellsite.com


If you do it my way then you only need to set up three things in your Google DNS zone.


Your DynDNS should then be updating the "A Record".




But again, there are many different ways to achieve this.
You just need to find out how you would like to do it.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: _Alchemist_ on July 14, 2021, 01:45:11 pm
Thanks a lot for the write up, I will try this out as soon as I can :)

The only thing that could be added on Part 4.3 is to use an Alias for Port 80 and 443 to only use one Firewall Rule ;)
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 14, 2021, 03:40:05 pm
Thanks a lot for the write up, I will try this out as soon as I can :)

The only thing that could be added on Part 4.3 is to use an Alias for Port 80 and 443 to only use one Firewall Rule ;)

Good Idea!
Will add this in my next update.
Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: saarko on July 27, 2021, 11:44:25 pm

I also have certain domains I don't want reachable from the Internet so I use two map file rules, one for internal domains along with a condition that checks that source is RFC1918.

And one for external domains where I also require additional authentication.

Dear @TheHellSite,
thanks for the great tutorial! It works well.

Dear @sorano,

thanks for your input. The hint with map file works well. However, I am unable to create a rule with multiple "OR" conditions for various sub-domains to match and check it with an "AND" condition to test if it is an internal IP. It shall cover your described rule.

Currently I try to create a rule like:
use map file 1
IF
condition 1 "subdomain1" OR condition 2 "subdomain2"
AND condition 3 "local IP (RFC1918)" is matched


How did you solve this with the conditions and rules within OPNsense HAProxy plugin?

thanks in advance for your help and reply.
Saarko
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 28, 2021, 09:18:12 pm
Thanks for the feedback!

I had a chat with @sorano and changed my config a little bit in the meantime.

I will update my post in a few days, hopefully. Very busy with work atm.
This will also solve your problem @saarko.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 29, 2021, 07:34:44 pm
I followed this, however, decided against using the LE and now not getting 100% A+. is there something I am missing...
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 29, 2021, 08:07:09 pm
I followed this, however, decided against using the LE and now not getting 100% A+. is there something I am missing...

Since you didn't use "LE"? (I guess you mean Let's Encrypt certificates) you should probably check out this page.
https://github.com/ssllabs/research/wiki/SSL-Server-Rating-Guide

So if I would have had to guess it is most likely due to the fact that the key length of your certificate is too short!

If you followed the rest of my guide (OPNsense settings + HAProxy settings) accordingly and didn't miss any configuration steps, you should be getting 100% A+ at SSLLabs!
Again, provided that your certificate has a long enough key length...



I hope you can understand that it is very hard to help you out, since you didn't share any more details about your config, the SSLLabs test results, the certificate you are using ...
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 29, 2021, 10:23:29 pm


@saarko
I didn't manage to finish part 7 "local-access-only subdomains" today.
I will try to do this tomorrow!

In the meantime you can already change your config over to map files.  :)
See part 5, step 7 to step 10. This is the only thing that has changed for using map files.
Map files will also make part 7 very easy for you!
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 30, 2021, 05:43:22 pm
ggetting this from hap:

[WARNING] 210/114212 (27105) : Proxy '1_HTTPS_Frontend': no-sslv3/no-tlsv1x are ignored for bind '192.168.1.50:443' at [/usr/local/etc/haproxy.conf.staging:71]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
Warnings were found.
Configuration file is valid
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 30, 2021, 06:21:48 pm
ggetting this from hap:

[WARNING] 210/114212 (27105) : Proxy '1_HTTPS_Frontend': no-sslv3/no-tlsv1x are ignored for bind '192.168.1.50:443' at [/usr/local/etc/haproxy.conf.staging:71]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
Warnings were found.
Configuration file is valid

Please post a screenshot of your entire HTTPS_frontend config, including advanced.
You can redact your rules.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 30, 2021, 06:51:11 pm


Code: [Select]
# Frontend: 1_HTTPS_Frontend
frontend 1_HTTPS_Frontend
    # WARNING: ciphersuites cannot be used with flavour libressl.
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 10.10.1.50:5555 name 10.10.1.50:555 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-min-ver TLSv1.2 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60f9db54XXX3488.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 30, 2021, 07:08:35 pm


Code: [Select]
# Frontend: 1_HTTPS_Frontend
frontend 1_HTTPS_Frontend
    # WARNING: ciphersuites cannot be used with flavour libressl.
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 10.10.1.50:5555 name 10.10.1.50:555 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-min-ver TLSv1.2 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60f9db54XXX3488.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

I was already assuming what you did (wrong) and your config is confirming it.  ;D

First of all. This is not an error but only a warning which won't make HAProxy refuse to start.
Your setup should still be working if you did everything else correctly.

This warning is appearing because you set "ssl-min-version = TLS1.2" in your HTTPS_frontend config.
Which is pretty much pointless since my config example is already blocking "sslv3 tls1.0 tls1.1" using the "Bind options" in the HTTPS_frontend field. This is why HAProxy is giving you a warning.
You can of course also remove my bind options and only use "min-ssl-version = TLS1.2", this is up to you.
Either way will work and allow only TLS1.3 connections.


Code: [Select]
# Frontend: 1_HTTPS_Frontend
frontend 1_HTTPS_Frontend
    # WARNING: ciphersuites cannot be used with flavour libressl.
...
What you should be aware of is that I specifically mentioned that my guide might not work well on LibreSSL due to the fact that not all features of the last HAProxy version is supported by the LibreSSL firmware variant of OPNsense, yet!
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 30, 2021, 07:13:04 pm
Dear @TheHellSite,
thanks for the great tutorial! It works well.

Dear @sorano,

thanks for your input. The hint with map file works well. However, I am unable to create a rule with multiple "OR" conditions for various sub-domains to match and check it with an "AND" condition to test if it is an internal IP. It shall cover your described rule.

Currently I try to create a rule like:
use map file 1
IF
condition 1 "subdomain1" OR condition 2 "subdomain2"
AND condition 3 "local IP (RFC1918)" is matched


How did you solve this with the conditions and rules within OPNsense HAProxy plugin?

thanks in advance for your help and reply.
Saarko

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 30, 2021, 08:08:34 pm
so not sure what you are referring to...

I used the cloudflare... what option is needed for the cert that's different from the  norm.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 30, 2021, 09:26:40 pm
So not sure if you understand what I am trying to tell you.........

I followed this, however, decided against using the LE and now not getting 100% A+. is there something I am missing...
Firstly, when you made your one-liner about the bad SSLLabs result, I asked you for details about your config and the SSLLabs result.
You gave NONE.
How could anyone be able to help you with that little information given?!  :o
This makes me even a bit angry.

Secondly, you asked why HAProxy is giving you a warning, so I explained to you in a very detailed manner why that is.
And now you are saying that you don't know what I am referring to?
Like, I even quoted the issue.

Thirdly, I am really willing to help anyone that follows my guide.
I am doing this in my free time free of charge.
But in return the least you could do is to say thank you, before asking for help about an issue that is most likely due to the fact that you didn't read my tutorial correctly and that you are not using Let's Encrypt certificates.


So with that being sad.
Maybe you should think about what I just said.
And then, if you are willing to, share just a tiny little bit more details about: your certificate, your SLLLabs result, ... like anything that could be of help and not just one-liners without any context.
Your issue could have already been solved if you had provided those information in the first place.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 30, 2021, 09:47:41 pm
here you go...


Protocols
TLS 1.3   Yes
TLS 1.2   Yes
TLS 1.1   Yes
TLS 1.0   Yes
SSL 3   No
SSL 2   No


Cipher Suites
# TLS 1.3 (server has no preference)
TLS_AES_128_GCM_SHA256 (0x1301)   ECDH x25519 (eq. 3072 bits RSA)   FS   128
TLS_AES_256_GCM_SHA384 (0x1302)   ECDH x25519 (eq. 3072 bits RSA)   FS   256
TLS_CHACHA20_POLY1305_SHA256 (0x1303)   ECDH x25519 (eq. 3072 bits RSA)   FS   256
# TLS 1.2 (suites in server-preferred order)
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)   ECDH x25519 (eq. 3072 bits RSA)   FS   128
OLD_TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcc14)   ECDH x25519 (eq. 3072 bits RSA)   FS   256P
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)   ECDH x25519 (eq. 3072 bits RSA)   FS   256P
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)   ECDH x25519 (eq. 3072 bits RSA)   FS   WEAK   128
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)   ECDH x25519 (eq. 3072 bits RSA)   FS   WEAK   128
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)   ECDH x25519 (eq. 3072 bits RSA)   FS   256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)   ECDH x25519 (eq. 3072 bits RSA)   FS   WEAK   256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)   ECDH x25519 (eq. 3072 bits RSA)   FS   WEAK   256
# TLS 1.1 (suites in server-preferred order)
# TLS 1.0 (suites in server-preferred order)
(P) This server prefers ChaCha20 suites with clients that don't have AES-NI (e.g., Android devices)   
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 31, 2021, 12:40:03 am
Please don't get me wrong, but...

You know what it's like trying to help you?
It's like trying to teach a stone how to sing.
You can try as long as you want and still fail.
Because...
A) He doesn't understand you.
and
B) He is not answering your questions.




This is the last time I am asking you.
Give a screenshot of your ENTIRE SSLLabs result.
A SCREENSHOT. But with your domain name and IP blacked out.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: saarko on July 31, 2021, 02:47:32 pm
  • 20210730
    • Added an explanation on how to configure local-access-only subdomains in HAProxy.

thanks, works like a charm.
My "mistake" was that I thought to need a condition to trigger a map rule. Since it is not necessary, it is even easier, except for the RFC1918 condition of course. :)
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Crappysauce on August 01, 2021, 07:26:04 am
First off, thank you so much for this guide. Really helped.

I was having issues connecting to my server due to handshake errors which I think got fixed after generating new ciphers using the Mozilla SSL Config generator and changing the HAProxy and OpenSSL versions to match my setup.

After that, HAProxy seemed to refuse to redirect me to my Vaultwarden server, unless I turned off the SSL option in my Real Server setting. It still shows that I'm secured with the proper (wildcard cert from Let's Encrypt).

Do I need the SSL option enabled? The SSL test still gave me an A+...
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 01, 2021, 10:05:57 am
I was having issues connecting to my server due to handshake errors which I think got fixed after generating new ciphers using the Mozilla SSL Config generator and changing the HAProxy and OpenSSL versions to match my setup.

Are you on the latest version of OPNsense and are the installed plugins up to date?
Just out of interest, which versions of OPNsense, HAProxy and Let's Encrypt are you running?

After that, HAProxy seemed to refuse to redirect me to my Vaultwarden server, unless I turned off the SSL option in my Real Server setting. It still shows that I'm secured with the proper (wildcard cert from Let's Encrypt).

Do I need the SSL option enabled? The SSL test still gave me an A+...

The reason you couldn't connect was due to a misconfiguration in your real server, as you figured out yourself.
You enabled the "SSL -  Enable or disable SSL communication with this server. " checkbox in your real server for Vaultwarden even though the port used to connect doesn't offer SSL encryption.
https://github.com/dani-garcia/vaultwarden/wiki/Enabling-HTTPS
https://github.com/dani-garcia/vaultwarden/wiki/Private-CA-and-self-signed-certs-that-work-with-Chrome



You need to think of a reverse proxy setup like this.

WWW ---Stage 1---> yourdomain.tld ---Stage 2---> OPNsense + HAProxy + LE ---Stage 3---> internal services

Stage 1 + 2
Public facing external traffic. Traffic in these stages is now always encrypted with a verified SSL certificate. In this case it is created and verified by Let's Encrypt.

Stage 3
Local facing internal traffic. Traffic in this stage can or can not be encrypted, depending on your service setup. This is the traffic from HAProxy to your internal service. It doesn't need to be encrypt because you can consider your internal network as trusted.
However it is still strongly advised to also run this traffic encrypted.
In HAProxy you only need to check the "SSL" box in your real server setting for this.
But then you also need to actually enable SSL encryption on that service, f.e. by installing a self-signed certificate on that service and enabling HTTPS. Even though using a self-signed certificate will give you a warning by your browser when accessing the service directly and not through the reverse proxy, the traffic is still encrypted, the certificate is just unverified.
How to actually do this this depends on the service but this should be covered somewhere in its manual.

You can read more about this here: https://www.globalsign.com/en/ssl-information-center/dangers-self-signed-certificates



I will add this explanation to the FAQ.
Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Crappysauce on August 02, 2021, 07:37:30 am
Are you on the latest version of OPNsense and are the installed plugins up to date?
Just out of interest, which versions of OPNsense, HAProxy and Let's Encrypt are you running?

Everything is at the latest version:
OPNSense: 21.7
HAProxy Plugin: 3.4
Let's Encrypt: 2.6


The reason you couldn't connect was due to a misconfiguration in your real server, as you figured out yourself.
You enabled the "SSL -  Enable or disable SSL communication with this server. " checkbox in your real server for Vaultwarden even though the port used to connect doesn't offer SSL encryption.
https://github.com/dani-garcia/vaultwarden/wiki/Enabling-HTTPS
https://github.com/dani-garcia/vaultwarden/wiki/Private-CA-and-self-signed-certs-that-work-with-Chrome

I knew I was forgetting something  :(
I'll read up on his quides and get it all sorted.

Thank you again for your guide and help  ;D
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 09, 2021, 04:31:33 pm
Hi

@TheHellsite Thank you so much for your time and knowledge

If you have a fixed IP, does the DynDNS Configuration step need to be done? if skipped is there other settings i should put in?

If it is a must when we signup, there are 2 options :

configure your own domain
or
register under dyn.io

i have my own domain names about 10. Do i add each one to there system to get certs then duplicate the process to reverse proxy and cert the other domains?

If you use your real domain eg. www.123.com Do i need to go to my current domain registrar and change name servers to point to desec??

Thank you for the help. just want to get these vms up so i can programme again :( going to cry.
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 09, 2021, 06:29:22 pm
Hi

If you have a fixed IP, does the DynDNS Configuration step need to be done? if skipped is there other settings i should put in?

If you have a fixed IP that for sure never ever changes.
Then yes, you can safely skip setting up DynDNS on your OPNsense.
You will then only need to configure an A-Record in the DNS zone of your domains / subdomains pointing to your static IP.
You can set this up at your domain hosting provider.

Feel free to share a bit more about your current domain set up.
This will make it easier for me to help you.
- What are your domains?
- What are your subdomains?
- What is your domain hosting provider?

If it is a must when we signup, there are 2 options :

configure your own domain
or
register under dyn.io

i have my own domain names about 10. Do i add each one to there system to get certs then duplicate the process to reverse proxy and cert the other domains?

update: i used my domain name. i think that was wrong.

Seems like deSEC now also supports managing domains that are "hosted / registered" at a different hosting provider.
If I understood this correctly it allows you to manage the DNS zone of your domain at deSEC without actually transfering your domain away from you current hosting provider.

Since you are saying that you already have some domains / subdomains registered.
Something like "sub1.yourdomain1.com" ... "sub4.yourdomain5.com" and so on...

In this case you should check if your current domain hosting provider supports the DNS challenge.
And that the Let's Encrypt Plugin on OPNsense supports the DNS challenge for your hosting provider.
If not, then you have two options if you would like to use wildcard certificates...

Option 1 - Proceed setting up the managed DNS for your desired domains at deSEC. Then follow my tutorial beginning with part 2 step 3.

Option 2 - Transfer your domains to a hosting provider that supports the DNS challenge and that is also supported by the Let's Encrypt plugin.





Feel free to share more information about your domain / subdomain scenario along with the name of your hosting provider.
Of course don't expose your real domain names.
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 09, 2021, 06:45:54 pm
Wow thank you

Yes IP is fixed 100%.
- What are your domains? .com and .co.uk eg. example.com is my primary one.
- What are your subdomains? cloud.example.com dev.example.com
- What is your domain hosting provider? 123reg.co.uk

Have got to step Part 2 step 7. setting up opnsense dynamic DNS. So far have followed all steps par i registered example.com and not "anything.dedyn.io".

So before continuing i will check 123reg.co.uk options
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 09, 2021, 06:54:15 pm
For your own safety please replace your real domain name with "example", if the above is your real domain name!

I quickly checked 123reg and it seems like they do not offer an API so you can't use the DNS challenge.

Which gives you only 2 options.

1 - From now on managing the DNS zone of your 123reg domains at deSEC.
https://www.123-reg.co.uk/support/domains/how-do-i-change-the-nameservers-for-my-domain-name/
+ see image attached.

2 - Moving your domain to another registrar.
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 09, 2021, 07:10:56 pm
Thank you,

name servers updated. I have added A and MX records (set MX prefence to "10", is that right?).

In the opnsense Dynamic DNS, stuck on update URL. just type update.example.com?
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 09, 2021, 07:38:08 pm
Is dynamic dns still needed for fixed IP. You did say start from part 2 step 3. This update URL makes me think?
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 09, 2021, 08:23:33 pm
Is dynamic dns still needed for fixed IP. You did say start from part 2 step 3. This update URL makes me think?

Not needed! A record is enough.
Dynamic DNS is basically only updating the A record in your DNS Zone.
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 10, 2021, 05:24:42 pm
Hi,

So was setting up all night and all morning.

I skipped and removed dyndns plugin.
I completed the rest of the steps, except for the map as no subdomains setup yet.
I also didnt add a wild card for subdomains to the cert, when i did add it the cert failed. once removed the cert installed fine.
When i checked the local ip in browser, it works. When i type www.example.com it fails and cant ping address.

Had a hunt for what it could be, in the end decided to reboot opnsense and see if it shows errors.

Now rebooted HaProxy status is down and will not start.
I have looked in every log and there is only this:

2021-08-10T15:15:37   root[90544]   /usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy   
2021-08-10T15:15:30   root[91171]   /usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy   
2021-08-10T15:00:29   root[26043]   /usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy   
2021-08-10T14:59:58   syslog-ng[29102]   syslog-ng starting up; version='3.33.2'   
2021-08-10T14:59:56   syslog-ng[70806]   syslog-ng shutting down; version='3.33.2'   
2021-08-10T14:59:55   root[49764]   /usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy

thinking to remove HAProxy and start again?
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 10, 2021, 06:07:04 pm
No need to.
Look in your PMs.
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 01, 2021, 11:49:00 am
Hi there,
First of all, thank you very much for the tutorial. I still have two questions. I hope you are so nice to answer me :).

I want to make OPENvpn and various websites accessible via 443.

1st question:
I create a subdomain for each service. Can this subdomain then point to the same DYNDNS cname entries?
For example:
openvpn.domain.com -> CNAME -> 123.dyndns.com
seafile.domain.com -> CNAME -> 123.dyndns.com

2nd question:
How to configure HAproxy for openvpn.
Could you explain it to me as well as you did in the tutorial or add the point to the tutorial?
Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 01, 2021, 04:39:55 pm
1st question:
I create a subdomain for each service. Can this subdomain then point to the same DYNDNS cname entries?
For example:
openvpn.domain.com -> CNAME -> 123.dyndns.com
seafile.domain.com -> CNAME -> 123.dyndns.com
Yes, this is exactly what the CNAME record is for! https://en.wikipedia.org/wiki/CNAME_record

If you follow my guide and use desec.io as your DynDNS provider then you can even use a wildcard CNAME record. As you can see in the attached picture.
"*.tutorial.dedyn.io" means that "a.tutorial.dedyn.io, b.tutorial.dedyn.io, c.tutorial.dedyn.io, ..." will all point to "tutorial.dedyn.io" which points to "8.8.8.8".

So if you decide to use a wildcard subdomain, then you will only need to create that single subdomain "*.domain.com".
But keep in mind that not all DNS providers / domain registrars support this!


2nd question:
How to configure HAproxy for openvpn.
Could you explain it to me as well as you did in the tutorial or add the point to the tutorial?
For this your OpenVPN server needs to run in TCP mode. (Your clients need to connect to your OpenVPN server using a TCP tunnel.)
This is because HAProxy doesn't support UDP load balancing!

If you are using OpenVPN over TCP, then the configuration is pretty straight forward.
The only difference to my example Plex configuration is that you will have to set the backend "OpenVPN_backend" to TCP mode instead of HTTP mode.
See the FAQ "Why are we doing 2-Level-SNI?" and "How can we load balance TCP traffic that we don't want to get SSL offloaded, f.e. OpenVPN over TCP?".
This should explain how to set this up.
I suggest that you first make your Seafile server working and after that proceed with your OpenVPN server.

As I am currently not using OpenVPN or any other service that requires TCP load balancing, I won't be adding this to the tutorial any time soon.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 04, 2021, 06:20:01 pm
Thanks for your answer.

I have now followed your instructions exactly before I play with a changed configuration. When I try to call up the page cloudserver.xyz.dedyn.io, I always get a 503 Service Unavailable message.

Do you have any idea what I'm doing wrong?
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 05, 2021, 10:41:02 am
Post your HAProxy config. But remove your public IP or anything that reveals your identity. But leave all local IPs in it.
There is an export option in HAProxy.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 08, 2021, 04:21:35 pm
Hi,

this is my HAproxy config:

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    option tcplog

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_6138b110159553.96461818 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6138b110159553.96461818

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6138b32401a006.77997133.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6138b15d48a964.28077676.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: SEAFILE_backend ()
backend SEAFILE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server SEAFILE_server 192.168.1.11:81 ssl verify none

Mapfile:
Code: [Select]
Name: PUBLIC_SUBDOMAINS_map
Content: cloudserver SEAFILE_backend

Do you need something else?
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sorano on September 08, 2021, 08:14:51 pm
Dropping this in here to make sure noone misses it:

https://forum.opnsense.org/index.php?topic=24668.0
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 08, 2021, 09:17:48 pm
This could be my Problem. How can i install the fix? Sorry for this dump question i did this never befoe.


Gesendet von iPhone mit Tapatalk
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sorano on September 08, 2021, 09:23:08 pm
This could be my Problem. How can i install the fix?

Lol no!! Not at all. That is a security vulnerability and not your problem.

If the information you posted is correct your problem is that you are not using FQDN in your map file.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 08, 2021, 09:28:56 pm
Oh ok.

you mean i must change the mapfile to

cloudserver.domain.com SEAFILE_backend


Gesendet von iPhone mit Tapatalk
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sorano on September 08, 2021, 09:41:58 pm
Oh ok.

you mean i must change the mapfile to

cloudserver.domain.com SEAFILE_backend


Gesendet von iPhone mit Tapatalk

Yeah
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 09, 2021, 11:49:52 am
Dropping this in here to make sure noone misses it:

https://forum.opnsense.org/index.php?topic=24668.0
Thanks!



If the information you posted is correct your problem is that you are not using FQDN in your map file.

Not necessarily, I think!
I also didn't place the whole FQDN in my mapfiles, instead I am just using the subdomains.
This works just fine. But I guess it is arguable if using the complete FQDN is the better option.
https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/syntax/

Code: [Select]
plex PLEX_backend
iot IOT_backend
...



Code: [Select]
# Backend: SEAFILE_backend ()
backend SEAFILE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server SEAFILE_server 192.168.1.11:81 ssl verify none

Your map file should still work even without using FQDNs.

Your HAProxy config also looks fine.
The only thing that makes me curious is your "SEAFILE_server" configuration!
You might wanna take a look at the "configured port" or the "SSL checkbox"!
I highly doubt that your seafile server is listening on port 81 for HTTPS traffic, which is what you have configured by ticking the SSL checkbox.
The SSL checkbox means that the server is using HTTPS/SSL on the specified port. However if the server isn't actually using HTTPS/SSL on that port then the connection will obviously not work.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 09, 2021, 04:11:09 pm
Many thanks for the help. FQDN is not necessary.

By changing to no ssl, the Seafile server now works.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 10, 2021, 09:27:30 am
Let me know if you need anymore help with your OpenVPN config.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 10, 2021, 08:09:51 pm
I need your help again.   ;D

Code: [Select]
root@OPNsense:~ # cat /usr/local/etc/haproxy.conf
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    option tcplog

    # ACTION: NOSSLservice_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/613b963c5f0851.94679524.txt)]

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_6138b110159553.96461818 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6138b110159553.96461818

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6138b32401a006.77997133.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6138b15d48a964.28077676.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: SEAFILE_backend ()
backend SEAFILE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server SEAFILE_server 192.168.30.16:80

# Backend: OPENVPN_backend ()
backend OPENVPN_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server OPENVPN_server 127.0.0.1:1194

Code: [Select]
2021-09-10T20:00:44 haproxy[11387] 192.168.1.231:51903 [10/Sep/2021:20:00:44.614] 0_SNI_frontend SSL_backend/SSL_server 1/0/4 0 -- 1/1/0/0/0 0/0
2021-09-10T20:00:44 haproxy[11387] 192.168.1.231:51903 [10/Sep/2021:20:00:44.615] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T20:00:40 haproxy[11387] 192.168.1.231:51902 [10/Sep/2021:20:00:40.526] 0_SNI_frontend SSL_backend/SSL_server 1/0/5 0 -- 1/1/0/0/0 0/0
2021-09-10T20:00:40 haproxy[11387] 192.168.1.231:51902 [10/Sep/2021:20:00:40.527] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 0_SNI_frontend SSL_backend/SSL_server 1/0/39 0 -- 1/1/0/0/0 0/0
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T19:59:26 haproxy[11387] xx.xx.xx.162:25707 [10/Sep/2021:19:59:26.004] 0_SNI_frontend SSL_backend/SSL_server 1/0/35 0 -- 1/1/0/0/0 0/0

MAP_file:
(https://i.ibb.co/wBVxxYG/map.jpg) (https://ibb.co/pbc99nV)

OPENVPN_settings:
(https://i.ibb.co/VDRxbhg/opnvpn-server-conf.jpg) (https://ibb.co/1K1MBpr)
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 12, 2021, 10:42:17 pm
Okay, first you should change the listening interface of the OpenVPN plugin from "any" to "localhost".
Because this is where HAProxy is sending the traffic to since you defined your "OpenVPN real server = 127.0.0.1".
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 13, 2021, 12:34:49 pm
Hi, I changed the interface, the problem remains the same. It looks like the VPN access is not forwarded by the SNI but by the HTTPS server, right? Unfortunately I don't know how to work around the problem.

Code: [Select]
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 0_SNI_frontend SSL_backend/SSL_server 1/0/39 0 -- 1/1/0/0/0 0/0
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T19:59:26 haproxy[11387] xx.xx.xx.162:25707 [10/Sep/2021:19:59:26.004] 0_SNI_frontend SSL_backend/SSL_server 1/0/35 0 -- 1/1/0/0/0 0/0
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 13, 2021, 06:38:48 pm
Is your OpenVPN client config correct?
- mode tcp
- port 443
- server vpn.domain.com

Some of this must be wrong.
Otherwise your SNI_frontend wouldn't pass the traffic to the configured default backend (SSL_backend) which is what it is currently doing.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 13, 2021, 07:43:52 pm
here is the extract of my configuration. I think that is all correct.

Code: [Select]
dev tun
persist-tun
persist-key
proto tcp-client
cipher AES-256-CBC
auth SHA256
client
resolv-retry infinite
remote vpn.xxxxx.dedyn.io 443 tcp
lport 0
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nullex on September 13, 2021, 10:05:21 pm
Hello,

I'm having trouble with this guide.. Very strange trouble...
I have followed the steps to a T.. except I am using cloudflare for Dynamic DNS and ACME/let's encrypt Certs. Also, I am using localhost/127.0.0.1 instead of the virtual IP option. Although I have tried it both ways with the same results...

The problem is.. weird.. So, like I said, followed the steps.. But only a few backend servers work. I have set up Plex, Sonarr, Radarr, Ombi, Cams(blueIris) and some other stuff. However, only Plex, Cams, and Radarr work. I've been fighting with it for a while, and previously only Plex and Cams were working.. not sure what I changed to make Radarr work.. but everything else just gives me Error 503 Service Unavailable.. Even though the service is up and running and is accessible through LAN.

My opnsense is fully updated, 21.7.2_1 and os-haproxy 3.5

My HAProxy config is as follows:

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug
cache opnsense-haproxy-cache
    total-max-size 512
    max-age 60

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on localhost:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_613eabd9cb19a0.51810931 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_613eabd9cb19a0.51810931

# Frontend: 1_HTTPS_frontend (Listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/613eae5151edb0.32207081.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/613eac85c00a60.86291436.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: Radarr_backend ()
backend Radarr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Radarr_server 192.168.1.111:7878

# Backend: Plex_backend ()
backend Plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Plex_server 192.168.1.159:32400

# Backend: Cams_backend ()
backend Cams_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Cams_server 192.168.1.10:81

# Backend: Ombi_backend ()
backend Ombi_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Ombi_server 192.168.1.159:5055

# Backend: Sonarr_backend ()
backend Sonarr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Sonarr_server 192.168.1.111:8989

# Backend: Tautulli_backend ()
backend Tautulli_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Tautulli_server 192.168.1.7:8181

My map File currently looks like:

Code: [Select]
sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
Ombi Ombi_backend
Tautulli Tautulli_backend

Any advice as to what I'm doing wrong? Why would some services work and others don't when they're using the same exact config?

What's also funny is that if I reconfigure the port and IP on the one of the Real Servers that works (For example, Plex_Server has IP 192.168.1.159:32400, Ombi_server has 192.168.1.159:5055, and I replace Plex_Server port to 5055 and go to Plexi.DOMAIN.com, Ombi pops up like it should. I've cloned new Real servers and Backend Pools while updating the PUBLIC_SUBDOMAINS_map  from the working Plex one, but still no go for ALL of my services.

I've also discovered that if I modify HAProxy Rules & Checks > Rules > Public_subdomains_map-rule > Default backend pool... and change it to a service that DOESN'T work with the map file.. when I hit apply I'm able to access that service on ANY rendition of my domain, as well as the root domain.com address... And if I leave the services that DO work in the map file (Plex), plex.domain.com displays plex as it should, while the rest of the domain is showing the service that doesn't work on it's own.. which further doesn't make any sense.. The map file is working for some services but not others?

Thanks in advance, I'm stumped.

Final edit:

LMFAO I think I figured out why it wasn't working... so apparently for the map file to work you have to have the first part all lowercase, cannot use any uppercase...

So when I changed my map file to:

Code: [Select]
sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
ombi Ombi_backend
tautulli Tautulli_backend


everything started working...
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: budimanjojo on September 14, 2021, 08:56:11 am
Thank you so much for this tutorial! But I have a problem and I don't know what should I do.  :'(
Maybe somebody can help me here.

I already have a Traefik reverse proxy running outside OPNSense listening on 192.168.200.244 port 80 and 443. That Traefik reverse proxying all my kubernetes cluster services including certificates handling for 2 domain names.

Now, what I want to is to have HAProxy in OPNSense to be the reverse proxy for my Traefik. For example:
- My domain names are 1stdomain.com and 2nddomain.com.
- Have a rule that: if the client go to opnsense.1stdomain.com, route it to localhost:55443 (OPNSense itself), else if the client go to *.1stdomain.com or *.2nddomain.com, route it to 192.168.200.244:443.

I don't know where to start, I tested using the tuturial but instead of SNI and HTTP frontend, I created 2 SNI frontend services but HAProxy refused to start at all.

There is an easier route for me though, that is to just create a route in Traefik to my OPNSense and be done with it. But I prefer having a firewall level reverse proxy so I can have another layer to let's say block external access like in this tutorial. Thanks before.  :)
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 01:08:53 pm
@TheHellSite

I think the problem is with the SNI frontend. Here the SSL backend is specified as the default backend. He doesn't even look at the MAP file. he forwards everything to the SSL backend. When I set the openvpn backend as default Backend for a test in the SNI frontend, openvpn work but the other things not.

do you have an idea how I can solve this?
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 02:31:50 pm
@TheHellSite

I think the problem is with the SNI frontend. Here the SSL backend is specified as the default backend. He doesn't even look at the MAP file. he forwards everything to the SSL backend. When I set the openvpn backend as default Backend for a test in the SNI frontend, openvpn work but the other things not.

do you have an idea how I can solve this?
I was just about to write you exactly this!  ;D
Your reply confirmed my guess.

Looking through the manual pages of HAProxy it seems that the "Default Backend" setting can only be overwritten by a "Use Backend" rule! Which a "Use map file" rule isn't able to.
https://www.haproxy.com/de/blog/the-four-essential-sections-of-an-haproxy-configuration/
But I can't imagine that this is the intended behaviour.

Anyways... you simply need to create a VPN_condition "host starts with vpn" and a "use backend OPENVPN_backend if VPN_condition=true" rule.
Add this rule to the SNI_frontend and set the default backend back to the SSL_backend.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 05:35:30 pm
My map File currently looks like:

Code: [Select]
sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
Ombi Ombi_backend
Tautulli Tautulli_backend

Any advice as to what I'm doing wrong? Why would some services work and others don't when they're using the same exact config?

What's also funny is that if I reconfigure the port and IP on the one of the Real Servers that works (For example, Plex_Server has IP 192.168.1.159:32400, Ombi_server has 192.168.1.159:5055, and I replace Plex_Server port to 5055 and go to Plexi.DOMAIN.com, Ombi pops up like it should. I've cloned new Real servers and Backend Pools while updating the PUBLIC_SUBDOMAINS_map  from the working Plex one, but still no go for ALL of my services.

I've also discovered that if I modify HAProxy Rules & Checks > Rules > Public_subdomains_map-rule > Default backend pool... and change it to a service that DOESN'T work with the map file.. when I hit apply I'm able to access that service on ANY rendition of my domain, as well as the root domain.com address... And if I leave the services that DO work in the map file (Plex), plex.domain.com displays plex as it should, while the rest of the domain is showing the service that doesn't work on it's own.. which further doesn't make any sense.. The map file is working for some services but not others?

Thanks in advance, I'm stumped.

Final edit:

LMFAO I think I figured out why it wasn't working... so apparently for the map file to work you have to have the first part all lowercase, cannot use any uppercase...

So when I changed my map file to:

Code: [Select]
sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
ombi Ombi_backend
tautulli Tautulli_backend


everything started working...

Glad it is working for you know.
My first guess where some misconfigured real servers (ports, ssl, ssl-verify).

BTW: Your map file is exposing your domain name! You should remove it from the forum post.

Also I did a quick scan of your domain using https://dnsdumpster.com (https://dnsdumpster.com). It lists all your subdomains since you created a single "A Record" for each of them. Consider switching to a "Wildcard A Record" in order to hide them!
If an attacker can see what services you are running it makes it easier for them to find an attack surface.

You can then still create individual a records, f.e. www.domain.tld, since the wildcard a record is resolved after all other a records have been resolved.

This is why my tutorial is using a "Wildcard A Record / Subdomain" in the form of "*.domain.tld".
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 08:23:42 pm
Still the same  :(. Any other idea :)
(https://i.ibb.co/BnKQ9sr/rule.jpg) (https://ibb.co/XYkKqJ3)
(https://i.ibb.co/nf1gxv3/condition.jpg) (https://ibb.co/VDCjfZH)
(https://i.ibb.co/37kGNLD/SNI-Frontend.jpg) (https://ibb.co/DtLsCSx)
(https://i.ibb.co/PgLz2SH/Public-Service.jpg) (https://ibb.co/bX0R4f9)
(https://i.ibb.co/FB0pX4G/logfile.jpg) (https://ibb.co/rZc932n)
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 08:39:39 pm
Please remove the default backend but leave the vpn_rule enabled.

I have another guess...
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 08:49:09 pm
does not work too. i don't think he matched the names
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 08:57:24 pm
Invert the vpn condition and name it "NoVPN_condition".
Rename the vpn rule to "NoVPN_rule" and change it to use SSL_backend.

Test it out.
1. Only the NoVPN_rule on the SNI_frontend.
2. NoVPN_rule on the SNI_frontend along with SSL_backend as default.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 09:23:55 pm
Still the Same :)

Code: [Select]
2021-09-14T21:21:05 haproxy[3256] 80.187.80.8:10670 [14/Sep/2021:21:21:05.818] 0_SNI_frontend SSL_backend/SSL_server 1/0/37 0 -- 1/1/0/0/0 0/0
2021-09-14T21:21:05 haproxy[3256] 80.187.80.8:10670 [14/Sep/2021:21:21:05.818] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-14T21:21:01 haproxy[3256] 80.187.80.8:10495 [14/Sep/2021:21:21:01.658] 0_SNI_frontend SSL_backend/SSL_server 1/0/36 0 -- 1/1/0/0/0 0/0
2021-09-14T21:21:01 haproxy[3256] 80.187.80.8:10495 [14/Sep/2021:21:21:01.658] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 09:30:49 pm
For which try?
1st or second?

If you remove the VPN_backend as default backend and place the novpn_rule in the frontend. Can you still access your seafile Server?
It should be accessible. Is it?
I want to test the sni functionality with this.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 09:41:28 pm
Your first question:
For both.

The second:
Yes i can access the Seafile server with any config. The haproxy give every request to the ssl backend.


Gesendet von iPhone mit Tapatalk
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 09:43:05 pm
It looks like that sni don’t work by OpenVPN


Gesendet von iPhone mit Tapatalk
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nullex on September 15, 2021, 12:06:38 am
My map File currently looks like:

Code: [Select]
sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
Ombi Ombi_backend
Tautulli Tautulli_backend

Any advice as to what I'm doing wrong? Why would some services work and others don't when they're using the same exact config?

What's also funny is that if I reconfigure the port and IP on the one of the Real Servers that works (For example, Plex_Server has IP 192.168.1.159:32400, Ombi_server has 192.168.1.159:5055, and I replace Plex_Server port to 5055 and go to Plexi.DOMAIN.com, Ombi pops up like it should. I've cloned new Real servers and Backend Pools while updating the PUBLIC_SUBDOMAINS_map  from the working Plex one, but still no go for ALL of my services.

I've also discovered that if I modify HAProxy Rules & Checks > Rules > Public_subdomains_map-rule > Default backend pool... and change it to a service that DOESN'T work with the map file.. when I hit apply I'm able to access that service on ANY rendition of my domain, as well as the root domain.com address... And if I leave the services that DO work in the map file (Plex), plex.domain.com displays plex as it should, while the rest of the domain is showing the service that doesn't work on it's own.. which further doesn't make any sense.. The map file is working for some services but not others?

Thanks in advance, I'm stumped.

Final edit:

LMFAO I think I figured out why it wasn't working... so apparently for the map file to work you have to have the first part all lowercase, cannot use any uppercase...

So when I changed my map file to:

Code: [Select]
sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
ombi Ombi_backend
tautulli Tautulli_backend


everything started working...

Glad it is working for you know.
My first guess where some misconfigured real servers (ports, ssl, ssl-verify).

BTW: Your map file is exposing your domain name! You should remove it from the forum post.

Also I did a quick scan of your domain using https://dnsdumpster.com (https://dnsdumpster.com). It lists all your subdomains since you created a single "A Record" for each of them. Consider switching to a "Wildcard A Record" in order to hide them!
If an attacker can see what services you are running it makes it easier for them to find an attack surface.

You can then still create individual a records, f.e. www.domain.tld, since the wildcard a record is resolved after all other a records have been resolved.

This is why my tutorial is using a "Wildcard A Record / Subdomain" in the form of "*.domain.tld".

Thanks for catching that, thought I removed all PII but apparently not lol

So I think DNSDumpster is showing those domains because I was previously using dynamic DNS on EACH of those domains and using caddy as a reverse proxy.. which all those WERE direct A records, but since following your guide I switched to wildcard domain so hopefully it should be fixed once DNSDumpster's Database is updated.. I'll keep an eye on it. Thanks again!
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Prismatic on October 06, 2021, 09:35:05 pm
Hello. First of all, many thanks for your tutorial.... working like a charm, even for an (enlightened)  newbie like me....

Im using a Synology NAS + Docker  with different services. So they are available on same LAN IP adress, but different Ports....

I would llike to setup the "access from internatl network" as on your part 6. I do understand that with my setup I cannot use the unbound split DNS option, as this doesnt work with ports...

So I'm relying on option A "NAT reflexion", but I'm unable to find it, inside the created "HAproxy rule", in Firewall/WAN section..... I do find this option in Firewall/NAT section rules, but we didnt created nothing there....

Could you please help ? and update your tutorial with a screenshot of this ?

Many thanks !


Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 06, 2021, 11:17:36 pm
Im using a Synology NAS + Docker  with different services. So they are available on same LAN IP adress, but different Ports....

I would llike to setup the "access from internatl network" as on your part 6. I do understand that with my setup I cannot use the unbound split DNS option, as this doesnt work with ports...

Of course you can use split DNS!
haproxy is handling the port scenario.

Example:
10.1.23.55:4456 = NEXTCLOUD = cloud.yourdomain.tld
10.1.23.55:4457 = PLEX = plex.yourdomain.tld

Read the Split dns part again!
You have to rewrite ALL of your 1st/2nd-level-subdomains with same IP of your OPNsense that HAProxy is listening on, f.e. the lan ip.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Prismatic on October 07, 2021, 12:55:24 am
Thanks for your answer !

Apologies, but I still don't understand something....

So my services IP & port are configured inHAproxy map file. (everything working fine, comming from WAN with certificate)..
.
this server IP for me on LANis 192.168.1.20
only using 1st level subdomains like:

audio.mydomain.tld
photo.mydomain.tld
video.mydomain.tld
etc...

My "LAN" port for opnsense on appliance is 192.168.1.1 and I did used a "virtual IP" as per your tutorial which is 192.168.50.1...

Using "hosts overrides in unbound" (not "domain") should my configuration look like this :

HOST : audio
DOMAIN: mydomain.tld
IP : 192.168.50.1 or 192.168.1.1  or 192.168.1.20 ??

tried all solutions, but I don't know if it works.... a "tracert" shows that I am always hitting my static public IP adress....

sorry for my approximative english and many thanks for your answer.
cheers !


 


Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 08, 2021, 08:46:00 am
So my services IP & port are configured inHAproxy map file. (everything working fine, comming from WAN with certificate)..
.
this server IP for me on LANis 192.168.1.20
only using 1st level subdomains like:

audio.mydomain.tld
photo.mydomain.tld
video.mydomain.tld
etc...

My "LAN" port for opnsense on appliance is 192.168.1.1 and I did used a "virtual IP" as per your tutorial which is 192.168.50.1...

Using "hosts overrides in unbound" (not "domain") should my configuration look like this :

HOST : audio
DOMAIN: mydomain.tld
IP: 192.168.1.1

IP has to be the lan ip of your opnsense.
Which I very well tell to do in my tutorial.
You should probably read it word by word again!

Also post your HAProxy config export in a code box.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 08, 2021, 05:45:47 pm
Hey all,

Is there a way to tie this in with Cloudflare?
I know ACME can use Cloudflare, but can someone add in where I need to do this for wildcards?

Also I installed the plugins and I don't see LetsEncrypt, rather I see ACME Client, and the settings are a bit different.
Any chance you can update the images?

Also I've tried to follow this to the best of my abilities.
I am getting: 503 Service Unavailable
No server is available to handle this request.

If I click on the padlock it is showing my certificate so I think its hitting right.
I'm doing things for:
homeassistant
jellyfin
radarr
sonarr

All of them say 503.

If I set a backend to maintenance I get Safari can't find the server.
So seems to be hitting it correctly.

Ok not sure what changed but now its no dice. Just server cannot be found
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 08, 2021, 11:45:41 pm
Sorry but I can't help with individual domain / dns registrars, this would simply take way to much of my time. You will have to study cloudflares documentation on your own to make it work with them.
Only difference should be the Dynamic DNS and ACME configuration, to get your certificate.
HAProxy settings are the same for every registrar.

The Let's Encrypt plugin has been renamed to ACME Client in one of the recent OPNsense updates. Apart from that nothing else has changed so all pictures are still valid. No need to change them.
I will update the text when I find the time.

Your error: Post your HAProxy export in a code box. Remove sensitive information (Public IP, Domain Name).
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 09, 2021, 10:46:19 am
Thank for taking the time to help.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_Frontend ()
frontend SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTP_Frontend ()
frontend HTTP_Frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: No_SSL_condition
    acl acl_6160768c129757.05678189 req.ssl_ver gt 0

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_6160768c129757.05678189

# Frontend: HTTPS_Frontend ()
frontend HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6160790e8358e2.93807756.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: Public_Subdomain_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61607700a205d2.15896401.txt)]

# Backend: SSL_backend (HAProxy SSL Backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: homeassistant_backend (Home Assistant Backend)
backend homeassistant_backend
    # health check: homeassistant_tcp_check
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server homeassistant 10.0.0.9:8123 check inter 2s port 8123  ssl verify none

# Backend: jellyfin_backend (Jellyfin Backend)
backend jellyfin_backend
    # health check: jellyfin_http_check
    option httpchk
    http-check send meth OPTIONS uri / ver HTTP/1.0
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server jellyfin 10.0.0.10:8090 check inter 2s port 8090  ssl verify none

The only difference I think I made was to add health checks so I can see if service is UP.
It is showing as UP so that looks good.

And my MAP looks like:
Code: [Select]
# public access subdomains
homeassistant homeassistant_backend
jellyfin jellyfin_backend

Oh one other thing to note is I have dual WAN setup.
I had this on pfSense too and HAProxy was working perfectly fine.
I just had all the external connections go through 1 WAN only which I did again here.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 09, 2021, 02:57:49 pm
Please confirm you are accessing your servers internally using:

http://10.0.0.10:8090/
http://10.0.0.9:8123/
So no https?

Even though it shouldn't matter test the access with health checks disabled.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 09, 2021, 03:17:58 pm
Please confirm you are accessing your servers internally using:

http://10.0.0.10:8090/
http://10.0.0.9:8123/
So no https?

Even though it shouldn't matter test the access with health checks disabled.

Correct if I hit those 2 IP's above internally I get the service correctly.
I did have it running with health checks disabled and same result 503 errors
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 09, 2021, 03:39:32 pm
Is opnsense the only router / firewall in your network?
To me it looks like your servers are in a totally different subnet / vlan.

Please give a little info about your local network layout.
I think your opnsense has no access to your server network!
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 09, 2021, 04:07:09 pm
Yes OPNsense is my only router/firewall.

I have dual WAN created a gateway group to double internet speed.
I created the rule on only 1 WAN gateway tho, and my a records point to the same IP.

I have 3 LANs (not VLANS) 10.0.0.1 (Main LAN) - 192.168.107.1 (IoT) - 192.168.200.1 (Guest)
My NAS Server (10.0.0.10) and OPNsense (10.0.0.1) are on 10.0.0.1/24 LAN, so no going through anything different there.

I wonder if with the dual WAN it needs a specific rule?
As with dual WAN I have to change the default gateway on the Allow LAN to any rule to the specific Multi_Gateway...
Like my LAN can no longer talk to my IOT Lan without having a specific rule to say allow...

I've just added the LAN Net to Loopback but still no dice
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 09, 2021, 05:08:43 pm
The error is not related to your dual WAN setup.  ;D
I misread something in your HAProxy config...

Code: [Select]
server jellyfin 10.0.0.10:8090 check inter 2s port 8090  ssl verify noneYou see that "ssl" word in your server config?
SSL and HTTP don't like each other!  ;)

Since your services are NOT using HTTPS but only HTTP locally... you need to UNCHECK the SSL checkbox in the HAProxy real server config for both of your servers.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 09, 2021, 05:58:49 pm
The error is not related to your dual WAN setup.  ;D
I misread something in your HAProxy config...

Code: [Select]
server jellyfin 10.0.0.10:8090 check inter 2s port 8090  ssl verify noneYou see that "ssl" word in your server config?
SSL and HTTP don't like each other!  ;)

Since your services are NOT using HTTPS but only HTTP locally... you need to UNCHECK the SSL checkbox in the HAProxy real server config for both of your servers.

Oh my days!!!
That worked! I spent all night trying to figure it out.

Thank you so very much!
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: blackhand27 on October 15, 2021, 06:06:51 am
Great Guide! Took me a bit since im using Godaddy for my domain name but figured out the cert mess and got my plex server working. Been hoping to get this setup so I can run a Synapse server at home.

I do seem to be having issues with my bitwarden site, I have the proper port set in HAProxy however I have a feeling its something with its configuration still using the internal cert vs the offload. Would anyone have some advice to see what I can look for?

figured it out, I reinstalled the server using self signed certs, for whatever reason it was still using the builtin lets encrypt certs and failing to connect.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on October 15, 2021, 07:25:02 pm
@TheHellSite

I think the problem is with the SNI frontend. Here the SSL backend is specified as the default backend. He doesn't even look at the MAP file. he forwards everything to the SSL backend. When I set the openvpn backend as default Backend for a test in the SNI frontend, openvpn work but the other things not.

do you have an idea how I can solve this?
I was just about to write you exactly this!  ;D
Your reply confirmed my guess.

Looking through the manual pages of HAProxy it seems that the "Default Backend" setting can only be overwritten by a "Use Backend" rule! Which a "Use map file" rule isn't able to.
https://www.haproxy.com/de/blog/the-four-essential-sections-of-an-haproxy-configuration/
But I can't imagine that this is the intended behaviour.

Anyways... you simply need to create a VPN_condition "host starts with vpn" and a "use backend OPENVPN_backend if VPN_condition=true" rule.
Add this rule to the SNI_frontend and set the default backend back to the SSL_backend.

Am I confused about this?

It looks like the current set up on page one will not work due to "default backend"??

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 19, 2021, 06:06:55 pm
I was just about to write you exactly this!  ;D
Your reply confirmed my guess.

Looking through the manual pages of HAProxy it seems that the "Default Backend" setting can only be overwritten by a "Use Backend" rule! Which a "Use map file" rule isn't able to.
https://www.haproxy.com/de/blog/the-four-essential-sections-of-an-haproxy-configuration/
But I can't imagine that this is the intended behaviour.

Anyways... you simply need to create a VPN_condition "host starts with vpn" and a "use backend OPENVPN_backend if VPN_condition=true" rule.
Add this rule to the SNI_frontend and set the default backend back to the SSL_backend.

Am I confused about this?

It looks like the current set up on page one will not work due to "default backend"??

I am not sure what you mean?
The setup still works as described. I also got OpenVPN over TCP working but haven't had the time yet to add it to my tutorial.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: thegriffen on October 27, 2021, 12:37:59 am
Hi everyone

First of all.. Awesome guide and even so, that you update it with new stuff as it comes along.

Iv been kinda hitting my head for a day or two now and gotta throw in the towel, and put my troubles in here.

im running a small test setup, where im gonna have a couple of web services running, but the thing is here, they are running traefik with there own LE ssl validation and so. but i cant get HAproxy to work propper.

My config looking like this:

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend ()
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:8888 name 0.0.0.0:8888
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options
    option tcplog

    # ACTION: Public_Domain_Map_Rule_2
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6178606dd3b431.48344137.txt)]
    # ACTION: Public_Domain_Map_Rule_3
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6178704b1d59f6.07557436.txt)]

# Frontend: 1_Http_Frontend ()
frontend 1_Http_Frontend
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: NoSSL_Condition
    acl acl_61781026415d99.64392811 req.ssl_ver gt 0

    # ACTION: HttpToHttps_Rule
    http-request redirect scheme https code 301 if !acl_61781026415d99.64392811

# Frontend: 1_Https_Frontend ()
frontend 1_Https_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    option httplog

    # ACTION: Public_Domain_Map_Rule_1
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/617812015049f9.00974692.txt)]

# Backend: SSL_Backend (Loopback)
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: Central_OpenSense_FW_Backend (OpenSense Firewal)
backend Central_OpenSense_FW_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Central_OpenSense_FW_SRV 10.10.5.2:55443 ssl verify none

# Backend: FRCD_Backend (domain1.com)
backend FRCD_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server DMZ_domain1_SRV 10.10.20.40:443 ssl alpn h2,http/1.1,http/1.0 verify none

# Backend: FRCD_2_Backend (domain2:8888)
backend FRCD_2_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server DMZ_domain2_SRV 10.10.20.41:8888

Best regrads
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 31, 2021, 10:44:49 am
Hi everyone

First of all.. Awesome guide and even so, that you update it with new stuff as it comes along.

Iv been kinda hitting my head for a day or two now and gotta throw in the towel, and put my troubles in here.

im running a small test setup, where im gonna have a couple of web services running, but the thing is here, they are running traefik with there own LE ssl validation and so. but i cant get HAproxy to work propper.

I don't really understand what you need haproxy for if you are already using traefik?
Can you please elaborate further what...

1. You are trying to achieve by using HAProxy?

2. What issue you are facing with HAProxy right now?

3. Explain your local network layout further!
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bringha on November 04, 2021, 10:46:42 am
Hi there,

first of all a big big thank you for this awesome and comprehensive tutorial. Very helpfull and a great contribution.

I have an additional question and I am not sure whether I suffer from a big misuderstanding.

I configured my Dyndns as suggested with dedyn.io and have now a domain.dedyn.io properly working. Your tutorial now assumes to create wildcard certificates for the *.domain.dedyn.io (in my case)

I have a main domain registered with a poster somewhere else which is domain.com. Historically I reach my dyndns based subdomains via CNAME DNS entries at my main domain provider's DNS systems, eg home.domain.com points then to home.domain.dedyn.io.

It is now possible to let the acme client generate wildcard certificates also for *.domain.com accordingly in addition/replacing the wildcard certs for *.domain.dedyn.io?

Looking forward to your reply.

Br br
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 04, 2021, 07:15:20 pm
I configured my Dyndns as suggested with dedyn.io and have now a domain.dedyn.io properly working. Your tutorial now assumes to create wildcard certificates for the *.domain.dedyn.io (in my case)

I have a main domain registered with a poster somewhere else which is domain.com. Historically I reach my dyndns based subdomains via CNAME DNS entries at my main domain provider's DNS systems, eg home.domain.com points then to home.domain.dedyn.io.

It is now possible to let the acme client generate wildcard certificates also for *.domain.com accordingly in addition/replacing the wildcard certs for *.domain.dedyn.io?

Basically there are two options to solve this

1. You replace "deSEC" as your DynDNS provider and use the DynDNS of your main domain registrar. If he supports this...
--> What is your domain registrar? (Where your real "domain.com" is registered...)

2. You transfer the DNS zone from your main "domain.com" to deSEC. Again, if your current domain registrar supports this.
This way your domain is still hosted at your current domain registrar, but deSEC will be managing all the DNS related stuff. (A records, ...)
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bringha on November 04, 2021, 07:44:54 pm
Thank you for your answer!

My domain Registrar is Ionos. Ionos has also a dyndns service and a dns management api and is contained in the opnsense dyndns list. I would like to leave the services of the main Domain there (mail, Webhosting) and have only the subdomains with my local dyndns based site.

Br br
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 04, 2021, 07:59:11 pm
IONOS works exactly like deSEC, just adopt the deSEC stuff to the IONOS settings.
get a dns API key, set up dyn dns and get your certs.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on November 05, 2021, 09:29:43 pm
Hey @TheHellSite, Do you know a way I can add Basic Auth to one of the sub domains?

I have a dashboard which just runs without login and I would really like it secured behind just basic auth if possible.
I tried creating a condition and a rule and applying it to the sub domain but it doesnt seem to work.

I also saw on the backend there is an option on the domain for basic auth checkbox, so I tried that but no dice either.

Any chance you can add a section to the guide about securing singular domains with basic auth?
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 06, 2021, 11:00:20 am
Hey @TheHellSite, Do you know a way I can add Basic Auth to one of the sub domains?

I have a dashboard which just runs without login and I would really like it secured behind just basic auth if possible.
I tried creating a condition and a rule and applying it to the sub domain but it doesnt seem to work.

I also saw on the backend there is an option on the domain for basic auth checkbox, so I tried that but no dice either.

Any chance you can add a section to the guide about securing singular domains with basic auth?

Definitely will be looking into this. You can expect this in the coming weeks, hopefully.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on November 06, 2021, 11:29:59 am
AMAZING! Thank you so much
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bringha on November 06, 2021, 10:15:27 pm
Hi there,

I tried to activate DDNS via the API with ionos as suggested but miserably failed yet.  ??? Might be somewhat unusual but I only want to activate DDNS for two subdomains, not my entire domain ‚domain.com‘ (only ‚sub.domain.com‘).

 I come up to the step that I have created my update URL, the server responds with 200 properly on the Ionos API page. This should then have activated DDNS for the requested subdomains according to the doc. However there is the no entry for these subdomain on the DNS page.

When I  apply Update URL  in the opnsense dyndns config and press ‚save and enforce update‘, my public IP is properly shown in the opnsense dyndns config page.  But still the subdomain is not shown as a DynDNS subdomain at ionos ..
as to be feared there is also no DNS with my public IP for the subdomain distributed …

Do i miss a step ?? I am looking forward to any idea.

Br br
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on November 08, 2021, 10:13:16 am
FYI I got it working the Basic Auth.

So 21.7.4 has an issue with HAProxy where changes to config are not being saved: https://forum.opnsense.org/index.php?topic=25480.0
After I applied the patch it is working.

For Basic Auth all I did was create a user under User Management
(There does seem to be some restrictions around password length or complexity, I didnt spend a load of time testing)
My 20+ Char passwords would not work until I dumbed it down a bit.

So once you have a user go to your Backend Pools, choose your desired service, scroll down to Basic Authentication -> tick the box and add the username you just created.

Boom it works.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 10, 2021, 03:33:49 pm
Hi there,

I tried to activate DDNS via the API with ionos as suggested but miserably failed yet.  ??? Might be somewhat unusual but I only want to activate DDNS for two subdomains, not my entire domain ‚domain.com‘ (only ‚sub.domain.com‘).

 I come up to the step that I have created my update URL, the server responds with 200 properly on the Ionos API page. This should then have activated DDNS for the requested subdomains according to the doc. However there is the no entry for these subdomain on the DNS page.

When I  apply Update URL  in the opnsense dyndns config and press ‚save and enforce update‘, my public IP is properly shown in the opnsense dyndns config page.  But still the subdomain is not shown as a DynDNS subdomain at ionos ..
as to be feared there is also no DNS with my public IP for the subdomain distributed …

Do i miss a step ?? I am looking forward to any idea.

Br br

Your issue is related to DynDNS I think you should fix that first and ask for help at the right place.
Maybe use google "how to set up a DynDNS subdomain at IONOS".
https://www.ionos.de/hilfe/domains/ip-adresse-konfigurieren/dynamisches-dns-ddns-einrichten-bei-company-name/

I am sorry, but since I am not using IONOS you will have to figure out their specifics on your own.
The only differences to my guide will be the DynDNS setup and the DNS challenge specifics for the ACME Client. Everything else is the same.



FYI I got it working the Basic Auth.

So 21.7.4 has an issue with HAProxy where changes to config are not being saved: https://forum.opnsense.org/index.php?topic=25480.0
After I applied the patch it is working.

For Basic Auth all I did was create a user under User Management
(There does seem to be some restrictions around password length or complexity, I didnt spend a load of time testing)
My 20+ Char passwords would not work until I dumbed it down a bit.

So once you have a user go to your Backend Pools, choose your desired service, scroll down to Basic Authentication -> tick the box and add the username you just created.

Boom it works.

Just tested it out myself. Basic Auth is so easy to set up that I am not really willing to cover it in this guide.
First create the user(s) in HAProxy. Then in the relevant backends activate basic auth and select the user(s).

The issue you are facing with your password is the way you included them in your config!
You will need to SHOULD use a password hash not the plain text password in the user management box.
Otherwise your passwords are visible in plain text in the config export.

Generate password hash using OpenSSL binary
1. (WINDOWS ONLY) Download the openssl binary, f.e. from here: http://wiki.overbyte.eu/wiki/index.php/ICS_Download#Download_OpenSSL_Binaries_.28required_for_SSL-enabled_components.29
2. (Windows) Open CMD in the directory of the binary file / (Linux) open the shell.
3. Run "openssl passwd -6" and enter your password.
4. Enter the generated password hash in the password field of the user in HAProxy Settings.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 12, 2021, 04:09:12 pm
I am having issues with modifying config which will not take effect.

as an example when attempting to modify default-dh-param from 2048 to 4096, the save and apply works but the config export is not updated.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 15, 2021, 04:41:55 pm
It was fixed after the latest update, however, I am still having connections issues... I am getting handshake failures.
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin expose-fd listeners
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend (listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_Frontend (Listening on 192.168.1.50:80)
frontend 1_HTTP_Frontend
    bind 192.168.1.50:80 name 192.168.1.50:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NOSSL_Condition
    acl acl_60f9d6d0118252.11362730 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_Rule
    http-request redirect scheme https code 301 if !acl_60f9d6d0118252.11362730

# Frontend: 1_HTTPS_Frontend (listening on 192.168.1.50:443)
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.1.50:443 name 192.168.1.50:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60f9db5421ce96.24863488.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: Public_Sub_MapRule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61698448328ff6.66158166.txt)]

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server HAP_VIP 192.168.1.50 send-proxy-v2 check-send-proxy

# Backend: TruePlex_Backend ()
backend TruePlex_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server TruePlex 192.168.1.12:32400 ssl verify none
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 16, 2021, 10:01:22 am
It was fixed after the latest update, however, I am still having connections issues... I am getting handshake failures.

From all client devices or just your smart TV(s)?
Does web browser access work?
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 16, 2021, 11:41:05 am
From all clients, nothing works.

I can connect to the plex with the local IP without an issue.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 16, 2021, 12:44:25 pm
From all clients, nothing works.

I can connect to the plex with the local IP without an issue.

1. Post a screenshot of your entire SSL Labs result. But black out any personal information from it, i.e. domain name, public IP,... You can also send me a personal message with the result link.
https://www.ssllabs.com/ssltest/


2. Go to "OPNsense --> System --> Trust --> Authorities" and post a screenshot of that page. If you want, you can black out all lines that don't have "Let's Encrypt" the name.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 16, 2021, 01:47:20 pm
There's an unknown cert and Cloudflare. I'll sent it in a bit.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 16, 2021, 02:26:15 pm
I thought you were asking about an HAProxy + Let's Encrypt issue...

Since you are using cloudflare certificates I am unable to help you. You are better off asking for help in the HAProxy forums or the cloudflare support regarding your issues.

Let me finish by giving you these informations:
1. The SSL Labs test pictures you sent me indicate that your certificate content (cn + alt name) seems to be wrong.
2. Your HAProxy HTTPS frontend settings do not match the ones I provide in my guide.


You could however just follow my tutorial step by step and would end up with a working setup. ;)
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on November 17, 2021, 03:04:01 pm
Thanks TheHellSite for the time taken to share the setup and help people.
This is my first foray into HA because I wanted to open traffic to a webserver on my lan for the first time. First I was going to just forward the port on OPN but then I saw your post and realised we can get so much more functionality from HA and your setup suggestion.

I have setup my webserver (nginix) to listen on custom port say 8082, and there is where I got the letsencrypt & dedyn.io setup first, as a kid of test before moving to OPN.
I've now done the cert as per your guide, essentially moved cert infrastrucure from the real server to OPN. All is good.

I've done the "pre-requisites" in my mind. I'm getting ready to string it together and then my question if I may.

Part 4, step 1. It says to change OPN Admin interface to another port(s) for http and https, usual 80, 443 because HA would expect traffic there.
Is it obligatory or could I keep my real server on 8082 and setup HA to exect that and keep OPN as is on 80 & 443?
I expect to use https with a self signed cert on the real server -instead of the current cert now in OPN-, or maybe only http on that port at the beginning.
Thanks.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 17, 2021, 07:16:00 pm
I thought you were asking about an HAProxy + Let's Encrypt issue...

Since you are using cloudflare certificates I am unable to help you. You are better off asking for help in the HAProxy forums or the cloudflare support regarding your issues.

Let me finish by giving you these informations:
1. The SSL Labs test pictures you sent me indicate that your certificate content (cn + alt name) seems to be wrong.
2. Your HAProxy HTTPS frontend settings do not match the ones I provide in my guide.


You could however just follow my tutorial step by step and would end up with a working setup. ;)

I am not sure what I am missing... I went over the HA HTTPS Frontend a dozen times and I am not seeing what's not matching... :(
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 18, 2021, 11:08:11 am
Part 4, step 1. It says to change OPN Admin interface to another port(s) for http and https, usual 80, 443 because HA would expect traffic there.
Is it obligatory or could I keep my real server on 8082 and setup HA to exect that and keep OPN as is on 80 & 443?
I expect to use https with a self signed cert on the real server -instead of the current cert now in OPN-, or maybe only http on that port at the beginning.
Thanks.

You can run your real servers (f.e. nginx web server) on whatever port you like (80, 443, 50443, ......) that doesn't matter at all.
The reason why you have to change the OPNsense web server port is quite simple.

OPNsense is obviously running a built in webserver by default (LAN INTERFACE / PORT 443), otherwise you wouldn't be able to access the web interface at all.
HAProxy is listening on ALL INTERFACES by default.

Web Server = Interface = IP:Port
HAProxy = ALL = 0.0.0.0:0 or 0.0.0.0:80+443
OPNsense = LAN = 192.168.1.1:443

This is the conflict! You can't have two or more services listening on the same "Interface + IP:Port". How would they know which traffic belongs to whom? As soon as one of them grabs the traffic the other one will never get it.

TL;DR:
1. Yes, you can run your real servers on any port (f.e. 8082).
2. No you can't change the OPNsense back to port 443 because you wouldn't be able to reach the OPNsense web interface anymore and or HAProxy will refuse to start.
3. I strongly advise you to also run your real server(s) with a self-signed SSL certificate to increase security. It is however not necessary.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 18, 2021, 11:20:00 am
I am not sure what I am missing... I went over the HA HTTPS Frontend a dozen times and I am not seeing what's not matching... :(

As I already said, it has something to do with your cloudflare domain config!
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on November 18, 2021, 12:46:35 pm

Web Server = Interface = IP:Port
HAProxy = ALL = 0.0.0.0:0 or 0.0.0.0:80+443
OPNsense = LAN = 192.168.1.1:443

This is the conflict! You can't have two or more services listening on the same "Interface + IP:Port". How would they know which traffic belongs to whom? As soon as one of them grabs the traffic the other one will never get it.

TL;DR:
1. Yes, you can run your real servers on any port (f.e. 8082).
2. No you can't change the OPNsense back to port 443 because you wouldn't be able to reach the OPNsense web interface anymore and or HAProxy will refuse to start.
3. I strongly advise you to also run your real server(s) with a self-signed SSL certificate to increase security. It is however not necessary.
Thank you for taking the time to answer. The problem with conflicting ip+ports is one I understand but I failed to articulate my question to explain what I meant.
I have been following the guide step by step, back and forth to understand each part. I made a little diagram for me to understand it better. I hope you don't mind if I spoil your thread with it. It just works for me to understand and better ask a question.
(https://i.ibb.co/WNXCxbb/HAP-understanding.png)
Essentially my question was if I could bind the SNI front end internally to a custom port instead of the usual 80,443. I probably should think again if what I'm asking is a sane question.
I've used this diagram and read&re-read your guide and I think I have a better grasp on it.
The nginx in my picture is me trying to figure out where/how I need to do the setup for my use. I hope I have not misrepresented it.
Let me come back with a better question when I've tested things. Much obliged.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 18, 2021, 04:00:00 pm
(https://i.ibb.co/WNXCxbb/HAP-understanding.png)
Essentially my question was if I could bind the SNI front end internally to a custom port instead of the usual 80,443.

First off all: B in your picture. The HTTP frontend shouldn't have the SSL_backend as the backend!!! Instead you only place the HTTPtoHTTPS redirect rule on it and if you need it, the HTTP services that don't require SSL.
You are also missing the PLEX_backend containing the PLEX_server.


Short Answer: No, because then you would have to access your services using that custom port in addition to having a subdomain for them. Which is pretty pointless, imho.

Example with default http/https port = http(s)://plex.yourdomain.tld/
Example with custom Port = 1234 = http(s)://plex.yourdomain.tld:1234/


Long Answer: It might be possible if you create a port forward in your OPNsense that forwards any WAN traffic that hits on TCP 80/443 to 127.0.0.1:1234.



My question to you: Why do you want't to do this in the first place? It makes this setup even more complex, for no reason...
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on November 18, 2021, 08:26:40 pm
Thank you for the clarifications. They help. I'm revising it now.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 11:05:05 am
Hello,

when I started implementing HAProxy in my network I couldn't find any complete and well written guide out there. I had to puzzle everything together from various websites.
So I thought I would save many of you a lot of time and provide my ultimate HAProxy on OPNsense guide.  :)

This tutorial will show you how to configure HAProxy as a reverse proxy on OPNsense using wildcard certificates from Let's Encrypt.
It is going to be a step-by-step guide with images on how to set things up while also explaining why we set things up in a certain way.
I will try to make this as complete and detailed as possible.
If you think that there is anything wrong or missing, feel free to tell me about it and I will consider changing it.

If this guide was helpful to you then please leave me a thanks down below as it took me several days to write this down.

Kind Regards
TheHellSite

This guide was a life-saver. And while implementing, I finally had the chance to understand a little more how HAProxy works. Thank you so much for writing it.

Everything worked as documented, and in a couple of hours I was up and running, the only problem was getting 100% on the ssl quality test. The Cipher Strength was always 90%, so I got A+, with a 90% overall score.

After a couple of hours of tinkering with the ciphers, I figured out that the test doesn't like 128bit ciphers. So I removed those from the Cipher List (TLSv1.2) and Cipher Suites (TLSv1.3) of the HTTPS_frontend, and I finally got the A+ 100% score.

In case anyone faces the same issue, these are the cipher settings for HTTPS_frontend:

Quote
Cipher List: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305

Cipher Suites: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

Thanks again,

Alessandro
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 23, 2021, 11:32:25 am
After a couple of hours of tinkering with the ciphers, I figured out that the test doesn't like 128bit ciphers. So I removed those from the Cipher List (TLSv1.2) and Cipher Suites (TLSv1.3) of the HTTPS_frontend, and I finally got the A+ 100% score.

In case anyone faces the same issue, these are the cipher settings for HTTPS_frontend:

Quote
Cipher List: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305

Cipher Suites: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

Thanks again,

Alessandro

Happy to hear it is working for you.
I think you missed the part of my tutorial where I am giving needed ciphers and cipher suites to get an 100% A+ rating.

Quote
My Tutorial
Cipher List: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384
Cipher Suites: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

yours
Cipher List: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305
Cipher Suites: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

The only difference between my guide and your list is the "DHE-RSA-AES256-GCM-SHA384" cipher.
Both will be scoring 100 % A+ while mine offer even more client compatibility.

Seems like I forgot to mention it again during tutorial at the part where the HTTPS_frontend is created. I now put a reference in there pointing to the beginning of my post where I provide the current best cipher list and cipher suits.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 11:54:24 am
Happy to hear it is working for you.
I think you missed the part of my tutorial where I am giving needed ciphers and cipher suites to get an 100% A+ rating.

Indeed I missed it, even though I followed the guide with great attention. Problem is that it was not in the tutorial, but at the beginning of the post. Sorry about that, I paid attention mainly to the tutorial. Would have saved a lot of time tinkering with the ciphers. :)

Quote
The only difference between my guide and your list is the "DHE-RSA-AES256-GCM-SHA384" cipher.
Both will be scoring 100 % A+ while mine offer even more client compatibility.

Yes I noticed that single difference. I started from the default ciphers of HAProxy, and that one wasn't in there. I simply removed the AES128 ones from the defaults. The link you put to that  Mozilla Config is fantastic, never even thought something like that existed. Thank you again.

Quote
Seems like I forgot to mention it again during tutorial at the part where the HTTPS_frontend is created. I now put a reference in there pointing to the beginning of my post where I provide the current best cipher list and cipher suits.

That's good, because I missed it completely, would've saved me quite some time. My bad. :)

One thing I wanted to ask you: I followed your naming conventions and I noticed you had the 1_ prefix both for the HTTP and HTTPS frontend. I renamed the HTTPS to 2_HTTPS_frontend, don't know if it was intentional or not, but I interpreted it as a progressive number so that one was a 2.


Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 23, 2021, 11:57:42 am
One thing I wanted to ask you: I followed your naming conventions and I noticed you had the 1_ prefix both for the HTTP and HTTPS frontend. I renamed the HTTPS to 2_HTTPS_frontend, don't know if it was intentional or not, but I interpreted it as a progressive number so that one was a 2.

I did that on purpose to express the "level" of reverse proxying.

Level 1 - SNI traffic
Level 2 - HTTP + HTTPS traffic

However you can name it as you like. It doesn't matter in terms of functionality.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 12:01:57 pm
I did that on purpose to express the "level" of reverse proxying.

Level 1 - SNI traffic
Level 2 - HTTP + HTTPS traffic

However you can name it as you like. It doesn't matter in terms of functionality.

Thanks for the explanation.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 12:30:11 pm
Seems like I forgot to mention it again during tutorial at the part where the HTTPS_frontend is created. I now put a reference in there pointing to the beginning of my post where I provide the current best cipher list and cipher suits.

I moved the frontend cipher settings and bind options to the Global Parameters -> SSL Default Settings section, emptied the relevant fields in HTTPS_frontend, it works fine and I think it's simpler:

(https://i.imgur.com/6sqazDa.png)

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 23, 2021, 12:43:27 pm
Arguable! ;)
If you have many HTTPS frontends (on different ports) that might need different SSL settings then my way is better.

Otherwise it doesn't really matter where you put the settings. Just note that the SSL default settings get overwritten once you set anything in the associated boxes on the HTTPS frontends.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 12:49:39 pm
Arguable! ;)
If you have many HTTPS frontends (on different ports) that might need different SSL settings then my way is better.

Otherwise it doesn't really matter where you put the settings. Just note that the SSL default settings get overwritten once you set anything in the associated boxes on the HTTPS frontends.

Your second point is the answer to your first observation: Global settings are a "global default". If you need a custom config for a frontend, you can customize its config and it will supersede the global one. ;)

That's why I like it, it's the classic enterprise approach to organizing things, more intuitive for me (global...and custom as an exception).

But like you said...it's all very subjective obviously. Individual preferences.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: dima1002 on November 28, 2021, 09:01:48 am
Hello,

what is wrong here?

In Firefox I get the following error: PR_END_OF_FILE_ERROR

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log audit debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): LetsEncrypt_443 ()

# Frontend (DISABLED): LetsEncrypt_80 ()

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_61a24897421141.86617043 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_61a24897421141.86617043

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61a24a78aa9cc4.11915455.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61a249350142e3.01879320.txt)]

# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Backend: Mail ()
backend Mail
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Main 192.168.111.2:443 ssl verify none

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend: Nextcloud ()
backend Nextcloud
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Nextcloud 192.168.111.3:443 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: Bitwarden_Backend ()
backend Bitwarden_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Vaultwarden80 192.168.111.77:80
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 28, 2021, 11:16:09 am
Hello,

what is wrong here?

In Firefox I get the following error: PR_END_OF_FILE_ERROR

Are you only getting this error when using Firefox?
If so, which version of Firefox are you running?
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: SOUK on December 04, 2021, 03:56:42 pm
Delete
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: evathesalmon on December 12, 2021, 11:41:33 pm

In Firefox I get the following error: PR_END_OF_FILE_ERROR


Hello,

I have same issue in firefox and chrome. In logs it is said "Received something which does not look like a PROXY protocol header"

(https://i.imgur.com/sjyyqey.png)

I thought at first that it is a proxy problem and double checked your 20210613 update but my bind option pass-through set as accept-proxy

(https://i.imgur.com/tkMNypC.png)

(https://i.imgur.com/HrWTwSa.png)

My current HAProxy version is 3.7 and opnsense is 21.7.6
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on December 14, 2021, 09:27:56 am
In Part 6, NAT Reflection: it applies to port forwarding rules, but in the guide you switched to a simple filter rule.

So there's only one option remaining: split DNS.

HAProxy has been rock solid, thanks again for your guide. I'm having a hard time only for Uptime Kuma, it uses websockets, and it's the only service that doesn't work behind HAProxy. The dev published a guide for the configuration behind several reverse proxies, unfortunately the only one missing is HAProxy: https://github.com/louislam/uptime-kuma/wiki/Reverse-Proxy#nginx

I'm sure there's a way to make it work but I can't find it...
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 14, 2021, 10:35:36 am

In Firefox I get the following error: PR_END_OF_FILE_ERROR


Hello,

I have same issue in firefox and chrome. In logs it is said "Received something which does not look like a PROXY protocol header"

I thought at first that it is a proxy problem and double checked your 20210613 update but my bind option pass-through set as accept-proxy

My current HAProxy version is 3.7 and opnsense is 21.7.6

I am running the setup with the latest OPNsense updates just fine, using Firefox (mainly) but Chrome or Edge work fine either.
It seems you did misconfigure something!

Post your haproxy export in a code box. (redact any sensitive information, but leave in the local IPs!)
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 14, 2021, 11:05:13 am
In Part 6, NAT Reflection: it applies to port forwarding rules, but in the guide you switched to a simple filter rule.

So there's only one option remaining: split DNS.

Thank you for pointing this out! I changed the guide some time ago and forgot to update that part.
Just keep in mind that there is nothing wrong with Split DNS, it is even the preferred way of doing it!

HAProxy has been rock solid, thanks again for your guide. I'm having a hard time only for Uptime Kuma, it uses websockets, and it's the only service that doesn't work behind HAProxy. The dev published a guide for the configuration behind several reverse proxies, unfortunately the only one missing is HAProxy: https://github.com/louislam/uptime-kuma/wiki/Reverse-Proxy#nginx

I'm sure there's a way to make it work but I can't find it...

Isn't that just a simple web service displaying uptime of servers?
How are you accessing it on your local network? f.e. http://192.168.2.55:3001/ or https://192.168.2.55:3001/
If it is http then you will need to DISABLE SSL in the real server settings for uptime kuma.
If it is https then you will need to ENABLE SSL but DISABLE SSL verification in the real server settings for uptime kuma.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on December 14, 2021, 11:16:25 am
Thank you for pointing this out! I changed the guide some time ago and forgot to update that part.
Just keep in mind that there is nothing wrong with Split DNS, it is even the preferred way of doing it!

I know, it's my preferred way too. I love DNS. Was just pointing out that there's only one option. :)

Quote
Isn't that just a simple web service displaying uptime of servers?
How are you accessing it on your local network? f.e. http://192.168.2.55:3001/ or https://192.168.2.55:3001/
If it is http then you will need to DISABLE SSL in the real server settings for uptime kuma.
If it is https then you will need to ENABLE SSL but DISABLE SSL verification in the real server settings for uptime kuma.

I've already reverse-proxied a lot of services, I know how that works. The problem is how Uptime Kuma works: it uses ws:// (websockets) connections in addition to HTTP, so you connect in http first to auth, then it starts communicating through WS, through a sort of tunnel. If you check that link I provided, you will see that for many proxies there's some custom configs to support that. The only proxy that does one-line config magic is caddy...it's tempting me a lot...everybody told me that caddy is the simplest one and it simply works, without doing any hard config work. But I already have HAProxy in place, and would like to stick to it.

UPDATE: I found this article https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/#advanced-configuration

Now I have to understand where to put those things in the UI on OPNsense. Is there a way to edit the config files directly?
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 14, 2021, 11:36:16 am
I've already reverse-proxied a lot of services, I know how that works. The problem is how Uptime Kuma works: it uses ws:// (websockets) connections in addition to HTTP, so you connect in http first to auth, then it starts communicating through WS, through a sort of tunnel. If you check that link I provided, you will see that for many proxies there's some custom configs to support that. The only proxy that does one-line config magic is caddy...it's tempting me a lot...everybody told me that caddy is the simplest one and it simply works, without doing any hard config work. But I already have HAProxy in place, and would like to stick to it.

Did you try something like this?
https://stackoverflow.com/a/22735431/17193869
or this
https://discourse.haproxy.org/t/using-reverse-proxy-with-secured-web-sockets-wss/2917

Otherwise your best bet is to ask in the haproxy discourse forum including your config export.
Sadly, I can't help everyone to get specific services running.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on December 14, 2021, 01:14:45 pm
Did you try something like this?
https://stackoverflow.com/a/22735431/17193869
or this
https://discourse.haproxy.org/t/using-reverse-proxy-with-secured-web-sockets-wss/2917

I found this article from HAProxy guys:
https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/#simple-configuration

That's why I wanted to know if I can configure it via shell, working on the files directly. Do you know if that's possible?

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 14, 2021, 01:36:46 pm
I found this article from HAProxy guys:
https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/#simple-configuration

That's why I wanted to know if I can configure it via shell, working on the files directly. Do you know if that's possible?

Oh very well, of course there is!
You just need to place these settings (one per line) in the advanced settings of the frontend/backend.

Frontends --> yourfrontend --> top left, enable advanced mode --> Advanced settings --> Option pass-through

As far as I am aware settings these options on the frontend will apply them to ALL services that are going through it.
But you should also be able to set them on the corresponding backend so that it will only apply to the specific service.

Code: [Select]
## routing based on websocket protocol header
  acl hdr_connection_upgrade hdr(Connection)  -i upgrade
  acl hdr_upgrade_websocket  hdr(Upgrade)     -i websocket
  use_backend bk_ws if hdr_connection_upgrade hdr_upgrade_websocket
  default_backend bk_web

This however is also just a combination of acl=condition and rule=use_backend.
It is basically saying: IF HTTP_header=Connection+Upgrade THEN USE_BACKEND AKUMA_ws_backend
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on December 14, 2021, 02:46:51 pm
As far as I am aware settings these options on the frontend will apply them to ALL services that are going through it. But you should also be able to set them on the corresponding backend so that it will only apply to the specific service.

Not working, I don't think it's a config issue...

Seems like there were issues with websockets with older versions, newest v2.5 fixed several things regarding websockets support: https://www.haproxy.com/blog/announcing-haproxy-2-5/

Latest plugin has HAProxy v2.2.18, will have to wait for the upgrade...unless I can upgrade it manually...

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: evathesalmon on December 15, 2021, 12:07:26 am

Post your haproxy export in a code box. (redact any sensitive information, but leave in the local IPs!)


Thank you for reply.

I only have map for internal network because I don't pass-through external traffic. And I also set host binding via unbound overrides all to 192.168.64.1.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 127.0.0.1:80, 127.0.0.1:443)
frontend 0_SNI_frontend
    bind 127.0.0.1:443 name 127.0.0.1:443
    bind 127.0.0.1:80 name 127.0.0.1:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_61a3f4fdabe975.79392880 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_61a3f4fdabe975.79392880

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61a3f8200d98a7.85727127.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61a3f5af2446f3.11405534.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: unraid_backend ()
backend unraid_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server unraid_server 192.168.0.20:443 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: opnsense_backend ()
backend opnsense_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opnsense_server 192.168.0.1:1443 ssl verify none


(https://i.imgur.com/gD1BC97.png)
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 15, 2021, 01:02:57 am
Thank you for reply.

I only have map for internal network because I don't pass-through external traffic. And I also set host binding via unbound overrides all to 192.168.64.1.

Check the cipher list and cipher suites on your HTTPS frontend! They don't match the ones I provide in my first post!

That one thing wonders me and might be your issue... Why did you set the SNI_frontend to listen on the localhost address? This makes no sense at all!
You are allowing inbound traffic on your WAN address 80+443 with our firewall rule but your SNI_frontend is only listening on the localhost address. This way it will never catch any traffic at all!

I never said to do so in my guide! Apart from the SSL_server address which can safely use the localhost address.
Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: evathesalmon on December 15, 2021, 09:11:15 pm

You are allowing inbound traffic on your WAN address 80+443 with our firewall rule but your SNI_frontend is only listening on the localhost address. This way it will never catch any traffic at all!



UPD: nevermind, it was my local cache and unbound cache. After purging everything it works just fine without any specific binding. Thank you for your help. 

I have amended my config and compared again with the guide but I must be missing something.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_61a3f4fdabe975.79392880 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_61a3f4fdabe975.79392880

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61a3f8200d98a7.85727127.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNET_condition
    acl acl_61a3f9b4ed7092.44798843 src 192.168.0.0/24

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61a3f5af2446f3.11405534.txt)] if acl_61a3f9b4ed7092.44798843

# Backend: unraid_backend ()
backend unraid_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server unraid_server 192.168.0.20:443 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: opnsense_backend ()
backend opnsense_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opnsense_server 192.168.0.1:1443 ssl verify none

# Backend: qbittorrent_backend ()
backend qbittorrent_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server qbittorrent_server 192.168.0.20:8080

The setup is working only if I bind my host->ip in unbound overrides the way they are physically, in example unraid is binded to 192.168.0.20, opnsense is binded to 192.168.0.1 and other (see the pic). But if the redirects are set to 192.168.54.1 just like in the guide it just unable to connect.

In other words seems like 0_SNI_frontend doesn't listen to all addresses even if it is configured so.

Also a question: should I set any WAN rules even if I only routing the LAN traffic, cause now I didn't set any.

(https://i.imgur.com/by1yDJ0.png)
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: boredpanda on January 03, 2022, 01:22:51 pm
Hello,

when I started implementing HAProxy in my network I couldn't find any complete and well written guide out there. I had to puzzle everything together from various websites.
So I thought I would save many of you a lot of time and provide my ultimate HAProxy on OPNsense guide.  :)

Hello! And thank you very much for your well-written guide. It made my switch from pfSense to OPNsense far smoother! HAProxy in pfSense looks quite different from HAProxy in OPNsense.

I self-host a bunch of services on a local server, and all the services are in dockers, meaning they all have the same IP but different ports. Most of the services worked just fine following your guide, but I'm noticing a few odd things. For instance, on my Firefly III installation, I get the error in the attachment and I'm unable to log in. If I visit the IP:PORT address, it works fine. Any suggestions on changes I could maybe make to my configuration?

None of my services are publicly exposed and I'm using a FQDN just to not having to remember port numbers.

My config is below. Thanks! :)

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_61d029838380d8.68540995 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_61d029838380d8.68540995

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61d02d156c0846.98881851.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    # ACL: EXTERNAL_conditions
    acl acl_61d066f2cc9639.62892989 src 193.138.218.219
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_61d05461151001.71548589 src 192.168.1.0/24 192.168.70.0/24

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61d053aef188f4.27343600.txt)] if acl_61d066f2cc9639.62892989 || acl_61d05461151001.71548589
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61d029eb5a9da6.54806678.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: nextcloud_backend ()
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nextcloud_server 192.168.1.12:444 ssl verify none

# Backend: miniflux_backend ()
backend miniflux_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server miniflux_server 192.168.1.12:5600

# Backend: joplin_backend ()
backend joplin_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server joplin_server 192.168.1.12:22300

# Backend: calibre_backend ()
backend calibre_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server calibre_server 192.168.1.12:8083

# Backend: emby_backend ()
backend emby_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server emby_server 192.168.1.12:8096

# Backend: grocy_backend ()
backend grocy_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server grocy_server 192.168.1.12:9283

# Backend: hydra_backend ()
backend hydra_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hydra_server 192.168.1.12:5076

# Backend: piwigo_backend ()
backend piwigo_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server piwigo_server 192.168.1.12:8099

# Backend: collabora_backend ()
backend collabora_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server collabora_server 192.168.1.12:9980

# Backend: freshrss_backend ()
backend freshrss_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server freshrss_server 192.168.1.12:8066

# Backend: wallabag_backend ()
backend wallabag_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server wallabag_server 192.168.1.12:6500

# Backend: wikijs_backend ()
backend wikijs_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server wikijs_server 192.168.1.12:3000

# Backend: heimdall_backend ()
backend heimdall_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server heimdall_server 192.168.1.12:8538

# Backend: monica_backend ()
backend monica_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server monica_server 192.168.1.12:8956

# Backend: firefly_backend ()
backend firefly_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server firefly_server 192.168.1.12:8088

# Backend: paperless_1_backend ()
backend paperless_1_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server paperless_1_server 192.168.1.12:8016

# Backend: paperless_2_backend ()
backend paperless_2_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server paperless_2_server 192.168.1.12:8006
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 04, 2022, 01:16:56 am
First of all please use the correct cipher list and suites, see the beginning of my OP.
You are still using AES128 ciphers indicating you didn't read my tutorial correctly.

Next thing would be to clear your browser cache.
Is it only firefly that is not working or are others also affected?

Are you using a trusted lets encrypt cert or a selfsigned one?
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: boredpanda on January 05, 2022, 06:04:45 pm
First of all please use the correct cipher list and suites, see the beginning of my OP.
You are still using AES128 ciphers indicating you didn't read my tutorial correctly.

Next thing would be to clear your browser cache.
Is it only firefly that is not working or are others also affected?

Are you using a trusted lets encrypt cert or a selfsigned one?

That's fair enough! I have updated my configuration with the correct cipher list and suites. My browser cache has been cleared and I've tried multiple browsers on different computers. The issue still remains. I am using a trusted let's encrypt cert (wildcard domain managed by ACME on OPNsense).

Two of my services aren't working as expected at the moment, and that's Firefly III and Grocy. Everything else is working.

For Firefly, I came across this thread (https://github.com/firefly-iii/firefly-iii/discussions/5118) on Github discussing my exact issue. It was apparently fixed in this comment (https://github.com/firefly-iii/firefly-iii/discussions/5118#discussioncomment-1398790). My Docker env TRUSTED_PROXIES is set to ** already. Do I need to edit 1_HTTPS_frontend or 1_HTTP_frontend? I see that both of those have the option X-Forwarded-For header enabled in my HAProxy.

For Grocy, I'm having the issue described here (https://github.com/linuxserver/docker-grocy/issues/18), on Github. Someone using nginx mentioned they solved the issue by adding proxy_set_header X-Forwarded-Proto https; to their nginx config.

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 05, 2022, 06:34:51 pm
Two of my services aren't working as expected at the moment, and that's Firefly III and Grocy. Everything else is working.

For Firefly, I came across this thread (https://github.com/firefly-iii/firefly-iii/discussions/5118) on Github discussing my exact issue. It was apparently fixed in this comment (https://github.com/firefly-iii/firefly-iii/discussions/5118#discussioncomment-1398790). My Docker env TRUSTED_PROXIES is set to ** already. Do I need to edit 1_HTTPS_frontend or 1_HTTP_frontend? I see that both of those have the option X-Forwarded-For header enabled in my HAProxy.

For Grocy, I'm having the issue described here (https://github.com/linuxserver/docker-grocy/issues/18), on Github. Someone using nginx mentioned they solved the issue by adding proxy_set_header X-Forwarded-Proto https; to their nginx config.

You should (but don't have to) leave the x-forwarded-for header enabled. It is very useful since the real servers (firefly, grocy, ...) will get to know the original IP of the client trying to access it.

The links you posted both imply that your issues COULD be resolved by adding the following to your "HTTPS_frontend".

HAProxy --> Virtual Services --> Public Services --> 1_HTTPS_frontend --> Edit --> Enable "advanced mode" (top left corner) --> Scroll down to "Advanced settings" --> Option pass-through --> insert the below code --> Save --> Apply

Code: [Select]
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Real-IP %[src]

You might not need both lines so play around until you find the necessary line(s) and please post the final solution!

If both of your services are working now then I suggest to remove that setting from your HTTPS_frontend and set it on the firefly and grocy backends instead!

HAProxy --> Virtual Services --> Public Services --> firefly/grocy backend --> Edit --> Enable "advanced mode" (top left corner) --> Scroll down to "Tuning options" --> Option pass-through --> insert the above code --> Save --> Apply
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: boredpanda on January 05, 2022, 07:00:36 pm
The links you posted both imply that your issues COULD be resolved by adding the following to your "HTTPS_frontend".

HAProxy --> Virtual Services --> Public Services --> 1_HTTPS_frontend --> Edit --> Enable "advanced mode" (top left corner) --> Scroll down to "Advanced settings" --> Option pass-through --> insert the below code --> Save --> Apply

Code: [Select]
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Real-IP %[src]

You might not need both lines so play around until you find the necessary line(s) and please post the final solution!

The second line didn't work, but http-request set-header X-Forwarded-Proto https if { ssl_fc } solved the issue with both services, so I have added the line to both of the back ends. Thank you so much! ;)
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on January 10, 2022, 07:24:43 pm
I wonder if I could ask this question here. I've followed the instruction to the letter, all is good.
I have an additional requirement however, to proxy TCP mode to the real server after TLS offloading at HA.
Say the real server is listening on port 8053 and we can live without tls between HA and it to begin with, is in the internal LAN. The public port for the service however is different, say 853. So HA needs to receive in one and sent to another.
I understand we can't mix tcp and http modes and the tutorial explains it in the relevant part.

My current thinking is that since 0_SNI_frontend and SSL_backend are already in TCP mode, the first part is adding the port, ending with 0.0.0.0:80, 0.0.0.0:443: 0.0.0.0:851
- Since the TLS is being terminated at the 1_HTTPS_frontend but is not TCP, that is not the right place to go next.
I thought then I needed to create a new frontend in TCP mode. I duplicated 1_HTTPS_frontend and changed name to 1_TCP_frontend. Set it to listen on the vip:851. I left the map rule on it and disabled SSL offload.

Testing gives me error:
2022-01-10T17:11:39   haproxy[81388] externalip:port [10/Jan/2022:17:11:39.859] 1_TCP_frontend 1_TCP_frontend/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0   
2022-01-10T17:11:39   haproxy[81388] Connect from externalip:port to wanip:851 (0_SNI_frontend/TCP)


Makes me think I'm not too far.
My question to you knowledgeable people is if the setup of this tutorial can be expanded to do this, or I'm trying to do something totally that'll never work, and some pointers to what I need to do would be wonderful please.
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 10, 2022, 10:10:57 pm
You have to create a condition and rule for that service. And then place that rule on the 0_SNI_frontend.

However you CAN NOT do url conditions, f.e. service.domain.com ....


OR

You create a new TCP frontend that listens on that port.
Then create server + backend for the service and set this backend on the new tcp frontend as default backend.

OR

you simply use a port forwarding.....
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on January 11, 2022, 12:07:21 am
Option 2 was what I was hoping was possible. It is working and what I found was that I needed to add the port to the 0_SNI_frontend too, which I wasn't sure about. I had it from earlier attempts but once working, my new proxy would not work without it.
So in summary, added the port to 0_SNI_frontend; added the new default backend to the new front end I had created (that I had missing), as well as has the map file rule. I haven't tested without this map yet, that's next.
I'm going to read and re-read the config that is working against the documentation to understand how. I can't follow the new path at present due to my lack of knowledge on the HA options used.
Thank you TheHellSite.
Looks like this and allows me to put my subdomain in my android phone to get DNS over TLS on my own infra that filters domains with Adguard before going out on TLS to the open internet. That was the goal.
For anyone wondering is here:
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    1
    hard-stop-after             60s
    maxconn                     10
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 10
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening on 0.0.0.0:80, 0.0.0.0:443, 0.0.0.0:853)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:853 name 0.0.0.0:853
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (listening on 192.168.5.100:80 i.e. http only)
frontend 1_HTTP_frontend
    bind 192.168.5.100:80 name 192.168.5.100:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_619439805021f2.97978352 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_619439805021f2.97978352

# Frontend: 1_HTTPS_frontend (Listening on 192.168.5.100:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.5.100:443 name 192.168.5.100:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61952b9d47d700.25962675.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Frontend: 1_TCP_frontend (Listening on 192.168.5.100:853)
frontend 1_TCP_frontend
    bind 192.168.5.100:853 name 192.168.5.100:853 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61dc51606078d9.11258474.certlist
    mode tcp
    default_backend nginx_backend-tcp
    # tuning options
    timeout client 15m

    # logging options
    option tcplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.5.100 send-proxy-v2 check-send-proxy

# Backend: nginx_backend-tcp ()
backend nginx_backend-tcp
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server nginx_1 192.168.5.152:8053
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Tattoofreak on January 24, 2022, 02:18:58 pm
Thank you for this tutorial.
Unfortunately my requirements are a bit different.
Basically, what I want to do, is:

I want to set up HAProxy just for routing traffic based on URLs (https://xyz.domain.com goes to server 1 and https://abc.domain.com goes to server 2, etc...).
All SSL stuff for the destination web servers is being handled by a separate Linux certificate server and the web servers themselfes, independent from OPNsense/HAProxy. HAProxy is really only needed for routing traffic based on URLs, nothing more, nothing less.

The websites are all working when I use NAT rules, but I was not able yet to make it run with HAProxy in between.

For testing it with just one webserver, I have configured the following:

OPNsense settings:

System --> Settings --> Administration:
TCP port: 4443

Firewall --> Rules --> WAN:
Protocol: IPv4/TCP | Source: any | Port: Any | Destination: This Firewall | Port: 80 (HTTP) | Rule: Pass
Protocol: IPv4/TCP | Source: any | Port: Any | Destination: This Firewall | Port: 443 (HTTPS) | Rule: Pass

HAProxy settings (everthing not mentioned I left on default settings):

Services --> HAProxy --> Settings --> Settings:
Service:
Enable HAProxy: Checked

Services --> HAProxy --> Settings --> Real Servers:
Real Servers:
Name or Prefix: web01_http | FQDN/IP: 192.168.1.10 | Port: 80
Name or Prefix: web01_https | FQDN/IP: 192.168.1.10 | Port: 443

Services --> HAProxy --> Settings --> Virtual Services:
Backend Pools:
Name: web01_backendpool01 | Mode: TCP (Layer4) | Servers: web01_http, web01_https

Services --> HAProxy --> Settings --> Rules & Checks:
Conditions:
Name: myWebsite01_condition01 | Condition type: Host starts with | Host prefix: <mysubdomain>.<mydomain>.com

Services --> HAProxy --> Settings --> Rules & Checks:
Rules:
Name: myWebsite01_rule01 | Select conditions: myWebsite01_condition01 | Execute function: Use specific Backend Pool | Use backend pool: web01_backendpool01

Services --> HAProxy --> Settings --> Virtual Services:
Public Services:
Name: web01_publicService01 | Listen Addresses: <myPublicIP>:80, <myPublicIP>:443 | Type: TCP | Rules: myWebsite01_rule01


Any help would be very appreciated, thank you in advance!
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Tattoofreak on January 27, 2022, 01:26:31 pm
Anyone?

I'm wondering if it is even possible to catch URL based packets when Public Services Type is set to TCP? Would this work?
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 29, 2022, 09:37:27 am
You have to set your backends and frontends to HTTP Mode.
Also disable SSL offloading on the frontends.

But I can't guarantee for sure that it will work.
TCP Mode will never (with a few exceptions) work because there is no header in the packets that would tell HAProxy which service to send the traffic to.

HTTP Mode could work, but you might need to create some "http header contains..." conditions.
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: afall on January 31, 2022, 05:42:25 pm
First of all thank you for the Guide it was great and helped me out a lot.

I just have a question as I have to use a vpn adapter call for work called Zscaler and it try to make a tunnel using port 443. How do i tell HAproxy as a default to only route only thing in the map file and dont intercept anything else. So the vpn client can create a tunnel. 
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: vladnik on February 01, 2022, 04:16:24 pm
Hi @TheHellSite,

just wanted to say thank you for this excellent guide, worked like a charm and thaught me much about how HAProxy works. Thank you for a job well done!  :)

Cheers!
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 03, 2022, 01:37:56 pm
I have two Server with several vhosts behind a OPNsense Router/Firewall

Is this also possible to have two server which need certs? Or only one?

When if it’s possible, how I have to do it?
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Tattoofreak on February 10, 2022, 03:40:27 pm
@TheHellSite
Maybe you would also like on how to enable Websockets on your frontend(s) if your incoming clients are looking for such one(s). Websockets are basically used for example for streaming services over web. I was looking for so long on how to resolve my problem for making HAProxy work with Synology's DS Cam Android app which tries to connect from remote to the Synology Surveillance Station NAS behind HAProxy and I finally found out. You have to insert the following on your frontend (where you have to replace <myBackend(Pool)> with your according backend, of course):

Code: [Select]
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend <myBackend(Pool)> if is_websocket

Here's the link to my original and solved issue:
 https://www.synoforum.com/threads/connecting-synology-ds-cam-android-app-to-synology-surveillance-station-through-opnsense-haproxy-plugin.7969/ (https://www.synoforum.com/threads/connecting-synology-ds-cam-android-app-to-synology-surveillance-station-through-opnsense-haproxy-plugin.7969/)
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 12, 2022, 09:13:03 pm
I got this error
Quote
[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 02:38:21 pm
First of all thank you for the Guide it was great and helped me out a lot.

I just have a question as I have to use a vpn adapter call for work called Zscaler and it try to make a tunnel using port 443. How do i tell HAproxy as a default to only route only thing in the map file and dont intercept anything else. So the vpn client can create a tunnel.

Since your provided little to no info of the setup I am not really able to help you.
Also my guide is just to show beginners how things are done.

However if I understood your issue correctly, you can try the below. If that doesn't work you are better of asking in the HAProxy forums!

Add the following in the "Option pass-through" field of your 0_SNI_frontend.
Don't forget to create the backend along with the server for Zscaler. My example uses OpenVPN.
Code: [Select]
tcp-request inspect-delay 5s
use_backend OPENVPN_backend if req_ssl_hello_type 1
tcp-request content accept if !{ req_ssl_hello_type 1 }
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 02:46:13 pm
I have two Server with several vhosts behind a OPNsense Router/Firewall

Is this also possible to have two server which need certs? Or only one?

When if it’s possible, how I have to do it?

I got this error
Quote
[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

Never had this issue!

1. Go through my guide again. If you follow it step by step, you WILL HAVE a working setup. Otherwise you did something wrong.

2. Where is your config export? Nobody will be able to help you without a HAProxy config export!

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 15, 2022, 03:01:33 pm
I have following configuration:

NAS (192.168.1.118) → Apache → Nextcloud/DAV
/
ROUTER/Firewall/HAproxy (192.168.1.1)
\
Server (192.168.1.100) → Apache with severals vhost

I want to bring up one vhost as test with following configuration

Quote
<VirtualHost *:80>
     ServerAdmin mail@xxx.com
     ServerName xxx.ch
     ServerAlias www.xxxx.ch
     DocumentRoot /usr/share/webapps/blog/
     DirectoryIndex index.php
     RemoteIPProxyProtocol On
     <Directory /usr/share/webapps/blog>
        Options +Indexes +FollowSymLinks +MultiViews
        AllowOverride All
        Order allow,deny
        allow from all

    <FilesMatch \.php$>
      # For Apache version 2.4.10 and above, use SetHandler to run PHP as a fastCGI process server
      SetHandler "proxy:unix:/run/php-fpm/php-fpm.sock|fcgi://localhost"
    </FilesMatch>
    <Files "*.php">
      MultiviewsMatch Any
    </Files>
    </Directory>
     ErrorLog /var/log/httpd/blog_error.log
     CustomLog /var/log/httpd/blog_access.log combined
</VirtualHost>
My HAproxy.conf file looks like this

cat haproxy.conf

Quote
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: O_SNI_fronted (Listening 0.0.0.0:80 0.0.0.0:443)
frontend O_SNI_fronted
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NO_SSL_Rule
    acl acl_620808a860e296.91534155 req.ssl_ver gt 0

    # ACTION: HTTP_TO_HTTPS_RULE
    http-request redirect scheme https code 301 if !acl_620808a860e296.91534155

# Frontend: 1_HTTPS_frontend (Listening 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6208140971a7a3.08696099.certlist
    mode http
    option http-keep-alive
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_MAP_RULE
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/620809e036a6d1.87483247.txt)]

# Backend: SSL_backend (SSL  backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: 5erver_backend (Server backend)
backend 5erver_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80
# ERROR: server data not found (0b989d9b-eb50-4dff-8a2f-6bc56245fd74)

# Backend: NAS_backend (NAS backend)
backend NAS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nas_Server_80 192.168.1.118:80
# ERROR: server data not found (36c63574-bd94-43f7-836e-cd78c8edc6c0)
My map files looks like this
Quote
#public subdomains mapping
flood 5erver_backend
frank 5erver_backend
www 5erver_backend
torrent 5erver_backend
grafana 5erver_backend
nas 5erver_backend
kvm 5erver_backend
monitoring 5erver_backend
speedtest 5erver_backend
sync 5erver_backend
tracker 5erver_backend
cloud NAS_backend
dav NAS_backend
How I can fix this error
503 error?

Thanks for reply
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 03:25:08 pm
FIRST: You should remove your personal info from your post.

SECOND: Another issue from not properly reading my guide.

Your solution is in Part 5 - Step 6.

Quote
Now we create the backend that belongs to an actual service. You will need one backend for each service.
If you have multiple servers serving the exact same content than you will want to add all servers into a single backend so HAProxy can actually balance the load between the servers.

YOU NEED: ... one backend for each service.

YOU DID: ... one backend for each server hosting individual services.

Just think about it... How should HAProxy even be able to talk to one of your services when you are only pointing him to the IP:Port of the server virtually hosting the service!? This makes no sense...
It is like telling someone "Meet me in New York in a bar." without telling him which bar.
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 15, 2022, 04:17:31 pm
FIRST: You should remove your personal info from your post.

SECOND: Another issue from not properly reading my guide.

Your solution is in Part 5 - Step 6.

Quote
Now we create the backend that belongs to an actual service. You will need one backend for each service.
If you have multiple servers serving the exact same content than you will want to add all servers into a single backend so HAProxy can actually balance the load between the servers.

YOU NEED: ... one backend for each service.

YOU DID: ... one backend for each server hosting individual services.

Just think about it... How should HAProxy even be able to talk to one of your services when you are only pointing him to the IP:Port of the server virtually hosting the service!? This makes no sense...
It is like telling someone "Meet me in New York in a bar." without telling him which bar.

Ok, thanks for finding the error.

English isn't my mother language so can you little bit clarify the steps.

So I have to copy the backend and adjust it for every virtual host of Apache even if they are on the same port?

I have one apache on both SERVER(192.168.1.100 and 192.168.1.118) on port 80 an only one webserver with several vhosts. I thought the map file will do the split of the to the vhosts...

For me is Apache one service on which is on both SERVER and so It's has one backend each for this two server?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 15, 2022, 04:29:47 pm
(https://abload.de/img/haproxydgj5s.png)

So I have to do this for every single vhost?

So it's looking so?

(https://abload.de/img/haproxy_backendlgk4r.png)

So the maps file is also wrong I have to change nas_backend to nas_server_80 and 5erver_backend to 5erver_server_80 ?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 05:37:21 pm
This is not a question of your mother language not being english.  ;D
This is a question of you not understanding the layout of HAProxy.

It is really hard for me to help here since you don't seem to understand the basics of a reverse proxy.
A reverse proxy is not a web server to host your sites! It is meant to route traffic to a web server.

Rule 1: You need at least ONE server for each backend.
With server I mean server created in HAProxy. A server in HAProxy represents the service (f.e. nextcloud, plex, ...) running on one of your physical servers.

Rule 2: Each backend will then present your individual services to your frontend.

HTTPS Frontend --> Mapfile --> PLEX_backend --> PLEX_server --> 192.168.2.30:32400
HTTPS Frontend --> Mapfile --> BLOG_backend --> BLOG_server --> 192.168.2.30:80
HTTPS Frontend --> Mapfile --> CLOUD_backend --> CLOUD_server --> 192.168.2.40:43569

Rule 3: A mapfile is simply mapping an access URL, f.e. www.yourdomain.com, against a backend.


To be fair it seems I misunderstood your setup/goal. But since you still didn't explain exactly what you are trying to achieve I am assuming you are trying to something like this.

Apache is your webserver. (www.yourdomain.com)
These vhost seem to provide subdirectories or individual websites. (f.e. service.yourdomain.com/subdir/)

Code: [Select]
#public subdomains mapping
flood 5erver_backend
frank 5erver_backend
www 5erver_backend
torrent 5erver_backend
grafana 5erver_backend
nas 5erver_backend
kvm 5erver_backend
monitoring 5erver_backend
speedtest 5erver_backend
sync 5erver_backend
tracker 5erver_backend
cloud NAS_backend
dav NAS_backend

Let me further explain to you the map file. I mean actually my first post does it, but you don't seem to understand or didn't read...

Code: [Select]
nas 5erver_backendnas.yourdomain.com --> NAS_backend

Code: [Select]
www 5erver_backendwww.yourdomain.com --> 5erver_backend

Code: [Select]
grafana 5erver_backendgrafana.yourdomain.com --> 5erver_backend
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 15, 2022, 06:00:06 pm
Thanks for your help. I appreciated it!

Quote
cat haproxy.conf
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: O_SNI_fronted (Listening MYIP:80 MYIP:443)
frontend O_SNI_fronted
    bind MY IP:80 name MYIP:80
    bind MY IP:443 name MY IP:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening 192.168.1.1:80)
frontend 1_HTTP_frontend
    bind 192.168.1.1:80 name 192.168.1.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NO_SSL_Rule
    acl acl_620808a860e296.91534155 req.ssl_ver gt 0

    # ACTION: HTTP_TO_HTTPS_RULE
    http-request redirect scheme https code 301 if !acl_620808a860e296.91534155

# Frontend: 1_HTTPS_frontend (Listening 192.168.1.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.1.1:443 name 192.168.1.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6208140971a7a3.08696099.certlist
    mode http
    option http-keep-alive
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_MAP_RULE
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/620809e036a6d1.87483247.txt)]

# Backend: SSL_backend (SSL  backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 192.168.1.1 send-proxy-v2 check-send-proxy

# Backend: blog_server_backend (Server backend blog)
backend blog_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: cloud_nas_backend (cloud  backend NAS)
backend cloud_nas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nas_Server_80 192.168.1.118:80 send-proxy-v2 check-send-proxy

# Backend: dav_nas_backend (dav  backend NAS)
backend dav_nas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nas_Server_80 192.168.1.118:80 send-proxy-v2 check-send-proxy

# Backend: frank_server_backend (Server backend frank)
backend frank_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: flood_server_backend (Server backend flood)
backend flood_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: sync_server_backend (Server backend sync)
backend sync_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: monitoring_server_backend (Server backend monitoring)
backend monitoring_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: kvm_server_backend (Server backend kvm)
backend kvm_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: nas_server_backend (Server backend nas)
backend nas_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: tracker_server_backend (Server backend tracker)
backend tracker_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

and my mapfile

Quote
#public subdomains mapping
flood flood_server_backend
frank frank_server_backend
grafana grafana_server_backend
nas  nas_server_backend
kvm kvm_server_backend
monitoring monitoring_server_backend
sync sync_server_backend
tracker tracker_server_backend
cloud cloud_nas_backend
dav dav_nas_backend

You see anymore errors?

EDIT1: Do you recommend to change to nginx who can handle as web and reverse proxy
EDIT2: After studying your latest post I got it.
EDIT3: Why can’t Haproxy multiple services on same port?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 06:27:48 pm
You still don't understand.  ???
Your first mapfile was correct.

Just do this and tell me if it works.

1. create a mapfile with the below content.

Code: [Select]
#public subdomains mapping
flood WEBSERVER_backend
frank WEBSERVER_backend
www WEBSERVER_backend
torrent WEBSERVER_backend
grafana WEBSERVER_backend
nas WEBSERVER_backend
kvm WEBSERVER_backend
monitoring WEBSERVER_backend
speedtest WEBSERVER_backend
sync WEBSERVER_backend
tracker WEBSERVER_backend
cloud NAS_backend
dav NAS_backend

2. create the following backends and servers
Code: [Select]
WEBSERVER_backend --> contains --> WEBSERVER_server

WEBSERVER_server=192.168.1.100:80


NAS_backend --> contains --> NAS_server

NAS_server=192.168.1.118:80

3. Try to access your webserver and nas.

Code: [Select]
# WEBSERVER_backend
flood.yourdomain.com
frank.yourdomain.com
www.yourdomain.com
torrent.yourdomain.com
grafana.yourdomain.com
nas.yourdomain.com
kvm.yourdomain.com
monitoring.yourdomain.com
speedtest.yourdomain.com
sync.yourdomain.com
tracker.yourdomain.com


# NAS_backend
cloud.yourdomain.com
dav.yourdomain.com



If it still doesn't work then I can only offer you paid support. I hope you understand. It is my free time and I can only help for free up to a certain point.
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opns_neuling on February 19, 2022, 01:17:00 pm
I got this error
Quote
[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

https://forum.opnsense.org/index.php?topic=27065.msg131206#msg131206

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 20, 2022, 05:25:18 pm
I have just tried TCP mode with map file, there is a few more steps to achieve the goal instead of placing the map rule directly to 0_SNI
(I checked the package and found SNI inside, however, haproxy doesn't recognize it in TCP mode, that's why we need to force it to recognize SNI)

1. Create a "Condition" to request client hello
Name: SSL_Hello
Condition type: Custom condition (option pass-through)
Option pass-through: req_ssl_hello_type 1
(https://i.postimg.cc/rD89fwvk/cond-hello.jpg) (https://postimg.cc/rD89fwvk)

2. Create a "Rule" to wait accept SSL hello
Name: tcp_request_content_accept_ssl
Select conditions: SSL_Hello
Execute function: tcp-request-content-accept
(https://i.postimg.cc/mcnNZNhL/rule-ssl.jpg) (https://postimg.cc/mcnNZNhL)

3. Create a "Rule" to wait for inspect
Name: tcp_request_inspect_delay
Optional condition: none
Execute function: tcp-request-inspect-delay
TCP inspection delay: 5s
(https://i.postimg.cc/gX6yxX6v/rule-delay.jpg) (https://postimg.cc/gX6yxX6v)

4. Place the Rule to 0_SNI_frontend in following order
tcp_request_inspect_delay
tcp_request_content_accept_ssl
map
(https://i.postimg.cc/sBx4R0jH/rule-order.jpg) (https://postimg.cc/sBx4R0jH)
(hmdir_ru is my map rule)


Update according to findings in #183 (https://forum.opnsense.org/index.php?topic=23339.msg131582#msg131582)

5. Change the no_SSL condition to Traffic is SSL (locally deciphered)
(https://i.postimg.cc/Cng6Mdtn/nossl.jpg) (https://postimg.cc/Cng6Mdtn)

*Remark
It is advised to use another map file for 1_HTTPS_frontend if necessary
If you really don't want to create another map file, use "SNI TLS extension matches (locally deciphered)" instead
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 20, 2022, 06:04:58 pm
I got this error
Quote
[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

https://forum.opnsense.org/index.php?topic=27065.msg131206#msg131206
Have you Disable web GUI redirect rule in Part 4-1?
If you haven't disable it, your web GUI will keep listening to port 80.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 20, 2022, 07:31:40 pm
I have http webui with port 4444 impossible that listen on 80
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 20, 2022, 09:03:34 pm
I have http webui with port 4444 impossible that listen on 80

(https://i.postimg.cc/YGBYddZD/Screenshot-20220221-035236-01.jpg) (https://postimg.cc/YGBYddZD)
That's what I get when I try to add another frontend with 192.168.6.1:80, which already exists in my running system.
You must have another process running which listening to port 80 when you get this error

That's why I asked rather you disabled the GUI redirect.
As this redirect process is listening to port 80 of your router IP, and help you redirect any port 80 to the HTTPS port you typed.
If you already disabled, you need to figure out what other plugin you have installed and using port 80.

Guess: GUI redirect, nginx, firewall port forward
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 21, 2022, 08:41:30 am
Just take a look at the config file, I feel strange for some part of it
1. You don't need to type WAN IP in 0_SNI_frontend
instead, it should be 0.0.0.0:80 and 0.0.0.0:443
0.0.0.0 means any IP that points to your router.

2. What is your router IP?
If your router is 192.168.1.1, then 1_HTTP_frontend and 1_HTTPS_frontend will obviously conflict with 0_SNI_fronted
Since 0_SNI_fronted is already listening to 80 and 443 port of your router, you won't able to listen it with 192.168.1.1
Please follow Part 4-2 to create Virtual IP, and set 1_HTTP_frontend and 1_HTTPS_frontend to the virtual IP

If you don't want to create any Virtual IP, please remove 0_SNI_frontend
set 1_HTTP_frontend with 0.0.0.0:80 and 1_HTTPS_frontend 0.0.0.0:443 instead
Since all of your servers are running in http mode, it should work for having no SNI frontend
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 21, 2022, 01:13:24 pm
I have just tried TCP mode with map file, there is a few more steps to achieve the goal instead of placing the map rule directly to 0_SNI
(I checked the package and found SNI inside, however, haproxy doesn't recognize it in TCP mode, that's why we need to force it to recognize SNI)

As I said in the tutorial I am not running any services that would require me to load balance / reverse proxy plain TCP traffic. This is why I never tested it and only could provide theoretical assumptions on how it could work.
So thank you very much for providing a good guide on this.

If you don't mind I will add this to the tutorial (credits to your post) when I find the time!

Just a quick question: Which service are you running in tcp mode? Does the service send the SNI header?
Or does you solution work with any TCP based service? Becuase as far as I am aware the service needs to add the SNI header otherwise the access URL is not beeing sent to HAProxy.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 21, 2022, 01:21:52 pm
Just take a look at the config file, I feel strange for some part of it
1. You don't need to type WAN IP in 0_SNI_frontend
instead, it should be 0.0.0.0:80 and 0.0.0.0:443
0.0.0.0 means any IP that points to your router.

2. What is your router IP?
If your router is 192.168.1.1, then 1_HTTP_frontend and 1_HTTPS_frontend will obviously conflict with 0_SNI_fronted
Since 0_SNI_fronted is already listening to 80 and 443 port of your router, you won't able to listen it with 192.168.1.1
Please follow Part 4-2 to create Virtual IP, and set 1_HTTP_frontend and 1_HTTPS_frontend to the virtual IP

If you don't want to create any Virtual IP, please remove 0_SNI_frontend
set 1_HTTP_frontend with 0.0.0.0:80 and 1_HTTPS_frontend 0.0.0.0:443 instead
Since all of your servers are running in http mode, it should work for having no SNI frontend

I am already on him with the fix.
Also explained to him that the HTTP(s)_frontend IPs could cause issues.
His Apache is also misconfigured badly.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 21, 2022, 03:27:41 pm
Sure, just add it to your tutorial if you like.
I have 2 TCP servers running. OpenVPN and v2ray
(both of them have SNI header with it)

I'm sure not all of the TCP services can use haproxy, for example minecraft server without additional tools.
(One of the ways is to add one more rule to redirect other SSL connections to SSL_backend, and set minecraft server as default backend of 0_SNI, as no conditions or rules in haproxy can catch connections that doesn't have SNI header).
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 21, 2022, 03:29:31 pm
Sure, just add it to your tutorial if you like.
I have 2 TCP servers running. OpenVPN and v2ray
(both of them have SNI header with it)

I'm sure not all of the TCP services can use haproxy, for example minecraft server without additional tools.
(One of the ways is to add one more rule to redirect other SSL connections to SSL_backend, and set minecraft server as default backend of 0_SNI, as no conditions or rules in haproxy can catch connections that doesn't have SNI header).

Thank you for confirming!
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 22, 2022, 08:30:30 am
Just checked the L6 warning a bit.
This is the code from haproxy github (https://github.com/haproxy/haproxy/blob/master/src/cfgparse.c)
Code: [Select]
int warnif_tcp_http_cond(const struct proxy *px, const struct acl_cond *cond)
{
if (!cond || px->mode != PR_MODE_HTTP)
return 0;

if (cond->use & (SMP_USE_L6REQ|SMP_USE_L6RES)) {
ha_warning("Proxy '%s': L6 sample fetches ignored on HTTP proxies (declared at %s:%d).\n",
   px->id, cond->file, cond->line);
return ERR_WARN;
}
return 0;
}

The condition: req.ssl_ver (Traffic is SSL....)
This condition might get error code, as http doesn't have tls packages.
Although negating this condition will still redirect packages to 443, haproxy will still run.
But we can simply use "Traffic is HTTP" here.
(Then, I don't know why I get such error when I create 2 port 80 frontend) ???
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 22, 2022, 09:29:18 am
Please repost your current config.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 22, 2022, 11:19:30 am
You mean my working config?

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if xray)
frontend tcp_front
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_6212326a7c07e4.28981163 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_6212326a7c07e4.28981163
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6214a3ae639096.17472719.txt)]

# Frontend: http_front (Listen to VIP:80 and redirect to 443)
frontend http_front
    bind 192.168.6.1:80 name 192.168.6.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: http
    acl acl_62123bbee27260.60165685 ssl_fc

    # ACTION: http_to_https
    http-request redirect scheme https code 301 if !acl_62123bbee27260.60165685

# Frontend: ssl_front (Listen to VIP 443, SSL offload cert)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.6.1:443 name 192.168.6.1:443 accept-proxy ssl no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621244f0de5919.36753000.certlist
    mode http
    option http-keep-alive
    default_backend hkbn_back
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: local
    acl acl_62123a1cebe813.09309501 src 192.168.3.0/24 192.168.5.0/24

    # ACTION: local_ru
    use_backend opn_back if acl_62123a1cebe813.09309501

# Backend: hmdir_back (Backend of v2ray)
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.3.3:443

# Backend: hkbn_back (Backend of HKBN)
backend hkbn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hkbn 192.168.4.2:443 ssl verify none

# Backend: SSL_back (Backend to redirect SSL servers)
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server ssl 192.168.6.1 send-proxy-v2 check-send-proxy

# Backend: opn_back (Backend of opnsense with SSL)
backend opn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opn 192.168.3.1:8443 ssl verify none

# Backend: open_back (Backend of OpenVPN)
backend open_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server open 192.168.3.1:10443

BTW, I know why I will get such warning after reading the code.
Since my TCP layer already received TLS packages and inspected it already.
For HTTP and HTTPS frontend, any more "request" for SSL/TLS will probably get such warning.

Thus, for anyone using map file in 0_SNI_frontend.
They not only need to create "tcp_request_inspect_delay", "tcp_request_content_accept_ssl" and "SSL_Hello" condition
but also need to change the non-HTTP traffic condition to either
Negate of Traffic is SSL (locally deciphered) (ssl_fc), which will not request another TLS package
or
Traffic is HTTP (req.proto_http)
but not
Traffic is SSL (TCP request content inspection) (req.ssl_ver)

If Someone already used map file in 0_SNI_frontend
And they have multiple HTTP server need to offload in HTTPS_frontend, they should either use another map file
or
SNI TLS extension matches (locally deciphered) (ssl_fc_sni)
but not
SNI TLS extension ... (TCP request content inspection) (req.ssl_sni)

*For Morta's case, this might also due to multiple inspection of TLS or another port 80 listener
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 22, 2022, 06:28:24 pm
I have looked wir netsocket but no other app is on port 80 and is working so far

I have also problem that the cert redirect automatically from www.xx.ch to xx.ch

When I insert xx.ch in the map file all in existing sub domains goes to xx.ch as examples buff.xx.ch goes to xx.ch

How can I map at least www.xx.ch to a service without redirect or better how can I map xx.ch and www.xx.ch without redirecting all subdomain to xx.ch vhost
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 22, 2022, 07:14:45 pm
Just set your Webserver as default backend on the https frontend.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 22, 2022, 07:32:33 pm
I have two Webserver. So I can took only one.
Which one? WEBSERVER_backend or NAS_backend?


Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 22, 2022, 08:35:47 pm
Obviously your Webserver and not your NAS.......  ::)

Also you just exposed your Domain again  :-\

Please don't get me wrong but I really hope you know what you are doing by self-hosting and exposing any services from your private internet connection!
There are many risks involved and you don't seem to me like someone that is aware of everything he is doing and why it needs to be done.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 23, 2022, 08:07:18 pm
Would be possible to adjust the tutorial with ipv6 support or a hint how to add ipv6 support to a existing configuration?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 24, 2022, 01:22:48 am
Would be possible to adjust the tutorial with ipv6 support or a hint how to add ipv6 support to a existing configuration?

For the frontend that using 0.0.0.0:80, "add" [::]:80
For the frontend that using 0.0.0.0:443, "add" [::]:443

In firewall rule, add one more rule to "allow" "ipv6" "TCP" to "this firewall" in "port 80 and 443"

Note: your DDNS need to update ipv6 too
ddclient 3.9.1 should be able to update ipv6 . However, I don't know rather OPNsense had implemented to update ipv6 automatically.

P.S. dyndns plugin is too old, it doesn't support ipv6
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: figadore on February 25, 2022, 07:27:07 am
When creating the virtual IP, there's a new option not included in the picture. Is there a correct setting for "Allow service binding"? Does it make a difference?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 27, 2022, 11:30:42 am
I mean it is self-explanatory.

To answer your question: You will want this enabled.
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opns_neuling on March 13, 2022, 09:42:06 pm
I got this error
Quote
[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

on Step:
Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
Here we will only create a "NoSSL_condition", which is necessary in order to identify non-HTTP traffic.


Please replace "req.ssl_ver gt 0" with "ssl_fc" also, select "custom" and on custom field enter "ssl_fc"

https://redmine.pfsense.org/issues/9261


Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 14, 2022, 01:23:40 pm
Can you make a screenshot? I don’t understand you instructions

I have no req.ssl_ver 0 in my configuration!

So?

(https://abload.de/img/auswahl_024oaj7m.png)
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 14, 2022, 08:22:32 pm
Mentioned in #183
Traffic is SSL (locally deciphered) is ssl_fc

And the condition need to be replaced is Part 5.7.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 15, 2022, 06:55:39 pm
Thanks for detailed instructions, I've follow step by step to make a web hosting running nginx with https support.

My HAProxy Config:

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_622eebaf197419.36314953 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_622eebaf197419.36314953

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/622eef9a9d7268.16491040.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Backend: backend_pool_abc ()
backend backend_pool_meet_huuich_vn
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server PLEX_server 192.168.82.11:32400 ssl verify none

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

My Map File content:

Code: [Select]
# public access subdomains
plex PLEX_backend

My nginx website config

Code: [Select]
server {
        listen 32400;
        listen [::]:32400;

        root /var/www/mywebsite.com/html;
        index index.html index.htm index.nginx-debian.html;

        server_name mywebsite.com;

        location / {
                try_files $uri $uri/ =404;
        }
}

I can go my website on internal http://192.168.82.11:32400 is ok but when I access http://mywebsite.com browser go to https://mywebsite.com and show error

Code: [Select]
503 Service Unavailable
No server is available to handle this request.

How can I fix this and show my website https://mywebsite.com ok? Thanks!
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 15, 2022, 07:15:48 pm
I can go my website on internal http://192.168.82.11:32400 is ok but when I access http://mywebsite.com browser go to https://mywebsite.com and show error

Code: [Select]
503 Service Unavailable
No server is available to handle this request.

How can I fix this and show my website https://mywebsite.com ok? Thanks!


First.
The entry "plex PLEX_backend" in the mapfile means that you will have to access it using the "plex" subdomain. --> In your case "plex.mywebsite.com"!

Alternatively just set the PLEX_backend as default backend on your HTTPS_frontend.

Second.
http will always get redirected to https. This is intended and you will most probably want this! This is configured using the HTTPtoHTTPS_rule and NoSSL_condition.

Third.
Apart from the above your config looks good. (just took a very short look at it)

Fourth.
You might have to disable the SSL checkbox in the PLEX_server settings.
But only if you are REALLY accessing it locally using http://IP:32400 and the service is NOT redirecting you to HTTPS. But I highly doubt this since Plex is running on a self-signed SSL cert by default...
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 15, 2022, 11:00:02 pm
First.
The entry "plex PLEX_backend" in the mapfile means that you will have to access it using the "plex" subdomain. --> In your case "plex.mywebsite.com"!

Alternatively just set the PLEX_backend as default backend on your HTTPS_frontend.

Second.
http will always get redirected to https. This is intended and you will most probably want this! This is configured using the HTTPtoHTTPS_rule and NoSSL_condition.

Third.
Apart from the above your config looks good. (just took a very short look at it)

Fourth.
You might have to disable the SSL checkbox in the PLEX_server settings.
But only if you are REALLY accessing it locally using http://IP:32400 and the service is NOT redirecting you to HTTPS. But I highly doubt this since Plex is running on a self-signed SSL cert by default...

You are right, mysubdomain.mywebsite.com, and follow your guide I've edit my Map File to "mysubdomain PLEX_backend" and "disable the SSL checkbox in the PLEX_server settings" and voila my subdomain website run https:// ok with A+ score, Thank you so much for quick and detailed reply!

P/s: I've follow your guide with cloudflare instead. (almost every settings are the same like your guide)
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 16, 2022, 02:47:59 am
I want to set up HAProxy just for routing traffic based on URLs (https://xyz.domain.com goes to server 1 and https://abc.domain.com goes to server 2, etc...).
All SSL stuff for the destination web servers is being handled by a separate Linux certificate server and the web servers themselfes, independent from OPNsense/HAProxy. HAProxy is really only needed for routing traffic based on URLs, nothing more, nothing less.I want to set up HAProxy just for routing traffic based on URLs (https://xyz.domain.com goes to server 1 and https://abc.domain.com goes to server 2, etc...).
All SSL stuff for the destination web servers is being handled by a separate Linux certificate server and the web servers themselfes, independent from OPNsense/HAProxy. HAProxy is really only needed for routing traffic based on URLs, nothing more, nothing less.
Did you find solution for your purpose? I wanna do same like you.
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 16, 2022, 03:52:23 am
You have to set your backends and frontends to HTTP Mode.
Also disable SSL offloading on the frontends.

But I can't guarantee for sure that it will work.
TCP Mode will never (with a few exceptions) work because there is no header in the packets that would tell HAProxy which service to send the traffic to.

HTTP Mode could work, but you might need to create some "http header contains..." conditions.
I can do that using SSL passthrough follow step by step this guide (https://forum.opnsense.org/index.php?topic=18538.msg84958#msg84958) from @alh

Every works ok except my Nginx log can not receive real ip from client, it's only show ip  local of my OPNSense server in file /var/log/nginx/access.log

Do you think can mix SSL passthrough with your guide (edit 0_SNI_frontend with rules from SSL passthrough and Default Backend Pool) or any solution to receive real ip from client with SSL passthrough? Thanks!

Code: [Select]
This should work for any TCP-based SSL/TLS encrypted service in passthrough (HAProxy: TCP) mode... It does NOT work for STARTTLS!

In this example I use TCP port 443.

HAProxy plugin: Create "Real Server" (enter name, IP/FQDN and port number if different from 443, the rest can be left at default)
HAProxy plugin: Create "Backend Pool" (enter name, set mode to TCP and select the real server from step 1)
HAProxy plugin: Create "Condition" (enter name ["traffic_ssl"], condition type is "custom condition (option pass-through)" with value "req_ssl_hello_type 1")
HAProxy plugin: Create "condition" (enter name ["myservice_sni"], condition type is "SNI TLS extension matches (TCP request content inspection)" with value "myservice.example.com" or whatever your FQDN is)
HAProxy plugin: Create "Rule" (enter name ["request_inspect_delay"], select no condition, function is "tcp-request inspect delay" with value "5s" or whatever suits you)
HAProxy plugin: Create "Rule" (enter name ["request_content_accept_ssl"], select condition of 3 ["traffic_ssl"], function is "tcp-request content accept")
HAProxy plugin: Create "Rule" (enter name ["myservice_sni"], select condition of 4 ["myservice_sni"], function is "Use specific backend pool" with your pool from 2)
HAProxy plugin: Create "Public service" (enter name ["https_passthrough"], choose a listen address [":443" for all], type is "TCP" and select the 3 rules created earlier)
HAProxy plugin: Enable plugin or test/apply
Firewall: allow incoming traffic to WAN (address) or whatever for TCP port 443.

That works at least for me. If you have double NAT you would need to disable port randomization for the proxied port...

Does that help you?
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 16, 2022, 11:51:30 am
If all SSL certificate are handled by webserver themselves.
Follow #176 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354), step 1-4 will allow 0_SNI_FRONTEND to recognize TLS package and redirect under TCP mode

Note: All backend redirect from 0_SNI_FRONTEND should be in TCP mode
HAProxy only work with server that using TLS package. For those server that doesn't contain TLS package (for example, game servers), HAProxy won't work. Although you can set default backend server to game server, but there's only one default backend server.

BTW, for backend server getting source IP, enabling X-Forwarded-For header for all frontend should work
However, this one only apply to HTTP.
reference (https://www.haproxy.com/fr/blog/preserve-source-ip-address-despite-reverse-proxies/)
 reference 2 (https://www.haproxy.com/blog/layer-7-load-balancing-transparent-proxy-mode/)
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 16, 2022, 09:20:26 pm
If all SSL certificate are handled by webserver themselves.
Follow #176 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354), step 1-4 will allow 0_SNI_FRONTEND to recognize TLS package and redirect under TCP mode

Note: All backend redirect from 0_SNI_FRONTEND should be in TCP mode
HAProxy only work with server that using TLS package. For those server that doesn't contain TLS package (for example, game servers), HAProxy won't work. Although you can set default backend server to game server, but there's only one default backend server.

BTW, for backend server getting source IP, enabling X-Forwarded-For header for all frontend should work
However, this one only apply to HTTP.
reference (https://www.haproxy.com/fr/blog/preserve-source-ip-address-despite-reverse-proxies/)
 reference 2 (https://www.haproxy.com/blog/layer-7-load-balancing-transparent-proxy-mode/)
I've follow your guide but though my backend server ctl_backend has code
Code: [Select]
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Real-IP %[src]

But it can not get source ip, here is my full config, could you show me how to get source ip? Or I have to have 2 OPNSense, one for SSL passthrough and one use Let's Encrypt Wildcard Certificates like this topic guide? Could 1  OPNSense sever run both nginx reverse proxy and HAProxy?

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    # tuning options
    timeout client 30s

    # logging options
    # ACL: traffic_ssl
    acl acl_601a842f14cee3.17646593 req_ssl_hello_type 1

    # ACTION: request_content_accept_ssl
    tcp-request content accept if acl_601a842f14cee3.17646593
    # ACTION: request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Frontend (DISABLED): 1_HTTP_frontend (Listening on 192.168.64.1:80)

# Frontend (DISABLED): 1_HTTPS_frontend (Listening on 192.168.64.1:443)

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Real-IP %[src]
    http-reuse safe
    server PLEX_server 192.168.82.11:32400

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend (DISABLED): SSL_backend ()

# Backend: ctl_backend ()
backend ctl_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Real-IP %[src]
    http-reuse safe
    server ctl_server 192.168.82.11:32401
Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 17, 2022, 04:15:01 am
Notice that you are using TCP mode frontend only.
X-forwarded-for will not simply work under any case that using TCP mode, as X-forwarded-for will only work under HTTP mode and need to enable in all layers (TCP mode will not forward any header)

For TCP mode, you need to set all backend that wants to get real client IP and redirected from this frontend to use Proxy Protocol (either version 1 or 2, while verion 1 sending ASCII data and version 2 sending binary)
You will find "send-proxy" under that backend, which means it will send another package via proxy protocol to your real server.

Your real server need to modify config to read proxy protocol package too.
 Nginx guide to set it accept, reading and logging proxy protocol address (https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/)
(Not 2 lines, tones should be modified whenever there is a layer of TCP mode)

P.S. Theoretically, setting like OP having (SNI frontend->send-proxy backend->HTTPS frontend with X-forwarded-for) should also forward proxy protocol package to real server too.

Example: my v2ray server is living in TCP mode (although I don't care about the IP log)
I will get router IP in log if I don't user proxy protocol.
I cannot reach v2ray server if I use proxy protocol in haproxy but not changing any config of v2ray server (probably rejected by the server itself)
I will get correct client IP if I use proxy protocol in haproxy and changing the config of v2ray to accept proxy protocol

My HAProxy config is as follow:
1. open_back is OPENVPN live in TCP mode, and will not get any real client IP (Always 192.168.3.1)
2. hmdir_back is v2ray (another proxy server) live in TCP mode, and will get real client IP (I need to modify v2ray config to make it read and log proxyprotocol
3. opn_back is opnsense page live in HTTP mode, will not get real client IP as I don't care.
4. unifi_back is unifi controller page live in HTTP mode, I don't care rather it get real IP too
5. hkbn_back is ISP router page, which has no access right to my LAN, just a dummy to fake ISP that I'm using their device, all unwanted access will reditect to this page (Their device always open 443 to public access)
5. opn_back and unifi_back can only be accessed with source IP as 192.168.3.0/24 and 192.168.5.0/24 and I have tested no one can access to these pages except my LAN clients. Thus, real client IP should be passed to SSL_frontend successfully too.
Just need to add backend with http-request set-header X-Real-IP %[src] (Although there will be warning, you can do it in frontend, as HAProxy config manual (http://cbonte.github.io/haproxy-dconv/2.0/configuration.html)mentioned both ways will work) and config webserver read and log X-Real-IP
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if v2ray)
frontend tcp_front
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_6212326a7c07e4.28981163 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_6212326a7c07e4.28981163
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6214a3ae639096.17472719.txt)]

# Frontend: http_front (Listen to VIP:80 and redirect to 443)
frontend http_front
    bind 192.168.6.1:80 name 192.168.6.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: http
    acl acl_62123bbee27260.60165685 ssl_fc

    # ACTION: http_to_https
    http-request redirect scheme https code 301 if !acl_62123bbee27260.60165685

# Frontend: ssl_front (Listen to VIP 443, SSL offload cert)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.6.1:443 name 192.168.6.1:443 accept-proxy ssl no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621244f0de5919.36753000.certlist
    mode http
    option http-keep-alive
    default_backend hkbn_back
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: local
    acl acl_62123a1cebe813.09309501 src 192.168.3.0/24 192.168.5.0/24

    # ACTION: local_map_ru
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623230d7bffd04.94702836.txt)] if acl_62123a1cebe813.09309501

# Backend: hmdir_back (Backend of v2ray)
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.3.3:443 send-proxy-v2 check-send-proxy

# Backend: hkbn_back (Backend of HKBN)
backend hkbn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hkbn 192.168.4.2:443 ssl verify none

# Backend: SSL_back (Backend to redirect SSL servers)
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server ssl 192.168.6.1 send-proxy-v2 check-send-proxy

# Backend: opn_back (Backend of opnsense with SSL)
backend opn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opn 192.168.3.1:8443 ssl verify none

# Backend: unifi_back (Backend of unifi with SSL)
backend unifi_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server unifi 192.168.3.4:8443 ssl verify none

# Backend: open_back (Backend of OpenVPN)
backend open_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server open 192.168.3.1:10443
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 17, 2022, 09:38:06 am
@Bunch Thank you for detailed guide and attached config. I've tried to simulate like your server but now my server not working

This is my config

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if v2ray))
frontend tcp_front
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: traffic_ssl
    acl acl_601a842f14cee3.17646593 req_ssl_hello_type 1

    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_601a842f14cee3.17646593
    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Frontend: http_front (Listening on 192.168.64.1:80)
frontend http_front
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_622eebaf197419.36314953 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_622eebaf197419.36314953

# Frontend: ssl_front (Listening on 192.168.64.1:443)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/622eef9a9d7268.16491040.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Backend: hmdir_back ()
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.82.11:32401 send-proxy-v2 check-send-proxy

# Backend (DISABLED): PLEX_backend ()

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend: SSL_back ()
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend (DISABLED): ctl_backend ()

My hmdir_ru is link to PUBLIC_SUBDOMAINS_map map file with content:
Code: [Select]
# public access subdomains
c hmdir_back

Could you share content of map file in local_map_ru in this config

Code: [Select]
# ACTION: local_map_ru
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623230d7bffd04.94702836.txt)] if acl_62123a1cebe813.09309501

And ACL http in this config

Code: [Select]
# logging options
    # ACL: http
    acl acl_62123bbee27260.60165685 ssl_fc

Thank you!
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 17, 2022, 10:07:30 am
My local_map_rule is:
Test type= if
Selection condition= local
Execute function= Map domains to backend pools using a map file
Map_file= local_map

Condition local:
Condition type= Source IP matches specified IP
Source IP=192.168.3.0/24 192.168.5.0/24

Local_map:
Code: [Select]
opn opn_back
unifi unifi_back

For ssl_fc, it is mentioned in #174 Step 5 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354)
And #183 (https://forum.opnsense.org/index.php?topic=23339.msg131582#msg131582)

For testing why it won't work in your case,
You can try not using Proxy Protocol first. If you can access your backend, that means your webserver config is not accepting proxy protocol (It won't work both way at the same time)
You need to use Proxy Protocol and modify webserver config ( for example, /etc/nginx/sites-enabled/default) at the same time

Similar test case mentioned in previous post
Quote
Example: my v2ray server is living in TCP mode (although I don't care about the IP log)
I will get router IP in log if I don't user proxy protocol.
I cannot reach v2ray server if I use proxy protocol in haproxy but not changing any config of v2ray server (probably rejected by the server itself)
I will get correct client IP if I use proxy protocol in haproxy and changing the config of v2ray to accept proxy protocol
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 17, 2022, 10:15:45 am

For testing why it won't work in your case,
You can try not using Proxy Protocol first. If you can access your backend, that means your webserver config is not accepting proxy protocol (It won't work both way at the same time)
You need to use Proxy Protocol and modify webserver config ( for example, /etc/nginx/sites-enabled/default) at the same time

Similar test case mentioned in previous post
Quote
Example: my v2ray server is living in TCP mode (although I don't care about the IP log)
I will get router IP in log if I don't user proxy protocol.
I cannot reach v2ray server if I use proxy protocol in haproxy but not changing any config of v2ray server (probably rejected by the server itself)
I will get correct client IP if I use proxy protocol in haproxy and changing the config of v2ray to accept proxy protocol

My /etc/nginx/sites-enabled is

Code: [Select]
server {
listen 32401;
listen [::]:32401;

server_name c.mywebsite.com;

root /var/www/html;

location / {
try_files $uri $uri/ /yourls-loader.php$is_args$args;
}

location ~ \.php$ {
include fastcgi.conf;

fastcgi_index index.php;

fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}

How could use Proxy Protocol and modify webserver config with my HAProxy config above?
Sorry for basic question. Thank you for detailed explaination.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 17, 2022, 11:04:16 am
That's not basic question IMO.  :D
The modification method is mentioned in Nginx guide (https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/)

Should be modified as below (haven't tested it)
Code: [Select]
server {
listen 32401 proxy_protocol;
listen [::]:32401 proxy_protocol;

server_name c.mywebsite.com;

root /var/www/html;

location / {
try_files $uri $uri/ /yourls-loader.php$is_args$args;
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr
}

location ~ \.php$ {
include fastcgi.conf;

fastcgi_index index.php;

fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}

For the log config, I forgot rather it is /etc/nginx/nginx.conf by default
You need to change the log format inside http{} to something like
Code: [Select]
log_format combined '$proxy_protocol_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent"';
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 18, 2022, 10:05:44 am
The modification method is mentioned in Nginx guide (https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/)

Should be modified as below (haven't tested it)

Your code "haven't tested but working for me with http website, but when I change to https with certbot

Code: [Select]
certbot --nginx -d c.mywebsite.com
My https website can not access though I've change my server port to 443, this is my nginx website config

Code: [Select]
server {
listen 32401 proxy_protocol;
listen [::]:32401 proxy_protocol;

listen [::]:443 ssl ipv6only=on proxy_protocol; # managed by Certbot
    listen 443 ssl proxy_protocol; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/c.mywebsite.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/c.mywebsite.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

server_name c.mywebsite.com;

root /var/www/html;

location / {
try_files $uri $uri/ /yourls-loader.php$is_args$args;
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr;
}

location ~ \.php$ {
include fastcgi.conf;

fastcgi_index index.php;

fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}
}

and my current HAProxy configure

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if v2ray))
frontend tcp_front
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: traffic_ssl
    acl acl_601a842f14cee3.17646593 req_ssl_hello_type 1

    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_601a842f14cee3.17646593
    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Frontend: http_front (Listening on 192.168.64.1:80)
frontend http_front
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_622eebaf197419.36314953 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_622eebaf197419.36314953

# Frontend: ssl_front (Listening on 192.168.64.1:443)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/622eef9a9d7268.16491040.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Backend: hmdir_back ()
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.82.11:443 ssl verify none send-proxy-v2 check-send-proxy

# Backend (DISABLED): PLEX_backend ()

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend: SSL_back ()
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend (DISABLED): ctl_backend ()

Could you take a look and guide me how to run https on my website?

For the log config, I forgot rather it is /etc/nginx/nginx.conf by default
You need to change the log format inside http{} to something like
Code: [Select]
log_format combined '$proxy_protocol_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent"';
Fire I use above code has a error, so I google and found a solution for working log, I post here for anyone need

Code: [Select]
log_format my_log '$proxy_protocol_addr - $remote_user [$time_local] '
                       '"$request" $status $body_bytes_sent '
                       '"$http_referer" "$http_user_agent"';

access_log /var/log/nginx/access.log my_log;
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 18, 2022, 12:59:38 pm
Better check /etc/letsencrypt/options-ssl-nginx.conf
Check rather the SSL protocol is too old (is it having something like TLSv1 TLSv1.1)
If the settings are too old, try to change those settings in that file according to this link (https://ssl-config.mozilla.org/#server=nginx&version=1.17.7&config=intermediate&openssl=1.1.1d&guideline=5.6)

Guess below lines need to be modified if case exists.
Code: [Select]
ssl_session_timeout 1d;
    ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
    ssl_session_tickets off;

    # intermediate configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    # HSTS (ngx_http_headers_module is required) (63072000 seconds)
    add_header Strict-Transport-Security "max-age=63072000" always;

    # OCSP stapling
    ssl_stapling on;
    ssl_stapling_verify on;

For real server in HAProxy, I have tried turning SSL on or off doesn't affect my servers under TCP mode, don't know rather it is the same in your case, please try it too.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on March 18, 2022, 01:06:28 pm
I'm probably out of place saying this, as is not my thread, but should't this discussion go to another thread and leave this one for it's original purpose?
It has branched off now to "how can I enable TLS on my website", from "how can I log the client ip not the proxy ip on the backend webserver" and "how do I use proxy_protocol".
What do you think?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 18, 2022, 01:13:51 pm
Yes, I think it is out of HAProxy scope too.
When send-proxy or X-forwarded-for part is done in HAProxy, nothing can be done in HAProxy afterwards.
All other settings are related to Nginx or some other server settings.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 18, 2022, 02:14:35 pm
@cookiemonster Thank for your comment about back to original purpose, I'll create a new topic relate my questions. Thanks.

@Bunch: I'm very thankful to you for your help. Best Regards!
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: The_Dave on March 18, 2022, 03:10:11 pm
I have a very serious problem with this haproxy config since I updated to 22.1.3. Suddenly haproxy didn't start anymore. On further investigation trying to start haproxy through the commandline showed that suddenly the ipadresses for the frontend cannot be bound anymore:

Code: [Select]
root@OPNsense:/home/David # /usr/local/etc/rc.d/haproxy start
Starting haproxy.
[ALERT]    (21351) : Starting frontend 1_HTTP_frontend: cannot bind socket (Can't assign requested address) [192.168.64.1:80]
[ALERT]    (21351) : Starting frontend 1_HTTPS_frontend: cannot bind socket (Can't assign requested address) [192.168.64.1:443]
[ALERT]    (21351) : [/usr/local/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.
/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy

Any help on how to fix this would be really appreciated
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 18, 2022, 06:12:04 pm
It seems that it is the same issue as This thread (https://forum.opnsense.org/index.php?topic=27547.msg133659#msg133659)
I have the same issue after update and reboot.
For temporary fix, edit the VIP, save without any changes, then apply.
You will able to start HAProxy again.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: The_Dave on March 18, 2022, 08:52:27 pm
For temporary fix, edit the VIP, save without any changes, then apply.

Thank you this actually works for now, I hope there will be a proper solution or fix for this soon though
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: balrog on March 20, 2022, 10:49:35 am
I had the problem after updating to OPNsense 22.1.3 that the HAProxy service did not start anymore. I was able to solve the problem by editing the Virtual IP and saving it again without adjusting it. After that the service could be started again.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 20, 2022, 02:13:15 pm
Code: [Select]
2022/03/20 15:00:53 [error] 1124599#1124599: *22208 upstream prematurely closed connection while reading response header from upstream, client: 192.168.1.1, server: sync.xxx.ch, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/run/uwsgi/mozilla-firefox-sync-server.sock:", host: "sync.xxx.ch"

I got a 502 Gateway error on this site with nginx.

Where is the option of keep-alive for the backends? Or anybody knows how to fix this error?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 22, 2022, 08:19:47 am
Check and change Services: HAProxy: Settings: Default Parameters
Check Tuning Options of your frontends rather you overwrite the setting too.
If it doesn't fix the issue, then it is Nginx setting problem
Check this link (https://stackoverflow.com/questions/36488688/nginx-upstream-prematurely-closed-connection-while-reading-response-header-from) too for Nginx setting
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 22, 2022, 09:37:57 am
Thanks for reply bunch!

The link is for the settings with a reverse proxy with nginx.

I will look at the tunables of Haproxy.

I found this link

https://serveanswer.com/questions/upstream-prematurely-closed-connection-while-reading-upstream-large-files (https://serveanswer.com/questions/upstream-prematurely-closed-connection-while-reading-upstream-large-files)

Also possible that is a connection error to the SQLite database.

I will give a try at home.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 22, 2022, 02:30:22 pm
I try to add full IPv6 support to my Haproxy configuration.
Now I have a problem with IPv6 localhost or loopback address

I did a real server ssl_server_ipv6 with IPv6 ::1

(https://abload.de/img/e083e5c0-101f-494d-984zjpo.png)
(https://abload.de/img/52010dc8-40b7-4474-abm6jvq.png)

I got following error when I add ssl_server_ipv6 to SSL_backend
(https://abload.de/img/ea69a72d-3153-4847-bb8fjuf.png)

How I can fix it or do a workaround?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 22, 2022, 03:41:30 pm
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 1000s
    timeout connect 1000s
    timeout server 1000s
    retries 3
    default-server init-addr libc,last
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend (Listening o)
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind :::80 name :::80
    bind :::443 name :::443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 1000s

    # logging options

# Frontend: HTTP_frontend (Listening 127.0.0.1:80)
frontend HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    bind [::1]:80 name [::1]:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 1000s

    # logging options
    # ACL: NoSSL_condition
    acl acl_621d0b77c74989.24704837 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_621d0b77c74989.24704837

# Frontend: HTTPS_frontend (Listinging on 127.0.0.1:443)
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621d11c7cad951.61400293.certlist
    bind [::1]:443 name [::1]:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621d11c7cad951.61400293.certlist
    mode http
    option http-keep-alive
    default_backend WEBSERVER_backend
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/621d0c7054ddb7.46420139.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: WEBSERVER_backend ()
backend WEBSERVER_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server WEBSERVER_server 192.168.1.100:80 send-proxy-v2 check-send-proxy
    server WEBSERVER_server_ipv6 2a02:XXX:XXX::2000:80 send-proxy-v2 check-send-proxy

# Backend: NAS_backend ()
backend NAS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server NAS_server 192.168.1.118:80
    server NAS_server_ipv6 2a02:XXX:XXX::1000:80

# Backend: WEBSERVER_SSL_backend ()
backend WEBSERVER_SSL_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server WEBSERVER_server_ssl 192.168.1.100:443
    server WEBSERVER_server_ssl_ipv6 2a02:XXX:XXX::2000:443

It's this a correct and possible configuration?
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 22, 2022, 04:01:34 pm
I don't think you need to create another ipv6 real server, as long as it is the same sever in ipv4
You only need to add :::443 and :::80 to frontend listener (in frontend, [::]:80 is the same as :::80, in case you confused with the syntax)
That will be ipv6 to 4 setup.

If you add 2 real server to the same backend, you are load balancing them.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 22, 2022, 05:35:53 pm
I don't think you need to create another ipv6 real server, as long as it is the same sever in ipv4
You only need to add :::443 and :::80 to frontend listener (in frontend, [::]:80 is the same as :::80, in case you confused with the syntax)
That will be ipv6 to 4 setup.

If you add 2 real server to the same backend, you are load balancing them.

Thanks for the Input.
Real IPv6 or Dual Stack support of HAproxy would be nice but so I have a fallback opportunity if IPv4 or IPv6 are out of service by me or the ISP! 
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: os_admin on March 27, 2022, 08:59:47 pm
I have a very serious problem with this haproxy config since I updated to 22.1.3. Suddenly haproxy didn't start anymore. On further investigation trying to start haproxy through the commandline showed that suddenly the ipadresses for the frontend cannot be bound anymore:


This behaviour seems to be gone since 22.1.4_1. No guarantee.
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: os_admin on March 27, 2022, 09:43:26 pm
Just tested it out myself. Basic Auth is so easy to set up that I am not really willing to cover it in this guide.
First create the user(s) in HAProxy. Then in the relevant backends activate basic auth and select the user(s).

Thanks for this tutorial. It saved my ass. I learned a lot about OPNsense and HAProxy. At last I enabled basic auth. on one of my backends. Anything was fine before, but after activating it I can't no longer login into the service web frontend itself. If I access the frontend browser asked for the basic auth, after that I see the login screen of the service, but after put in the service credentials the FE refresh and shows the login screen again.
If I disable basic auth, the service FE works as expected.

What do I miss? Maybe someone out there has a hint...
Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 31, 2022, 06:58:19 am
Strange,  have just tested one of my backend
Create user->Enable Basic Auth and select Allowed users in backend
It works as expected.

Edit: Just tested a bit deeper, for some pages like unifi controller, it will always redirect to wrong page
For some pages like opnsense web UI, in chrome(PC), it will keep prompting for auth, but in firefox(PC & mobile), everything works normally

BTW, nothing more can be done in haproxy too, as some site in some browser works normally. Thus, the problem is due to webserver and browser.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 01, 2022, 05:58:07 pm
I'm probably out of place saying this, as is not my thread, but should't this discussion go to another thread and leave this one for it's original purpose?
It has branched off now to "how can I enable TLS on my website", from "how can I log the client ip not the proxy ip on the backend webserver" and "how do I use proxy_protocol".
What do you think?

Absolutely true!  ;D

I don't want to sound like an asshole here, but this tutorial was intended to get the basics working for new users.
This is also why I stopped answering questions about issues like "my service_abc has the requirement_xyz how work????".

If there are any questions in that regard then people should consider posting them...
here: https://forum.opnsense.org/index.php?board=28.0
or here: https://discourse.haproxy.org/
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 01, 2022, 05:58:31 pm
It seems that it is the same issue as This thread (https://forum.opnsense.org/index.php?topic=27547.msg133659#msg133659)
I have the same issue after update and reboot.
For temporary fix, edit the VIP, save without any changes, then apply.
You will able to start HAProxy again.

Thank you for posting the workaround!
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 11, 2022, 09:23:22 pm
Just a quick notifcation for everyone following the thread.
It seems like Let's Encrypt changed something regarding wildcard certificates.

I updated the picture in Part 3 - Step 6 to reflect the changes necessary in order to obtain a certificate.

You will have to remove the alt name "*.yourdomain.tld" and change the common name to "*.yourdomain.tld".
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: ElliotAlderson on April 12, 2022, 03:54:12 am
Code: [Select]
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind website.com:443 name website.com:443
    bind website.com:80 name website.com:80
    mode tcp
    default_backend SSL_backend
    timeout client 30s

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 10.10.10.1:80 name 10.10.10.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 30s

    # ACL: NoSSL_cond
    acl acl_62548efaf067e6.21908045 req.ssl_ver gt 0
    # ACTION: HTTPupgrade_rule
    http-request redirect scheme https if !acl_62548efaf067e6.21908045

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 10.10.10.1:443 name 10.10.10.1:443 accept-proxy ssl ssl-min-ver TLSv1.3 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62549082216928.65241361.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 30s

    # ACTION: PUBLIC_SUBDOMAINS_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62548f2d97ef05.80304462.txt)]

# Backend: club_backend ()
backend club_backend
website.com    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server club_host 10.0.0.94:3000 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
website.com    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    timeout connect 30s
    timeout server 30s
    server SSL_host 10.10.10.1 send-proxy-v2 check-send-proxy

I switched over from pfSense to OPNSense months ago and I had to set my side projects to the side because I simply could not replicate my HAProxy setup from before. I'm thankful for this tutorial since it's seems like the closest to what I used to have.

I'm extremely lost here. I have gone through this tutorial many times to double check my steps, I have tried changing things on my own.
At the moment, the HTTP->HTTPS redirect doesn't seem to work at all (empty response, no redirected) and the https site gives a blank response as well.
I have checked it with tcpdump directly on the OPNSense shell and could see that packets do get exchanged between my host and the virtual IP.  (TCP handshake, TLSv1 Client Hello, End connection )
The site itself is definitely working correctly internally at the host specified in the config file.

In the config I posted, I used website.com:443 in the SNI frontend.
I have tried it with 0.0.0.0:443 and my public IP with no success.

10.0.0.94 is in my LAN.
10.10.10.1 is the virtual interface I created.

My wildcard certificate seems to be working correctly.

I would really appreciate some help  :'(


By the way, what would the process be for getting another domain and wildcard cert to work added to this setup?
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 12, 2022, 07:06:19 pm
You are missing "code 301" in HTTPupgrade_rule
(Part 5-9)

Explanation of the code can be found here (https://www.haproxy.com/blog/redirect-http-to-https-with-haproxy/)
Quote
This technique will only work when using mode http because it redirects at the HTTP layer using a 302 Found HTTP response status, which is known as a temporary redirect. Once you’re fully committed to using HTTPS and have tested it thoroughly on your website, you may wish to instruct the browser to cache the redirect, which will save one round trip between the browser and HAProxy, speeding up page load times. Set the code parameter to 301 to send a 301 Moved Permanently status back, which browsers can cache:

Although it should work with 302, just give it a try

BTW, I don't know why there is website.com in the code of both backend. There shouldn't be something like this.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: ElliotAlderson on April 12, 2022, 09:01:22 pm
Oh, yeah. I actually did have the code 301 there originally. It's still the same effect with that though. That's just something I forgot to change back when I was trying new things. (Saw an older forum post that didn't use the "code 301" part.)

Regarding about the website.com,
In the config I posted, I used website.com:443 in the SNI frontend.
I have tried it with 0.0.0.0:443 and my public IP as well with no success.

I just can't get it to work so I'm trying different things.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 13, 2022, 02:16:04 am
I mean
Quote
# Backend: SSL_backend ()
backend SSL_backend
website.com    mode tcp
And
Quote
# Backend: club_backend ()
backend club_backend
website.com    mode http

Usually the wouldn't have SNI or domain names hard code in backends
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 13, 2022, 10:36:48 am
This week i have moved away from pfSense, I had acme, cloudflare & HAProxy working prior to the switch. Installed opnsense while slowly getting my services back online I came across this well written tutorial which seems more in-depth than my old setup but run into issues while accessing the hosted web service, it is failing to load with a 522 error, the connection if timing out before a response I think?

I have not got any further in the guide than part 5, step 10. Accessing from outside of my network as this is not possible so far.

I have a static WAN IP.. in cloudflare a have [A record *.example.com > Static IP]

I have double checked all the settings in this tutorial and after some googling i came across a reddit post, suggesting they fixed the 522 error in opnsense because HAProxy wasn't listening on port 80 during the HTTPtoHTTPS redirect. Is there a way I can diagnose this issue and trace the route somehow.

Lastly

ACME do not show any error in the log files. 
Code: [Select]
2022-04-13T18:53:42 php AcmeClient: running automation (configd): Restart HAProxy
2022-04-13T18:53:42 php AcmeClient: running automations for certificate: *.example.com
2022-04-13T18:53:42 opnsense AcmeClient: updated ACME X.509 certificate: *.example.com
2022-04-13T18:53:42 opnsense AcmeClient: successfully issued/renewed certificate: *.example.com
2022-04-13T18:51:27 opnsense AcmeClient: using challenge type: CloudFlare_DNS-01
2022-04-13T18:51:27 opnsense AcmeClient: account is registered: example.com
2022-04-13T18:51:27 opnsense AcmeClient: using CA: letsencrypt
2022-04-13T18:51:27 opnsense AcmeClient: issue certificate: *.example.com

HAProxy has no errors in the log file either

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62565b172acae6.05588153 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62565b172acae6.05588153

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62565eb5d0ff12.02152772.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62565c00b116b3.27816426.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: MineOS_backend ()
backend MineOS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server MineOS_server 192.168.1.103:8443 ssl verify none
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: hopey on April 13, 2022, 03:34:46 pm
@TheHellSite: Thanks for your tutorial, it helped me to understand things a bit better!

I just had this problem: https://forum.opnsense.org/index.php?topic=27903.new#new. That's why I was reading your thread.

 What was strange to me, that you need to change the port of your opensense web interface! That's the point were I thought there might be a better solution 😬 I was reading and found this tutorial: https://schulnetzkonzept.de/opnsense. This guy just used a virtual IP, like you did. But instead of pointing to the lookup adress he just introduced a new adress and let haproxy listen on. And so did I - and it worked like charm! (detailes in the upper linked problem)

Maybe you wanna add it to your tutorial too?!

Have a nice day!
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: pottproll on April 13, 2022, 04:47:13 pm
Thanks for the tutorial! Its working great so far! I have one question: in my ACME Client log it says after renewal: php[2613]   AcmeClient: automation not supported: restart_haproxy
Is this just me or maybe because i didnt check HA-Proxy Integration?

Edit: never mind, worked perfectly the next renewal.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 14, 2022, 04:22:13 am
I usually trace the session in Services: HAProxy: Statistics, Counters
When saving HAProxy setting, it will reset the session stat.
So, we can check the session stop at which server if we try to access immediately after a reset.
For example, I expect a session goes through
1. Frontend: TCP_front
2. Frontend: SSL_front
3. Backend: opn_back (usually don't have problem if you type correctly)
4. Server: opn

If all of them have session counts, but you still failed to access the website.
That means, it should be problem between browser and server, but you can access to the page when you type the server IP and port directly. Then, there might be issue about your ciphers, certs, OCSP settings, etc. You might find handshake error in server log too.

If it stop at 2. That means, your haproxy is not recognizing SNI correctly. Check your map file, or you can try to create condition with "SNI TLS extension matches (locally deciphered)" and your full SNI (the.domain.com), then create a rule to "Use specified Backend Pool" when condition matches.
Removing all rules and set default backend to test server first can also be a choice. (At least you will know rather it fails only in SNI part or more parts suffers)

If it stop at 1. There might be issue with the VIP again. (The bug similar to this (https://forum.opnsense.org/index.php?topic=27547.msg133659#msg133659))

If no session count. The listener is not working. TCP_frontend have wrong Listen Addresses. If you try it from WAN, check your firewall rules too, or maybe DNS record issue.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 14, 2022, 05:03:24 am
Okay i can now access my webservices but in doing so a missed out the Virtual IP step.
After thinking about my issue some more I am listening HTTP & HTTPS traffic on 192.168.64.1 which I think is when the time out happens.

So I;
Service > HAProxy > Settings > Real Servers > SSL_server: changed FQDN or IP, from 192.168.64.1 to 192.168.1.1

Service > HAProxy > Settings > Virtual Services > 1_HTTPS_frontend: changed Listen Addresses, from 192.168.64.1:443 to 192.168.1.1:443

Service > HAProxy > Settings > Virtual Services > 1_HTTP_frontend: changed Listen Addresses, from 192.168.64.1:80 to 192.168.1.1:80

Now it is all working, What did I do wrong in setting up the Virtual IP I wonder.

0_SNI_frontend > Listen Addresses:0.0.0.0:80, 0.0.0.0:443
should this need to be the Virtual IP as opnsense runs on 192.168.1.1

^^fyi thankyou for the tips on tracing
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 14, 2022, 05:42:11 am
Quote
0_SNI_frontend > Listen Addresses:0.0.0.0:80, 0.0.0.0:443
should this need to be the Virtual IP as opnsense runs on 192.168.1.1

^^fyi thankyou for the tips on tracing

For this question, lets clear the package path that OP wants to do first.
Assume the following IP config:
WAN IP: 1.2.3.4
Firewall IP: 192.168.1.1
VIP: 192.168.64.1
Server IP:port and SNI: 192.168.1.2:80, the.website.com

A browser try to access https://the.website.com from internet.
It asked system to resolve from DNS server: the.website.com. DNS server replies it is 1.2.3.4
Browser try to access 1.2.3.4 with port 443, sending TLS package with SNI=the.website.com

Since haproxy SNI_frontend is listening to 0.0.0.0:443, that means it is listening to port 443 that all IP can represent the firewall. In this case, it is 1.2.3.4:443 and 192.168.1.1:443

SNI_frontend catches the TLS package in 1.2.3.4:443 and passes to SSL_backend(VIP) without changing port

Since SSL_frontend is listening to 192.168.64.1:443, it takes the TLS package and knows that it try to access 192.168.1.2:80. SSL_frontend communicate to the browser, exchanging the SSL cert and keys according to ciphers.

Session to webserver_backend start, SSL_frontend redirect remaining packages to webserver_backend.

I don't know why your VIP won't work, maybe missing opnsense update, misconfig of VIP or another new bug. (You can try to ping the VIP in LAN to check rather it reply. If the VIP is working normally, it should reply)
And SNI frontend wont use VIP unless you use hopey's method (He still need to add NAT rule to redirect packages to VIP as I mentioned in another thread)
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 14, 2022, 07:21:31 am
You guys are awesome, really appreciate you explaining the process. It is easy to follow along to a guide but to understand what is happening makes it that much easier down the line.

After pinging the VIP 192.168.64.1, it was timing out.

Checked the setting and all is correct to the tutorial.
Decided to changed the submask from 32 to 24.
Then I was able to ping the VIP and access my web services.
Reset the submask back to 32 and i am still able to ping the VIP and web services working with HTTP & HTTPS listening on the VIP. Very strange but it seems resolved.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 14, 2022, 07:43:34 am
The issue is really similar to this one (https://forum.opnsense.org/index.php?topic=27547.msg133659)
I guess the issue should be solved in 22.1.4.
I run the patch in 22.1.3, so I don't know rather it is really fixed. But I don't have such problem anymore (currently 22.1.5)

If your are in 22.1.4 or 22.1.5, having similar issue. But general log don't have something like
Code: [Select]
/firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '192.168.64.1' -alias' failed to executePlease create another thread under 22.1 Production Series (https://forum.opnsense.org/index.php?board=41.0)

If you have exactly the same log and you are in 22.1.4 or 22.1.5. Please try to reply to the thread (https://forum.opnsense.org/index.php?topic=27547.msg133659)
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 14, 2022, 08:05:12 am
I will make my way over to the other thread you linked then as it is a similar issue.

I am on 22.1.5 and after checking the general log I also have the below
Code: [Select]
2022-04-14T16:42:58 Error opnsense /firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '192.168.64.1' -alias' failed to execute


--EDIT--
After reading the linked thread, a patch was applied in 22.1.4 but you also need to untick "Allow service binding".
I can confirm this has fixed the issue. Thanks again for point me in the right direction.

---EDIT 2---
I am now interested in the last part of the tutorial, I did not have anything like this setup on my previous pfsense setup so it will be a great addition. It would make for remembering local ip/ ports easier when navigating to locally hosted services.
I have followed the steps to make these subdomains accessible only from my local network but getting a "503 service unavailable".

I think its not reading the local map file correctly. I followed the previious steps for tracing and can see the counters go up when I try to access local.website.com but nothing is being passed to the prism_backend.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62565b172acae6.05588153 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62565b172acae6.05588153

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62565eb5d0ff12.02152772.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6257dfacde7e16.43417850 src_is_local

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6257d684d34507.32920094.txt)] if acl_6257dfacde7e16.43417850
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62565c00b116b3.27816426.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: MineOS_backend ()
backend MineOS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server MineOS_server 192.168.1.103:8443 ssl verify none

# Backend: Prism_backend ()
backend Prism_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Prism_server 192.168.1.103:2342

PUBLIC_SUBDOMAINS_map
Code: [Select]
# public access subdomains
mineos MineOS_backend
LOCAL_SUBDOMAINS_map
Code: [Select]
# local access subdomains
prism Prism_backend

# public access subdomains
mineos MineOS_backend

just to confirm;
mineos.website.com > works locally and externally
prism.website.com > 503 error locally and externally
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 14, 2022, 03:41:11 pm
Quote
mineos.website.com > works locally and externally
prism.website.com > 503 error locally and externally
Sorry, I haven't read the error 503.

Try not to use "src_is_local"
I remember that there is issue with this function, especially under proxy protocol. (It's hyproxy issue since 1.7?)
Try to use "Source IP matches a specific IP" instead
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 15, 2022, 12:22:50 am
Try to use "Source IP matches a specific IP" instead

I actually set this first time around which gave the same output. So I then decided to try a broader range with "IP is local".

I tried again "Source IP matches a specific IP" of the specific subnet again with no luck. have even tried isolating it to the IP of the client I am currently using.

the counter still is not being passed on the final stage of the route. After checking the HAProxy log file, as I navigate to local.website.com i am receiving a external request from cloudflare. I suspect the issue lies within the browser redirecting the request out or HAproxy not grabbing it locally before it goes out??
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 17, 2022, 01:03:33 pm
Strange, there shouldn't be something related to cloudflare
Maybe try to create condition: "SNI TLS extension matches (locally deciphered)" with your full sni, prism.website.com
Create another rule that
Use specified Backend Pool: Prism_backend
When
Source IP matches a specific IP...
And
SNI TLS extension matches (locally deciphered)

Use this rule to replace the one with local map file first
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bothson on April 19, 2022, 05:28:18 pm
Hello and thank you for this tutorial.
It helped me alot.

My services are available from the outside.
But from inside they are not accessible.

In the logs i see:
Code: [Select]
Informational haproxy 10.10.10.206:63264 [19/Apr/2022:17:26:27.483] 1_HTTPS_frontend/10.12.0.1:443: Received something which does not look like a PROXY protocol header
I already checked the ciphers. They seem to be ok.
Does anybody know where i can search for the problem?

My config:
Code: [Select]
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 10.12.0.1:80 name 10.12.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_60d1a0c1b278f7.63252237 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_60d1a0c1b278f7.63252237

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 10.12.0.1:443 name 10.12.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6256591773a972.14047672.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/625655d89e4274.43878203.txt)]

# Backend: bitwarden_backend ()
backend bitwarden_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server bitwarden_host 10.10.10.11:8080

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 10.12.0.1 send-proxy-v2 check-send-proxy

Best
Mathias
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 19, 2022, 06:26:15 pm
My services are available from the outside.
But from inside they are not accessible.

Well, there you got the point of error.
You probably configured the wrong IP in your DNS overwrites.

What is your OPNsense LAN IP, what is the DNS Overwrite IP and what is the virtual IP of the "HAProxy SSL Server"?
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bothson on April 19, 2022, 08:50:05 pm
Quote
Well, there you got the point of error.
You probably configured the wrong IP in your DNS overwrites.

What is your OPNsense LAN IP, what is the DNS Overwrite IP and what is the virtual IP of the "HAProxy SSL Server"?

My LAN IP is 10.10.10.1/24
My DNS Override points to    10.12.0.1
Which is also my Virtual IP (Loopback).

I do not get the point.

Edit: Ok, changed the Override IP to my LAN IP (10.10.10.1). Now it works.
But to be honest, i do not understand why.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 19, 2022, 08:59:40 pm
Edit: Ok, changed the Override IP to my LAN IP (10.10.10.1). Now it works.
But to be honest, i do not understand why.

You error explains why!

Code: [Select]
Informational haproxy 10.10.10.206:63264 [19/Apr/2022:17:26:27.483] 1_HTTPS_frontend/10.12.0.1:443: Received something which does not look like a PROXY protocol header
The HTTPS_frontend expects that all data sent to it has the "proxy protocol header".
Since you pointed your internal requests directly to your HTTPS_frontend (HAProxy_VIP) instead of your SNI_frontend (any of the real local IPs of your OPNsense) the data didn't get the PROXY protocol header attached by the SSL_backend.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bothson on April 20, 2022, 11:31:06 am
Edit: Ok, changed the Override IP to my LAN IP (10.10.10.1). Now it works.
But to be honest, i do not understand why.

You error explains why!

Code: [Select]
Informational haproxy 10.10.10.206:63264 [19/Apr/2022:17:26:27.483] 1_HTTPS_frontend/10.12.0.1:443: Received something which does not look like a PROXY protocol header
The HTTPS_frontend expects that all data sent to it has the "proxy protocol header".
Since you pointed your internal requests directly to your HTTPS_frontend (HAProxy_VIP) instead of your SNI_frontend (any of the real local IPs of your OPNsense) the data didn't get the PROXY protocol header attached by the SSL_backend.

Ah ok, i get the point.
Thank you very much for the explanation.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 24, 2022, 05:45:11 am
I am not sure if this is the correct way to achieve multiple domains pointing to different backends but it seems to be working for me.
At first I ran into a issue were all domains could access the same subdomain, this is when I realized I just needed some extra conditions.

Here are the steps to achieve; service.example.com & service1.example1.com

Services --> ACME Client --> Certificates
Add the certificate for your extra domains and forcefully issue your certificate

Services --> HAProxy --> Settings --> Advanced --> Map Files
Here we will create a new map file for each domain "PUBLIC_SUBDOMAINS_map-example" & "PUBLIC_SUBDOMAINS_map-example1"

Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
Add a new condition for each domain that you have added.
Name = "example1_condition"
Description = "Traffic matches example1.com"
Condition type = "host contains"
Host Contains = "example1.com"

Services --> HAProxy --> Settings --> Rules & Checks --> Rules
Add a map rule for each domain while also selecting our newly created "example1_condition"
Name = "PUBLIC_SUBDOMAINS_map-rule-example1"
Select conditions = "example1_condition"
Map file = "PUBLIC_SUBDOMAINS_map-example1"

Services --> HAProxy --> Settings --> Virtual Services --> Public Services
Finally we edit our "1_HTTPS_frontend"
Add all extra domains in the "Certificates" input.
Scroll down and add each map-rule-example1 in the "Select Rules" input
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 24, 2022, 11:51:04 am
I am not sure if this is the correct way to achieve multiple domains pointing to different backends but it seems to be working for me.
At first I ran into a issue were all domains could access the same subdomain, this is when I realized I just needed some extra conditions.

Here are the steps to achieve; service.example.com & service1.example1.com

Services --> ACME Client --> Certificates
Add the certificate for your extra domains and forcefully issue your certificate

Services --> HAProxy --> Settings --> Virtual Services --> Public Services
Finally we edit our "1_HTTPS_frontend"
Add all extra domains in the "Certificates" input.
Just the steps above are necessary and the following step.

Then edit the "PUBLIC_SUBDOMAINS_map" and change the subdomain entries to full FQDNs.

Code: [Select]
old
===
nas NAS_backend

new
===
nas.domain1.com NAS_1_backend
nas.domain2.com NAS_2_backend

Of course you could also create dedicated map-files and rules for each domain, but in terms of functionality it is not necessary.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 25, 2022, 04:45:20 am
Although it has the same outcome, The steps you provided using only 1 map file is a lot cleaner and easier to follow.

Thanks again for pointing this out.
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: christian_domes on April 29, 2022, 09:19:57 am
@theHellSite
Hello
When I overide the dns Server will the haproxy still be used or not?
Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on May 11, 2022, 05:11:28 am
@theHellSite
Hello
When I overide the dns Server will the haproxy still be used or not?

It depends on how you override the dns record.
Assume that you set your SNI frontend with 0.0.0.0:80 and 0.0.0.0:443

For example, you added a DNS record in Cloudflare "abc.domain.com" pointing to your WAN IP, and your tested it and found HAProxy working both locally and externally.
Then you removed the DNS record from Cloudflare, and add one in unbounded "abc.domain.com" pointing to your OpnSense IP (either LAN or WAN, doesn't metter)

Then your HAProxy should work locally, but failed externally.
*If you want to override DNS record in unbound, always point to SNI frontend. If you set it to either HTTP frontend or HTTPS frontend, it will fail.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 04, 2022, 03:42:34 pm
I just updated the tutorial with a very important change to the DynDNS part. It therefore only affects users with a dynamic WAN IP.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: schnerring on June 10, 2022, 12:57:32 am
I've been following this excellent guide to a tee, but I get the following warning:

Quote
[WARNING] (51339) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:69).
Warnings were found.
Configuration file is valid

This is the affected part of the HAProxy config (the last line is 69):

Code: [Select]
# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_629f48c6073c95.86527303 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_629f48c6073c95.86527303

For reference, other people also encountered this issue:


The issue seems to be the NoSSL_condition. The suggested solution is to replace the condition type from Traffic is SSL (TCP request content inspection)  to Traffic is SSL (locally deciphered):

Code: [Select]
     # logging options
     # ACL: NoSSL_condition
-    acl acl_629f48c6073c95.86527303 req.ssl_ver gt 0
+    acl acl_629f48c6073c95.86527303 ssl_fc
 
     # ACTION: HTTPtoHTTPS_rule
     http-request redirect scheme https code 301 if !acl_629f48c6073c95.86527303

When doing so, the warning is gone. However, this is the first time I'm using HAProxy and I don't really know what I'm doing, so I wanted to check-in with you guys to ensure my solution is correct?

If so, I'd appreciate if you updated the screenshot in step 5.7 @TheHellSite
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 10, 2022, 02:14:12 pm
The issue seems to be the NoSSL_condition. The suggested solution is to replace the condition type from Traffic is SSL (TCP request content inspection)  to Traffic is SSL (locally deciphered):

Code: [Select]
     # logging options
     # ACL: NoSSL_condition
-    acl acl_629f48c6073c95.86527303 req.ssl_ver gt 0
+    acl acl_629f48c6073c95.86527303 ssl_fc
 
     # ACTION: HTTPtoHTTPS_rule
     http-request redirect scheme https code 301 if !acl_629f48c6073c95.86527303

When doing so, the warning is gone. However, this is the first time I'm using HAProxy and I don't really know what I'm doing, so I wanted to check-in with you guys to ensure my solution is correct?

If so, I'd appreciate if you updated the screenshot in step 5.7 @TheHellSite

Thanks for the detailed explanation!
I already had this on my to do list, but then couldn't find the time for it and in the end simply forgot about it.

The HAProxy docs are also stating to use the "ssl_fc" condition.
https://www.haproxy.com/blog/redirect-http-to-https-with-haproxy/

I will change it in the next couple of days!
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 11, 2022, 03:20:57 pm
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: PaulePils on June 14, 2022, 11:41:40 am
Hello,

I just recently upgraded my home network with an opnsense and want to reconfigure some stuff in the same run.
Luckily I found this tutorial which was really easy to follow through, especally because of the screenshots :-)
Sadly I must have done something wrong but I can't put my finger on it.
When I try to access from internal LAN (IP address or Name), I get the following error:
From external (via mobile data):
503 Service Unavailable

If I enter http://IP_ADRESS then it opens the Nextcloud Login page but after entering the login data, it switches to https:// and the error appears.

HAProxy config
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62a76f360f0732.68695084 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62a76f360f0732.68695084

# Frontend: 2_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 2_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets no-tlsv12 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62a772caaae308.49400660.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62a76fbf29df39.71162057.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Nextcloud_backend ()
backend Nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Nextcloud 192.168.10.40:443 ssl verify none


Map files
Code: [Select]
#public access subdomains
nextcloud NEXTCLOUD_backend

Log files
Code: [Select]
Informational haproxy 192.168.10.101:4054 [14/Jun/2022:11:11:05.082] 2_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
My Network:
ISP --> Modem --> OPNSense --> Proxmox --> Server 1 Nextcloud

Maybe you can push me in the right direction :-)
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on June 15, 2022, 05:01:20 am
Hello,

I just recently upgraded my home network with an opnsense and want to reconfigure some stuff in the same run.
Luckily I found this tutorial which was really easy to follow through, especally because of the screenshots :-)
Sadly I must have done something wrong but I can't put my finger on it.
When I try to access from internal LAN (IP address or Name), I get the following error:
  • Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
  • Edge:ERR_SSL_PROTOCOL_ERROR
From external (via mobile data):
503 Service Unavailable

If I enter http://IP_ADRESS then it opens the Nextcloud Login page but after entering the login data, it switches to https:// and the error appears.

HAProxy config
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62a76f360f0732.68695084 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62a76f360f0732.68695084

# Frontend: 2_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 2_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets no-tlsv12 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62a772caaae308.49400660.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62a76fbf29df39.71162057.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Nextcloud_backend ()
backend Nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Nextcloud 192.168.10.40:443 ssl verify none


Map files
Code: [Select]
#public access subdomains
nextcloud NEXTCLOUD_backend

Log files
Code: [Select]
Informational haproxy 192.168.10.101:4054 [14/Jun/2022:11:11:05.082] 2_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
My Network:
ISP --> Modem --> OPNSense --> Proxmox --> Server 1 Nextcloud

Maybe you can push me in the right direction :-)

Since you are forcing HAProxy to use TLS 1.3. (As you use no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets no-tlsv12)
There might be issue taking out TLS_AES_128_GCM_SHA256.

According to RFC 8446, S9.1 (https://www.rfc-editor.org/rfc/rfc8446#section-9.1)
Quote
A TLS-compliant application MUST implement the TLS_AES_128_GCM_SHA256 [GCM] cipher suite and SHOULD implement the TLS_AES_256_GCM_SHA384 [GCM] and TLS_CHACHA20_POLY1305_SHA256 [RFC8439] cipher suites (see Appendix B.4).

Although it shouldn't have any effect by implementing a Cipher List, but Cipher List should be removed under TLS 1.3.



From external (via mobile data):
503 Service Unavailable

one of the reasons can be: it cannot read the map file, so it doesn't redirect to target backend.
Some device is hard to load map files during boot time. (One of my setup under Esxi is having such issue)
Apply the HAProxy one more time after all services finished loading can make it read the map file again. Or you can implement condition and rule to ignore such issue
example:
Condition name: nextcloud_cond, condition type: host contains, Host Contains: nextcloud
Rule name: nextcloud_ru, Test type: IF, Select conditions: nextcloud_cond, Execute function: Use specified backend Pool, Use backend pool: NEXTCLOUD_backend
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: PaulePils on June 15, 2022, 11:26:53 am
Thanks for your input. I changed the cipher accordingly to https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6 (https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6) and also applied your rule but still no success.
I am just curios because in the main post the value TLS_AES_128_GCM_SHA256 is also left out  ???

Out of curiosity I tried do enter the only the main domain DOMAIN.dedyn.io and not SUBDOMAIN.DOMAIN.dedyn.io. Then I get a different error
Quote
Firefox vertraut dieser Website nicht, weil das von der Website verwendete Zertifikat nicht für DOMAIN.dedyn.io gilt. Das Zertifikat ist nur gültig für *.DOMAIN.dedyn.io.
SSL_ERROR_BAD_CERT_DOMAIN
I can accept the risk but this is something I need to do on all devices...

Could the problem be that I already had DynDNS by another provider and LE certificate? If yes how can I revoke it?
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 15, 2022, 03:15:28 pm
When I try to access from internal LAN (IP address or Name), I get the following error:
  • Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
  • Edge:ERR_SSL_PROTOCOL_ERROR

This error usually means that you tried to access a service using HTTPS that only supports HTTP.

Quote
internal LAN (IP address or Name)
What do you mean by that? You should use the FQDN and not the local hostname / IP in order to use the reverse proxy (HAProxy).

If I enter http://IP_ADRESS then it opens the Nextcloud Login page but after entering the login data, it switches to https:// and the error appears.
Again, which IP are you entering? The nextcloud local IP or your public IP?
Your public IP should ALWAYS forward HTTP to HTTPS and not display any webpages via HTTP whatsoever! (HTTPtoHTTPS_rule)

My Network:
ISP --> Modem --> OPNSense --> Proxmox --> Server 1 Nextcloud
Is the firewall in your modem disabled or is it still doing NAT?

Thanks for your input. I changed the cipher accordingly to https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6 (https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6) and also applied your rule but still no success.
I am just curios because in the main post the value TLS_AES_128_GCM_SHA256 is also left out  ???

You won't need any 128 bit ciphers unless you are using very very old devices to access your services which I highly doubt since you only want to enable TLS v1.3 anyway.
So you can safely use the cipher suites in my first post which are identical to the ones from the Mozilla SSL configurator but have the "insecure / weak" 128 bit ciphers removed.

Out of curiosity I tried do enter the only the main domain DOMAIN.dedyn.io and not SUBDOMAIN.DOMAIN.dedyn.io. Then I get a different error
Quote
Firefox vertraut dieser Website nicht, weil das von der Website verwendete Zertifikat nicht für DOMAIN.dedyn.io gilt. Das Zertifikat ist nur gültig für *.DOMAIN.dedyn.io.
SSL_ERROR_BAD_CERT_DOMAIN
I can accept the risk but this is something I need to do on all devices...

Could the problem be that I already had DynDNS by another provider and LE certificate? If yes how can I revoke it?

Well, the error is pretty much self-explanatory isn't it?
In my tutorial the wildcard certificate is only valid for the 2nd-level subdomains "*.DOMAIN.dedyn.io" but not for the 1st-level subdomain "DOMAIN.dedyn.io" itself.

If you want the certificate to also cover for "domain.dedyn.io" then you will have to change the certificate in the ACME client to match that. See Part 3 - Step 6.
You will however only need this if you are serving a website in the domain root without "www" infront of it.

Code: [Select]
# currently configured
Common Name = *.DOMAIN.dedyn.io

# what you MIGHT want
Common Name = DOMAIN.dedyn.io
Alt Names = *.DOMAIN.dedyn.io

This will cover the 1st-level subdomain including all 2nd-level subdomains.
Don't forget to reissue the certificate.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 15, 2022, 03:25:18 pm
Since you are forcing HAProxy to use TLS 1.3. (As you use no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets no-tlsv12)
There might be issue taking out TLS_AES_128_GCM_SHA256.

According to RFC 8446, S9.1 (https://www.rfc-editor.org/rfc/rfc8446#section-9.1)
Quote
A TLS-compliant application MUST implement the TLS_AES_128_GCM_SHA256 [GCM] cipher suite and SHOULD implement the TLS_AES_256_GCM_SHA384 [GCM] and TLS_CHACHA20_POLY1305_SHA256 [RFC8439] cipher suites (see Appendix B.4).

Well, in order to get an 100% A+ rating at SSL Labs it is mandatory to remove any ciphers below or equal to 128 bit. This is why I haven't included them in my cipher list and cipher suites.
So unless you have really old devices with outdated browsers I would suggest to only use ciphers with a strength of 256 bit or above.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: PaulePils on June 15, 2022, 05:28:26 pm
When I try to access from internal LAN (IP address or Name), I get the following error:
  • Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
  • Edge:ERR_SSL_PROTOCOL_ERROR

This error usually means that you tried to access a service using HTTPS that only supports HTTP.
Does this mean I need an additional vhost config for the Nextcloud? One for :80 and one for :443? But the certificate lies on a different machine (opnsense). How can I point it there? Or am I missing something?
By deleting the "overwirte" codes in the nextcloud config I at least got access in my internal LAN. I assume the HAProxy config is correct and I need to make corrections in the nextcloud config.

Quote
Quote
internal LAN (IP address or Name)
What do you mean by that? You should use the FQDN and not the local hostname / IP in order to use the reverse proxy (HAProxy).
It was the IP address of the Nextcloud machine. But if it doesn't work that way that is new for me but thanks for pointing it out.

Quote
If I enter http://IP_ADRESS then it opens the Nextcloud Login page but after entering the login data, it switches to https:// and the error appears.
Again, which IP are you entering? The nextcloud local IP or your public IP?
Your public IP should ALWAYS forward HTTP to HTTPS and not display any webpages via HTTP whatsoever! (HTTPtoHTTPS_rule)
I entered the local IP address of the nextcloud machine. But this also happens when I enter the FQDN.

Quote
My Network:
ISP --> Modem --> OPNSense --> Proxmox --> Server 1 Nextcloud
Is the firewall in your modem disabled or is it still doing NAT?
It is dumb modem (TC-4400) therefor it should not have NAT. It only provides the connection to my ISP (Vodafone Cable).

Quote
Thanks for your input. I changed the cipher accordingly to https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6 (https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6) and also applied your rule but still no success.
I am just curios because in the main post the value TLS_AES_128_GCM_SHA256 is also left out  ???
Quote
You won't need any 128 bit ciphers unless you are using very very old devices to access your services which I highly doubt since you only want to enable TLS v1.3 anyway.
So you can safely use the cipher suites in my first post which are identical to the ones from the Mozilla SSL configurator but have the "insecure / weak" 128 bit ciphers removed.
OK
Quote
Out of curiosity I tried do enter the only the main domain DOMAIN.dedyn.io and not SUBDOMAIN.DOMAIN.dedyn.io. Then I get a different error
Quote
Firefox vertraut dieser Website nicht, weil das von der Website verwendete Zertifikat nicht für DOMAIN.dedyn.io gilt. Das Zertifikat ist nur gültig für *.DOMAIN.dedyn.io.
SSL_ERROR_BAD_CERT_DOMAIN
I can accept the risk but this is something I need to do on all devices...

Could the problem be that I already had DynDNS by another provider and LE certificate? If yes how can I revoke it?

Well, the error is pretty much self-explanatory isn't it?
In my tutorial the wildcard certificate is only valid for the 2nd-level subdomains "*.DOMAIN.dedyn.io" but not for the 1st-level subdomain "DOMAIN.dedyn.io" itself.

If you want the certificate to also cover for "domain.dedyn.io" then you will have to change the certificate in the ACME client to match that. See Part 3 - Step 6.
You will however only need this if you are serving a website in the domain root without "www" infront of it.

Code: [Select]
# currently configured
Common Name = *.DOMAIN.dedyn.io

# what you MIGHT want
Common Name = DOMAIN.dedyn.io
Alt Names = *.DOMAIN.dedyn.io

This will cover the 1st-level subdomain including all 2nd-level subdomains.
Don't forget to reissue the certificate.
I don't need the coverage of "domain.dedyn.io". So this should be fine.[/quote]
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 15, 2022, 10:23:09 pm
When I try to access from internal LAN (IP address or Name), I get the following error:
  • Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
  • Edge:ERR_SSL_PROTOCOL_ERROR

This error usually means that you tried to access a service using HTTPS that only supports HTTP.
Does this mean I need an additional vhost config for the Nextcloud? One for :80 and one for :443? But the certificate lies on a different machine (opnsense). How can I point it there? Or am I missing something?
By deleting the "overwirte" codes in the nextcloud config I at least got access in my internal LAN. I assume the HAProxy config is correct and I need to make corrections in the nextcloud config.


Code: [Select]
    server Nextcloud 192.168.10.40:443 ssl verify noneThis line in your HAProxy config indicates that you are accessing your nextcloud webinterface from your local network using "https://192.168.10.40".
If the above is not true and you are accessing it using "http://192.168.10.40" instead then you should have figured your error after reading this sentence.

IF you can already access nextcloud from within your local using its Local_IP:Port then you won't have to change any settings at nextcloud.

Also... Please do yourself a favour and read about what a reverse proxy is, what it does and how it works.
No offense but I think you don't really know what you are doing here.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brynjolm on June 17, 2022, 09:07:30 pm
Hi! First off, i want to thank you for the detailed guide you posted. I am new to HAProxy and have some questions regarding some configs, do you still reply to this post? or should i ask or open another thread somewhere? Thanks!
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 17, 2022, 10:03:40 pm
Hi! First off, i want to thank you for the detailed guide you posted. I am new to HAProxy and have some questions regarding some configs, do you still reply to this post? or should i ask or open another thread somewhere? Thanks!

As long as the question is related to THIS tutorial then feel free to ask!

Otherwise please ask here: https://forum.opnsense.org/index.php?board=28.0
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brynjolm on June 17, 2022, 11:39:15 pm
Thanks! So, i set up HAP as per the guide, and am wondering if its a normal consequence to have 503 error on my truenas webui. Internally i can acces it no problems, outside of the home network i am unable to. Sidenote, i do not plan to expose the web ui or planned to. I was just using it as a sort of test run to see if it will resolve to my domain.truenas.com. I also did expose plex under port 32400. But seem to encounter some TLS handshake errors. It would be nice if you could take a look at my config and point me in the right direction. I did only a A and AAAA record for wildcard hostname i.e *.mydomain.com. I dont know if thats enough.
My general goal is to only expose plex and some other services like nextcloud, but thats for another section of the forums. Also my network is kinda flat. All services run through a single VLAN.
I created two Servers and two backends. One for Truenas and plex respectively.
i have been trying to make a stable connection to plex but it just seems to either time out or lose connection after a while.

Heres and output of my config file, i have some disabled stuff in the config file since i turn on and off stuff for testing. Also renamed some stuff, truenas is just the Physical server with the IP. (myloopbackip) is the virutal address.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

#
# NOTE: HAProxy is currently DISABLED
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr libc,last
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend (Listening on ip:80 / ip:443)
frontend SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind :::80 name :::80
    bind :::443 name :::443
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTP_frontend (Listening on 127.10.20.5)
frontend HTTP_frontend
    bind (myloopbackip):80 name (myloopbackip):80 accept-proxy
    bind :::80 name :::80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_62aa8dcf894a87.42381056 ssl_fc

    # ACTION: HTTPtoHTTPS
    http-request redirect scheme https code 301 if !acl_62aa8dcf894a87.42381056

# Frontend: HTTPS_frontend (Listening on (myloopbackip))
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind (myloopbackip):443 name (myloopbackip):443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62aad04a028639.71957640.certlist
    bind :::443 name :::443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62aad04a028639.71957640.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: Public_subdomains_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62aa8e31993357.88056717.txt)]
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server (myloopbackip) send-proxy-v2 check-send-proxy

# Backend (DISABLED): router_Backend (router Backend)

# Backend: truenas_Backend (truenas Backend)
backend truenas_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server truenas truenasip:443 ssl sni str(truenas) verify none send-proxy-v2 check-send-proxy

# Backend: plex_backend (plex Backend)
backend plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Plex truenasip:32400 send-proxy-v2 check-send-proxy


This is my map file:

Code: [Select]
# public access domains
truenas truenas_backend
plex plex_backend

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: droren on June 18, 2022, 06:17:16 am
I've followed the article and able to setup few subdomain to internal machines in my network.

I have dumb questions... when I created new real server and backend server for one my Synology packages, I initially used HTTPS port and received 400 bad request error The plain HTTP request was sent to HTTPS port

When I changed the port in real server settings to HTTP request, everything worked fine and I am able to access the internal server with SSL certificate.

Why?
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 21, 2022, 10:49:24 am
I've followed the article and able to setup few subdomain to internal machines in my network.

I have dumb questions... when I created new real server and backend server for one my Synology packages, I initially used HTTPS port and received 400 bad request error The plain HTTP request was sent to HTTPS port

When I changed the port in real server settings to HTTP request, everything worked fine and I am able to access the internal server with SSL certificate.

Why?

FAQ - Do I need to enable "SSL" in the Real Server configuration of a service?

Thanks for NOT reading my first post properly.  :(
I explained everything there very well.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 21, 2022, 10:58:08 am
Heres and output of my config file, i have some disabled stuff in the config file since i turn on and off stuff for testing. Also renamed some stuff, truenas is just the Physical server with the IP. (myloopbackip) is the virutal address.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

#
# NOTE: HAProxy is currently DISABLED
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr libc,last
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend (Listening on ip:80 / ip:443)
frontend SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind :::80 name :::80
    bind :::443 name :::443
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTP_frontend (Listening on 127.10.20.5)
frontend HTTP_frontend
    bind (myloopbackip):80 name (myloopbackip):80 accept-proxy
    bind :::80 name :::80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_62aa8dcf894a87.42381056 ssl_fc

    # ACTION: HTTPtoHTTPS
    http-request redirect scheme https code 301 if !acl_62aa8dcf894a87.42381056

# Frontend: HTTPS_frontend (Listening on (myloopbackip))
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind (myloopbackip):443 name (myloopbackip):443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62aad04a028639.71957640.certlist
    bind :::443 name :::443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62aad04a028639.71957640.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: Public_subdomains_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62aa8e31993357.88056717.txt)]
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server (myloopbackip) send-proxy-v2 check-send-proxy

# Backend (DISABLED): router_Backend (router Backend)

# Backend: truenas_Backend (truenas Backend)
backend truenas_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server truenas truenasip:443 ssl sni str(truenas) verify none send-proxy-v2 check-send-proxy

# Backend: plex_backend (plex Backend)
backend plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Plex truenasip:32400 send-proxy-v2 check-send-proxy


This is my map file:

Code: [Select]
# public access domains
truenas truenas_backend
plex plex_backend

1. The map file is case sensitive. Fix it.

2. Remove the "send-proxy-v2 check-send-proxy" directives from the backends of your actual services. These two options are only necessary on the "SSL_backend".

3. Why do you have the sni setting configured in your truenas real server? that shouldn't be necessary at all. Also remove the "send-proxy-v2 check-send-proxy" directives from the backend (see 1.).
Code: [Select]
server truenas truenasip:443 ssl sni str(truenas) verify none send-proxy-v2 check-send-proxy
4. Your "HTTP_frontend" and "HTTPS_frontend" should ONLY be listening to your SSL_server IP address. Not to "0.0.0.0:0" or "::::0". Just think about it and take a look at my diagram in the first post... You should quickly figure that this doesn't make any sense.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brynjolm on June 25, 2022, 12:16:30 am
Quote
1. The map file is case sensitive. Fix it.

2. Remove the "send-proxy-v2 check-send-proxy" directives from the backends of your actual services. These two options are only necessary on the "SSL_backend".
4. Your "HTTP_frontend" and "HTTPS_frontend" should ONLY be listening to your SSL_server IP address. Not to "0.0.0.0:0" or "::::0". Just think about it and take a look at my diagram in the first post... You should quickly figure that this doesn't make any sense.


Thanks! That solved a lot of the problems i had.

I also had to edit some settings on my cloudflare setup in order for it to work. I now have full access to everything aside from nextcloud and some other stuff. while still being proxied behind cloudflare. I dont know if it will be useful for other cloudflare users. But for me at least since i wanted to go cloudflare for everything. I ended up also using the cloudflare certs and uploading it to HAP while still being onf Full(Strict) mode. i dont know if its and intended sideffect but i didnt have to use split dns for this. I could just call up everything by subdomain.domain.com locally.
Also with regards to the map file for the local subdomanins. The format should look like this?
Quote
local loca_backend

plex plex_backend

Should there be a space in between?
Last questions. Is it possible to use the generated Cert for both truenas and opnsense instead of the self signed one? would it break the config? Also is there a specifi HAP sub, or should i just post in the General section?
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 27, 2022, 04:08:58 pm
I could just call up everything by subdomain.domain.com locally.
This is because your domains resolve to cloudflares IP and not your own public IP.

Also with regards to the map file for the local subdomanins. The format should look like this?
Quote
local loca_backend

plex plex_backend

Should there be a space in between?
https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/syntax/


Last questions. Is it possible to use the generated Cert for both truenas and opnsense instead of the self signed one? would it break the config? Also is there a specifi HAP sub, or should i just post in the General section?
Shouldn't break anything but isn't neccessary.
https://discourse.haproxy.org/
or
https://forum.opnsense.org/index.php?board=28.0
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: xxavarlonxx on June 29, 2022, 01:04:47 pm
Thnak you very much. Thats the best guide about HAProxy - Lets Encrypt I've ever found. Everything works and its not a mickey mouse setup, its a good setup for production..
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on July 04, 2022, 10:25:37 pm
I am fighting with this setup - what is most troubling is that it worked a few days ago, but for the life of me I cannot find what went wrong.

I now have a stripped-down version of the setup where I just want to use a map-based setup with SSL offloading for some internal servers (now my setup contains only one).

This is my setup:

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend (Listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_Frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_Frontend (Listening on 127.0.0.1:80)
frontend 1_HTTP_Frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: No_SSL_Condition
    acl acl_629b7d353dc6e8.95969175 ssl_fc

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_629b7d353dc6e8.95969175

# Frontend: 1_HTTPS_Frontend (Listening on 127.0.0.1:443)
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/629b82033c9ac6.13569566.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_Subdomains_Map_Rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/629b7dc0816c90.87321785.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend (DISABLED): PLEX_backend (PLEX Backend)

# Backend: BLOB_backend (BLOB Webserver Backend)
backend BLOB_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server BLOB_server 192.168.10.3 ssl sni str(ssl.xyz.de) verify none resolve-prefer ipv4

The corresponding map file would be:

Code: [Select]
ssl.xyz.de BLOB_backend

Symptoms are: I can see the certificate containing ssl.xyz.de (and others) when I connect to my OpnSense via 'openssl s_client'.

If I try 'wget -O- https://ssl.xyz.de', the connection hangs after initial TLS handshake - when I use one of the other domains like 'abc.xyz.de' which resolve to the same IP, I get the expected 503. I can even see the HTTP->HTTPS redirection when I use http instead of https.

So, I conclude that the certificate is O.K., I get through to 1_HTTP_Frontend (otherwise there would be no 503 when I use an unmapped URL). I seems like most of the time, the frontend does not connect to the backend (server) - it does sometimes, however.

I am at a loss what causes this because I also verified that OpnSense can reach BLOB server at 192.168.10.3. It also has a Let's Encrypt certificate on ssl.xyz.de, because before, I just port-forwarded port 443. I can get data via 'curl --connect-to 192.168.10.3:443 https://ssl.xyz.de' from it.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 04, 2022, 11:02:05 pm
So, I conclude that the certificate is O.K., I get through to 1_HTTP_Frontend (otherwise there would be no 503 when I use an unmapped URL). I seems like most of the time, the frontend does not connect to the backend (server) - it does sometimes, however.

I am at a loss what causes this because I also verified that OpnSense can reach BLOB server at 192.168.10.3. It also has a Let's Encrypt certificate on ssl.xyz.de, because before, I just port-forwarded port 443. I can get data via 'curl --connect-to 192.168.10.3:443 https://ssl.xyz.de' from it.

Code: [Select]
server BLOB_server 192.168.10.3 ssl sni str(ssl.xyz.de) verify none resolve-prefer ipv4
1. You have to include the server port in the config of the server, obviously...
2. Remove the SNI string in the server config, not neccessary at all. I also DID NOT use it in my tutorial, so why are you using it?
3. Remove the resolve-prefer directive. Again, I did not use it in my tutorial...
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on July 05, 2022, 12:33:32 am
1. I thought that the server port would be 443 per default, but I now set it and it did not change anything.
2. I need the SNI string because the server reacts to different names but I cannot use the "external" name since I want to contact the LAN IP. However, I set an alternative name so that I can leave out the SNI for testing and - no dice.
3. Disabling resolve-prefer did nothing as well.

4. I even tried without SSL, same result. So it must be something before that.

New config:

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend (Listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_Frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_Frontend (Listening on 127.0.0.1:80)
frontend 1_HTTP_Frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: No_SSL_Condition
    acl acl_629b7d353dc6e8.95969175 ssl_fc

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_629b7d353dc6e8.95969175

# Frontend: 1_HTTPS_Frontend (Listening on 127.0.0.1:443)
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/629b82033c9ac6.13569566.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_Subdomains_Map_Rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/629b7dc0816c90.87321785.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend (DISABLED): PLEX_backend (PLEX Backend)

# Backend: BLOB_backend (BLOB Webserver Backend)
backend BLOB_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server BLOB_server blob.xyz:80

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on July 05, 2022, 03:02:16 am
Why you use 127.0.0.1 as SSL_backend IP?
Please setup a VIP

For a server, 0.0.0.0 is ANY IP that can represent the server itself, including localhost
Thus, 0_SNI_frontend conflict with SSL_backend obviously.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on July 05, 2022, 08:16:57 am
I already tried using 127.4.4.3 for SSL_server, HTTP and HTTPS frontends as well with no change in behaviour. The tutorial explicitely states that this is optional (see FAQ #4) and besides, if you are right, there is no difference, because 0.0.0.0/0 would also overlap 127.4.4.3/32.

Thanks for any input, though. I tried several hours and as I said, this already worked...
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 05, 2022, 08:56:40 am
Your haproxy is configured correctly. VIP is indeed optional.

Your real server IP:Port (SSL enabled yes or no / SSL verify yes or no) is the only issue left that could cause it from the haproxy side. Otherwise your web server is misconfigured.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on July 05, 2022, 09:29:53 am
As I said, I triple-checked everything and I also cannot see what goes wrong (even sometimes, it works).

I also tried a non-SSL backend, including another server to eliminate problems on the backend.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 05, 2022, 03:24:45 pm
Here is my reference config.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends


# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60a6828680bca8.63910725.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/60bdf8931a97c9.33132019.txt)]

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_607ae66cdeaed1.61504267 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_607ae66cdeaed1.61504267

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 1h
    http-reuse safe
    server PLEX_server 192.168.215.60:32400 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy


If you compare it with yours. You will see that your real server is the problem!
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on July 05, 2022, 08:23:36 pm
Wow. This was a tough one...

I also believed that my real server is the problem - or to put it another way: ALL my real servers seemed to be the problem.

I replaced the real server by 127.0.0.1 (i.e. OpnSense itself) and guess what? It works.
Then with the LAN IP of my OpnSense - works. I replaced it with www.google.com - works!!!

None of my LAN servers work, however. First, I re-did all of the HAProxy configuration from scratch - with the same result.
I found out that curl has problems as well (most of the time) - so in effect, OpnSense cannot make (reliably) any outgoing TCP connections (while pings work!).

So I asked myself: Is that a firewall or NAT problem? I disabled all firewall rules, but no dice.
After that, I did a tcpdump on the receiving side, only to find out that packets were sent from the OpnSense, and answers also went out but seem to get eaten, although there were no firewall log entries.

Before that, I already had tried to disable all hardware offloading, but only in the global settings.
Looking into that again, I noticed a hint that you have to reconfigure the interfaces in order to take over the defaults, so I disabled all offloading on the LAN interface.

After that - hey, presto! Everything works!

Matter-of-fact, this is a very subtle error: On the AMD 10 GBe (ax) interfaces, the hardware CRC offloading is obviously broken, but you will notice this only in the direction which is rarely used.
I had no problems with "through" traffic. The problem must have surfaced when I trusted my hardware too much, which explains why the whole HAProxy setup once worked.

Thanks for all hints!
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: shuvitcrew on July 11, 2022, 03:39:32 pm
Hello,

first I have to say thank you for this perfect tutorial. I have setup my haproxy for my webservers and everything works fine for internal and external use. Now I've tried to implement OpenVPN on Port 443 in TCP mode. I added the configuration parts as mentioned in Reply #171. The config of haproxy seems to be corrrect, but I can't connect via vpn. I've tried to setup a second vpn-server on port 1194 with upd and i works staight away. Only the vpn in tcp-mode on port 443 refuses to work. Here is my haproxy config:
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 40s
    timeout server 40s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Hört auf 0.0.0.0:80 und 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_62c874b4f2fdc4.23213917 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_62c874b4f2fdc4.23213917
    # ACTION: VPN_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62c87ba1538c16.11776198.txt)]

# Frontend: 1_HTTP_frontend (Hört auf 192.168.161.1:80)
frontend 1_HTTP_frontend
    bind 192.168.161.1:80 name 192.168.161.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62360185bf9055.41837138 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62360185bf9055.41837138

# Frontend: 1_HTTPS_frontend (Hört auf 192.168.161.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.161.1:443 name 192.168.161.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62360bcec06250.52672470.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_FQDN_condition
    acl acl_62361ba046b312.42897137 src darkstar.example.xyz
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_62361a89a23796.93721092 src 192.168.110.0/24

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623619b6c7da11.06632077.txt)] if acl_62361ba046b312.42897137 || acl_62361a89a23796.93721092
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623607a0728a46.68273508.txt)]

# Backend: nextcloud_backend (Nextcloud Backend)
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    # ACL: nextcloud-caldav
    acl acl_6236326bbeed09.73911658 path_beg -i /.well-known/caldav
    # ACL: nextcloud-carddav
    acl acl_6236329a31b372.83647612 path_beg -i /.well-known/carddav

    # ACTION: nextcloud-caldav-carddav
    http-request set-path /remote.php/dav if acl_6236326bbeed09.73911658 || acl_6236329a31b372.83647612
    http-reuse safe
    server nextcloud_host 192.168.160.10:443 ssl verify none

# Backend: SSL_backend (SSL Backend TCP)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    server SSL_Server 192.168.161.1 send-proxy-v2 check-send-proxy

# Backend: bitwarden_backend (Bitwarden Backend)
backend bitwarden_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    # ACL: bitwarden-admin
    acl acl_62362f799a0826.60491269 path_beg -i /admin

    # ACTION: bitwarden-admin_block
    http-request deny if acl_62362f799a0826.60491269
    http-reuse safe
    server bitwarden_host 192.168.160.20:80

# Backend: zyxel-1_backend (Zyxel-1 Backend)
backend zyxel-1_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server zyxel-1 192.168.150.8:443 ssl verify none

# Backend: zyxel-2_backend (Zyxel-2 Backend)
backend zyxel-2_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server zyxel-2 192.168.150.9:443 ssl verify none

# Backend: checkmk_backend (CheckMK Backend)
backend checkmk_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server checkmk_host 192.168.150.21:8080

# Backend: ampache_backend (Ampache Backend)
backend ampache_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server ampache_host 192.168.160.15:443 ssl verify none

# Backend: jellyfin_backend (Jellyfin Backend)
backend jellyfin_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server jellyfin_host 192.168.160.16:8096

# Backend: guacamole_backend (Guacamole Backend)
backend guacamole_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server guacamole_host 192.168.150.22:8080

# Backend: vpn_backend (OpenVPN Backend)
backend vpn_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    server vpn_host 192.168.110.1:1195
Here the OpenVPN config:
Code: [Select]
dev ovpns1
verb 1
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_server1.pid
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto tcp-server
cipher AES-256-CBC
auth SHA512
up /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkup
down /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkdown
local 192.168.110.1
client-disconnect "/usr/local/etc/inc/plugins.inc.d/openvpn/attributes.sh server1"
tls-server
server 10.10.1.0 255.255.255.0
client-config-dir /var/etc/openvpn-csc/1
username-as-common-name
auth-user-pass-verify "/usr/local/etc/inc/plugins.inc.d/openvpn/ovpn_auth_verify user 'Local Database' 'false' 'server1'" via-env
tls-verify "/usr/local/etc/inc/plugins.inc.d/openvpn/ovpn_auth_verify tls 'vpn.shuvit.space' 1"
lport 1195
management /var/etc/openvpn/server1.sock unix
max-clients 5
push "route 192.168.110.0 255.255.255.0"
ca /var/etc/openvpn/server1.ca
cert /var/etc/openvpn/server1.cert
key /var/etc/openvpn/server1.key
dh /usr/local/etc/dh-parameters.4096.sample
tls-auth /var/etc/openvpn/server1.tls-auth 0
persist-remote-ip
float
I hope, that someone could help! Thanks in advance.
Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 11, 2022, 04:24:06 pm
Hello,

first I have to say thank you for this perfect tutorial. I have setup my haproxy for my webservers and everything works fine for internal and external use. Now I've tried to implement OpenVPN on Port 443 in TCP mode. I added the configuration parts as mentioned in Reply #171. The config of haproxy seems to be corrrect, but I can't connect via vpn. I've tried to setup a second vpn-server on port 1194 with upd and i works staight away. Only the vpn in tcp-mode on port 443 refuses to work. Here is my haproxy config:
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 40s
    timeout server 40s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Hört auf 0.0.0.0:80 und 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_62c874b4f2fdc4.23213917 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_62c874b4f2fdc4.23213917
    # ACTION: VPN_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62c87ba1538c16.11776198.txt)]

# Frontend: 1_HTTP_frontend (Hört auf 192.168.161.1:80)
frontend 1_HTTP_frontend
    bind 192.168.161.1:80 name 192.168.161.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62360185bf9055.41837138 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62360185bf9055.41837138

# Frontend: 1_HTTPS_frontend (Hört auf 192.168.161.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.161.1:443 name 192.168.161.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62360bcec06250.52672470.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_FQDN_condition
    acl acl_62361ba046b312.42897137 src darkstar.example.xyz
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_62361a89a23796.93721092 src 192.168.110.0/24

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623619b6c7da11.06632077.txt)] if acl_62361ba046b312.42897137 || acl_62361a89a23796.93721092
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623607a0728a46.68273508.txt)]

# Backend: nextcloud_backend (Nextcloud Backend)
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    # ACL: nextcloud-caldav
    acl acl_6236326bbeed09.73911658 path_beg -i /.well-known/caldav
    # ACL: nextcloud-carddav
    acl acl_6236329a31b372.83647612 path_beg -i /.well-known/carddav

    # ACTION: nextcloud-caldav-carddav
    http-request set-path /remote.php/dav if acl_6236326bbeed09.73911658 || acl_6236329a31b372.83647612
    http-reuse safe
    server nextcloud_host 192.168.160.10:443 ssl verify none

# Backend: SSL_backend (SSL Backend TCP)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    server SSL_Server 192.168.161.1 send-proxy-v2 check-send-proxy

# Backend: bitwarden_backend (Bitwarden Backend)
backend bitwarden_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    # ACL: bitwarden-admin
    acl acl_62362f799a0826.60491269 path_beg -i /admin

    # ACTION: bitwarden-admin_block
    http-request deny if acl_62362f799a0826.60491269
    http-reuse safe
    server bitwarden_host 192.168.160.20:80

# Backend: zyxel-1_backend (Zyxel-1 Backend)
backend zyxel-1_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server zyxel-1 192.168.150.8:443 ssl verify none

# Backend: zyxel-2_backend (Zyxel-2 Backend)
backend zyxel-2_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server zyxel-2 192.168.150.9:443 ssl verify none

# Backend: checkmk_backend (CheckMK Backend)
backend checkmk_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server checkmk_host 192.168.150.21:8080

# Backend: ampache_backend (Ampache Backend)
backend ampache_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server ampache_host 192.168.160.15:443 ssl verify none

# Backend: jellyfin_backend (Jellyfin Backend)
backend jellyfin_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server jellyfin_host 192.168.160.16:8096

# Backend: guacamole_backend (Guacamole Backend)
backend guacamole_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server guacamole_host 192.168.150.22:8080

# Backend: vpn_backend (OpenVPN Backend)
backend vpn_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    server vpn_host 192.168.110.1:1195
Here the OpenVPN config:
Code: [Select]
dev ovpns1
verb 1
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_server1.pid
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto tcp-server
cipher AES-256-CBC
auth SHA512
up /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkup
down /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkdown
local 192.168.110.1
client-disconnect "/usr/local/etc/inc/plugins.inc.d/openvpn/attributes.sh server1"
tls-server
server 10.10.1.0 255.255.255.0
client-config-dir /var/etc/openvpn-csc/1
username-as-common-name
auth-user-pass-verify "/usr/local/etc/inc/plugins.inc.d/openvpn/ovpn_auth_verify user 'Local Database' 'false' 'server1'" via-env
tls-verify "/usr/local/etc/inc/plugins.inc.d/openvpn/ovpn_auth_verify tls 'vpn.shuvit.space' 1"
lport 1195
management /var/etc/openvpn/server1.sock unix
max-clients 5
push "route 192.168.110.0 255.255.255.0"
ca /var/etc/openvpn/server1.ca
cert /var/etc/openvpn/server1.cert
key /var/etc/openvpn/server1.key
dh /usr/local/etc/dh-parameters.4096.sample
tls-auth /var/etc/openvpn/server1.tls-auth 0
persist-remote-ip
float
I hope, that someone could help! Thanks in advance.

I have just tried TCP mode with map file, there is a few more steps to achieve the goal instead of placing the map rule directly to 0_SNI
(I checked the package and found SNI inside, however, haproxy doesn't recognize it in TCP mode, that's why we need to force it to recognize SNI)

1. Create a "Condition" to request client hello
Name: SSL_Hello
Condition type: Custom condition (option pass-through)
Option pass-through: req_ssl_hello_type 1
(https://i.postimg.cc/rD89fwvk/cond-hello.jpg) (https://postimg.cc/rD89fwvk)

2. Create a "Rule" to wait accept SSL hello
Name: tcp_request_content_accept_ssl
Select conditions: SSL_Hello
Execute function: tcp-request-content-accept
(https://i.postimg.cc/mcnNZNhL/rule-ssl.jpg) (https://postimg.cc/mcnNZNhL)

3. Create a "Rule" to wait for inspect
Name: tcp_request_inspect_delay
Optional condition: none
Execute function: tcp-request-inspect-delay
TCP inspection delay: 5s
(https://i.postimg.cc/gX6yxX6v/rule-delay.jpg) (https://postimg.cc/gX6yxX6v)

4. Place the Rule to 0_SNI_frontend in following order
tcp_request_inspect_delay
tcp_request_content_accept_ssl
map
(https://i.postimg.cc/sBx4R0jH/rule-order.jpg) (https://postimg.cc/sBx4R0jH)
(hmdir_ru is my map rule)


Update according to findings in #183 (https://forum.opnsense.org/index.php?topic=23339.msg131582#msg131582)

5. Change the no_SSL condition to Traffic is SSL (locally deciphered)
(https://i.postimg.cc/Cng6Mdtn/nossl.jpg) (https://postimg.cc/Cng6Mdtn)

*Remark
It is advised to use another map file for 1_HTTPS_frontend if necessary
If you really don't want to create another map file, use "SNI TLS extension matches (locally deciphered)" instead

Might want to ask @Bunch for help, since I am still not using any non SSL services.

I can only give you the below config. I figured that one out a while back with another user (@Lip90 (https://forum.opnsense.org/index.php?action=profile;u=23531)).
As far as I can remeber it worked fine but it was neccessary to establish the connection twice because the first time (always) failed.
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    option log-separate-errors
    option tcplog
    # ACL: OPENVPN_condition
    acl acl_6143a3ff7e6bf2.30491250 req_ssl_hello_type 1

    # ACTION: OPENVPN_rule
    use_backend OPENVPN_backend if !acl_6143a3ff7e6bf2.30491250
    # WARNING: pass through options below this line
    tcp-request inspect-delay 5s
    tcp-request content accept if !{ req_ssl_hello_type 1 }

# Frontend: 1_HTTP_frontend (Listening on 0.0.0.0:80)
frontend 1_HTTP_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_6138b110159553.96461818 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6138b110159553.96461818

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6138b32401a006.77997133.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    option httplog
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6141ef8f0a8841.88130105 src 192.168.0.0/16

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6141ef3275d630.55285385.txt)] if acl_6141ef8f0a8841.88130105
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6138b15d48a964.28077676.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: OPENVPN_backend ()
backend OPENVPN_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server OPENVPN_server 127.0.0.1:1194
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: xkpx on July 23, 2022, 06:28:37 pm
Lovely , Thanks for hard work !
Question: is it possbile to cover somehow  multi domain wildcard (for www.firewall.network.com ) -

I got problem with this settings it covers the subdomains but not www.
Common Name: *.network.com
Multidomain name: network.com

Any idea how to issue one cert for all services with subdomains and 1st level domain and www.
Or what is the right way to do this , or maybe to redirect www -> *.network.com without it?

** So far i issued new cert and added in HaProxy and its working so i guess this is the way
www.dev.network.com
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 27, 2022, 12:11:08 am
Lovely , Thanks for hard work !
Question: is it possbile to cover somehow  multi domain wildcard (for www.firewall.network.com ) -

I got problem with this settings it covers the subdomains but not www.
Common Name: *.network.com
Multidomain name: network.com

Any idea how to issue one cert for all services with subdomains and 1st level domain and www.
Or what is the right way to do this , or maybe to redirect www -> *.network.com without it?

** So far i issued new cert and added in HaProxy and its working so i guess this is the way
www.dev.network.com

If you want to cover also the base domain and not only the subdomains of it, then you will have to change the certificate settings to:

Common Name: yourdomain.com
Alt names: *.yourdomain.com

You will also have to create a rule in HAProxy to respond to requests on your base domain (yourdomain.com). Alternatively just set the desired backend for your basedomain (i.e. WWW_backend) as default backend on the 1_HTTPS_frontend.

In the end you should have a working certificate and HAProxy redirection for all your subdomains (i.e. cloud.yourdomain.com) and your base domain (yourdomain.com).
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on July 28, 2022, 11:50:22 am
Hello,

Thank you so much for writing this guide.

I were able to get this working and got the A+ authentication as well as access my server from outside network. I tested using VPN and everything work including the lock on https.

However, I'm running in to problem with the very last part which is accessing my server using https within my network.

Attatched is the screenshot of my setting in unbound dns. I also made sure that unbound dns service is running as well.

Thank you again.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 28, 2022, 01:00:42 pm
Check that the client devices in you LAN are actually using unbound dns resolver.

Edit: You have to put the OPNsense LAN IP in the DNS overide. Not the IP of the service. I explicitly say this in the tutorial.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: schnerring on July 28, 2022, 07:04:43 pm
I upgraded to 22.7. HAProxy spits out some deprecation warnings (https://forum.opnsense.org/index.php?topic=29515.msg142527#msg142527), but my config seems to be working fine.

edit: a PR with a fix has already been merged, so we just have to wait for a new haproxy plugin release
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on August 03, 2022, 09:02:59 am
Hello,

Sorry for bothering you again. I have to restart my opnsense because my piHole were messing it up so I did a clean install. However, this time I am getting stuck at the last step verifying SSL to get the A+ score.

I received an error "Assessment failed: No secure protocols supported" I've tried to went back and double check my setting and I couldn't find any error. Checked my ACME and registered, my cert is verified okay. The ip of my domain is updated automatically on the server.

My goal is to aim to get this certified so I could use my domain to add to my adguard to implement DNS over HTTPS to block ad that route as well.

I'm also included the attachments you have requested.

THank you so much for your time.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on August 05, 2022, 05:21:10 am
Hello,

I've tried to fresh re-installed Opnsense and followed your step again and finally i got A rating. not A plus but i think it worked.

Thank you so much.

This time I read your comment and got to accessed the local as well. However, for some reason I'm still getting blocked by my ISP router.

If I connected from external network I received 503 Service Unavailble. I think this make sense since I didn't allow external IP to connect my server yet (which is one of your last step)

When I connected from my lan network, my ISP router log-in page keep popping up. I've tried to put my Opnsense router to the DMZ port and tried to port forward 80 and 443 of my router internal LAN ipaddress but still didn't work. I couldn't get pass my ISP router's log in page.

Thank you again for taking your time and write this.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on August 05, 2022, 06:02:03 am
Sorry, I forgot to add my config log
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on August 05, 2022, 09:48:57 am
Hello,

I think I figured it out.

My mistake were at the very last step where you now have everything setup and wildcard which is *.zzzz.com. I copied the screenshot without understanding what I'm doing so I remove my host and kept it as zzzz where my domain is .com and I were able to access my TrueNas.

Then I understand about the public and local domain if I put my map at my local domain then i can only access it via local network. if I put it at the public map files then I can  access it at the external network and local network. Do I have it understand correctly?

However I have this one last problem I hope you can help me point out. I have adguard installed on the same IP as my opnsense. I changed my port https of my opnsense according to your guide and the port adguard's web UI listening is also different. However, when I add them in the Real server according to their port which they currently listening to. I cannot get them working. It still happening where my external network connect to it, I have the 503 error which make sense since I am only allow local. But when I access them locally I hit the ISP main router log-in page.

I hope what i wrote make sense. I'm so close, I hope someone can guide me to the right direction.

Thank you all.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: 8dgrpsu on August 06, 2022, 12:28:31 am
Thanks for this guide saved me after 2 days, the next bit is passing remote desktop through, i saw this Reddit post but I am not sure how i add to your setup or do I need to create new?

https://www.reddit.com/r/OPNsenseFirewall/comments/l2usx5/opnsense_haproxy_remote_desktop_gateway/
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 06, 2022, 07:12:14 pm
Then I understand about the public and local domain if I put my map at my local domain then i can only access it via local network. if I put it at the public map files then I can  access it at the external network and local network. Do I have it understand correctly?

Yes.

However I have this one last problem I hope you can help me point out. I have adguard installed on the same IP as my opnsense. I changed my port https of my opnsense according to your guide and the port adguard's web UI listening is also different. However, when I add them in the Real server according to their port which they currently listening to. I cannot get them working. It still happening where my external network connect to it, I have the 503 error which make sense since I am only allow local. But when I access them locally I hit the ISP main router log-in page.

Well I can't help you there... If your other services are working then you probably have your internal network misconfigured, given that you have another router in front of your opnsense.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 06, 2022, 07:13:23 pm
Thanks for this guide saved me after 2 days, the next bit is passing remote desktop through, i saw this Reddit post but I am not sure how i add to your setup or do I need to create new?

https://www.reddit.com/r/OPNsenseFirewall/comments/l2usx5/opnsense_haproxy_remote_desktop_gateway/

1. You can easily add this to my/your current setup. Just follow the guide in the reddit thread.

2. Not related to my tutorial so I won't be helping here.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on August 07, 2022, 01:51:38 am
Hello,

I finally got it.

I think my problem were that my firefox browser keep pusing the connection to http thus my ISP router log-in page keep pop up. However, when I use edge or chrome the https connection pushed through and I were able to access the service.

May I ask how would I fix this problem? I tried to delete the certificate from firefox and tried to re-install the firefox but when I tried to access my service it keep asking me this connection is not secured and forced me to use http.

Thank you again.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 08, 2022, 08:37:48 pm
Delete all of the firefox history (cache, cookies, website settings ...).
if that doesn't work, it is your network, not your browser.
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Aphid667 on August 08, 2022, 09:25:21 pm
First of all, thank you for taking the time and effort to write this impressive guide. Despite this guide I still run into problems  :-[

I have a few web servers running that each have their own subdomain name. I am now trying to make the switch from pfsense to opnsense and have followed your guide to set up haproxy. Currently there is no service running on the domain name. However, when I now try to access my web server via both lan and wan I kept getting error 503 service not available. These web servers are all visualized on a proxmox server.

A second question I have, single post above you talk about "You have to put the OPNsense LAN IP in the DNS overide. Not the IP of the service." I am confused about this piece, is it possible to explain a little more about this.

Thanks in advance for feedback
Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 08, 2022, 10:21:09 pm
Currently there is no service running on the domain name. However, when I now try to access my web server via both lan and wan I kept getting error 503 service not available.

Well, if there is no service running, then HAProxy will spit out the 503 message because it can not reach the service.  ???

Code: [Select]
# Backend: cloud_backend ()
backend cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server cloud_server 192.168.7.5:80 ssl verify required ca-file /etc/ssl/cert.pem

Code: [Select]
    server cloud_server 192.168.7.5:80 ssl verify required ca-file /etc/ssl/cert.pemAre you really uploading the self-signed cert of the service to the OPNsense and checking it with HAProxy?
Otherwise that line is wrong.

A second question I have, single post above you talk about "You have to put the OPNsense LAN IP in the DNS overide. Not the IP of the service." I am confused about this piece, is it possible to explain a little more about this.

It is very well explained in part 6 of the post. Nothing I could really add here.
I also provided documentation links that explain the solutions.
But to maybe explain your question.
If you put the service IP in the DNS override, the client will connect to the service and therefore WON'T be able to use the Let's Encrypt - HAProxy cert.
If you put the OPNsense LAN IP in the DNS override, the client will first connect to HAProxy using a valid cert and then HAProxy talks to the client using either no or the self-signed cert of the service.

I hope you understand that I don't have the time to teach everyone how something works.
Thanks to the internet and search machines you should be able to find that out yourself.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: underwriter on August 10, 2022, 12:44:55 pm
Thank you much for this amazing tutorial. I have referred a few people from reddit to this.

Questions I hope someone can help me with: If I have xdomain.com, xczxdomain.com and ltsdomain.com;
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 10, 2022, 02:26:45 pm
Thank you much for this amazing tutorial. I have referred a few people from reddit to this.

Questions I hope someone can help me with: If I have xdomain.com, xczxdomain.com and ltsdomain.com;
  • does this support multi domain usage?
    can I use this tutorial to assign a particular domain for a given service?
    do I need to recreate the whole entries for each or at which point do I make the adjustment?

Of course it does! Just make sure the domains point to any public IP of your OPNsense.

You will have to make three changes to the setup.

1. Let's Encrypt: Here you will have to add one certificate for each domain.
If their DNS Zones are managed at different domain registrars you will also have to create the corresponding DNS-01 challenges for each registrar or move their DNS Zones to deSEC (Managed DNS).

2. HAProxy HTTPS Frontend: Add the newly created certificates for each individual domain.

3. HAProxy Public Subdomain Map File: Change the map file content from f.e. "plex PLEX_backend" to "plex.xdomain.com PLEX_backend", "cloud.xczxdomain.com CLOUD_backend" and so on. This way HAProxy can map each subdomain to the correct domain and backend.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: underwriter on August 11, 2022, 12:04:53 am
Thank you so much. I appreciate your support.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on August 19, 2022, 04:02:03 pm
I was going to open a separate thread to ask for help but I think I might try and see if it is a legitimate variation to the tutorial.
First things first. My setup is exactly as per the tutorial.
I am now investigating the use of an application that uses http transport on port 80 or a custom port but without TLS. The encryption instead is done on the app layers on top. It's called rport.
I'm trying custom port 5000. Looking for the last 10 or so pages of the thread I can only see something similar in Bunch's input "Reply #171 on: February 20, 2022, 05:25:18 pm ". Not quite the same.
I've tried creating conditions matching on the hostname i.e. my subdomain part of mysubdomain.mydomain.dedyn.io and matching on http traffic. Then tried the rule on the SNI frontend, the http frontend and  https frontends. Essentially all frontends trying to make the exception there but in all cases after the sni, the http to https rule gets evaluated first, defeating any exception I've tried.

If it's not too much a deviation, could I have a suggestion on how to approach it? In sum, I'm looking for a way to route my http custom port to a back end as an exception in this Tutorial setup.

My config (you'll notice my port 5000 already in the SNI and the real server listening on that port).

Code: [Select]
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    1
    hard-stop-after             60s
    no strict-limits
    maxconn                     10
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 10
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listens on 80, 443, 853, 5000)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:853 name 0.0.0.0:853
    bind 0.0.0.0:5000 name 0.0.0.0:5000
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (listening on 192.168.5.100:80 i.e. http only)
frontend 1_HTTP_frontend
    bind 192.168.5.100:80 name 192.168.5.100:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_619439805021f2.97978352 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_619439805021f2.97978352

# Frontend: 1_HTTPS_frontend (Listening on 192.168.5.100:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.5.100:443 name 192.168.5.100:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61952b9d47d700.25962675.certlist
    bind 192.168.5.100:5000 name 192.168.5.100:5000 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61952b9d47d700.25962675.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Frontend: 1_TCP_frontend (Listening on 192.168.5.100:853)
frontend 1_TCP_frontend
    bind 192.168.5.100:853 name 192.168.5.100:853 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61dc51606078d9.11258474.certlist
    mode tcp
    default_backend nginx_backend-tcp
    # tuning options
    timeout client 15m

    # logging options
    option tcplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.5.100 send-proxy-v2 check-send-proxy

# Backend: nginx_backend-tcp ()
backend nginx_backend-tcp
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server nginx_2 192.168.5.1:8054 resolve-prefer ipv4 send-proxy check-send-proxy

# Backend: bastion_backend (bastion_backend)
backend bastion_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server bastion-1 192.168.5.157:5000
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: authelia on August 30, 2022, 06:55:39 am
I am trying to follow the instructions to enable HAProxy for internal domains. However, I can't seem to get the frontend listener for the virtual ip to work. Service binding is disabled for the virtual ip.

When the frontend listener for the virtual ip is enabled:

1. haproxy cannot start (when webgui is running).
2. webgui cannot start (when haproxy is running).

I have tried various things such as assigning the virtual ip from a brand new subnet etc. However the frontend listener for virtual ip seems to conflict with lighttpd no matter what I do. The only way I can get both services to start is to remove the virtual ip from /var/etc/lighty-webConfigurator.conf.

Virtual IP in LAN Subnet (192.168.1.0/24)

Code: [Select]
root@OPNsense:~ # sockstat -4 -l | grep lighttpd
root     lighttpd   28364 6  tcp4   192.168.1.65:443      *:*
root     lighttpd   28364 8  tcp4   192.168.1.1:443       *:*
root     lighttpd   28364 10 tcp4   192.168.1.65:80       *:*
root     lighttpd   28364 12 tcp4   192.168.1.1:80        *:*
root     sshd       84263 5  tcp4   192.168.1.1:22        *:*

root@OPNsense:~ # /usr/local/etc/rc.d/haproxy start
Starting haproxy.
[ALERT]    (2036) : Starting frontend 1_HTTP_frontend: cannot bind socket (Can't assign requested address) [192.168.1.65:80]
[ALERT]    (2036) : Starting frontend 1_HTTPS_frontend: cannot bind socket (Can't assign requested address) [192.168.1.65:443]
[ALERT]    (2036) : [/usr/local/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.
/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy


Virtual IP in Brand New Subnet (192.168.10.0/32)

Code: [Select]
root@OPNsense:~ # /usr/local/etc/rc.restart_webgui
Starting web GUI...done.
Generating RRD graphs...done.

root@OPNsense:~ # sockstat -4 -l | grep lighttpd
root     lighttpd   64654 6  tcp4   192.168.10.65:443     *:*
root     lighttpd   64654 8  tcp4   192.168.1.1:443       *:*
root     lighttpd   64654 10 tcp4   192.168.10.65:80      *:*
root     lighttpd   64654 12 tcp4   192.168.1.1:80        *:*
root     sshd       84263 5  tcp4   192.168.1.1:22        *:*

root@OPNsense:~ # /usr/local/etc/rc.d/haproxy start
Starting haproxy.
[ALERT]    (18033) : Starting frontend 1_HTTP_frontend: cannot bind socket (Address already in use) [192.168.10.65:80]
[ALERT]    (18033) : Starting frontend 1_HTTPS_frontend: cannot bind socket (Address already in use) [192.168.10.65:443]
[ALERT]    (18033) : [/usr/local/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 30, 2022, 01:37:07 pm
I am trying to follow the instructions to enable HAProxy for internal domains. However, I can't seem to get the frontend listener for the virtual ip to work. Service binding is disabled for the virtual ip.

When the frontend listener for the virtual ip is enabled:

1. haproxy cannot start (when webgui is running).
2. webgui cannot start (when haproxy is running).

Part 4 - Step 1.

If you would have followed the tutorial STEP BY STEP you wouldn't have any issues... Just stick to the tutorial and don't skip a single step.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 30, 2022, 01:41:06 pm
Then tried the rule on the SNI frontend, the http frontend and  https frontends. Essentially all frontends trying to make the exception there but in all cases after the sni, the http to https rule gets evaluated first, defeating any exception I've tried.

If it's not too much a deviation, could I have a suggestion on how to approach it? In sum, I'm looking for a way to route my http custom port to a back end as an exception in this Tutorial setup.

The order of the rules is important! Make sure that all "http-redirect-to-backend" rules are placed BEFORE the HTTPtoHTTPS rule on the HTTP_frontend.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: boredpanda on August 30, 2022, 03:29:34 pm
Hello! I've used this guide a while back and my self-hosted services are running rock solid. There hasn't been any problems at all with HAProxy - so thank you very much for this guide! I haven't exposed any of my services publicly and mainly use HAProxy to be able to use FQDN at home, and when I VPN in using my phone or personal laptop.

However, there is an instance where it would be very nice to be able to white-list one (or a couple) of specific IPs, so that I could access my services at home from my office. I am not able to install software at the office, and there are other restrictions preventing me from using a VPN.

How would I go about white-listing a single IP, allowing access to some of my internal services? Please let me know if my question is out-of-scope for this tutorial and I'll ask elsewhere. ;)
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 30, 2022, 03:39:32 pm
However, there is an instance where it would be very nice to be able to white-list one (or a couple) of specific IPs, so that I could access my services at home from my office. I am not able to install software at the office, and there are other restrictions preventing me from using a VPN.

This is a very easy task, given that the IPs are static!

I don't know how you restricted local access but if you followed my tutorial you will just have to do this.
Take a look at part 7 of the tutorial.

Create the public subdomains map file, create a condition containing all the whitelist public IPs and create the corresponding redirect rule just as I did with the local access subdomains map file.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheEPOCH on August 30, 2022, 11:18:52 pm
Hello,

my previous question is canceled. I build up my whole OPNsense from scratch and now the Tutorial worked very fine.

Thanks for writing this!
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on August 30, 2022, 11:52:21 pm
Then tried the rule on the SNI frontend, the http frontend and  https frontends. Essentially all frontends trying to make the exception there but in all cases after the sni, the http to https rule gets evaluated first, defeating any exception I've tried.

If it's not too much a deviation, could I have a suggestion on how to approach it? In sum, I'm looking for a way to route my http custom port to a back end as an exception in this Tutorial setup.

The order of the rules is important! Make sure that all "http-redirect-to-backend" rules are placed BEFORE the HTTPtoHTTPS rule on the HTTP_frontend.
Thank you. Unfortunately I haven't been able to do this. The exact warning is:
Code: [Select]
[WARNING] (96704) : parsing [/usr/local/etc/haproxy.conf.staging:74] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.
Warnings were found.
Configuration file is valid
Any other ideas are welcome :)
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on September 01, 2022, 12:39:57 pm
Then tried the rule on the SNI frontend, the http frontend and  https frontends. Essentially all frontends trying to make the exception there but in all cases after the sni, the http to https rule gets evaluated first, defeating any exception I've tried.

If it's not too much a deviation, could I have a suggestion on how to approach it? In sum, I'm looking for a way to route my http custom port to a back end as an exception in this Tutorial setup.

The order of the rules is important! Make sure that all "http-redirect-to-backend" rules are placed BEFORE the HTTPtoHTTPS rule on the HTTP_frontend.
Thank you. Unfortunately I haven't been able to do this. The exact warning is:
Code: [Select]
[WARNING] (96704) : parsing [/usr/local/etc/haproxy.conf.staging:74] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.
Warnings were found.
Configuration file is valid
Any other ideas are welcome :)
I've found what I think is a workaround with the service in question, leaving the haproxy setup still as per this tutorial's. Thanks for the earlier suggestion.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brooklynmind on September 03, 2022, 01:35:04 pm
I've been following this wonderfully crafted tutorial, so "THANK YOU" to the op for this.
Question (I know this might outside the scope of this tutorial):
Is that something this setting can help to implement?
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 06, 2022, 09:16:08 am
I've been following this wonderfully crafted tutorial, so "THANK YOU" to the op for this.
Question (I know this might outside the scope of this tutorial):
  • If I want HAProxy to handle *.my1stdomain.xyz which would be for specific services (already have this working flawlessly),
    but I would like to forward *.my2nddomain.xyz to nginx proxy manager running on docker so that nginx proxy manager will be used to manage that.
Is that something this setting can help to implement?

This has been answered 12 messages back.
https://forum.opnsense.org/index.php?topic=23339.msg143886#msg143886
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brooklynmind on September 08, 2022, 02:46:39 am
This has been answered 12 messages back.
https://forum.opnsense.org/index.php?topic=23339.msg143886#msg143886
[/quote]

Not really, as that person's question was different from mine.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on September 08, 2022, 04:52:40 am
I've been following this wonderfully crafted tutorial, so "THANK YOU" to the op for this.
Question (I know this might outside the scope of this tutorial):
  • If I want HAProxy to handle *.my1stdomain.xyz which would be for specific services (already have this working flawlessly),
    but I would like to forward *.my2nddomain.xyz to nginx proxy manager running on docker so that nginx proxy manager will be used to manage that.
Is that something this setting can help to implement?

If you want nginx to handle TLS of *.my2nddomain.xyz, then the backend of *.my2nddomain.xyz will be in TCP mode.
And you need to set SNI_FRONTEND to something like #176 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354)

If you want keeping HAPROXY to handle TLS, you need to change config of nginx to accept proxy protocol, which is really out of scope
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brooklynmind on September 08, 2022, 12:44:01 pm


If you want nginx to handle TLS of *.my2nddomain.xyz, then the backend of *.my2nddomain.xyz will be in TCP mode.
And you need to set SNI_FRONTEND to something like #176 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354)

If you want keeping HAPROXY to handle TLS, you need to change config of nginx to accept proxy protocol, which is really out of scope
[/quote]

Thanks a lot for your input. I'll play around with that and look at #176 as you pointed out.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stevoni on September 09, 2022, 05:15:16 pm
Great tutorial!

I'm running into a problem accessing the sites within the network after following this tutorial and enabling Cloudflare proxy. Without the Cloudflare proxy I can access the sites both externally and internally but when I enable the Cloudflare proxy I'm unable to access the sites from the internal network.

This post, https://vitobotta.com/2019/12/23/real-ip-haproxy-ingress-behind-cloudflare/ (https://vitobotta.com/2019/12/23/real-ip-haproxy-ingress-behind-cloudflare/), explains how to get the correct IP but I'm not clear on how to implement that in the OPNsense HAProxy implementation. I found a similar question on the forums, https://forum.opnsense.org/index.php?topic=26419.msg127542#msg127542 (https://forum.opnsense.org/index.php?topic=26419.msg127542#msg127542), but there wasn't any answer.

I created a condition with the Cloudflare IPs but I don't know where to go from there, any suggestions?
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 20, 2022, 08:47:40 pm
Great tutorial!

I'm running into a problem accessing the sites within the network after following this tutorial and enabling Cloudflare proxy. Without the Cloudflare proxy I can access the sites both externally and internally but when I enable the Cloudflare proxy I'm unable to access the sites from the internal network.

This post, https://vitobotta.com/2019/12/23/real-ip-haproxy-ingress-behind-cloudflare/ (https://vitobotta.com/2019/12/23/real-ip-haproxy-ingress-behind-cloudflare/), explains how to get the correct IP but I'm not clear on how to implement that in the OPNsense HAProxy implementation. I found a similar question on the forums, https://forum.opnsense.org/index.php?topic=26419.msg127542#msg127542 (https://forum.opnsense.org/index.php?topic=26419.msg127542#msg127542), but there wasn't any answer.

I created a condition with the Cloudflare IPs but I don't know where to go from there, any suggestions?

Sorry, but this is out of scope of this tutorial.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mozart on September 23, 2022, 01:18:11 pm
I followed the tutorial and added the necessary config to enable OpenVPN on port 443 but somehow it always keeps sending everything to the SSL_Backend.

My config file:
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_632625326b34a3.00256787 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_632625326b34a3.00256787
    # ACTION: Openvpn_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6329deb37cfb97.45093681.txt)] if acl_632625326b34a3.00256787

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_632463bc8a4e03.38927091 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_632463bc8a4e03.38927091

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 crt-list /tmp/haproxy/ssl/632498ac5e6503.54058036.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6324642dba2f56.47189800.txt)]

# Backend: wiki ()
backend wiki
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server apache03 192.168.254.4:80

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Zoneminder_backend ()
backend Zoneminder_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server ZoneMinder 192.168.254.22:80 source 192.168.254.2

# Backend: Nextcloud_Backend ()
backend Nextcloud_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server owncloud 192.168.254.23:443 ssl verify none

# Backend: KH_backend ()
backend KH_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server kh 192.168.10.50:80

# Backend: HA_backend ()
backend HA_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server ha 192.168.0.51:80

# Backend: HASS_backend ()
backend HASS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hass 192.168.254.13:8123

# Backend: BITWARDEN_backend ()
backend BITWARDEN_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server bitwarden 192.168.254.4:81

# Backend: Webmin_Backend ()
backend Webmin_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server sshgateway 192.168.254.34:10000 ssl verify none

# Backend: OPENVPN_backend ()
backend OPENVPN_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server OPENVPN_server 127.4.4.3:10194

When I try to connect to the OpenVPN server on port 443 I get this in the haproxy logs:
Code: [Select]
2022-09-23T13:16:23 Informational haproxy 1.2.3.4:43265 [23/Sep/2022:13:16:23.981] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
2022-09-23T13:16:23 Informational haproxy Connect from 1.2.3.4:43265 to 4.3.2.1:443 (0_SNI_frontend/TCP)
2022-09-23T13:16:19 Informational haproxy 1.2.3.4:43264 [23/Sep/2022:13:16:19.866] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
2022-09-23T13:16:19 Informational haproxy Connect from 1.2.3.4:43264 to 4.3.2.1:443 (0_SNI_frontend/TCP)

Any ideas?
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 23, 2022, 02:26:19 pm
I followed the tutorial and added the necessary config to enable OpenVPN on port 443 but somehow it always keeps sending everything to the SSL_Backend.

Any ideas?

Configure your SNI_frontend like below and it should work.

Code: [Select]
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    option log-separate-errors
    option tcplog
    # ACL: OPENVPN_condition
    acl acl_6143a3ff7e6bf2.30491250 req_ssl_hello_type 1

    # ACTION: OPENVPN_rule
    use_backend OPENVPN_backend if !acl_6143a3ff7e6bf2.30491250
    # WARNING: pass through options below this line
    tcp-request inspect-delay 5s
    tcp-request content accept if !{ req_ssl_hello_type 1 }
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mozart on September 23, 2022, 02:56:22 pm
Thanks, I updated my frontend config:
Code: [Select]
# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_632625326b34a3.00256787 req_ssl_hello_type 1

    # ACTION: Openvpn_map-rule
    use_backend OPENVPN_backend if acl_632625326b34a3.00256787
    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request content accept if !{ req_ssl_hello_type 1 }


I think it matches your example but I still get errors in the haproxy log:
Code: [Select]
2022-09-23T14:52:32 Informational haproxy 1.2.3.4:42250 [23/Sep/2022:14:52:32.904] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
2022-09-23T14:52:32 Informational haproxy Connect from 1.2.3.4:42250 to 4.3.2.1:443 (0_SNI_frontend/TCP)
2022-09-23T14:52:31 Informational haproxy Connect from 192.168.254.13:52460 to 192.168.254.1:443 (0_SNI_frontend/TCP)

I looked a bit further and now I do see entries appearing in the OpenVPN logs so I will check those out first.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on September 24, 2022, 04:02:14 am
hi guys,

I am trying to setup a SSO using keycloak, openLDAP and other providers. I have it all working locally but now wanting to communicate to the LDAP server via ldap://ldap.mydomain.com instead of ldap://192.168.1.104:1389.

The only difference that i have tried is changing the mode on the backend server from HTTP to TCP. I just recieve a timeout error when testing.

Code: [Select]
2022-09-24 13:40:25,284 ERROR [org.keycloak.services] (executor-thread-39) KC-SERVICES0055: Error when connecting to LDAP: ldap.mydomain.com:389: javax.naming.CommunicationException: ldap.mydomain.com:389 [Root exception is java.net.SocketTimeoutException: connect timed out]

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     100000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62bbec3b1189e7.31090598 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62bbec3b1189e7.31090598

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62bbef8e4ab6b5.77631912.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62bbecc24b7a71.66647551.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: PRISM_backend ()
backend PRISM_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server PRISM_server 192.168.1.103:2342

# Backend: REQUEST_backend ()
backend REQUEST_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server REQUEST_server 192.168.1.104:5055

# Backend: LDAP_backend ()
backend LDAP_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server LDAP_server 192.168.1.104:1636 ssl verify none
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bunchofreeds on September 27, 2022, 01:22:56 am
Thanks @TheHellSite for an awesome tutorial and support to us all with OPNsense & HAproxy
I have a perfectly working HAproxy thanks to you with dynamic DNS, LetsEncrypt and multiple reverse proxied sites.

However, I have not changed the default port for OPNsense away from 443

I have three externally accessible sites all via my single public IP, which are directed to three different backend servers using HAproxy
These public urls are specified in the 'listening addresses' of my public service using their ww.example.com:443 format. I have not specified a broad 0.0.0.0:443 for example

What's your view on this setup and its possible conflict with the OPNsesne webserver. Am I correct to specify the url's and therefore HAproxy is only listening for these and not the url or IP for OPNsense?

Thanks for any advice


Title: Thanks very much for this post - I have a quick question if you don't mind
Post by: phib3r on September 28, 2022, 04:30:17 pm
My config is working great thanks to this post ! I do have a quick question that you may be able to expand on - requiring client certificates from a specific sub domain. I have tried out a few things, but all that seems to happen is that all the sub domains request client certs. I have tried to create a new public https frontend and have that look up from a different mapping file - but all that seems to happen is that the original https front end just now requires client certs. I would post my config - but it is currently working and to the letter what you have posted with just some more sub domains added. Any help or pointing in the right direction would be fantastic.

M
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phib3r on September 28, 2022, 10:53:16 pm
To the LDAP question above might be that LDAP is using UDP and not tcp
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phib3r on September 28, 2022, 10:53:50 pm
Might be - sorry
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on October 06, 2022, 07:53:44 pm
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 1000s
    timeout connect 1000s
    timeout server 1000s
    retries 3
    default-server init-addr libc,last
    default-server maxconn 5000

# autogenerated entries for ACLs

# userlists generated from groups
userlist Allowedusers
    user joel insecure-password XXX
    user mopidy insecure-password XXX
    # NOTE: UserlistAddUsers called with empty group data


# autogenerated entries for config in backends/frontends
userlist list_6245eeb66d3ab2.08976803
    # Origin: MOPIDY_backend
    user mopidy insecure-password XXX
    user joel insecure-password XXX
    # WARNING: skipping duplicate username (mopidy)


# autogenerated entries for stats




# Frontend: SNI_frontend (Listening on http&amp;https)
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind :::80 name :::80
    bind :::443 name :::443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 1000s

    # logging options

# Frontend: HTTP_frontend (Listening 127.0.0.1:80)
frontend HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    bind [::1]:80 name [::1]:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 1000s

    # logging options
    # ACL: NoSSL_condition
    acl acl_621d0b77c74989.24704837 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_621d0b77c74989.24704837

# Frontend: HTTPS_frontend (Listinging on 127.0.0.1:443)
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621d11c7cad951.61400293.certlist
    bind [::1]:443 name [::1]:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621d11c7cad951.61400293.certlist
    mode http
    option http-keep-alive
    default_backend WEBSERVER_backend
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/621d0c7054ddb7.46420139.txt)]
    # WARNING: pass through options below this line
      # Matrix client traffic
      acl matrix-host hdr(host) -i chat.XXX.ch chat.XXX.ch:443
      acl matrix-path path_beg /_matrix
      acl matrix-path path_beg /_synapse/client
   
      use_backend MATRIX_backend if matrix-host matrix-path

# Frontend: MATRIX_frontend (Listining * Port 8448)
frontend MATRIX_frontend
    bind *:8448 name *:8448 alpn h2,http/1.1 ssl  crt-list /tmp/haproxy/ssl/6256daae2378c2.17892750.certlist
    bind [::]:8448 name [::]:8448 alpn h2,http/1.1 ssl  crt-list /tmp/haproxy/ssl/6256daae2378c2.17892750.certlist
    mode http
    option http-keep-alive
    default_backend MATRIX_backend
    # tuning options
    timeout client 1000s

    # logging options
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
      http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
      http-request set-header X-Forwarded-For %[src]

# Frontend: SSH_frontend (Listining * Port 22)
frontend SSH_frontend
    bind *:22 name *:22 alpn h2,http/1.1
    bind [::]:22 name [::]:22 alpn h2,http/1.1
    mode tcp
    # tuning options
    timeout client 1000s

    # logging options

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: WEBSERVER_backend ()
backend WEBSERVER_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    acl restricted_page path_beg /wp-admin
    acl auth_ok http_auth(Allowedusers)
    http-request auth if restricted_page !auth_ok
   
    http-reuse safe
    server WEBSERVER_server 192.168.1.100:80 send-proxy-v2 check-send-proxy
    server WEBSERVER_server_ipv6 XXX:168:a774::2000:80 send-proxy-v2 check-send-proxy

# Backend: NAS_backend ()
backend NAS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server NAS_server 192.168.1.118:80
    server NAS_server_ipv6 XXX:168:a774::1000:80

# Backend: WEBSERVER_SSL_backend ()
backend WEBSERVER_SSL_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
   
    http-reuse safe
    server WEBSERVER_server_ssl 192.168.1.100:443
    server WEBSERVER_server_ssl_ipv6 XXX:168:a774::2000:443

# Backend: MOPIDY_backend ()
backend MOPIDY_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    acl auth_ok http_auth(list_6245eeb66d3ab2.08976803)
    http-request auth if !auth_ok
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    acl is_root path -i /
    redirect code 301 location /iris if is_root
    http-reuse safe
    server MOPIDY_server 192.168.1.100:6680

# Backend: MATRIX_backend ()
backend MATRIX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
      http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
      http-request set-header X-Forwarded-For %[src]
    http-reuse safe
    server MATRIX_server 192.168.1.100:8008
    server MATRIX_server_ipv6 XXX:168:a774::2000:8008

# Backend: KVM_backend ()
backend KVM_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server KVM_server 192.168.1.105:80

# Backend: SYNC_backend ()
backend SYNC_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
      http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
      http-request set-header X-Forwarded-For %[src]
    http-reuse safe
    server SYNC_server 192.168.1.100:5050

# Backend: ROUTER_SSH_backend ()
backend ROUTER_SSH_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server ROUTER_SSH_Server 192.168.1.1:22
    server ROUTER_SSH_Server_ipv6 XXX:168:a774::1000:22

# Backend: NAS_SSH_backend ()
backend NAS_SSH_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server NAS_server_ipv6 XXX:168:a774::1000:80
    server NAS_SSH_server 192.168.1.118:22

# Backend: KVM_SSH_backend ()
backend KVM_SSH_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server KVM_SSH_server 192.168.1.105:22

# Backend: SERVER_SSH_backend ()
backend SERVER_SSH_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server SERVER_SSH_server 192.168.1.100:22
    server SERVER_SSH_server_ipv6 XXX:168:a774::2000:22

Could someone say why my SSH service points everytime to the Router and doesn't split to ssh, ssh.kvm, ssh.server, ssh.nas?

My map file looks like
Code: [Select]
#public access subdomains
flood WEBSERVER_backend
kvm KVM_backend
nas WEBSERVER_backend
grafana WEBSERVER_backend
phpmyadmin WEBSERVER_backend
speedtestserver WEBERSERVER_backend
cloud NAS_backend
dav NAS_backend
stefan NAS_backend
mopidy MOPIDY_backend
git WEBSERVER_backend
chat MATRIX_backend
admin WEBSERVER_backend
sync SYNC_backend
ssh.nas NAS_SSH_backend
ssh.server SERVER_SSH_backend
ssh ROUTER_SSH_backend
ssh.kvm KVM_SSH_backend

Is a frontend for port 22 necessary?
Thanks for advices.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on October 10, 2022, 06:01:41 pm
Just think a little bit deeper:
I'm afraid SSH_frontend will never able to bind if your router is 192.168.1.1
As it always using port 22, you will never able to take port 22 with HAProxy.
Thus, everything related to SSH_frontend will never work until you change SSH of your router to different port
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on October 25, 2022, 07:10:49 pm
Ok. I changed OPNsene Port to 222 and SSH_frontend to 22.

Now I can access all my clients over haproxy with ssh,ipv4 and port 22.

Thanks for the Input.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meelokun on November 08, 2022, 09:02:33 am
deleted post
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: techsolo12 on November 10, 2022, 01:46:14 pm
Hello Guys!

Today its my first post here at this forum. At first @TheHellSite THANK YOU for your tutorial it helps my a lot! Before i used nginx proxy manager which was a lot easier than haproxy :)

I had one for my big problem and need the help from you all, please. I want to configure vaultwarden with websocket support in haproxy. The normal redirect to vaultwarden is no problem, but to add websocket support is still driven my crazy!

https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples (https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples)
Here are some examples how the proxy setup should, but i dont understand were my problem is.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_636976fd9d4d71.97561865 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_636976fd9d4d71.97561865

# Frontend: 1_HTTPS_frontend (Listening on 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/636aad8d3cbe18.58884679.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: nc_carddav
    acl acl_636ba4e5b6aa82.28881573 path_end -i /.well-known/carddav
    # ACL: nc_caldav
    acl acl_636ba2d9f14933.27250118 path_end -i /.well-known/caldav
    # ACL: vw_ws_acl01_condition
    acl acl_636c2f2b5accd9.55827620 path_beg -i /notifications/hub
    # ACL: vw_ws_acl02_condition
    acl acl_636cc909734817.72974823 path_beg -i /notifications/hub/negotiate
    # ACL: vw_ws_acl03_condition
    acl acl_636ccac64fcd74.27409543 path_beg -i /notifications/hub
    # ACL: vw_ws_acl04_condition
    acl acl_636ccae443ca48.73072029 path_beg -i /notifications/hub/negotiate

    # ACTION: nc_carddav_rule
    http-request redirect code 301 location /remote.php/dav if acl_636ba4e5b6aa82.28881573
    # ACTION: nc_caldav_rule
    http-request redirect code 301 location /remote.php/dav if acl_636ba2d9f14933.27250118
    # ACTION: PUBLIC_SUBDOMAINS-map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63653d33935cd3.47503593.txt)]
    # ACTION: vw_ws_acl01_rule
    use_backend vw_backend if !acl_636c2f2b5accd9.55827620
    # ACTION: vw_ws_acl02_rule
    use_backend vw_backend if acl_636cc909734817.72974823
    # ACTION: vw_ws_acl03_rule
    use_backend vw_ws_backend if acl_636ccac64fcd74.27409543
    # ACTION: vw_ws_acl04_rule
    use_backend vw_ws_backend if !acl_636ccae443ca48.73072029

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: office_backend (Onlyoffice)
backend office_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server office_server 10.10.20.8:80

# Backend: vw_backend (Vaultwarden)
backend vw_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server vw_server 10.10.20.7:80

# Backend: mc_backend (Minecraft Server)
backend mc_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server mc_server 10.10.40.4:80

# Backend: cloud_backend (Nextcloud01)
backend cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server cloud_server 10.10.20.5:80

# Backend: demo_backend (Nextcloud02)
backend demo_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server demo_server 10.10.20.6:80

# Backend: kunden_backend (Nextcloud03)
backend kunden_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server kunden_server 10.10.20.11:80

# Backend: vw_ws_backend (Vaultwarden Websocket)
backend vw_ws_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server vw_ws_server 10.10.20.7:3012



# statistics are DISABLED

Code: [Select]
2022-11-10T13:33:48 Informational haproxy Connect from 10.10.10.239:54010 to PUBLICIP:443 (1_HTTPS_frontend/HTTP)
2022-11-10T13:33:48 Informational haproxy Connect from 10.10.10.239:54010 to PUBLICIP:443 (1_HTTPS_frontend/HTTP)
2022-11-10T13:33:48 Informational haproxy Connect from 10.10.10.239:54010 to PUBLICIP:443 (1_HTTPS_frontend/HTTP)
2022-11-10T13:33:48 Informational haproxy Connect from 10.10.10.239:54010 to PUBLICIP:443 (0_SNI_frontend/TCP)

With best regard,
techsolo12
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 10, 2022, 02:36:49 pm
Hello Guys!

Today its my first post here at this forum. At first @TheHellSite THANK YOU for your tutorial it helps my a lot! Before i used nginx proxy manager which was a lot easier than haproxy :)

I had one for my big problem and need the help from you all, please. I want to configure vaultwarden with websocket support in haproxy. The normal redirect to vaultwarden is no problem, but to add websocket support is still driven my crazy!

Sorry, but out of scope of this tutorial. Please ask in the official HAProxy forum.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meelokun on November 11, 2022, 02:52:45 am
Alright! Im back with a clean baseline. Lets try this again... And apologies if anything below sounds dumb, im still faily new to understanding Certs, and how wildcard domains work. (although i had it working in NGINX with another domain of mine, i want to upgrade from a B score to A+ Score, and part of that was moving to HAProxy) - also i have no interest in utilizing 2nd-level-subdomains. I will only be using 1st-level-subdomains for all my services.

Heres where im getting stuck (Part 3, Step 6): Cert Validation - fails
Since i use Cloudflare, I tried my best to adapt your DynDNS setup to Cloudflare (DynDNS confirmed working)


Cloudflare account DNS Management

(https://i.imgur.com/5Fi4xnT.png)

Dynamic DNS Settings
(https://i.imgur.com/iW3UIgY.png)

ACME Settings
(https://i.imgur.com/Sf7cWdM.png)
(https://i.imgur.com/kWNSVLy.png)
(https://i.imgur.com/cB2MevV.png)

What am i doing wrong?
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: robbie11r1 on November 11, 2022, 04:33:04 am
Good Evening,

Thank you for the excellent tutorial! I have setup HAProxy + Wildcard Certificates following this tutorial, but am experiencing a 503 error when trying to access jellyfin.example.com using the setup here. I do not have any of these services accessible from outside my network (I.E. no WAN 443/80 ports open) and am only interested in being able to access "jellyfin.example.com" --> 192.168.5.88:8096 with a valid Let's Encrypt cert from inside my network. Right now when I access jellyfin.example.com, my browser shows a valid cert but throws a 503 error. Double and triple checking the tutorial has left me without any further options to explore.

Does anyone have any suggestions where I can look? Thank you very much in advance!

Unbound Host Override: *.example.com --> 192.168.5.1 (OpnSense LAN IP)


Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: HTTPS_Frontend (Listening on 127.4.4.3:443)
frontend HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63687fb14df779.98297035.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: Local_Only_Subnet
    acl acl_63687bc7cf9331.77802781 src 192.168.5.0/24

    # ACTION: Local_subdomain_map
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63687b6d92a544.19528694.txt)] if acl_63687bc7cf9331.77802781

# Frontend: SNI_Frontend (Listening to 0.0.0.0:80; 0.0.0.0:443)
frontend SNI_Frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTP_Frontend (Listening on 127.4.4.3:80)
frontend HTTP_Frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_63687974def2f4.69235454 ssl_fc

    # ACTION: HTTPtoHTTPS
    http-request redirect scheme https code 301 if !acl_63687974def2f4.69235454

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Jellyfin_Backend ()
backend Jellyfin_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Jellyfin 192.168.5.88:8096 ssl verify none

# Backend: TPLink_Backend ()
backend TPLink_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server TPLinkSwitch 192.168.5.5:80

# Backend: Opnsense_Backend ()
backend Opnsense_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Opnsense 192.168.5.1:8100 ssl verify none

# Backend: Proxmox_Backend ()
backend Proxmox_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Proxmox 192.168.5.201:8006 ssl verify none



# statistics are DISABLED

Map File:
Code: [Select]
jellyfin Jellyfin_Backend
tplink TPLink_Backend
opnsense Opnsense_Backend
proxmox Proxmox_Backend
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 11, 2022, 11:11:04 am
Alright! Im back with a clean baseline. Lets try this again... And apologies if anything below sounds dumb, im still faily new to understanding Certs, and how wildcard domains work. (although i had it working in NGINX with another domain of mine, i want to upgrade from a B score to A+ Score, and part of that was moving to HAProxy) - also i have no interest in utilizing 2nd-level-subdomains. I will only be using 1st-level-subdomains for all my services.

Heres where im getting stuck (Part 3, Step 6): Cert Validation - fails
Since i use Cloudflare, I tried my best to adapt your DynDNS setup to Cloudflare (DynDNS confirmed working)

Out of scope of this tutorial!
As a hint: Your certificate configuration looks fine. Probably wrong DNS-01 settings. Please search on Google for OPNsense Cloudflare ACME guide.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 11, 2022, 11:15:15 am
Good Evening,

Thank you for the excellent tutorial! I have setup HAProxy + Wildcard Certificates following this tutorial, but am experiencing a 503 error when trying to access jellyfin.example.com using the setup here. I do not have any of these services accessible from outside my network (I.E. no WAN 443/80 ports open) and am only interested in being able to access "jellyfin.example.com" --> 192.168.5.88:8096 with a valid Let's Encrypt cert from inside my network. Right now when I access jellyfin.example.com, my browser shows a valid cert but throws a 503 error. Double and triple checking the tutorial has left me without any further options to explore.

Does anyone have any suggestions where I can look? Thank you very much in advance!

Unbound Host Override: *.example.com --> 192.168.5.1 (OpnSense LAN IP)

Map File:
Code: [Select]
jellyfin Jellyfin_Backend
tplink TPLink_Backend
opnsense Opnsense_Backend
proxmox Proxmox_Backend


Since you get a 503 with the right certificate your overall setups seems to be working.

Is only Jellyfin not working or does it affect all services mentioned in the map file?
If it only affects Jellyfin: You might want to check the server settings for it in regards of SSL (yes or no). By default Jellyfin comes WITHOUT a self-signed SSL certificate so you will want to disable SSL for this server in HAProxy. Also from what I know the Jellyfin port 8096 is HTTP so no SSL. But you have SSL enabled in the server settings.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: robbie11r1 on November 11, 2022, 01:01:32 pm
@TheHellSite, I saw some other posts over thr last year with similar problems to me (503 error) and solved it by de-selecting SSL in the Real Server section. I tried that with Jellyfin and nothing changed for me (still same 503 with valid cert).

I get the same error for every service, no matter if SSL is selected or not. Is there anything else I should be changing aside from just de-selecting SSL?

Thanj you again!

EDIT: As of this morning (and changing nothing), everything started to work. Not sure what did it, but awesome tutorial and thanks for the feedback/troubleshooting with everyone, helped immensely.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meelokun on November 12, 2022, 11:19:38 am
Im up and running (Everything works!) I Figured out my Cloudflare DNS challenge issue (you have to make sure to get the Token that is displayed AFTER you create a Cloudflare API Token for the Zone.DNS read and Zone.DNS Edit... I mistakenly thought it was the Global ID Key..)

Now the only thing i cant get to work is the Unbound DNS Override.

You stated
Quote
If you are running all of your services on your 1st level subdomain "your_subdomain.dedyn.io" than you will just need to override this one.

Since im utilizing a wildcard, i figured it should work this way, so that any subdomain i enter, will be redirected to HAProxy's SNI_Frontend. And since its listening on 0.0.0.0, i figured the virtual IP should work - i also tried the Firewalls IP address with no luck.

(https://i.imgur.com/tU7X9xH.png)

And yes - the virtual ip is set to loopback.

Also - the Unbound DNS Overrides section looks different, now theres 2 tabs (Host Overrides), a main entry and an aliases entry,
(https://i.imgur.com/nhr5gHq.png)
then a (Domain Overrides) tab.
(https://i.imgur.com/1TmbDeX.png)
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 12, 2022, 12:04:40 pm

You stated
Quote
If you are running all of your services on your 1st level subdomain "your_subdomain.dedyn.io" than you will just need to override this one.

Since im utilizing a wildcard, i figured it should work this way, so that any subdomain i enter, will be redirected to HAProxy's SNI_Frontend. And since its listening on 0.0.0.0, i figured the virtual IP should work - i also tried the Firewalls IP address with no luck.

(https://i.imgur.com/tU7X9xH.png)

My tutorial clearly states that you have to use the OPNsense LAN IP in the DNS override.

How on earth would the lan devices be able to talk to a virtual IP created on the loopback device of the OPNsense. You should Google what the localhost is.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: techsolo12 on November 13, 2022, 04:08:24 pm
Hello Guys!

Today its my first post here at this forum. At first @TheHellSite THANK YOU for your tutorial it helps my a lot! Before i used nginx proxy manager which was a lot easier than haproxy :)

I had one for my big problem and need the help from you all, please. I want to configure vaultwarden with websocket support in haproxy. The normal redirect to vaultwarden is no problem, but to add websocket support is still driven my crazy!

Sorry, but out of scope of this tutorial. Please ask in the official HAProxy forum.

Hello Guys!

Unfortunally nobody in the other forums can help me with this situation. Anybody in vaultwarden or haproxy forum. Is here nobody who had vaultwarden getting worked? :(

With best regards;
techsolo12
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 14, 2022, 12:53:41 am
Hello Guys!

Unfortunally nobody in the other forums can help me with this situation. Anybody in vaultwarden or haproxy forum. Is here nobody who had vaultwarden getting worked? :(

With best regards;
techsolo12

As I already explained a dozen times: This tutorial is about getting HAProxy up and running with a LE cert. I won't be helping here with service specific settings, issues and what not!

That beeing said... I am also running Vaultwarden and it is working fine. However I don't use its websockets as I simply have no need for the additional features.

So please find this one out by yourself and feel free to post the solution here. :)
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meelokun on November 14, 2022, 06:50:25 am
apologies, my tired eyes at 4am missed this part

  • But if you would like to do it my way then you will need to create a virtual IP that is in a different subnet than any of your other networks. Preferably you would chose an IP that belongs to the localhost subnet in order to avoid IP conflicts in your local network.

i also misunderstand that setting a virtual IP that has a "loopback" does not automatically mean that it serves to "Loopback" to the firewall. that one was on me.

I updated the virtual ip to be 127.4.4.3 (mirroring your setup)

and updated SSL_Server to 127.4.4.3

HTTPS and HTTP Front Ends to listen on 127.4.4.3.

Updated unbound overide IP value to 10.0.1.1 (Firewall/OPNSense IP)

Restarted HAProxy - and its still not working. i wonder what i did wrong...

Update: Rebooted the firewall and that fixed it...
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 02, 2022, 06:55:54 am
Okay, I've been through the instructions at least 3 times and cannot find why it's not working. Can someone please take a look? Other than it being currently disabled, obviously.

Firewall rule is:
IPv4 TCP Src* Port* Dest WAN address Port AliasforHTTP/HTTPS Gateway* Schedule*

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

#
# NOTE: HAProxy is currently DISABLED
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch 1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend ()
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 15m

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_63859d8c6a7b81.10799804 ssl_fc

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_63859d8c6a7b81.10799804

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6385a4c7e68d06.81674833.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63859df5259306.89264162.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: homeassistant_backend ()
backend homeassistant_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server HomeAssistant 192.168.0.3:8123 check inter 30s port 8123

# Backend: web_backend ()
backend web_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server web 192.168.0.4:80

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: factorio_backend ()
backend factorio_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server factorio 192.168.0.17:80

# Backend: jira_backend ()
backend jira_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server jira 192.168.0.20:80

# Backend: meshcentral_backend ()
backend meshcentral_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server meshcentral 192.168.0.41:443

# Backend: nextcloud_backend ()
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nextcloud 192.168.0.38:443 ssl alpn h2,http/1.1 verify none


# statistics are DISABLED

And the mapping file, which I have tried with the full FQDN and without the periods as well.
Code: [Select]
# public access subdomains
hass. homeassistant_backend
factorio. factorio_backend
jira. jira_backend
mesh. meshcentral_backend
nextcloud. nextcloud_backend
web_backend

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 03, 2022, 09:27:22 pm
Okay, I've been through the instructions at least 3 times and cannot find why it's not working. Can someone please take a look? Other than it being currently disabled, obviously.

Firewall rule is:
IPv4 TCP Src* Port* Dest WAN address Port AliasforHTTP/HTTPS Gateway* Schedule*

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

#
# NOTE: HAProxy is currently DISABLED
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch 1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend ()
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 15m

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_63859d8c6a7b81.10799804 ssl_fc

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_63859d8c6a7b81.10799804

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6385a4c7e68d06.81674833.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63859df5259306.89264162.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: homeassistant_backend ()
backend homeassistant_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server HomeAssistant 192.168.0.3:8123 check inter 30s port 8123

# Backend: web_backend ()
backend web_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server web 192.168.0.4:80

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: factorio_backend ()
backend factorio_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server factorio 192.168.0.17:80

# Backend: jira_backend ()
backend jira_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server jira 192.168.0.20:80

# Backend: meshcentral_backend ()
backend meshcentral_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server meshcentral 192.168.0.41:443

# Backend: nextcloud_backend ()
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nextcloud 192.168.0.38:443 ssl alpn h2,http/1.1 verify none


# statistics are DISABLED

And the mapping file, which I have tried with the full FQDN and without the periods as well.
Code: [Select]
# public access subdomains
hass. homeassistant_backend
factorio. factorio_backend
jira. jira_backend
mesh. meshcentral_backend
nextcloud. nextcloud_backend
web_backend

1. No real error description. What is the error? What is not working?

2. Your map file seems off. Remove the dots after each service, I didn't have them in my example config so why do you have them in yours?

3. At the end of your mapfile, you have "web_backend" without any matching scheme before it.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 03, 2022, 10:44:19 pm
The dots were just an attempt to see if the default backend was an issue by listing my default in the map, but I wanted to be sure nothing else would conflict with the same 'startswith'. web_backend had my domain and I missed that it was left empty when I cleaned up PII. I have went through the config again, removing the periods and defining the default on the mapping rule again.

Rejected connections every time, but I know the traffic is making it because my port forward to an internal server I am using with Nginx Proxy Manager (and am seeking to replace with this setup) works when I disable HAProxy and put back its port forward. I've completely removed the port forward and rebooted the router to ensure that it isn't interfering. My Opnsense WebGUI port was already changed to 81.

If I attempt to browse to my IP from outside my network, http shows ERR_EMPTY_RESPONSE in Chrome, https shows ERR_CONNECTION_CLOSED.

One thing I find really odd is I'm not getting anything in the log file until I disable the service, then I see the stopping messages. I setup a health check that's working and writes to the log, but that was just more troubleshooting to be sure it was able to see my internal service from the router.


Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch 1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend ()
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 15m

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_63859d8c6a7b81.10799804 ssl_fc

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_63859d8c6a7b81.10799804

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6385a4c7e68d06.81674833.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63859df5259306.89264162.txt,web_backend)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: homeassistant_backend ()
backend homeassistant_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server HomeAssistant 192.168.0.3:8123

# Backend: web_backend ()
backend web_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server web 192.168.0.4:80

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: factorio_backend ()
backend factorio_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server factorio 192.168.0.17:80

# Backend: jira_backend ()
backend jira_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server jira 192.168.0.20:80

# Backend: meshcentral_backend ()
backend meshcentral_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server meshcentral 192.168.0.41:443

# Backend: nextcloud_backend ()
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nextcloud 192.168.0.38:443 ssl alpn h2,http/1.1 verify none

# Backend: gallery_backend ()
backend gallery_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server gallery 192.168.0.12:80



# statistics are DISABLED

SNI_Mapping
Code: [Select]
# public access subdomains
hass homeassistant_backend
factorio factorio_backend
jira jira_backend
mesh meshcentral_backend
nextcloud nextcloud_backend
gallery gallery_backend
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 04, 2022, 01:32:38 am
If I attempt to browse to my IP from outside my network, http shows ERR_EMPTY_RESPONSE in Chrome, https shows ERR_CONNECTION_CLOSED.

If you don't even get any 503s with a blank white Page and the HAProxy Log is not indicating any traffic, then your firewall rule is configured wrong.

Also if you are not willing to share the HAProxy log then I am unable to help. You have to set it to "Informational" in the top right corner!
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 04, 2022, 01:52:45 am
If you don't even get any 503s with a blank white Page and the HAProxy Log is not indicating any traffic, then your firewall rule is configured wrong.

Also if you are not willing to share the HAProxy log then I am unable to help. You have to set it to "Informational" in the top right corner!


Shouldn't the HAProxy log show startups as well? This is fresh after a reboot (empty log), restarting the HAProxy service from System, Diagnostics, Services (still empty), then unchecking Enable HAProxy and applying (Stop messages appear), then rechecking Enable HAProxy and applying (nothing new added).

Confirmed haproxy is listening. I've also tried the DNS redirection to my opnsense internal IP with the same results from inside the network.

Code: [Select]
root@OPNsense:~ # sockstat -l | grep '443\|80'
www      haproxy    3539  4  tcp4   *:443                 *:*
www      haproxy    3539  5  tcp4   *:80                  *:*
www      haproxy    3539  6  tcp4   127.4.4.3:80          *:*
www      haproxy    3539  7  tcp4   127.4.4.3:443         *:*

I've also compared the firewall policy once again, and it is exactly like https://postimg.cc/VS3DKGPg other than I named my alias HTTP_HTTPS.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 04, 2022, 03:26:45 pm
Please just post the overview of YOUR WAN rules page and YOUR wan rule. Troubleshooting with pictures of my tutorial won't get you any further...
If you don't expose your WAN IP or public domain name then you have nothing to worry about.

Again since your HAProxy log is empty this means that there is NO traffic reaching HAProxy and maybe not even your firewall. You would have to check the firewall logs for this though.

Also is there another firewall / router placed before your OPNsense (double NAT)?
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 04, 2022, 05:19:18 pm
Ignore the HTTP_HTTPS Port Forward. I've been removing it and rebooting to ensure a clean test, but didn't want to mess with it this morning. The port forwards all work though, so the firewall is definitely receiving the traffic. There's an ATT Gateway in front of my router in passthrough mode.

Also grabbed a packet capture of the traffic, after disabling the port forward and confirming haproxy is bound to 80/443. Phone was off the network and packet capture was filtered to its public ip.

Code: [Select]
10:15:09.123024 IP PHONE.25700 > ROUTER.443: Flags [SEW], seq 642215500, win 65535, options [mss 1460,nop,wscale 12,sackOK,TS val 2478600365 ecr 0], length 0
10:15:09.123057 IP ROUTER.443 > PHONE.25700: Flags [S.E], seq 639098840, ack 642215501, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 3059434307 ecr 2478600365], length 0
10:15:09.141384 IP PHONE.25700 > ROUTER.443: Flags [.], ack 1, win 256, options [nop,nop,TS val 2478600384 ecr 3059434307], length 0
10:15:09.142758 IP PHONE.25700 > ROUTER.443: Flags [P.], seq 1:518, ack 1, win 256, options [nop,nop,TS val 2478600384 ecr 3059434307], length 517
10:15:09.142790 IP ROUTER.443 > PHONE.25700: Flags [.], ack 518, win 510, options [nop,nop,TS val 3059434328 ecr 2478600384], length 0
10:15:09.142818 IP ROUTER.443 > PHONE.25700: Flags [F.], seq 1, ack 518, win 514, options [nop,nop,TS val 3059434328 ecr 2478600384], length 0
10:15:09.161122 IP PHONE.25700 > ROUTER.443: Flags [.], ack 2, win 256, options [nop,nop,TS val 2478600404 ecr 3059434328], length 0
10:15:09.161149 IP PHONE.25700 > ROUTER.443: Flags [F.], seq 518, ack 2, win 256, options [nop,nop,TS val 2478600404 ecr 3059434328], length 0
10:15:09.161163 IP ROUTER.443 > PHONE.25700: Flags [.], ack 519, win 513, options [nop,nop,TS val 3059434346 ecr 2478600404], length 0
10:15:15.141831 IP PHONE.26438 > ROUTER.443: Flags [SEW], seq 3285634286, win 65535, options [mss 1460,nop,wscale 12,sackOK,TS val 4185299120 ecr 0], length 0
10:15:15.141883 IP ROUTER.443 > PHONE.26438: Flags [S.E], seq 4283526657, ack 3285634287, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 95186048 ecr 4185299120], length 0
10:15:15.160570 IP PHONE.26438 > ROUTER.443: Flags [.], ack 1, win 256, options [nop,nop,TS val 4185299139 ecr 95186048], length 0
10:15:15.161943 IP PHONE.26438 > ROUTER.443: Flags [P.], seq 1:518, ack 1, win 256, options [nop,nop,TS val 4185299139 ecr 95186048], length 517
10:15:15.161977 IP ROUTER.443 > PHONE.26438: Flags [.], ack 518, win 510, options [nop,nop,TS val 95186067 ecr 4185299139], length 0
10:15:15.162008 IP ROUTER.443 > PHONE.26438: Flags [F.], seq 1, ack 518, win 514, options [nop,nop,TS val 95186067 ecr 4185299139], length 0
10:15:15.181057 IP PHONE.26438 > ROUTER.443: Flags [.], ack 2, win 256, options [nop,nop,TS val 4185299159 ecr 95186067], length 0
10:15:15.181181 IP PHONE.26438 > ROUTER.443: Flags [F.], seq 518, ack 2, win 256, options [nop,nop,TS val 4185299159 ecr 95186067], length 0
10:15:15.181199 IP ROUTER.443 > PHONE.26438: Flags [.], ack 519, win 513, options [nop,nop,TS val 95186086 ecr 4185299159], length 0
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 04, 2022, 10:20:24 pm
Shouldn't the HAProxy log show startups as well? This is fresh after a reboot (empty log), restarting the HAProxy service from System, Diagnostics, Services (still empty), then unchecking Enable HAProxy and applying (Stop messages appear), then rechecking Enable HAProxy and applying (nothing new added).

HAProxy shouldn't even print a stop message in the haproxy log at all. Only if there are errors, f.e. misconfiguration of your firewall.

After enabling HAProxy and hitting "Apply" then waiting for 5sec and reloading the HAProxy settings page. Is there a green Play icon in the top right corner when you are on the HAProxy Settings page?

If not, then you have something misconfigured or another service is listening on the same ip:port.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 05, 2022, 12:21:54 am
Yes, the green play button is there.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 05, 2022, 01:16:17 am
Please reboot the firewall, then post an updated haproxy config export and haproxy log export (after trying to access your services using an FQDN) .
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 05, 2022, 01:29:26 am
Log is empty. Green play button is present.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 05, 2022, 01:40:36 am
I strongly believe it is not empty... AGAIN You have to set it to informational.
Post a screenshot of your PUBLIC_SUBDOMAINS_map-rule.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 05, 2022, 02:26:20 am
Here's my log page and settings and the map-rule.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 05, 2022, 03:55:45 pm
Based on this the traffic is clearly NOT reaching HAProxy. Why? I can't tell you.
You will have to troubleshoot here on your own. Your HAProxy config and wan firewall rule looks fine.

I can only imagine that you have some leftover port forwards or whatever that are intercepting the traffic.
If you enable the logging on the WAN "HAProxy" rule and go to Firewall --> Log Files --> Live View and filter for the WAN rule... You should see the traffic beeing green.
However this does not guarantee that any old port forward rules are interfering.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 05, 2022, 05:07:01 pm
Alright. Thanks for the attempt.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 06, 2022, 12:32:58 am
Well there has to be something wrong with either you ATT modem or you overall OPNsense settings.

Since the SNI_frontend is listening on all IPs, interfaces and 80+443 port (0.0.0.0:80+443) it should at least spit out something to the HAProxy logs when there is a connection to your OPNsense WAN IP on any of the two ports. Even if the SSL handshake or anything after it fails, the SNI_frontend will always make log entries.

If it doens't then there is something wrong with your OPNsense in general or some other network device in your WAN facing setup.

Take a look at my HAProxy log example. The first hit always goes to the SNI_frontend, from there to the SSL_server and is then catched by the HTTP(S)_frontend.
Code: [Select]
2022-12-06T00:23:39 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:34677 to OPNSENSE_WAN_IP:443 (1_HTTPS_frontend/HTTP)
2022-12-06T00:23:39 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:34677 to OPNSENSE_WAN_IP:443 (0_SNI_frontend/TCP)
2022-12-06T00:23:39 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:9659 to OPNSENSE_WAN_IP:443 (1_HTTPS_frontend/HTTP)
2022-12-06T00:23:39 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:9659 to OPNSENSE_WAN_IP:443 (0_SNI_frontend/TCP)
2022-12-06T00:23:03 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:62798 to OPNSENSE_WAN_IP:443 (1_HTTPS_frontend/HTTP)
2022-12-06T00:23:03 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:62798 to OPNSENSE_WAN_IP:443 (0_SNI_frontend/TCP)
2022-12-06T00:23:01 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:62797 to OPNSENSE_WAN_IP:443 (1_HTTPS_frontend/HTTP)
2022-12-06T00:23:01 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:62797 to OPNSENSE_WAN_IP:443 (0_SNI_frontend/TCP)

You really should consider testing if your HTTP(S) WAN rule is working properly (see my previous reply) and or if there is any other service on your OPNsense or network device in general messing with the data stream.

I am sadly unable to help here since the traffic is not hitting your firewall and or haproxy.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: vladnik on December 15, 2022, 01:33:46 pm
Hey all,

first of all, thank you for the amazing guide @TheHellSite, I have set it up over a year ago and everything works like a charm.

Today, I have a small problem. I have a Mikrotik switch that can only be accessed via Port 80 and requires HTTP basic auth.
I have setup my backend and map file like always, the site is reachable fine, however I cannot login. I enter my credentials and press enter, and the dialog for entering my credentials just shows back up infinitely. The credentials are correct, if I go to the switch via IP-address I can log in normally.

(https://i.ibb.co/5WTzqPb/haproxy-mikrotik.png)

I have tried adding a user and password to 'User management' & ticking the box for Basic auth in the backend and selecting my user, no luck sadly. I'm pretty sure I just have to add an option somewhere to pass the auth header, however, I can't figure it out. Googling for the problem just leads to info on how to setup HAproxy to do basic auth, which I don't need... The logs also don't seem to show anything useful.

Anyone has any ideas? Thanks in advance.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 15, 2022, 02:09:56 pm
I have tried adding a user and password to 'User management' & ticking the box for Basic auth in the backend and selecting my user, no luck sadly. I'm pretty sure I just have to add an option somewhere to pass the auth header, however, I can't figure it out. Googling for the problem just leads to info on how to setup HAproxy to do basic auth, which I don't need... The logs also don't seem to show anything useful.

Anyone has any ideas? Thanks in advance.

The user management in HAProxy has nothing to do at all with any login forms of services that are behind HAProxy! You can use this to add a login form that pops up before the client can even connect to the service that is behind HAProxy. So unrelated to your issue.

Apart from that please ask in the official HAProxy forums about your issue since it is not related to my tutorial.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: vladnik on December 15, 2022, 02:21:49 pm
The user management in HAProxy has nothing to do at all with any login forms of services that are behind HAProxy! You can use this to add a login form that pops up before the client can even connect to the service that is behind HAProxy. So unrelated to your issue.

Figured as much. Was worth a try though.

Apart from that please ask in the official HAProxy forums about your issue since it is not related to my tutorial.

Alright, will do. Thanks anyways!
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: EuleMitKeule on December 25, 2022, 03:34:26 pm
I know this is not really part of the (great) tutorial, but I wanted to ask if it is possible to change the SSL certificate that is being used based on whether the traffic is local or public. I already have setup the rule for subdomains that are only accessible from local IPs.

The reason for this is that I want to enable Full (Strict) mode in Cloudflare. That means I have to use the Cloudflare Origin Server Certificate for public access to my HAProxy. I already uploaded the certificate to OPNsense and selected it along with the Let's Encrypt certificate for the HTTPS frontend. However it seems only the LE certificate is being used, so public access via Cloudflare fails. I looked for an HAProxy function that chooses a specific certificate, but it does not seem to exist.

Can anyone point me in the right direction?
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on December 25, 2022, 07:44:32 pm
Use different set of frontends to handle it.
For example, my current setup is
WAN_TCP_frontend->WAN_SSL_frontend, with 192.168.5.1, 192.168.6.1 as listening IP
LAN_TCP_frontend->LAN_SSL_frontend, with 192.168.7.1 and 192.168.8.1 as listening IP

WAN NAT port forward to 192.169.5.1
While Unbound DNS overwrite domain name to 192.168.7.1
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DeWilde on December 28, 2022, 01:50:44 pm
Hello,

and thank you for this great tutorial!
I managed to get the HAProxy running and I am able to access the desired services from the internet side.
But when i want to access them from the internal netwerk i am unable to reach them.
i tried the DNS override as explained but this does not work. I suppose i am doing something wrong here.
As far as I understood, i must create an "override" for each host i also want to reach internally and assign it the IP adres of the LAN interface of the OPNsense. Is that correct?
My default LAN interface has "192.168.10.1/24" so i created a host override f.e. "mynas.mydomain.com" pointing to 192.168.10.1.
I assume the HAProxy is also listening on the LAN interface?

thank you for your help!
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 28, 2022, 10:48:41 pm
I assume the HAProxy is also listening on the LAN interface?

If configured correctly, yes. However no way to know since...

thank you for your help!

No logs, no haproxy config export, no other details...
Until provided, no help.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DeWilde on December 29, 2022, 12:16:31 pm
Hi, attached you can find my haproxy.conf and map file.
I replicated your tutorial 1:1

The OPNsense firewall LAN IP is 192.168.10.1
for the 2 items in the map file i created a DNS override for those FQDN's pointing to 192.168.10.1

access from the internet works fine. The wildcard ssl cert is being used and the port redirection works and is not visible. (ex :55443 for the firewall interface and :5000 for the NAS)

internally the browser says "ERR_CONNECTION_TIMED_OUT" and nothing is displayed.

If you need more info or details i'll be happy to provide them to you.
thank you for your help!
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: matemrc on December 30, 2022, 12:47:08 am
Hey y'all,

First of all, truly appreciate the energy you put into this thorough guide @TheHellSite! Managed to make everything work, thank you!

There's one thing I can't sort out and your guide doesn't touch the topic either. HAProxy does not forward client IP to the server as expected. The server in this case is a Synology NAS which is only aware of the OPNSense IP if the connection is made via the reverse proxy.

Is this an expected behavior or could I have some settings wrong? My settings follow your guide entirely and the "X-Forwarded-For header" checkboxes are checked in both HTTP and HTTPS Frontends under the Public Services tab. Is it possible that the SNI Frontend doesn't forward the client IP "downstream" so the other two can't properly forward it?

I have found some forum posts that explain how this should be working in the HAProxy config file but I can't really translate that into the user interface as I lack the appropriate level of knowledge here.
https://access.redhat.com/solutions/3552581 (https://access.redhat.com/solutions/3552581)
https://serverfault.com/questions/722151/haproxy-how-to-append-client-ip-in-x-client-ip-and-x-forwarded-for-headers (https://serverfault.com/questions/722151/haproxy-how-to-append-client-ip-in-x-client-ip-and-x-forwarded-for-headers)
https://community.synology.com/enu/forum/1/post/150860?reply=488269 (https://community.synology.com/enu/forum/1/post/150860?reply=488269)
https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/haproxy_pass_clientip_to_webserver (https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/haproxy_pass_clientip_to_webserver)

I was hoping you guys could help me find a solution to this that can be achieved via the OPNSense UI.

Thank you in advance!
------------------

Update:
So anyone bumping into this issue on a Synology NAS, know it can be easily resolved. Apparently the server should be aware that a reverse proxy is forwarding the connections so it can resolve the original client IP.
The solution to my problem was posted here: https://www.reddit.com/r/synology/comments/mmubnv/tip_for_using_dsm_reverse_proxy_logging_correct/ (https://www.reddit.com/r/synology/comments/mmubnv/tip_for_using_dsm_reverse_proxy_logging_correct/)

Quote
In the DSM control panel, go to security, at the bottom is "trusted proxies". Add the dsm ip address HAProxy IP address, and boom! The correct external IP address is logged at a connection attempt and you will get notified about new login behavior if you have that turned on, and ip address blocking should now work if you have that turned on.
------------------

Update 2:
Apparently this above solution comes with a caveat, which may not make it a satisfying solution at all.
In order to establish connection between the client and the Synology the proxy must be allowed in the Synology's firewall. And as it appears even though the client IP is passed on the connection itself is made through the proxy's IP address. Even if the client's IP is specifically blocked, through the the proxy it is allowed to connect to every service the proxy is allowed to.

This effectively renders the the Synology Firewall useless.

I'm going to open a new topic for this issue for better visibility.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mintchipmadness on January 07, 2023, 11:49:57 pm
Hello and thank you very much for this guide. I am definitely learning a lot as I am setting up HAProxy on my opensense for public and local access. I have hopefully a quick question. Does this setup work to access the opnsense GUI or is that a special item that I need to setup? I am still troubleshooting but I'd thought I'd ask just in case. Thank you for your help.

Edit: Addition domain name structure information for my setup and some additional troubleshooting

Public Services: service.example.com
Internal Services: service.internal.example.com

I have been troubleshooting my setup and had a small breakthrough with my unbound settings. Under the general settings of unbound I needed to check "Do not register system A/AAAA records" to get the overrides to work when I ping. Before all my interfaces were getting registered under the firewall domain name causing a random interface IP to get pulled when I pinged opnsense.local.example.com of the firewall. Only certain IPs are allowed to access the GUI from my lan. Now my override works appropriately.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mike0000 on January 08, 2023, 01:06:23 am
TheHellSite, thanks a lot for all the work that you've put into this tutorial. I have followed every step of it and almost everything is working well.

One issue I am facing is that when I ping a local domain (e.g. opnsense.mydomain.com (router/fw box), nas.mydomain.com (qnap nas)) the IP gets resolved as my external WAN IP address.

When I direct my browser at one of my internal domains I'm not getting the same result as when I simply go to the corresponding IP.

I am not sure whether there is a misconfiguration in the HAProxy setup, or whether it is in fact unrelated to your tutorial. Below are the outputs I see from accessing two different subdomains from both my phone on cell service, as well as from internal wifi:

When accessing my OPNSense webui on opnsense.mydomain.com:55443 (internal IP 192.168.5.1):
When accessing my QNAP NAS on nas.mydomain.com (internal IP 192.168.5.60):
Code: [Select]
user@OPNsense:~ $ wget --save-headers http://nas.mydomain.com:8080
--2023-01-08 12:47:22--  http://nas.mydomain.com:8080/
Resolving nas.mydomain.com (nas.mydomain.com)... 192.168.5.60
Connecting to nas.mydomain.com (nas.mydomain.com)|192.168.5.60|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 580 [text/html]
Saving to: 'index.html'
index.html 100%[=====================================================================================>] 580  --.-KB/s    in 0s
2023-01-08 12:47:22 (139 MB/s) - 'index.html' saved [580/580]

When checking the response on https request an error comes (with the '--no-check-certificate' parameter yields the same result/output as the http request):

Code: [Select]
user@OPNsense:~ $ wget --save-headers https://nas.mydomain.com
--2023-01-08 13:01:21--  https://nas.mydomain.com/
Resolving nas.mydomain.com (nas.mydomain.com)... 192.168.5.60
Connecting to nas.mydomain.com (nas.mydomain.com)|192.168.5.60|:443... connected.
ERROR: cannot verify nas.mydomain.com's certificate, issued by 'CN=R3,O=Let\'s Encrypt,C=US':
  Unable to locally verify the issuer's authority.
To connect to nas.mydomain.com insecurely, use `--no-check-certificate'.

There seem to be two issues here: 1) there is a certificate error (I imported the acme/LE wildcard .crt and .key into the NAS).

But 2) more generally as observed with the pings thrown at the WAN IP rather than the correct internal IP the request from the browser is also being forwarded to the WAN (mydomain.com) - this is shown by wget on mydomain.com which also returns a 503, the same that the browser does:

Code: [Select]
user@OPNsense:~ $ wget --save-headers https://mydomain.com
--2023-01-08 13:13:59--  https://mydomain.com/
Resolving mydomain.com (mydomain.com)... 185.176.xxx.xxx [WAN IP]
Connecting to mydomain.com (mydomain.com)|185.176.xxx.xxx [WAN IP]|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2023-01-08 13:13:59 ERROR 503: Service Unavailable.

Any ideas or can you recommend any tools to do further troubleshooting or does anyone spot what the issue is?

The purpose of my setup is that all subdomains should only be accessible from the LAN or through VPN (this is set up correctly, I can VPN in via OpenVPN).

-------------------------

edit1: inserted code view of wget header response for nas.mydomain.com access - seems to be working in shell but not from browser

-------------------------
Many thanks
Michael

Attached the config files requested
- HAProxy Config Export
- HAProxy errors and/or log entries
- Details about setup: above, but happy to elaborate further if unclear
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: dirkscheck on January 08, 2023, 04:17:49 pm
@TheHellSide
First thank you for this wonderful guide. I learned a lot of and it helps to understand everything a little bit better.
At the moment I play around with these features to understand it even better.

My setup is a bit different so I need to play around with it and see what happens.

At the moment lets encrypt is working and HAProxy is configured. And here I have some questions in general:

1. Why do I have to open port 80 and 443 when using DNS-01 challenge ? I thought that is NOT needed and that was the reason why I choose DNS01
2. I DIDNT make the internal procedure BUT I can access the configured backend internally without any problems via the name !!! BUT
a. in Safari i get an ssl certificate (it is a self signed ssl certificate from my router)
b. in firefox it is still unsecured

Why is that ? If I have to send to you some information please let me know but I think that are general questions and no files / screenshots are needed or ?
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 09, 2023, 12:24:34 am
My default LAN interface has "192.168.10.1/24" so i created a host override f.e. "mynas.mydomain.com" pointing to 192.168.10.1.
I assume the HAProxy is also listening on the LAN interface?
Yes, your OPNsense LAN IP is the correct DNS Override target, as explained in the tutorial.
Yes, HAProxy is also listening on that interface since the SNI_frontend is listening on ALL IPs:Ports (0.0.0.0:0).

Hi, attached you can find my haproxy.conf and map file.
I replicated your tutorial 1:1

The OPNsense firewall LAN IP is 192.168.10.1
for the 2 items in the map file i created a DNS override for those FQDN's pointing to 192.168.10.1

access from the internet works fine. The wildcard ssl cert is being used and the port redirection works and is not visible. (ex :55443 for the firewall interface and :5000 for the NAS)

internally the browser says "ERR_CONNECTION_TIMED_OUT" and nothing is displayed.

If you need more info or details i'll be happy to provide them to you.
thank you for your help!

If access from external network is working fine, then your DNS override isn't taking effect.
Check this using nslookup/ping from any PC within your network.
Also check if you are actually using Unbound as your DNS resolver.

Then there is your HAProxy config...
Your SFINX and MYNAS both have the exact same IP:Port in the server settings, double check this.
BTW: I would never expose my OPNsense Web UI externally, but this is up to you.
Apart from that the HAProxy config looks fine.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 09, 2023, 12:33:54 am
TheHellSite, thanks a lot for all the work that you've put into this tutorial. I have followed every step of it and almost everything is working well.

One issue I am facing is that when I ping a local domain (e.g. opnsense.mydomain.com (router/fw box), nas.mydomain.com (qnap nas)) the IP gets resolved as my external WAN IP address.

If you also followed "Part 6 - Access from internal networks" of my guide this should be working.
If not, then check if the DNS overrides are working using nslookup/ping.

Oh and your HAProxy config looks wrong / incomplete.
There are no actual services configured (server+backend).

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 09, 2023, 12:46:21 am
@TheHellSide
First thank you for this wonderful guide. I learned a lot of and it helps to understand everything a little bit better.
At the moment I play around with these features to understand it even better.

My setup is a bit different so I need to play around with it and see what happens.

At the moment lets encrypt is working and HAProxy is configured. And here I have some questions in general:

1. Why do I have to open port 80 and 443 when using DNS-01 challenge ? I thought that is NOT needed and that was the reason why I choose DNS01
2. I DIDNT make the internal procedure BUT I can access the configured backend internally without any problems via the name !!! BUT
a. in Safari i get an ssl certificate (it is a self signed ssl certificate from my router)
b. in firefox it is still unsecured

Why is that ? If I have to send to you some information please let me know but I think that are general questions and no files / screenshots are needed or ?

1. Maybe because the internet works almost entirely over TCP ports 80(HTTP) and 443(HTTPS)?
So if we don't open WAN port 80+443 you will never be able to access your services over your WAN IP.
If you would have understood what you are doing, which you actually should have since I explained it pretty detailed in the tutorial, you would know that 80 is only open so any unencrypted traffic hitting port 80 gets redirected to the encryption required port 443.

2. I am assuming there is another router/firewall in front of your OPNsense that handles NAT, DNS and DHCP.
"My setup is a bit different so I need to play around with it and see what happens."
See my answer to your 3rd question.

3. "Why is that ? If I have to send to you some information please let me know but I think that are general questions and no files / screenshots are needed or ?"
WRONG.
You are stating that your setup is a bit different and then ask a question about some issues because you didn't follow the tutorial which is why you are probably having these issues in the first place.
I am pretty confused.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DeWilde on January 10, 2023, 09:46:16 am
@TheHellSide

MYNAS vs SFINX was for testing purpose because i was doubting if i configured something wrong. So i created MYNAS.
About OPNsense GUI being accessible from the internet, i completely understand your concern.
I was looking for a webinterface to test the config with, but indeed, not realy a good thing to do.

Some extra troubleshooting.
Interfaces: Diagnostics: DNS Lookup
Code: [Select]
host: feniks.domain.net
server: 192.168.10.1
response: A feniks.domain.net. 3600 IN A 192.168.10.200 192.168.10.1 0 msec

Interfaces: Diagnostics: Trace Route
Code: [Select]
# /usr/sbin/traceroute -w 2 -n  -m '18'  'feniks.domain.net'
traceroute to feniks.domain.net (192.168.10.200), 18 hops max, 40 byte packets
 1  192.168.10.200  0.787 ms  0.462 ms  0.475 ms

on network client:
Code: [Select]
Pinging feniks.domain.net [192.168.10.200] with 32 bytes of data:
Reply from 192.168.10.200: bytes=32 time<1ms TTL=64

Code: [Select]
Tracing route to feniks.domain.net [192.168.10.200]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  FENIKS.domain.net [192.168.10.200]

Trace complete.

so now i'm lost  ???
i'll try to find/set-up another internal website to test with.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 10, 2023, 08:34:29 pm
Some extra troubleshooting.
Interfaces: Diagnostics: DNS Lookup
host: feniks.domain.net
server: 192.168.10.1
response: A   feniks.domain.net. 3600 IN A 192.168.10.200   192.168.10.1   0 msec

so now i'm lost  ???
i'll try to find/set-up another internal website to test with.

Please use codeboxes when posting such results. This makes it a lot easier to read them!

Your issue is mostlikely related to misconfigured DNS overwrites or another DNS resolver that is controling the DNS replies in your local network.

Question 1: Is Unbound your only DNS resolver in your network or are you running something like piHole?

Question 2: Are your client devices (f.e. iPhone, Notebook, ...) using your OPNsense as their DNS resolver inside your network or are they configured to use something like Google DNS, Cloudflare DNS, AdGuard, ...?

Also please post/attach screenshots of your configured DNS overwrites.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DeWilde on January 20, 2023, 10:50:19 am
Hi,
as far as i am aware of, Unbound is my primary DNS resolver.
I do have Zenarmor installed on the OPNsense. But this is only a web filter, not a DNS resolver.

i checked my smartphone, laptop, kids computer, ... all of them are using OPNsense as there DNS resolver.

I attached some screenshots for you.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DeWilde on January 20, 2023, 10:50:51 am
and some more  ;)
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 20, 2023, 11:32:02 am
Actually I shouldn't offer this free support since you are clearly requesting it for business use.
That being said...

If I where you, I would remove "opnsense_04.jpg" asap from your post!
I just got direct access to your "opnsense.yourdomain.com" and was presented with the webinterface. Seriously don't expose it via HAProxy. Use WireGuard for this!!!

However, this screenshot also might point out your issue. Please try and write the hostnames in the host overrides in lowercase letters only.
Your "opnsense" override (lowercase) is working, but none of the others (all uppercase).
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: AlexisM on January 20, 2023, 07:48:33 pm
Many thank's for your tutorial. It real help me.

I'd need to throughout for my Synology Nas so I use the informations found in your topic , https://forum.opnsense.org/index.php?topic=18538.msg84958#msg84958 and https://forum.opnsense.org/index.php?topic=22630.msg118934#msg118934

I don't need to throughout admin console of my Nas but the services with port 433 (exemple https://drive.xxxx.synology.me, https://video.xxxx.synology.me etc.)

That I'm doing in completion of your tutorial (in order):

Then, when I'm going with my mobile device to "plex.mydomain.com", it use backend with SSL from OpnSense
And when I use "drive.xxxx.synology.me", it throughout the ssl and use SSL from my Synology NAS
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 20, 2023, 07:51:21 pm
Please provide haproxy config export
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: AlexisM on January 20, 2023, 10:56:09 pm
Oops. In fact, 'Create map file "throughout_ssl_map_domain" with content :' doesn't work (because SNI work on TCP).

Replace : "HAProxy plugin: Create map file "throughout_ssl_map_domain" with content : ..."
By Create Condition "SNI_synology_me", condition "SNI TLS extension ends with (TCP request content inspection), suffixe SNI ".synologe.me"

Change : HAProxy plugin: Create "Rule" (enter name ["sni_throughout_ssl-rule"], select condition "SNI_synology_me", execute function : "use backend", Backend :"Synology_backend"

*** haproxy config export :

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 500

# autogenerated entries for ACLs

# autogenerated entries for config in backends/frontends

# autogenerated entries for stats

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    # tuning options
    timeout client 30s

    # logging options
    option tcplog
    # ACL: traffic_ssl
    acl acl_63c840bdd3f440.07842774 req_ssl_hello_type 1
    # ACL: SNI_synology_me
    acl acl_63c826ed0527a7.29957165 req.ssl_sni -m end -i .synology.me

    # ACTION: request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5
    # ACTION: request_content_accept_ssl
    tcp-request content accept if acl_63c840bdd3f440.07842774
    # ACTION: sni_throughout_ssl-rule
    use_backend Synology_backend if acl_63c826ed0527a7.29957165

# Frontend: 1_HTTP_frontend (Listening on 127.74.0.0:80)
frontend 1_HTTP_frontend
    bind 127.74.0.0:80 name 127.74.0.0:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_63c813f73e3ac8.56482289 ssl_fc

    # ACTION: HTTP_to_HTTPS-rule
    http-request redirect scheme https code 301 if !acl_63c813f73e3ac8.56482289

# Frontend: 1_HTTPS_frontend (Listening on 127.74.0.0:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 127.74.0.0:443 name 127.74.0.0:443 accept-proxy ssl prefer-client-ciphers ssl-min-ver TLSv1.2 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63c817f31748b0.16739019.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63c814d7b1ebe0.58772734.txt)]

# Backend: OpnSense_backend (OpnSense Pool)
backend OpnSense_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Opnsense 192.168.74.1:444 ssl verify required ca-file /etc/ssl/cert.pem

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.74.0.0 send-proxy-v2 check-send-proxy

# Backend: Synology_backend ()
backend Synology_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server Synology 192.168.74.4:443 ##

# statistics are DISABLED
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 20, 2023, 11:44:15 pm
I am unable to help here. Please ask the people that already did the things you mentioned.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on January 21, 2023, 08:01:28 pm
I am unable to help here. Please ask the people that already did the things you mentioned.

Well, I guess he is not asking question, but to update how he manage redirecting package to NAS in TCP mode by adding conditions and rules for recognizing SNI

(I have read his config and compare with mine one, and guess his config should be working flawlessly)
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: AlexisM on January 24, 2023, 12:12:52 pm
yes  ;)
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brynjolm on February 01, 2023, 07:07:39 am
Hello again Mr.Hellsite the guide you provided has been rock solid for a year now. no hiccups or whatsoever. Im writing back in this post because i wanted to know exactly what you meant on NR.6 on the faq page as im interested in managing a traefik instance behind haproxy

Quote
How can we load balance TCP traffic that we don't want to get SSL offloaded, f.e. OpenVPN over TCP?
In my tutorial I only explain how to "redirect+load balance SSL offloaded traffic".
This is because I myself don't have (yet) the need to actually load balance any non SSL traffic.
However balancing non SSL traffic is pretty much the same as balancing SSL traffic.
You only have to make sure that your "NOSSLservice_rule" or "NOSSLservices_map-file_rule" is placed on the "SNI_frontend" instead of the "HTTPS_frontend" and that the backend that belongs to your

Would this kind of setup be applicable to do traefik behind haproxy? also what do you exactly mean by NOSSL_service_rule NOSSL_services_map_file_rule?
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 03, 2023, 05:06:38 pm
Please refer to this post about it. Be warned I can not provide help for this since I am not using such a setup.

https://forum.opnsense.org/index.php?topic=18538.msg84958#msg84958
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: tomdh76 on February 04, 2023, 08:19:08 pm
Thx you @TheHellSite for this tutorial. Unfortunately I cannot get it to work. I always get a "503 service unavaible status"

Here is my config
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 1_HTTP_Frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_Frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_63de5470175f22.54470191 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_63de5470175f22.54470191

# Frontend: 0_SNI_Frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_Backend

    # logging options

# Frontend: 1_HTTPS_Frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63de597c094f01.72503480.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63de5520a92049.75714996.txt)]

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: BITWARDEN_backend ()
backend BITWARDEN_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server BITWARDEN 192.168.2.55:80 ssl verify none

# Backend: CALIBRE_backend ()
backend CALIBRE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server CALIBRE 192.168.2.40:8083 ssl verify none



# statistics are DISABLED

If I use the internal addresses of the websites (192.168.2.40:8083 for example) I can login. But using the website.domain.com I get a 503 error. In the logs I see that the client is accessing the public Ip on port 443.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 04, 2023, 09:48:07 pm
Thx you @TheHellSite for this tutorial. Unfortunately I cannot get it to work. I always get a "503 service unavaible status"

If I use the internal addresses of the websites (192.168.2.40:8083 for example) I can login. But using the website.domain.com I get a 503 error. In the logs I see that the client is accessing the public Ip on port 443.

Please give further details on what is and whet it is not working.

Are you able to access your services via their domain name from a device outside of your local network?

Did you configure the DNS overrides for the local clients?

Also your Bitwarden server seems to be misconfigured are you sure it is serving SSL on the HTTP port? Also verify this for your other service.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: tomdh76 on February 04, 2023, 11:11:11 pm
Thx you @TheHellSite for this tutorial. Unfortunately I cannot get it to work. I always get a "503 service unavaible status"

If I use the internal addresses of the websites (192.168.2.40:8083 for example) I can login. But using the website.domain.com I get a 503 error. In the logs I see that the client is accessing the public Ip on port 443.

Please give further details on what is and whet it is not working.

Are you able to access your services via their domain name from a device outside of your local network?

Did you configure the DNS overrides for the local clients?

Also your Bitwarden server seems to be misconfigured are you sure it is serving SSL on the HTTP port? Also verify this for your other service.

Well, nothing is working, both not from within the local network and also not from outside.

I did configure the DNS override, but I first try to access the services from my mobile device.

I do not understand what you mean by serving SSL on the HTTP port. I think I followed your tutorial to the letter (except for using a Let's encrypt certificate by using cloudflare API from my domain)

Edit: I found it, I needed to uncheck the SSL tickbox in the real server settings. In your tutorial you have it checked and I saw in this forum someone else who had the same problem...

Thx alot!!
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 04, 2023, 11:30:25 pm
I do not understand what you mean by serving SSL on the HTTP port. I think I followed your tutorial to the letter (except for using a Let's encrypt certificate by using cloudflare API from my domain)

It is dangerous to do things like exposing services to the internet when you don't even understand this simple question from me!  :-\

Read step 9 of my FAQ. You should also really read the explanation of the "SSL checkbox" in the server setup page!
I bet you are not accessing your services by their local ip using HTTPS you are likely accessing them using HTTP.
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: tomdh76 on February 04, 2023, 11:32:41 pm
I do not understand what you mean by serving SSL on the HTTP port. I think I followed your tutorial to the letter (except for using a Let's encrypt certificate by using cloudflare API from my domain)

It is dangerous to do things like exposing services to the internet when you don't even understand this simple question from me!  :-\

Read step 9 of my FAQ. You should also really read the explanation of the "SSL checkbox" in the server setup page!
I bet you are not accessing your services by their local ip using HTTPS you are likely accessing them using HTTP.

Yes that was the problem...
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: xkpx on February 05, 2023, 09:54:51 am
Strange but 503 error appear to me as well.
I tested with apache,nodejs,wamp nothing worked. They i try to redirect to my switch to see if my windows is not the problem... but nope.
DynamicDNS is configured and working fine,
All gui redirections disabled and opnsense gui port changed.
Added firewall rule to WAN , and no additional LAN rules added ( it's almost fresh install )
Acme - generated fine cert via dns. ( 2/4/2023, 7:23:39 PM   OK   2/4/2023, 7:23:40 PM )

(https://i.ibb.co/rtPW5S4/rule.png)
Tested from external network via smartphone on cellular data.

One thing is that i am using proxmox to virtualize opnsense as "routerOnStick/Forbidden Router" and i pass two ports from quad NIC on promox-server as LAN/WAN for opnsense , and lan is going to dumb switch that transfer vlans/lan to rest of my house , so far not a single problem with that but maybe just maybe..

Code: [Select]
OPNsense 23.1_6-amd64
FreeBSD 13.1-RELEASE-p5
OpenSSL 1.1.1s 1 Nov 2022

Code: [Select]
HAProxy version 2.6.7-c55bfdb 2022/12/02 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.7.html
Running on: FreeBSD 13.1-RELEASE-p5 FreeBSD 13.1-RELEASE-p5 stable/23.1-n250372-c4ad069e50a SMP amd64
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:443 and 0.0.0.0:80)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option log-separate-errors
    option tcplog

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option log-separate-errors
    option httplog
    # ACL: NoSSL_condition
    acl acl_63dea06740dee5.93056632 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_63dea06740dee5.93056632

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63dea303583a84.37941891.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option log-separate-errors
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63dea0bbafdf17.31648976.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: XKP_backend ()
backend XKP_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server XKP_server 192.168.1.104:80 ssl verify none



# statistics are DISABLED

Code: [Select]
2023-02-05T10:45:38 Error haproxy ********:31073 [05/Feb/2023:10:45:38.668] 1_HTTPS_frontend~ 1_HTTPS_frontend/<NOSRV> -1/-1/-1/-1/0 503 217 - - SC-- 2/1/0/0/0 0/0 "GET https://[********:/favicon.ico HTTP/2.0"
Code: [Select]
root@firewall:~ # haproxy -f /usr/local/etc/haproxy.conf -d
Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.

Available filters :
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace
Using kqueue() as the polling mechanism.
00000000:0_SNI_frontend.accept(0004)=0014 from [********:31207] ALPN=<none>
00000001:1_HTTPS_frontend.accept(0007)=0017 from [********:31207] ALPN=h2
00000001:1_HTTPS_frontend.clireq[0017:ffffffff]: GET https://********/ HTTP/2.0
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: host: ********
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: cache-control: max-age=0
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-mobile: ?1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-platform: "Android"
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: save-data: on
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: dnt: 1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: upgrade-insecure-requests: 1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: user-agent: Mozilla/5.0 (Linux; Android 13; Mi 9T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-site: cross-site
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-mode: navigate
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-user: ?1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-dest: document
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-encoding: gzip, deflate, br
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-language: en-US,en;q=0.9
00000001:1_HTTPS_frontend.clicls[0017:ffff]
00000001:1_HTTPS_frontend.closed[0017:ffff]
00000002:1_HTTPS_frontend.accept(0007)=0017 from [********:31207] ALPN=h2
00000002:1_HTTPS_frontend.clireq[0017:ffffffff]: GET https://********/favicon.ico HTTP/2.0
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: host: ********
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: dnt: 1
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-mobile: ?1
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: save-data: on
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: user-agent: Mozilla/5.0 (Linux; Android 13; Mi 9T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-platform: "Android"
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-site: same-origin
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-mode: no-cors
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-dest: image
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: referer: https://********/
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-encoding: gzip, deflate, br
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-language: en-US,en;q=0.9
00000002:1_HTTPS_frontend.clicls[0017:ffff]
00000002:1_HTTPS_frontend.closed[0017:ffff]
00000000:SSL_backend.srvcls[0014:ffff]
00000000:SSL_backend.clicls[ffff:ffff]
00000000:SSL_backend.closed[ffff:ffff]
Code: [Select]
Interesting is that from opnsense ssh via wget i managed to download from server, and from windows too..
wget --save-headers http://ccc.network.ccc
This was with DNS override , but still not accessible by browser

image - https://i.ibb.co/bL8Wgbj/34.png
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: tomdh76 on February 05, 2023, 10:47:01 am
Strange but 503 error appear to me as well.
I tested with apache,nodejs,wamp nothing worked. They i try to redirect to my switch to see if my windows is not the problem... but nope.
DynamicDNS is configured and working fine,
All gui redirections disabled and opnsense gui port changed.
Added firewall rule to WAN , and no additional LAN rules added ( it's almost fresh install )
Acme - generated fine cert via dns. ( 2/4/2023, 7:23:39 PM   OK   2/4/2023, 7:23:40 PM )

Tested from external network via smartphone on cellular data.

One thing is that i am using proxmox to virtualize opnsense as "routerOnStick/Forbidden Router" and i pass two ports from quad NIC on promox-server as LAN/WAN for opnsense , and lan is going to dumb switch that transfer vlans/lan to rest of my house , so far not a single problem with that but maybe just maybe..

Code: [Select]
OPNsense 23.1_6-amd64
FreeBSD 13.1-RELEASE-p5
OpenSSL 1.1.1s 1 Nov 2022

Code: [Select]
HAProxy version 2.6.7-c55bfdb 2022/12/02 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.7.html
Running on: FreeBSD 13.1-RELEASE-p5 FreeBSD 13.1-RELEASE-p5 stable/23.1-n250372-c4ad069e50a SMP amd64
Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:443 and 0.0.0.0:80)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option log-separate-errors
    option tcplog

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option log-separate-errors
    option httplog
    # ACL: NoSSL_condition
    acl acl_63dea06740dee5.93056632 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_63dea06740dee5.93056632

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63dea303583a84.37941891.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option log-separate-errors
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63dea0bbafdf17.31648976.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: XKP_backend ()
backend XKP_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server XKP_server 192.168.1.104:80 ssl verify none



# statistics are DISABLED

Code: [Select]
2023-02-05T10:45:38 Error haproxy ********:31073 [05/Feb/2023:10:45:38.668] 1_HTTPS_frontend~ 1_HTTPS_frontend/<NOSRV> -1/-1/-1/-1/0 503 217 - - SC-- 2/1/0/0/0 0/0 "GET https://[********:/favicon.ico HTTP/2.0"
Code: [Select]
root@firewall:~ # haproxy -f /usr/local/etc/haproxy.conf -d
Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.

Available filters :
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace
Using kqueue() as the polling mechanism.
00000000:0_SNI_frontend.accept(0004)=0014 from [********:31207] ALPN=<none>
00000001:1_HTTPS_frontend.accept(0007)=0017 from [********:31207] ALPN=h2
00000001:1_HTTPS_frontend.clireq[0017:ffffffff]: GET https://********/ HTTP/2.0
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: host: ********
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: cache-control: max-age=0
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-mobile: ?1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-platform: "Android"
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: save-data: on
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: dnt: 1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: upgrade-insecure-requests: 1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: user-agent: Mozilla/5.0 (Linux; Android 13; Mi 9T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-site: cross-site
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-mode: navigate
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-user: ?1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-dest: document
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-encoding: gzip, deflate, br
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-language: en-US,en;q=0.9
00000001:1_HTTPS_frontend.clicls[0017:ffff]
00000001:1_HTTPS_frontend.closed[0017:ffff]
00000002:1_HTTPS_frontend.accept(0007)=0017 from [********:31207] ALPN=h2
00000002:1_HTTPS_frontend.clireq[0017:ffffffff]: GET https://********/favicon.ico HTTP/2.0
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: host: ********
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: dnt: 1
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-mobile: ?1
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: save-data: on
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: user-agent: Mozilla/5.0 (Linux; Android 13; Mi 9T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-platform: "Android"
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-site: same-origin
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-mode: no-cors
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-dest: image
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: referer: https://********/
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-encoding: gzip, deflate, br
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-language: en-US,en;q=0.9
00000002:1_HTTPS_frontend.clicls[0017:ffff]
00000002:1_HTTPS_frontend.closed[0017:ffff]
00000000:SSL_backend.srvcls[0014:ffff]
00000000:SSL_backend.clicls[ffff:ffff]
00000000:SSL_backend.closed[ffff:ffff]

I see also in your backend for XKP SSL is checked. Could you try by unchecking the SSL?
Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: xkpx on February 05, 2023, 10:56:55 am
Totally my mistake of course, i somehow manage to confuse myself to put in map file xkp XKP_Server, instead of xkp XKP_backend, damn i did this tutorial 3 times already today to realize it.
Also forgot in ACME , OCSP must staple.
So far everything works.