OPNsense Forum

English Forums => Tutorials and FAQs => Topic started by: TheHellSite on May 31, 2021, 01:06:11 pm

Title: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 31, 2021, 01:06:11 pm

Hello,

when I started implementing HAProxy in my network I couldn't find any complete and well written guide out there. I had to puzzle everything together from various websites.
So I thought I would save many of you a lot of time and provide my ultimate HAProxy on OPNsense tutorial. :)

This tutorial will show you how to configure HAProxy as a reverse proxy on OPNsense using wildcard certificates from Let's Encrypt.
It is going to be a step-by-step guide with images on how to set things up while also explaining why we set things up in a certain way.
I will try to make this as complete and detailed as possible.
If you think that there is anything wrong or missing, feel free to tell me about it and I will consider changing it.

---

Consider Supporting Free Resources

(https://storage.ko-fi.com/cdn/kofi1.png) (https://ko-fi.com/thehellsite) (https://i.postimg.cc/htR5wWr0/buymeabeer.png) (https://www.buymeacoffee.com/thehellsite)

If you find this tutorial helpful and it saves you a significant amount of time, please consider showing your appreciation by buying me a beer. Your contribution will be a valuable recognition of the time and effort I have put into creating this content and will help me to continue providing quality resources for others.

Thank you for your consideration,
TheHellSite

---

No More Free Support

Due to the increasing number of support requests I've been receiving, both directly in the topic and via DM, I regret to inform that I can no longer provide free assistance. Balancing my real job and personal life has become extremely challenging. While I genuinely want to help everyone resolve their issues to get things up and running smoothly, I find it difficult to allocate the necessary time without sacrificing my personal commitments.

In addition, it has come to my attention that some individuals seeking help are not thoroughly reading the provided tutorial or lack the fundamental knowledge of networking. This has been a recurring issue and has made the support process increasingly frustrating.

I sincerely appreciate your interest in my expertise and if you would like to receive my assistance, I am more than happy to provide you with the details via DM.

Thank you for your understanding in this matter,
TheHellSite

---

This configuration is tested to be working on OPNsense 24.1.x with the latest updates as of 20240207.

---

How To Ask For Help

When asking for help in the topic ALWAYS include the below information:

1. ALWAYS include the HAProxy Config Export.
2. ALWAYS include relevant HAProxy errors and/or log entries.
3. ALWAYS include details about your setup, your goal, the service, ... anything relevant to the issue.

Again, please note that I can no longer provide free support.

---

Changelog

• 20210603
  
  • Fixes and some enhancements

• 20210611
  
  • Implemented @sorano (https://forum.opnsense.org/index.php?topic=23339.msg111339#msg111339)'s enhancements

• 20210613
  
  • The tutorial is now using a wildcard CNAME record.
  • Enabled Proxy Protocol in the "SSL_backend", "HTTPS_frontend" and "HTTP_frontend" configuration so that the IPs of clients accessing HAProxy will now no longer be overwritten with the "SSL_server" IP.
    https://www.haproxy.com/blog/using-haproxy-with-the-proxy-protocol-to-better-secure-your-database/

• 20210729
  
  • Added an alias for the HAProxy ports and updated the WAN interface firewall rule with it.
    This leaves us with only one firewall rule instead of two and makes even more sense if one is using additional frontends on different ports.
    Thanks @_Alchemist_ (https://forum.opnsense.org/index.php?topic=23339.msg113979#msg113979) for the suggestion.
  • The tutorial is now using a map file instead of "condition + rule" for service configuration.

• 20210730
  
  • Added an explanation on how to configure local-access-only subdomains in HAProxy.

• 20210801
  
  • Added an explanation about using self-signed certificates for internal communication to the FAQ.

• 20211110
  
  • Updated the Let's Encrypt part since the service has been renamed to ACME client.

• 20220411
  
  • Updated the Let's Encrypt part because of changes to the wildcard certificate generation.

• 20220604
  
  • Updated the DynDNS part to use the newer "Dynamic DNS Client" (os-ddclient) plugin.
    The previous "Dynamic DNS Support" (os-dyndns) plugin will be removed with the release of OPNsense 22.7.x.

• 20220611
  
  • Changed the IP address of the "SSL_server" to one that belongs to the localhost subnet.
  • Updated the "NoSSL_condition" based on the HAProxy docs.

• 20230223
  
  • Updated Part 7 to avoid confusion.

• 20230304
  
  • Added Part 8 to hide the certificate on access by IP.

• 20240201
  
  • Simplified and improved Part 8. The "INVALID_SNI" certificate is no longer needed, instead we use the "strict-sni" parameter to achieve an even better result.
    Thanks @mnaiman (https://github.com/opnsense/plugins/issues/3779#issuecomment-1920084820) for the suggestion.
    See here for instructions on how to update your HAProxy configs (https://forum.opnsense.org/index.php?topic=23339.msg188524#msg188524).
  • Updated Part 5 - OCSP updates are now built into HAProxy. No external Cron job is necessary anymore.
    Everyone please take note of the warning from @vladnik (https://forum.opnsense.org/index.php?topic=23339.msg188872#msg188872).
    

---

Current Ciphers and Cipher Suites for a 100% A+ SSLLabs rating

Last updated/verified on 20230223 using Mozilla SSL Configuration Generator.
https://ssl-config.mozilla.org/#server=haproxy&version=2.4.17&config=intermediate&openssl=1.1.1o&guideline=5.6

All ciphers with a strength of 128 bit or below have been removed in order to get a 100% A+ rating at SSL Labs.

Code: [Select]

Cipher List
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384

Cipher Suites
TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

---

What will the end result look like?

We will have a wildcard SSL certificate from Let's Encrypt that gets an A+ rating with a score of 100% in each category in SSLLabs SSL Server Test.
https://www.ssllabs.com/ssltest/

We will also have two levels of load balancing our services.

Level 0 - SSL Offloading disabled
WWW --> WAN interface --> OPNsense --> HAProxy SNI Frontend --> internal servers / services

Level 1 - SSL Offloading enabled
WWW --> WAN interface --> OPNsense --> HAProxy SNI Frontend --> HAProxy SSL Frontend --> internal servers / services

Visual Schematic
(Idea: @cookiemonster (https://forum.opnsense.org/index.php?topic=23339.msg123295#msg123295))
(https://i.postimg.cc/MM2dj6zQ/P000-001-Schematic.png) (https://postimg.cc/MM2dj6zQ)

---

What are we going to do?

• We will install the necessary plugins.
• We will use a free DynDNS provider (https://desec.io/) that supports the DNS challenge which is mandatory in order to obtain a wildcard certificate.
  If you already have your own domain and your hosting provider offers an API that is supported by the Let's Encrypt (ACME Client) plugin for OPNsense, you can use it instead.
• We will create a wildcard SSL certificate using Let's Encrypt.
• We will configure the necessary firewall rules and change some OPNsense settings in order for HAProxy to function properly.
• We will use HAProxy to do SNI (explanation below) and SSL offloading.
• We will enable access to HAProxy from the internal network.

---

FAQ

• Why are we using wildcard certificates and not just regular certificates?
  For me the main reason is simplicity, there is no need to set up multiple certificates for multiple subdomains.
  Also you don't need to add any rules in HAProxy or your firewall for the ACME plugin to function correctly as the DNS challenge doesn't need this.
  If you want to read more about the differences follow the links below.
  https://sectigostore.com/page/ssl-vs-wildcard-ssl-certificate/
  https://comodosslstore.com/resources/whats-the-difference-wildcard-certificate-vs-regular-ssl-certificate/
• How does the DNS challenge work?
  https://letsencrypt.org/docs/challenge-types/
• What is SNI?
  https://en.wikipedia.org/wiki/Server_Name_Indication
  As you already saw above we are going to do it in two stages and not just one.
• Why are we using a virtual IP?
  You don't have to!
  I am only doing it because I don't want to use the localhost as my "SSL_server". I just prefer to keep things isolated from each other.
  You can of course simply use the localhost (127.0.0.1) as your "SSL_server" and let your "HTTP_frontend" / "HTTPS_frontend" listen to that IP.
• Why are we doing 2-Level-SNI?
  The main reason for this is so we can load balance services that don't require additional SSL offloading, f.e. OpenVPN over TCP.
  Basically we will have two frontends listening on port 443, one with and one without SSL offloading.
• How can we load balance TCP traffic that we don't want to get SSL offloaded, f.e. OpenVPN over TCP?
  In my tutorial I only explain how to "redirect+load balance SSL offloaded traffic".
  This is because I myself don't have (yet) the need to actually load balance any non SSL traffic.
  However balancing non SSL traffic is pretty much the same as balancing SSL traffic.
  You only have to make sure that your "NOSSLservice_rule" or "NOSSLservices_mapfile_rule" is placed on the "SNI_frontend" instead of the "HTTPS_frontend" and that the backend that belongs to your "NOSSLservice_server" is running in TCP mode.
• How can I add new services in HAProxy?
  It is almost everytime the same procedure to configure / add a new service in HAProxy.
  The long way is to...
  
  • Create the server or servers, f.e. "NEW_server".
  • Create the corresponding backend containing "NEW_server", f.e. "NEW_backend".
  • Create the condition that matches the traffic of "NEW_server", f.e. "NEW_condition".
  • Create the rule that redirects traffic to "NEW_backend", f.e. "NEW_rule".
  • Add "NEW_rule" to your "SNI_frontend" (if you don't need SSL offloading) or to your "HTTPS_frontend" (if you need SSL offloading).
• Is there a faster way of adding new services if I have a lot of subdomains and services?
  Yes there is! Thank you @sorano (https://forum.opnsense.org/index.php?topic=23339.msg111339#msg111339) for pointing this out.
  https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/overview/
  https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/syntax/
  Map files are a great way of adding new services to your HAProxy configuration.
  It is most of the time much faster than creating a individual condition + rule for each service and then adding each rule to your frontend.
  To sum this up, the short way is to...
  
  • Create the server or servers, f.e. "NEW_server".
  • Create the corresponding backend containing "NEW_server", f.e. "NEW_backend".
  • Create a map file containing all of your subdomains (of your services) with the corresponding backend.
  • Create a rule that uses that map file.
  • Add this rule to your desired frontend.
  You might be asking yourself now: Why is this faster?
  Well, because next time you add a new service you will only need to create the server, the backend and add the combination of "subdomain to backend" to your map file.
• Do I need to enable "SSL" in the Real Server configuration of a service?
  You need to think of a reverse proxy setup like this.
  
  WWW ---Stage 1---> yourdomain.tld ---Stage 2---> OPNsense + HAProxy + LE ---Stage 3---> internal services
  
  Stage 1 + 2
  Public facing external traffic. Traffic in these stages is now always encrypted with a verified SSL certificate. In this case it is created and verified by Let's Encrypt.
  
  Stage 3
  Local facing internal traffic. Traffic in this stage can or can not be encrypted, depending on your service setup. This is the traffic from HAProxy to your internal service. It doesn't need to be encrypted because you can consider your internal network as trusted.
  However it is still strongly advised to also run this traffic encrypted.
  In HAProxy you only need to check the "SSL" box in your real server setting for this.
  But then you also need to actually enable SSL encryption on that service, f.e. by installing a self-signed certificate on that service and enabling HTTPS. Even though using a self-signed certificate will give you a warning by your browser when accessing the service directly and not through the reverse proxy, the traffic is still encrypted. The certificate is just unverified, which isn't that much of an issue since we are using our reverse proxy in front of it anyway.
  How to actually do this this depends on the service but this should be covered somewhere in its manual.
  
  You can read more about this here: https://www.globalsign.com/en/ssl-information-center/dangers-self-signed-certificates
  

---

The Configuration

Part 1 - Plugin Installation

• Make sure your OPNsense is up to date and that you are using the OpenSSL firmware flavour as the LibreSSL version doesn't support TLS_1.3 as of writing this.
  In your OPNsense, go to "System --> Firmware --> Updates" and install all updates.
• Go to "System --> Firmware --> Plugins" and install the following plugins: os-acme-client, os-ddclient, os-haproxy

Part 2 - DynDNS configuration

• Visit https://desec.io/signup, create your account and verify your email address.
• Visit https://desec.io/domains and create your domain by clicking on the "round yellow + icon" in the top right corner.
  Your domain has to be in the form of "your_subdomain.dedyn.io". I will use "tutorial.dedyn.io" as an example.
  Note: Please don't get confused by the word "domain" here as you are not really creating a proper domain but actually just a subdomain of the domain "dedyn.io".
  (https://i.postimg.cc/XGCWRbPF/P002-001-Dyn-DNS-Domain.png) (https://postimg.cc/XGCWRbPF)
• Visit https://desec.io/tokens, create a token for your domain and name it accordingly.
  (https://i.postimg.cc/ZvPSqL45/P002-002-Dyn-DNS-Token-0.png) (https://postimg.cc/ZvPSqL45)
• Save the token somewhere secure as we will need it twice during this tutorial and you also might need it again in the future.
  This token will allow the DynDNS service and the Let's Encrypt plugin of your OPNsense to authenticate to the API of deSEC.
  (https://i.postimg.cc/HVHmhF9c/P002-003-Dyn-DNS-Token-1.png) (https://postimg.cc/HVHmhF9c)
• The next step is to add a CAA record containing the information on which certification authority (CA) is allowed to issue SSL certificates for your domain.
  This is important in order to get an A+ rating from SSLLabs.
  Visit https://desec.io/domains/your_subdomain.dedyn.io and add the CAA recoard by clicking on the "round yellow + icon" in the top right corner.
  (https://i.postimg.cc/5Ycxsrd4/P002-005-Dyn-DNS-CAA.png) (https://postimg.cc/5Ycxsrd4)
• The next step is to add a CNAME record for your services.
  If you don't need or want to do this because you only have one service, just skip this step. But in my opinion it makes sense to do this as it simplifies loadbalancing the services with HAProxy.
  You can either add an individual CNAME record per service, f.e. "plex.your_subdomain.dedyn.io", "mail.your_subdomain.dedyn.io" and so on.
  OR
  You just create a single wildcard CNAME record "*.your_subdomain.dedyn.io" for all your services.
  With the latter "any_string.your_subdomain.dedyn.io" will resolve to the IP of "your_subdomain.dedyn.io".
  (https://i.postimg.cc/jwzRmhtz/P002-004-Dyn-DNS-CNAME.png) (https://postimg.cc/jwzRmhtz)
• In your OPNsense go to: Services --> Dynamic DNS --> General settings
  Check that "Enable" is ticked.
• In your OPNsense go to: Services --> Dynamic DNS --> Accounts
  Click on "Add", fill out the information accordingly, hit "Save" and then "Apply".
  (The password is your token.)
  (https://i.postimg.cc/HJZMmWVm/P002-006-Dyn-DNS-OPNsense.png) (https://postimg.cc/HJZMmWVm)
• If everything went right "your_subdomain.dedyn.io" and "any_string.your_subdomain.dedyn.io" should resolve to your public IP.
  You can test this by going to: Interfaces --> Diagnostics --> Ping
  If it doesn't work wait a few minutes as it can take some time before all the DNS providers around the world have your hostname in their database.
  If it still doesn't work check your configuration.
• Now that we have our DynDNS provider all set up we will want to obtain our SSL certificate.

Part 3 - Let's Encrypt (ACME Client)

• In your OPNsense go to: Services --> ACME Client --> Settings
  Change the settings according to my image. We don't need the HAProxy integration as we are obtaining our certificates using the DNS challenge.
  (https://i.postimg.cc/8JDkMCd5/P003-001-ACME-Settings.png) (https://postimg.cc/8JDkMCd5)
• Next go to: Services --> ACME Client --> Settings --> Update Schedule
  Here we will configure at which time of the day our certificates are renewed.
  They won't be renewed everyday as the ACME client will first check if the certificates are close to expiration and if they are not they won't get renewed.
  You want this to happen at a time of the day where there is not much load on your services as the ACME plugin restarts HAProxy so it can use your new certificates which results in a very short downtime of HAProxy.
  You will also want this to NOT happen at any full hours (f.e. 3 am) because these are times when the servers of Let's Encrypt could be so busy that your certificate renewal fails.
  Change the settings according to my image.
  (https://i.postimg.cc/2bwkwfTx/P003-002-ACME-Schedule.png) (https://postimg.cc/2bwkwfTx)
• Next go to: Services --> ACME Client --> Accounts
  Create a new account. You can name it whatever you like, I usually use the domain name (if it is a real domain) or in this case the FQDN.
  Note: We will want to use the staging environment "Let's Encrypt Test CA" for now. (https://letsencrypt.org/docs/staging-environment/)
  (https://i.postimg.cc/zLTXyhg6/P003-003-ACME-Accounts.png) (https://postimg.cc/zLTXyhg6)
• Next go to: Services --> ACME Client --> Automations
  Create the automation to restart HAProxy after our certificates have been renewed.
  (https://i.postimg.cc/RWZBMf8v/P003-004-ACME-Automations.png) (https://postimg.cc/RWZBMf8v)
• Next go to: Services --> ACME Client --> Challenge Types
  Add the DNS challenge for deSEC.
  (https://i.postimg.cc/ns6bYWLL/P003-005-ACME-Challenge-Types.png) (https://postimg.cc/ns6bYWLL)
• Next go to: Services --> ACME Client --> Certificates
  Add the certificate for your domain according to the image below.
  I prefer to use Elliptic Curve Cryptography (ECC). (https://en.wikipedia.org/wiki/Elliptic-curve_cryptography)
  But you can of course also use RSA keys just make sure to set the key length as high as possible in order to get an A+ rating from SSLLabs.
  (https://i.postimg.cc/1nYFfjFL/P003-006-ACME-Certificates.png) (https://postimg.cc/1nYFfjFL)
• Next go to: Services --> ACME Client --> Certificates
  Now we need to forcefully issue our staging certificate so we can test things out and don't have to wait for the next update schedule.
  To do this click on the button marked in the image.
  (https://i.postimg.cc/G4Wm3KHv/P003-007-ACME-Certificates.png) (https://postimg.cc/G4Wm3KHv)
• Next go to: Services --> ACME Client --> Log Files --> ACME Log
  If your certificate is issued successful your log should look similiar and you can proceed to the next step.
  If it doesn't check your configuration.
  (https://i.postimg.cc/dDRPr4cq/P003-008-ACME-Log.png) (https://postimg.cc/dDRPr4cq)
• Next go to: Services --> ACME Client --> Accounts
  Since we successfully issued our staging certificate we can now leave the test environment and issue our production certificate.
  To do this edit your account and change the ACME CA from "Let's Encrypt Test CA" to "Let's Encrypt [default]".
• Next go to: Services --> ACME Client --> Certificates
  Now again forcefully issue your certificate like we did before.
  This time it you should receive a valid and trusted SSL certificate. Make sure to check the ACME log for any errors though!
• If this went well, we can now proceed to the prepare everything for HAProxy. To do this we have to configure some things in your OPNsense.

Part 4 - System preparation

• In your OPNsense go to: System --> Settings --> Administration
  You only need to check "Disable web GUI redirect rule" and change the "Web GUI TCP port" to a custom one.
  Otherwise HAProxy will not function correctly as you will propably want to access your services from the WWW using the default HTTPS port (TCP 443).
  (https://i.postimg.cc/svVFc735/P004-001-System-Settings-Administration.png) (https://postimg.cc/svVFc735)
• Next go to: Interfaces --> Virtual IPs --> Settings
  NOTE: This step is optional, see FAQ! You can safely skip this step and use the localhost instead for your "SSL_server".
  But if you would like to do it my way then you will need to create a virtual IP that is in a different subnet than any of your other networks.
  Preferably you would chose an IP that belongs to the localhost subnet in order to avoid IP conflicts in your local network. (https://en.wikipedia.org/wiki/Localhost)
  In this tutorial "127.4.4.3/32" is going to be the IP on which the "HTTP_frontend" and "HTTPS_frontend" will be listening on.
  (https://i.postimg.cc/ZCv7gSGM/P004-002-Interfaces-Virtual-IPs-Settings.png) (https://postimg.cc/ZCv7gSGM)
• Next go to: Firewall --> Aliases
  Now we are going to create an alias for the ports that HAProxy will be listening on.
  In most setups you will probably need at least 80 and 443.
  (https://i.postimg.cc/fJKqMfYW/P004-003-Firewall-Aliases-HAProxy.png) (https://postimg.cc/fJKqMfYW)
• Next go to: Firewall --> Rules --> WAN
  Now we are going to allow any inbound traffic hitting our WAN interface on the ports specified in the "HAProxy_ports" alias.
  Create a new firewall rule with the content below.
  (https://i.postimg.cc/VS3DKGPg/P004-004-Firewall-Rules-WAN-HAProxy.png) (https://postimg.cc/VS3DKGPg)
  If you did it right it should look like this, make sure that the rule is above all other rules in the list.
  (https://i.postimg.cc/yDpfZ6fw/P004-005-Firewall-Rules-WAN.png) (https://postimg.cc/yDpfZ6fw)
• Now we can finally configure HAProxy and make our services available on WAN.

Part 5 - HAProxy configuration

• In your OPNsense go to: Services --> HAProxy --> Settings --> Service
  Change the settings according to the image below.
  (https://i.postimg.cc/5XRZbrLm/P005-001-HAProxy-Settings.png) (https://postimg.cc/5XRZbrLm)
• Next go to: Services --> HAProxy --> Settings --> Global Parameters
  Change the settings according to the image below.
  As our certificate has the OCSP Must Staple extension we need to update HAProxy's OCSP data regularly. If we don't do this clients connecting to our services will get security warnings or won't connect at all.
  Note: The number of HAProxy threads should not exceed the number of CPU threads of your OPNsense.
  (https://i.postimg.cc/18495c48/P005-002-HAProxy-Settings.png) (https://postimg.cc/18495c48)
• Next go to: Services --> HAProxy --> Settings --> Default Parameters
  Change the settings according to the image below.
  (https://i.postimg.cc/GBT62jjn/P005-003-HAProxy-Settings.png) (https://postimg.cc/GBT62jjn)
• Next go to: Services --> HAProxy --> Settings --> Real Servers
  First we will add our "SSL_server" using our virtual IP (or localhost IP if you wish) to do the SSL offloading.
  (https://i.postimg.cc/MnkPJLxK/P005-005-HAProxy-Servers-SSL.png) (https://postimg.cc/MnkPJLxK)
  Then you can add all your other services according to their individual configuration.
  (https://i.postimg.cc/CBjtCLtv/P005-006-HAProxy-Servers-PLEX.png) (https://postimg.cc/CBjtCLtv)
• Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Backend Pools
  Here we first create our "SSL_backend". This is the backend to which our "SNI_frontend" sends most of its traffic to.
  As you can't mix HTTP mode and TCP mode in a frontend to backend relation, make sure that the "SSL_backend" is set to TCP mode since our "SNI_frontend" is also running in TCP mode.
  (https://i.postimg.cc/fkVZ8jkP/P005-007-HAProxy-Backends-SSL.png) (https://postimg.cc/fkVZ8jkP)
  Now we create the backend that belongs to an actual service. You will need one backend for each service.
  If you have multiple servers serving the exact same content than you will want to add all servers into a single backend so HAProxy can actually balance the load between the servers.
  (https://i.postimg.cc/TL6GvSv4/P005-008-HAProxy-Backends-HTTP.png) (https://postimg.cc/TL6GvSv4)
• Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
  Here we will only create a "NoSSL_condition", which is necessary in order to identify non-HTTPS traffic.
  (https://i.postimg.cc/XrHK4dNs/P005-009-HAProxy-Conditions-No-SSL.png) (https://postimg.cc/XrHK4dNs)
• Next go to: Services --> HAProxy --> Settings --> Advanced --> Map Files
  Here we will create a new map file "PUBLIC_SUBDOMAINS_mapfile" for our public subdomains that we want to access from outside of our network.
  Please read the FAQ about Map Files first!
  This map file is telling HAProxy that any FQDN that starts with "plex" belongs to our "PLEX_backend" (which belongs to our "PLEX_server").
  (https://i.postimg.cc/D8Q18nsH/P005-010-HAProxy-Map-Files-Public-Subdomains.png) (https://postimg.cc/D8Q18nsH)
• Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Rules
  Here we add the rules that decide what to do with the traffic based on our map files (or conditions if necessary).
  The "HTTPtoHTTPS_rule" upgrades the HTTP connection to HTTPS so that the client connects to the "HTTPS_frontend" instead.
  (https://i.postimg.cc/218bvBTT/P005-011-HAProxy-Rules-HTTPto-HTTPS.png) (https://postimg.cc/218bvBTT)
  The "PUBLIC_SUBDOMAINS_rule" maps our subdomains to our backends using the map file we created in the previous step.
  (https://i.postimg.cc/2bBZmDrH/P005-012-HAProxy-Rules-Public-Subdomains.png) (https://postimg.cc/2bBZmDrH)
• Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Public Services
  Here we will create the frontends that are listening on our interface IPs and the virtual IP we created earlier.
  At first we will create our "SNI_frontend" which will decide wether the traffic is going to be SSL offloaded or not.
  You will have to place the rules for all of your services that you don't want to get SSL offloaded in here.
  Our default backend in this frontend will be the "SSL_backend" that redirects all traffic to our virtual "SSL_server" which is actually our "HTTPS_frontend".
  (https://i.postimg.cc/bsJPPk4p/P005-013-HAProxy-Frontends-SNI.png) (https://postimg.cc/bsJPPk4p)
  No we will create our "HTTP_frontend". Make sure to place the "HTTPtoHTTPS_rule" in this frontend!
  This frontend is necessary in order to redirect HTTP traffic to HTTPS. But you could also use it to serve non SSL encrypted services on port 80.
  (https://i.postimg.cc/tZgDzQdD/P005-014-HAProxy-Frontends-HTTP.png) (https://postimg.cc/tZgDzQdD)
  No we will create our "HTTPS_frontend".
  This will be our primary frontend which is doing the SSL offloading using our earlier created Let's Encrypt certificate.
  You will have to place the "PUBLIC_SUBDOMAINS_rule" and all other rules of your services that you want to get SSL offloaded in here.
  Don't forget to change the "Cipher List" and "Cipher Suites" with the ones at the top of this tutorial "Current Ciphers and Cipher Suites for a 100% A+ rating at SSLLabs".
  (https://i.postimg.cc/RJ7J6L3K/P005-015-HAProxy-Frontends-HTTPS.png) (https://postimg.cc/RJ7J6L3K)
• Done.
  Access from external networks should now already be working.
  Just try to access your URL "your_service.your_subdomain.dedyn.io" from any device that is not connected to your local network, f.e. a smartphone on cellular data.
• You can now also verify your SSL settings: https://www.ssllabs.com/ssltest/
• If you configured everything in HAProxy and the ACME Client according to my template you should get an A+ rating with a score of 100% in each category.
  (https://i.postimg.cc/2Lk3Fq7c/P005-016-SSLLabs-Rating.png) (https://postimg.cc/2Lk3Fq7c)
• The last thing for us to do now is enable access from internal networks.

Part 6 - Access from internal networks
If you try to access your URL "your_service.your_subdomain.dedyn.io" from a device in your internal network, it should fail.
There are two ways of fixing this. I will cover both options but keep in mind that Split DNS (Option A) is the suggested way of doing it.
NAT Reflection (Option B) is an inferior solution since you lose the ability to track originating source IP in HAProxy when going through NAT. (@sorano (https://forum.opnsense.org/index.php?topic=23339.msg111439#msg111439))

Option A - Split DNS (https://docs.opnsense.org/manual/unbound.html#overrides)
Option B - NAT Reflection (https://docs.opnsense.org/manual/nat.html)

Option A - Split DNS (DNS Overrides)
Since you are using OPNsense you are probably also using the Unbound DNS plugin as your local DNS server.
Because of that you can easily set up DNS overrides.

• In your OPNsense go to: Services --> Unbound DNS --> Overrides
• Here you will need to create "Host Overrides" for each of your services. At least if you are using 2nd level subdomains "your_service.your_subdomain.dedyn.io" for you services.
  If you are running all of your services on your 1st level subdomain "your_subdomain.dedyn.io" than you will just need to override this one.
• The IP address can be any LAN (or VLAN) interface IP of your OPNsense, I am using the LAN IP on which the "SNI_frontend" is also listening on because we set it to "0.0.0.0".
  (https://i.postimg.cc/GT3CHy2P/P006-001-Unbound-DNS-Overrides.png) (https://postimg.cc/GT3CHy2P)
• Access from internal networks should now be working.

Option B - NAT Reflection
Please note that NAT Reflection is only applicable on port forwarding rules so you will have to change the "Allow HAProxy" rule to a port forwarding rule with the localhost (or some random virtual IP on the localhost) as target.

• In your OPNsense go to: Firewall --> NAT --> Port Forward
• Here you will have to edit the "Allow HAProxy" rule we created in Part 4 - Step 3 of this tutorial.
  At the bottom of each rule there is a setting called "NAT reflection = Use system default".
  You will want to change this to "NAT reflection = Enable".
• Access from internal networks should now be working.

Part 7 - Advanced Configuration: local-access-only subdomains
Imagine you have a service that you would like to access / protect using your brand new reverse proxy without making it available on the internet?
Well, HAProxy has got you covered!

• In your OPNsense go to: Services --> HAProxy --> Settings --> Advanced --> Map Files
  Here you need to clone the "PUBLIC_SUBDOMAINS_mapfile", rename it to f.e. "LOCAL_SUBDOMAINS_mapfile" and add all your local-access-only subdomains along with their corresponding backends.
  Keep in mind that the content of your "PUBLIC_SUBDOMAINS_mapfile" also has to be put in the "LOCAL_SUBDOMAINS_mapfile"! I will explain why later.
  (https://i.postimg.cc/hJvmymwR/P007-001-HAProxy-Map-Files-Local-Subdomains.png) (https://postimg.cc/hJvmymwR)
• Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
  Now you need a condition that detects if the source of the request is a local IP or a FQDN.
  You can of course also use the predefined "Source IP is local" condition.
  I am however using only specific subnets since the predefined condition is using the entire RFC1918 IP range, which I don't need!
  (https://i.postimg.cc/PPMcSxX5/P007-002-HAProxy-Conditions-Local-Subdomains-Subnets.png) (https://postimg.cc/PPMcSxX5)
  
  As I just said you can also check for a FQDN.
  But please keep in mind that HAProxy resolves those hostnames to their IPs and then checks them. But the resolving is only done once during the start / restart of HAProxy.
  So if the IP of your FQDN is changing regularly this won't work very well, except if you restart your HAProxy using a cron job like every 24 hours or so.
  (https://i.postimg.cc/3dhVjpHq/P007-003-HAProxy-Conditions-Local-Subdomains-FQDN.png) (https://postimg.cc/3dhVjpHq)
• Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Rules
  Here you need to clone the "PUBLIC_SUBDOMAINS_rule", rename it to f.e. "LOCAL_SUBDOMAINS_rule", select your "LOCAL_SUBDOMAINS_SUBNETS_condition" and select your "LOCAL_SUBDOMAINS_mapfile".
  If you are also using a FQDN condition, like I do, you will need to select both your FQDN and your subnet condition together with the logical "or" operator!
  (https://i.postimg.cc/64JGpmQB/P007-004-HAProxy-Rules-Local-Subdomains.png) (https://postimg.cc/64JGpmQB)
• Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Public Services
  The last thing left to do is to place the "LOCAL_SUBDOMAINS_rule" before your "PUBLIC_SUBDOMAINS_rule" in your "HTTPS_frontend".
  
  Attention!
  Remember that I told you to also put the content of your "PUBLIC_SUBDOMAINS_mapfile" in the "LOCAL_SUBDOMAINS_mapfile"?
  This is because HAProxy is processing the rules in the frontends based on the order they appear!
  So if you place your "PUBLIC_SUBDOMAINS_rule" before your "LOCAL_SUBDOMAINS_rule" in the frontend configuration, you won't get access to your local-access-only subdomains.
  Vice versa this will also happen and you will no longer have access to your public subdomains.
  To avoid this you have to also put the content of your "PUBLIC_SUBDOMAINS_mapfile" in the "LOCAL_SUBDOMAINS_mapfile" and place their rules in the correct order.
  
  The correct way of placing both rules is like this.
  (https://i.postimg.cc/4KMmdS3m/P007-005-HAProxy-Frontends-HTTPS-Rule-Order.png) (https://postimg.cc/4KMmdS3m)
• Done!
  You should now still have access to your public subdomains from any network and also have access to your local-access-only subdomains from the locations you defined.

Part 8 - Advanced Configuration: Hide your certificate on access by IP
You might have noticed that if you now access your OPNsense using your public WAN IP (https://YOUR_PUBLIC_IP/) the connection will be secured and upon further inspection you will see that your Let's Encrypt certificate is beeing used.
While this is not a major security problem it still presents at least some privacy issues.
To fix this we can add the "strict-sni" parameter to the "HTTPS_frontend" so the connection will be closed when HAProxy is directly accessed using your public WAN IP (https://YOUR_PUBLIC_IP/).

• In your OPNsense go to: Services --> HAProxy --> Settings --> Virtual Services --> Public Services
  Edit your "HTTPS_frontend" and enable the "advanced mode" in the top left corner.
  You should now be able to see the "SSL option pass-through" field in the "SSL Offloading" section, here we already added the parameter "curves secp384r1" to make use of ECC.
  Change this field from "curves secp384r1" to "curves secp384r1 strict-sni", save the changes to the "HTTPS_frontend" and apply them to HAProxy.
• Done!
  You should now no longer get presented with your trusted Let's Encrypt certificate when accessing "https://YOUR_PUBLIC_IP/" but instead with a "SSL_ERROR_UNRECOGNIZED_NAME_ALERT".

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: browne on June 01, 2021, 10:35:23 am

Thank you very much!
This helped me switching from regular certificates to wildcard certificates!
I now also do score 100% A+ in the SSL test.

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: skittlebrau on June 01, 2021, 04:23:25 pm

Thank you so much for this guide! I was completely lost in the new UI layout before.

Do you mind showing in Step 5 an example of how you've configured 'PLEX_backend'? I noticed you referred to it, but there wasn't a screenshot for it.

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 01, 2021, 06:52:28 pm

The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different. :)

--> I will add it to the tutorial.
Reason: https://forum.opnsense.org/index.php?topic=23339.msg111143#msg111143


It is also almost everytime the same procedure to configure / add a new service in HAProxy.

• Create the server or servers, f.e. NEW_server.
• Create the corresponding backend containing NEW_server, f.e. NEW_backend.
• Create the condition that matches the traffic of NEW_server, f.e. NEW_condition.
• Create the rule that redirects traffic to NEW_backend, f.e. NEW_rule.
• Add NEW_rule to your SNI_frontend if you don't need SSL offloading or to your HTTPS_frontend if you need SSL offloading.

--> I will add this to the FAQ.

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: ejball02 on June 01, 2021, 08:55:28 pm

I'm using a self-signed cert for HTTPS inspection for content filtering. I've got OPT2 configured as a guest network on my Protectli, and content filtering, using shallalist works great. Only downside, is that when I try to access am HTTPS site, Firefox/Chrome always give a warning page: "Your connection is not private" "ERR_CERT_AUTHORITY_INVALID". After much Googling, I came across an old post, that said Let's Encrypt can give public certs which would get rid of the message.

I added the LE plugin but couldn't figure out, how to create a cert for use on the Foward Proxy "CA to use" field, required for SSL inspection. Looking through this walkthrough, I'm wondering if there is something here that can help achieve creating a cert for content filtering. Anyone have any experience with this?

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 01, 2021, 11:36:13 pm

Quote from: ejball02 on June 01, 2021, 08:55:28 pm

I'm wondering if there is something here that can help achieve creating a cert for content filtering. Anyone have any experience with this?

No offense, but this is meant to be a tutorial related to HAProxy in a reverse proxy setup.  :)
As this is the tutorial sub-forum it is not really meant to ask for help and especially not regarding different topics.
You can of course follow this guide to the part where you get your Let's Encrypt certificate, but from there on you will be on your own.

I think you should probably open up your own thread in the help forums and ask for help there.

https://forum.opnsense.org/index.php?board=27.0
or here
https://forum.opnsense.org/index.php?board=28.0
or here
https://forum.opnsense.org/index.php?board=1.0

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: skittlebrau on June 02, 2021, 02:24:48 am

Quote from: TheHellSite on June 01, 2021, 06:52:28 pm

The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different. :)

I assumed that it would be the case, so thanks for confirming  ;D

One part that's tripping me up is in my plex_backend pool and other ones I set for my internal services like Resilio Sync, if I set the mode to TCP (Layer 4) then I get a syntax error below. Switching the mode to HTTP for those two backends resolves the syntax error, but produces blank pages. Stats area in HAProxy shows the services as being UP.

[NOTICE] 152/082107 (17536) : haproxy version is 2.2.14-a07ac36
[ALERT] 152/082107 (17536) : http frontend '1_HTTPS_frontend' (/usr/local/etc/haproxy.conf.staging:70) tries to use incompatible tcp backend 'sync1_backend' (/usr/local/etc/haproxy.conf.staging:117) in a 'use_backend' rule (see 'mode').
[ALERT] 152/082107 (17536) : http frontend '1_HTTPS_frontend' (/usr/local/etc/haproxy.conf.staging:70) tries to use incompatible tcp backend 'plex_backend' (/usr/local/etc/haproxy.conf.staging:104) in a 'use_backend' rule (see 'mode').
[ALERT] 152/082107 (17536) : Fatal errors found in configuration.

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 02, 2021, 08:08:18 am

Quote from: skittlebrau on June 02, 2021, 02:24:48 am

Quote from: TheHellSite on June 01, 2021, 06:52:28 pm

The PLEX_backend looks very similiar to the SSL_backend. Only "Name: PLEX_backend" and "Servers: PLEX_server" are different. :)

I assumed that it would be the case, so thanks for confirming  ;D

One part that's tripping me up is in my plex_backend pool and other ones I set for my internal services like Resilio Sync, if I set the mode to TCP (Layer 4) then I get a syntax error below. Switching the mode to HTTP [...]
[/font]

Now I remember correctly.
The SSL_backend is different from most other backends since it is NOT running in HTTP mode.
But HTTP mode is usually the one you want for a web based service.
I changed that part in the tutorial. Please take a look at the backend creation.

My bad! I wrote the tutorial from brain memory. So I guess you are my beta tester.  ;D

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: skittlebrau on June 02, 2021, 08:54:51 am

All good, you've been so helpful.

I checked through everything again and realised that when setting up the NAT port forward, I forgot to enter the dropdown menu at the end to add the associated filter rule. I was too used to pfSense automatically selecting that by default, so no wonder it wasn't working despite changing from TCP to HTTP mode for the backend services!

For posterity sake, I've saved your guide for my own reference and archived it to PDF as well.

If there's a particular charity you support, send me a private message and I'll happily donate to one in your name, along with a donation to the OPNsense project.  :D

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: michaelgo on June 02, 2021, 03:19:36 pm

Hi,
thank you for an amazing guide.

in part 5.5 the picture is broken, can you please re-post (it's working now)
also, 5.7 "plex condition" is it a rule? (5.7 and 5.8 screenshots are vice versa)

also, i don't have "Backends" only "Backend pools" there
Services --> HAProxy --> Settings --> Virtual Services --> Backends
is it what you mean?
thanx

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 03, 2021, 01:01:13 am

Yep, those pictures are in the wrong order and yes, I meant Backend Pools.
I will change this.

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 03, 2021, 01:24:05 pm

Quote from: skittlebrau on June 02, 2021, 08:54:51 am

If there's a particular charity you support, send me a private message and I'll happily donate to one in your name, along with a donation to the OPNsense project.  :D

A donation to OPNsense would be happily welcome I guess. :)
I myself don't want any money for this. We are all more or less using OPNsense free of charge so the least anyone can do is help the community or donate to the project.

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: sorano on June 05, 2021, 03:00:21 pm

@TheHellSite Great guide! I'm sure it will help alot of people trying to get this kind of setup up and running.

I'm running a similar setup and have some suggestions for improvement to your guide.

1. You dont need to use virtual IP's.
If you bind the HAProxy frontends to 0.0.0.0:80 & 0.0.0.0:443 it will bind to you WAN interface (even if it's dynamic). And when you do there is no need for NAT forwarding to the virtual IP's so a simple firewall rule for 80/443 on the WAN interface is enough.
Then you can bind the SSL terminating frontend to 127.0.0.1:[port] and use that IP for your SSL terminating "real server".
With the added bonus of that it performs better in a CARP setup.

2. Use map files {Advanced --> Map files}
Using map files to map domains to backends will keep your config rules alot less cluttered, especially when you have many subdomains to match. With map files 1 rule is enough to map all of your domains.

Example for map file:
plex.mydomain.tld Backend_plex

Example for rule:
(Execute function: Map domains to backend pools using a map file
Map file: [name of your map file]

Then apply the map file rule to your SSL terminating frontend.

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 07, 2021, 11:29:22 am

Quote from: sorano on June 05, 2021, 03:00:21 pm

1. You dont need to use virtual IP's.
2. Use map files {Advanced --> Map files}

1. You dont need to use virtual IP's.
I totally get your point! This makes indeed sense but I think only if you have a static WAN IP.
As it would break the access from internal networks to the external URLs "service.subdomain.mydomain.tld" if one enabled that access using DNS rewrite rules. I am not aware of a way to rewrite DNS entries in Unbound to the WAN interface address.

With NAT reflection your way of setting this up can of course work.


2. Use map files {Advanced --> Map files}
I haven't used those yet but looks very promising!
This really makes sense in a big environment with lots of subdomains.
Thank you for pointing this out! I will add it to the FAQ.  :)

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: sorano on June 07, 2021, 02:21:02 pm

Quote from: TheHellSite on June 07, 2021, 11:29:22 am

1. You dont need to use virtual IP's.
I totally get your point! This makes indeed sense but I think only if you have a static WAN IP.

Well, I only have dynamic IP's for my WAN interfaces. MultiWAN consisting of a fiber primary with LTE failover, two OPNsense hosts running CARP on all interfaces except for WAN (since I cannot get proper stateful failover with dynamic WAN IP's).

Quote from: TheHellSite on June 07, 2021, 11:29:22 am

As it would break the access from internal networks to the external URLs "service.subdomain.mydomain.tld" if one enabled that access using DNS rewrite rules. I am not aware of a way to rewrite DNS entries in Unbound to the WAN interface address.

With NAT reflection your way of setting this up can of course work.

The way I'm doing access from internal networks is with Split DNS (DNS override as you call it).
In my opinion NAT reflection is an inferior solution since you lose the ability to track originating source IP in HAProxy when going through NAT.

Since HAProxy is already listening on 0.0.0.0 (all available IPv4 interfaces) I resolve the Split DNS to the internal IP of my DMZ CARP IP (but any internal IPv4 interface will do as long as you allow 80/443).

I also have certain domains I don't want reachable from the Internet so I use two map file rules, one for internal domains along with a condition that checks that source is RFC1918.

And one for external domains where I also require additional authentication.

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on June 09, 2021, 12:03:57 pm

Quote from: sorano on June 07, 2021, 02:21:02 pm

Since HAProxy is already listening on 0.0.0.0 (all available IPv4 interfaces) I resolve the Split DNS to the internal IP of my DMZ CARP IP (but any internal IPv4 interface will do as long as you allow 80/443).

So this means you are actually also using sort of a virtual IP.  :D
Okay so you say the easier way is like this:
- Firewall rule on WAN allowing 443+80
- SNI_frontend listening on 0.0.0.0:80+443 (Which covers all interfaces)
- SSL_frontend listening on 127.0.0.1:443
- Split DNS pointing to f.e. the LAN IP of OPNsense.

You are right, with that you could skip the virtual IPs.
I would however still prefer to use a virtual IP for the SSL_frontend as I don't want to hijack the localhost for it. I just prefer to keep things seperated.

Quote from: sorano on June 07, 2021, 02:21:02 pm

I also have certain domains I don't want reachable from the Internet so I use two map file rules, one for internal domains along with a condition that checks that source is RFC1918.

For this I guess you have to use 0.0.0.0 on the SNI_frontend otherwise you would need another NAT rule forwarding 443-LAN traffic to the virtual IP.

EDIT 20210611:
Implemented @sorano (https://forum.opnsense.org/index.php?topic=23339.msg111339#msg111339)'s enhancements  :)

Title: Re: Tutorial: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ SSLLabs Rating
Post by: TheHellSite on June 13, 2021, 04:13:00 pm

• Changelog 20210613
  
  • The tutorial is now using a wildcard CNAME record.
  • Enabled Proxy Protocol in the "SSL_backend", "HTTPS_frontend" and "HTTP_frontend" configuration so that the IPs of clients accessing HAProxy will now no longer be overwritten with the "SSL_server" IP.
    https://www.haproxy.com/blog/using-haproxy-with-the-proxy-protocol-to-better-secure-your-database/

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: pintu1228 on July 08, 2021, 04:49:36 pm

This is exactly what I was looking for, have had trouble coming from pfsense to opnsense to setup haproxy/let's encrypt.  I have cloudflare setup to use DNS.

I am using google domain, how do I go about setting up the 1st part (Dynamic DNS), do I need to create 3 custom records:

domain.com (A type)
*.domain.com (A type)
www.domain.com (CNAME)

And also I created separate dynamicDNS for plex.domain.com to use for part 7 (configure Dynamic DNS on opnsense).

Thanks

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 09, 2021, 12:28:11 pm

Quote from: pintu1228 on July 08, 2021, 04:49:36 pm

I am using google domain, how do I go about setting up the 1st part (Dynamic DNS), do I need to create 3 custom records:

domain.com (A type)
*.domain.com (A type)
www.domain.com (CNAME)

It all depends on what you would like to achieve!
You can set your DynDNS to update the IP of "domain.com" and then have a CNAME record in the form of "subdomain.domain.com" pointing to the IP of "domain.com".

However since you might also want to host a website on "domain.com / www.domain.com", you will probably want a subdomain for your domain first!
I myself always create individual 1st-level-subdomains for each physical location that is sitting in a different building / city / country having there own public IP.

f.e. location.domain.tld = earth.thehellsite.com / moon.thehellsite.com / mars.thehellsite.com and so on

And then (as I already explained in my tutorial) a wildcard 2nd-level-subdomain.
Which means that any string in the 2nd-level-subdomain is pointing to my 1st-level-subdomain.

f.e. any_string.location.domain.tld = *.earth.thehellsite.com = abcxyz.earth.thehellsite.com / 123.earth.thehellsite.com


If you do it my way then you only need to set up three things in your Google DNS zone.

• An "A Record", f.e. location.domain.tld
• A wildcard "CNAME Record", f.e. *.location.domain.tld
• A "CAA Record", as explained in the tutorial

Your DynDNS should then be updating the "A Record".




But again, there are many different ways to achieve this.
You just need to find out how you would like to do it.

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: _Alchemist_ on July 14, 2021, 01:45:11 pm

Thanks a lot for the write up, I will try this out as soon as I can :)

The only thing that could be added on Part 4.3 is to use an Alias for Port 80 and 443 to only use one Firewall Rule ;)

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 14, 2021, 03:40:05 pm

Quote from: _Alchemist_ on July 14, 2021, 01:45:11 pm

Thanks a lot for the write up, I will try this out as soon as I can :)

The only thing that could be added on Part 4.3 is to use an Alias for Port 80 and 443 to only use one Firewall Rule ;)

Good Idea!
Will add this in my next update.

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: saarko on July 27, 2021, 11:44:25 pm

Quote from: sorano on June 07, 2021, 02:21:02 pm

I also have certain domains I don't want reachable from the Internet so I use two map file rules, one for internal domains along with a condition that checks that source is RFC1918.

And one for external domains where I also require additional authentication.

Dear @TheHellSite,
thanks for the great tutorial! It works well.

Dear @sorano,

thanks for your input. The hint with map file works well. However, I am unable to create a rule with multiple "OR" conditions for various sub-domains to match and check it with an "AND" condition to test if it is an internal IP. It shall cover your described rule.

Currently I try to create a rule like:
use map file 1
IF
condition 1 "subdomain1" OR condition 2 "subdomain2"
AND condition 3 "local IP (RFC1918)" is matched


How did you solve this with the conditions and rules within OPNsense HAProxy plugin?

thanks in advance for your help and reply.
Saarko

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 28, 2021, 09:18:12 pm

Thanks for the feedback!

I had a chat with @sorano and changed my config a little bit in the meantime.

I will update my post in a few days, hopefully. Very busy with work atm.
This will also solve your problem @saarko.

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 29, 2021, 07:34:44 pm

I followed this, however, decided against using the LE and now not getting 100% A+. is there something I am missing...

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 29, 2021, 08:07:09 pm

Quote from: lilsense on July 29, 2021, 07:34:44 pm

I followed this, however, decided against using the LE and now not getting 100% A+. is there something I am missing...

Since you didn't use "LE"? (I guess you mean Let's Encrypt certificates) you should probably check out this page.
https://github.com/ssllabs/research/wiki/SSL-Server-Rating-Guide

So if I would have had to guess it is most likely due to the fact that the key length of your certificate is too short!

If you followed the rest of my guide (OPNsense settings + HAProxy settings) accordingly and didn't miss any configuration steps, you should be getting 100% A+ at SSLLabs!
Again, provided that your certificate has a long enough key length...



I hope you can understand that it is very hard to help you out, since you didn't share any more details about your config, the SSLLabs test results, the certificate you are using ...

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 29, 2021, 10:23:29 pm

• Changelog 20210729
  
  • Added an alias for the HAProxy ports and updated the WAN interface firewall rule with it.
    This leaves us with only one firewall rule instead of two and makes even more sense if one is using additional frontends on different ports.
    Thanks @_Alchemist_ (https://forum.opnsense.org/index.php?topic=23339.msg113979#msg113979) for the suggestion.
  • The tutorial is now using a map file instead of "condition + rule" for service configuration.

@saarko
I didn't manage to finish part 7 "local-access-only subdomains" today.
I will try to do this tomorrow!

In the meantime you can already change your config over to map files.  :)
See part 5, step 7 to step 10. This is the only thing that has changed for using map files.
Map files will also make part 7 very easy for you!

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 30, 2021, 05:43:22 pm

ggetting this from hap:

[WARNING] 210/114212 (27105) : Proxy '1_HTTPS_Frontend': no-sslv3/no-tlsv1x are ignored for bind '192.168.1.50:443' at [/usr/local/etc/haproxy.conf.staging:71]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
Warnings were found.
Configuration file is valid

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 30, 2021, 06:21:48 pm

Quote from: lilsense on July 30, 2021, 05:43:22 pm

ggetting this from hap:

[WARNING] 210/114212 (27105) : Proxy '1_HTTPS_Frontend': no-sslv3/no-tlsv1x are ignored for bind '192.168.1.50:443' at [/usr/local/etc/haproxy.conf.staging:71]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
Warnings were found.
Configuration file is valid

Please post a screenshot of your entire HTTPS_frontend config, including advanced.
You can redact your rules.

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 30, 2021, 06:51:11 pm

Code: [Select]

# Frontend: 1_HTTPS_Frontend
frontend 1_HTTPS_Frontend
    # WARNING: ciphersuites cannot be used with flavour libressl.
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 10.10.1.50:5555 name 10.10.1.50:555 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-min-ver TLSv1.2 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60f9db54XXX3488.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s


Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 30, 2021, 07:08:35 pm

Quote from: lilsense on July 30, 2021, 06:51:11 pm

Code: [Select]

# Frontend: 1_HTTPS_Frontend
frontend 1_HTTPS_Frontend
    # WARNING: ciphersuites cannot be used with flavour libressl.
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 10.10.1.50:5555 name 10.10.1.50:555 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-min-ver TLSv1.2 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60f9db54XXX3488.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s


I was already assuming what you did (wrong) and your config is confirming it.  ;D

First of all. This is not an error but only a warning which won't make HAProxy refuse to start.
Your setup should still be working if you did everything else correctly.

This warning is appearing because you set "ssl-min-version = TLS1.2" in your HTTPS_frontend config.
Which is pretty much pointless since my config example is already blocking "sslv3 tls1.0 tls1.1" using the "Bind options" in the HTTPS_frontend field. This is why HAProxy is giving you a warning.
You can of course also remove my bind options and only use "min-ssl-version = TLS1.2", this is up to you.
Either way will work and allow only TLS1.3 connections.

Quote from: lilsense on July 30, 2021, 06:51:11 pm

Code: [Select]

# Frontend: 1_HTTPS_Frontend
frontend 1_HTTPS_Frontend
    # WARNING: ciphersuites cannot be used with flavour libressl.
...


What you should be aware of is that I specifically mentioned that my guide might not work well on LibreSSL due to the fact that not all features of the last HAProxy version is supported by the LibreSSL firmware variant of OPNsense, yet!

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 30, 2021, 07:13:04 pm

Quote from: saarko on July 27, 2021, 11:44:25 pm

Dear @TheHellSite,
thanks for the great tutorial! It works well.

Dear @sorano,

thanks for your input. The hint with map file works well. However, I am unable to create a rule with multiple "OR" conditions for various sub-domains to match and check it with an "AND" condition to test if it is an internal IP. It shall cover your described rule.

Currently I try to create a rule like:
use map file 1
IF
condition 1 "subdomain1" OR condition 2 "subdomain2"
AND condition 3 "local IP (RFC1918)" is matched


How did you solve this with the conditions and rules within OPNsense HAProxy plugin?

thanks in advance for your help and reply.
Saarko

• 20210730
  
  • Added an explanation on how to configure local-access-only subdomains in HAProxy.

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 30, 2021, 08:08:34 pm

so not sure what you are referring to...

I used the cloudflare... what option is needed for the cert that's different from the  norm.

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 30, 2021, 09:26:40 pm

So not sure if you understand what I am trying to tell you.........

Quote from: lilsense on July 29, 2021, 07:34:44 pm

I followed this, however, decided against using the LE and now not getting 100% A+. is there something I am missing...

Firstly, when you made your one-liner about the bad SSLLabs result, I asked you for details about your config and the SSLLabs result.
You gave NONE.
How could anyone be able to help you with that little information given?!  :o
This makes me even a bit angry.

Secondly, you asked why HAProxy is giving you a warning, so I explained to you in a very detailed manner why that is.
And now you are saying that you don't know what I am referring to?
Like, I even quoted the issue.

Thirdly, I am really willing to help anyone that follows my guide.
I am doing this in my free time free of charge.
But in return the least you could do is to say thank you, before asking for help about an issue that is most likely due to the fact that you didn't read my tutorial correctly and that you are not using Let's Encrypt certificates.


So with that being sad.
Maybe you should think about what I just said.
And then, if you are willing to, share just a tiny little bit more details about: your certificate, your SLLLabs result, ... like anything that could be of help and not just one-liners without any context.
Your issue could have already been solved if you had provided those information in the first place.

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on July 30, 2021, 09:47:41 pm

here you go...


Protocols
TLS 1.3   Yes
TLS 1.2   Yes
TLS 1.1   Yes
TLS 1.0   Yes
SSL 3   No
SSL 2   No


Cipher Suites
# TLS 1.3 (server has no preference)
TLS_AES_128_GCM_SHA256 (0x1301)   ECDH x25519 (eq. 3072 bits RSA)   FS   128
TLS_AES_256_GCM_SHA384 (0x1302)   ECDH x25519 (eq. 3072 bits RSA)   FS   256
TLS_CHACHA20_POLY1305_SHA256 (0x1303)   ECDH x25519 (eq. 3072 bits RSA)   FS   256
# TLS 1.2 (suites in server-preferred order)
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)   ECDH x25519 (eq. 3072 bits RSA)   FS   128
OLD_TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcc14)   ECDH x25519 (eq. 3072 bits RSA)   FS   256P
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)   ECDH x25519 (eq. 3072 bits RSA)   FS   256P
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)   ECDH x25519 (eq. 3072 bits RSA)   FS   WEAK   128
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)   ECDH x25519 (eq. 3072 bits RSA)   FS   WEAK   128
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)   ECDH x25519 (eq. 3072 bits RSA)   FS   256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)   ECDH x25519 (eq. 3072 bits RSA)   FS   WEAK   256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)   ECDH x25519 (eq. 3072 bits RSA)   FS   WEAK   256
# TLS 1.1 (suites in server-preferred order)
# TLS 1.0 (suites in server-preferred order)
(P) This server prefers ChaCha20 suites with clients that don't have AES-NI (e.g., Android devices)   

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 31, 2021, 12:40:03 am

Please don't get me wrong, but...

You know what it's like trying to help you?
It's like trying to teach a stone how to sing.
You can try as long as you want and still fail.
Because...
A) He doesn't understand you.
and
B) He is not answering your questions.




This is the last time I am asking you.
Give a screenshot of your ENTIRE SSLLabs result.
A SCREENSHOT. But with your domain name and IP blacked out.

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: saarko on July 31, 2021, 02:47:32 pm

Quote from: TheHellSite on July 30, 2021, 07:13:04 pm

• 20210730
  
  • Added an explanation on how to configure local-access-only subdomains in HAProxy.

thanks, works like a charm.
My "mistake" was that I thought to need a condition to trigger a map rule. Since it is not necessary, it is even easier, except for the RFC1918 condition of course. :)

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Crappysauce on August 01, 2021, 07:26:04 am

First off, thank you so much for this guide. Really helped.

I was having issues connecting to my server due to handshake errors which I think got fixed after generating new ciphers using the Mozilla SSL Config generator and changing the HAProxy and OpenSSL versions to match my setup.

After that, HAProxy seemed to refuse to redirect me to my Vaultwarden server, unless I turned off the SSL option in my Real Server setting. It still shows that I'm secured with the proper (wildcard cert from Let's Encrypt).

Do I need the SSL option enabled? The SSL test still gave me an A+...

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 01, 2021, 10:05:57 am

Quote from: Crappysauce on August 01, 2021, 07:26:04 am

I was having issues connecting to my server due to handshake errors which I think got fixed after generating new ciphers using the Mozilla SSL Config generator and changing the HAProxy and OpenSSL versions to match my setup.

Are you on the latest version of OPNsense and are the installed plugins up to date?
Just out of interest, which versions of OPNsense, HAProxy and Let's Encrypt are you running?

Quote from: Crappysauce on August 01, 2021, 07:26:04 am

After that, HAProxy seemed to refuse to redirect me to my Vaultwarden server, unless I turned off the SSL option in my Real Server setting. It still shows that I'm secured with the proper (wildcard cert from Let's Encrypt).

Do I need the SSL option enabled? The SSL test still gave me an A+...

The reason you couldn't connect was due to a misconfiguration in your real server, as you figured out yourself.
You enabled the "SSL -  Enable or disable SSL communication with this server. " checkbox in your real server for Vaultwarden even though the port used to connect doesn't offer SSL encryption.
https://github.com/dani-garcia/vaultwarden/wiki/Enabling-HTTPS
https://github.com/dani-garcia/vaultwarden/wiki/Private-CA-and-self-signed-certs-that-work-with-Chrome



You need to think of a reverse proxy setup like this.

WWW ---Stage 1---> yourdomain.tld ---Stage 2---> OPNsense + HAProxy + LE ---Stage 3---> internal services

Stage 1 + 2
Public facing external traffic. Traffic in these stages is now always encrypted with a verified SSL certificate. In this case it is created and verified by Let's Encrypt.

Stage 3
Local facing internal traffic. Traffic in this stage can or can not be encrypted, depending on your service setup. This is the traffic from HAProxy to your internal service. It doesn't need to be encrypt because you can consider your internal network as trusted.
However it is still strongly advised to also run this traffic encrypted.
In HAProxy you only need to check the "SSL" box in your real server setting for this.
But then you also need to actually enable SSL encryption on that service, f.e. by installing a self-signed certificate on that service and enabling HTTPS. Even though using a self-signed certificate will give you a warning by your browser when accessing the service directly and not through the reverse proxy, the traffic is still encrypted, the certificate is just unverified.
How to actually do this this depends on the service but this should be covered somewhere in its manual.

You can read more about this here: https://www.globalsign.com/en/ssl-information-center/dangers-self-signed-certificates



I will add this explanation to the FAQ.

Title: Re: Tutorial 2021/07: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Crappysauce on August 02, 2021, 07:37:30 am

Quote from: TheHellSite on August 01, 2021, 10:05:57 am

Are you on the latest version of OPNsense and are the installed plugins up to date?
Just out of interest, which versions of OPNsense, HAProxy and Let's Encrypt are you running?

Everything is at the latest version:
OPNSense: 21.7
HAProxy Plugin: 3.4
Let's Encrypt: 2.6

Quote from: TheHellSite on August 01, 2021, 10:05:57 am

The reason you couldn't connect was due to a misconfiguration in your real server, as you figured out yourself.
You enabled the "SSL -  Enable or disable SSL communication with this server. " checkbox in your real server for Vaultwarden even though the port used to connect doesn't offer SSL encryption.
https://github.com/dani-garcia/vaultwarden/wiki/Enabling-HTTPS
https://github.com/dani-garcia/vaultwarden/wiki/Private-CA-and-self-signed-certs-that-work-with-Chrome

I knew I was forgetting something  :(
I'll read up on his quides and get it all sorted.

Thank you again for your guide and help  ;D

Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 09, 2021, 04:31:33 pm

Hi

@TheHellsite Thank you so much for your time and knowledge

If you have a fixed IP, does the DynDNS Configuration step need to be done? if skipped is there other settings i should put in?

If it is a must when we signup, there are 2 options :

configure your own domain
or
register under dyn.io

i have my own domain names about 10. Do i add each one to there system to get certs then duplicate the process to reverse proxy and cert the other domains?

If you use your real domain eg. www.123.com Do i need to go to my current domain registrar and change name servers to point to desec??

Thank you for the help. just want to get these vms up so i can programme again :( going to cry.

Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 09, 2021, 06:29:22 pm

Quote from: newbee on August 09, 2021, 04:31:33 pm

Hi

If you have a fixed IP, does the DynDNS Configuration step need to be done? if skipped is there other settings i should put in?

If you have a fixed IP that for sure never ever changes.
Then yes, you can safely skip setting up DynDNS on your OPNsense.
You will then only need to configure an A-Record in the DNS zone of your domains / subdomains pointing to your static IP.
You can set this up at your domain hosting provider.

Feel free to share a bit more about your current domain set up.
This will make it easier for me to help you.
- What are your domains?
- What are your subdomains?
- What is your domain hosting provider?

Quote from: newbee on August 09, 2021, 04:31:33 pm

If it is a must when we signup, there are 2 options :

configure your own domain
or
register under dyn.io

i have my own domain names about 10. Do i add each one to there system to get certs then duplicate the process to reverse proxy and cert the other domains?

update: i used my domain name. i think that was wrong.

Seems like deSEC now also supports managing domains that are "hosted / registered" at a different hosting provider.
If I understood this correctly it allows you to manage the DNS zone of your domain at deSEC without actually transfering your domain away from you current hosting provider.

Since you are saying that you already have some domains / subdomains registered.
Something like "sub1.yourdomain1.com" ... "sub4.yourdomain5.com" and so on...

In this case you should check if your current domain hosting provider supports the DNS challenge.
And that the Let's Encrypt Plugin on OPNsense supports the DNS challenge for your hosting provider.
If not, then you have two options if you would like to use wildcard certificates...

Option 1 - Proceed setting up the managed DNS for your desired domains at deSEC. Then follow my tutorial beginning with part 2 step 3.

Option 2 - Transfer your domains to a hosting provider that supports the DNS challenge and that is also supported by the Let's Encrypt plugin.





Feel free to share more information about your domain / subdomain scenario along with the name of your hosting provider.
Of course don't expose your real domain names.

Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 09, 2021, 06:45:54 pm

Wow thank you

Yes IP is fixed 100%.
- What are your domains? .com and .co.uk eg. example.com is my primary one.
- What are your subdomains? cloud.example.com dev.example.com
- What is your domain hosting provider? 123reg.co.uk

Have got to step Part 2 step 7. setting up opnsense dynamic DNS. So far have followed all steps par i registered example.com and not "anything.dedyn.io".

So before continuing i will check 123reg.co.uk options

Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 09, 2021, 06:54:15 pm

For your own safety please replace your real domain name with "example", if the above is your real domain name!

I quickly checked 123reg and it seems like they do not offer an API so you can't use the DNS challenge.

Which gives you only 2 options.

1 - From now on managing the DNS zone of your 123reg domains at deSEC.
https://www.123-reg.co.uk/support/domains/how-do-i-change-the-nameservers-for-my-domain-name/
+ see image attached.

2 - Moving your domain to another registrar.

Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 09, 2021, 07:10:56 pm

Thank you,

name servers updated. I have added A and MX records (set MX prefence to "10", is that right?).

In the opnsense Dynamic DNS, stuck on update URL. just type update.example.com?

Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 09, 2021, 07:38:08 pm

Is dynamic dns still needed for fixed IP. You did say start from part 2 step 3. This update URL makes me think?

Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 09, 2021, 08:23:33 pm

Quote from: newbee on August 09, 2021, 07:38:08 pm

Is dynamic dns still needed for fixed IP. You did say start from part 2 step 3. This update URL makes me think?

Not needed! A record is enough.
Dynamic DNS is basically only updating the A record in your DNS Zone.

Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: newbee on August 10, 2021, 05:24:42 pm

Hi,

So was setting up all night and all morning.

I skipped and removed dyndns plugin.
I completed the rest of the steps, except for the map as no subdomains setup yet.
I also didnt add a wild card for subdomains to the cert, when i did add it the cert failed. once removed the cert installed fine.
When i checked the local ip in browser, it works. When i type www.example.com it fails and cant ping address.

Had a hunt for what it could be, in the end decided to reboot opnsense and see if it shows errors.

Now rebooted HaProxy status is down and will not start.
I have looked in every log and there is only this:

2021-08-10T15:15:37   root[90544]   /usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy   
2021-08-10T15:15:30   root[91171]   /usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy   
2021-08-10T15:00:29   root[26043]   /usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy   
2021-08-10T14:59:58   syslog-ng[29102]   syslog-ng starting up; version='3.33.2'   
2021-08-10T14:59:56   syslog-ng[70806]   syslog-ng shutting down; version='3.33.2'   
2021-08-10T14:59:55   root[49764]   /usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy

thinking to remove HAProxy and start again?

Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 10, 2021, 06:07:04 pm

No need to.
Look in your PMs.

Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 01, 2021, 11:49:00 am

Hi there,
First of all, thank you very much for the tutorial. I still have two questions. I hope you are so nice to answer me :).

I want to make OPENvpn and various websites accessible via 443.

1st question:
I create a subdomain for each service. Can this subdomain then point to the same DYNDNS cname entries?
For example:
openvpn.domain.com -> CNAME -> 123.dyndns.com
seafile.domain.com -> CNAME -> 123.dyndns.com

2nd question:
How to configure HAproxy for openvpn.
Could you explain it to me as well as you did in the tutorial or add the point to the tutorial?

Title: Re: Tutorial 2021/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 01, 2021, 04:39:55 pm

Quote from: Lip90 on September 01, 2021, 11:49:00 am

1st question:
I create a subdomain for each service. Can this subdomain then point to the same DYNDNS cname entries?
For example:
openvpn.domain.com -> CNAME -> 123.dyndns.com
seafile.domain.com -> CNAME -> 123.dyndns.com

Yes, this is exactly what the CNAME record is for! https://en.wikipedia.org/wiki/CNAME_record

If you follow my guide and use desec.io as your DynDNS provider then you can even use a wildcard CNAME record. As you can see in the attached picture.
"*.tutorial.dedyn.io" means that "a.tutorial.dedyn.io, b.tutorial.dedyn.io, c.tutorial.dedyn.io, ..." will all point to "tutorial.dedyn.io" which points to "8.8.8.8".

So if you decide to use a wildcard subdomain, then you will only need to create that single subdomain "*.domain.com".
But keep in mind that not all DNS providers / domain registrars support this!

Quote from: Lip90 on September 01, 2021, 11:49:00 am

2nd question:
How to configure HAproxy for openvpn.
Could you explain it to me as well as you did in the tutorial or add the point to the tutorial?

For this your OpenVPN server needs to run in TCP mode. (Your clients need to connect to your OpenVPN server using a TCP tunnel.)
This is because HAProxy doesn't support UDP load balancing!

If you are using OpenVPN over TCP, then the configuration is pretty straight forward.
The only difference to my example Plex configuration is that you will have to set the backend "OpenVPN_backend" to TCP mode instead of HTTP mode.
See the FAQ "Why are we doing 2-Level-SNI?" and "How can we load balance TCP traffic that we don't want to get SSL offloaded, f.e. OpenVPN over TCP?".
This should explain how to set this up.
I suggest that you first make your Seafile server working and after that proceed with your OpenVPN server.

As I am currently not using OpenVPN or any other service that requires TCP load balancing, I won't be adding this to the tutorial any time soon.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 04, 2021, 06:20:01 pm

Thanks for your answer.

I have now followed your instructions exactly before I play with a changed configuration. When I try to call up the page cloudserver.xyz.dedyn.io, I always get a 503 Service Unavailable message.

Do you have any idea what I'm doing wrong?

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 05, 2021, 10:41:02 am

Post your HAProxy config. But remove your public IP or anything that reveals your identity. But leave all local IPs in it.
There is an export option in HAProxy.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 08, 2021, 04:21:35 pm

Hi,

this is my HAproxy config:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    option tcplog

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_6138b110159553.96461818 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6138b110159553.96461818

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6138b32401a006.77997133.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6138b15d48a964.28077676.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: SEAFILE_backend ()
backend SEAFILE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server SEAFILE_server 192.168.1.11:81 ssl verify none
Mapfile:

Code: [Select]

Name: PUBLIC_SUBDOMAINS_map
Content: cloudserver SEAFILE_backend
Do you need something else?

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sorano on September 08, 2021, 08:14:51 pm

Dropping this in here to make sure noone misses it:

https://forum.opnsense.org/index.php?topic=24668.0

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 08, 2021, 09:17:48 pm

This could be my Problem. How can i install the fix? Sorry for this dump question i did this never befoe.


Gesendet von iPhone mit Tapatalk

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sorano on September 08, 2021, 09:23:08 pm

Quote from: Lip90 on September 08, 2021, 09:17:48 pm

This could be my Problem. How can i install the fix?

Lol no!! Not at all. That is a security vulnerability and not your problem.

If the information you posted is correct your problem is that you are not using FQDN in your map file.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 08, 2021, 09:28:56 pm

Oh ok.

you mean i must change the mapfile to

cloudserver.domain.com SEAFILE_backend


Gesendet von iPhone mit Tapatalk

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sorano on September 08, 2021, 09:41:58 pm

Quote from: Lip90 on September 08, 2021, 09:28:56 pm

Oh ok.

you mean i must change the mapfile to

cloudserver.domain.com SEAFILE_backend


Gesendet von iPhone mit Tapatalk

Yeah

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 09, 2021, 11:49:52 am

Quote from: sorano on September 08, 2021, 08:14:51 pm

Dropping this in here to make sure noone misses it:

https://forum.opnsense.org/index.php?topic=24668.0

Thanks!

---

Quote from: sorano on September 08, 2021, 09:23:08 pm

If the information you posted is correct your problem is that you are not using FQDN in your map file.

Not necessarily, I think!
I also didn't place the whole FQDN in my mapfiles, instead I am just using the subdomains.
This works just fine. But I guess it is arguable if using the complete FQDN is the better option.
https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/syntax/

Code: [Select]

plex PLEX_backend
iot IOT_backend
...

---

Quote from: Lip90 on September 08, 2021, 04:21:35 pm

Code: [Select]

# Backend: SEAFILE_backend ()
backend SEAFILE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server SEAFILE_server 192.168.1.11:81 ssl verify none

Your map file should still work even without using FQDNs.

Your HAProxy config also looks fine.
The only thing that makes me curious is your "SEAFILE_server" configuration!
You might wanna take a look at the "configured port" or the "SSL checkbox"!
I highly doubt that your seafile server is listening on port 81 for HTTPS traffic, which is what you have configured by ticking the SSL checkbox.
The SSL checkbox means that the server is using HTTPS/SSL on the specified port. However if the server isn't actually using HTTPS/SSL on that port then the connection will obviously not work.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 09, 2021, 04:11:09 pm

Many thanks for the help. FQDN is not necessary.

By changing to no ssl, the Seafile server now works.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 10, 2021, 09:27:30 am

Let me know if you need anymore help with your OpenVPN config.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 10, 2021, 08:09:51 pm

I need your help again.   ;D

Code: [Select]

root@OPNsense:~ # cat /usr/local/etc/haproxy.conf
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    option tcplog

    # ACTION: NOSSLservice_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/613b963c5f0851.94679524.txt)]

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_6138b110159553.96461818 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6138b110159553.96461818

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6138b32401a006.77997133.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6138b15d48a964.28077676.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: SEAFILE_backend ()
backend SEAFILE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server SEAFILE_server 192.168.30.16:80

# Backend: OPENVPN_backend ()
backend OPENVPN_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server OPENVPN_server 127.0.0.1:1194


Code: [Select]

2021-09-10T20:00:44 haproxy[11387] 192.168.1.231:51903 [10/Sep/2021:20:00:44.614] 0_SNI_frontend SSL_backend/SSL_server 1/0/4 0 -- 1/1/0/0/0 0/0
2021-09-10T20:00:44 haproxy[11387] 192.168.1.231:51903 [10/Sep/2021:20:00:44.615] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T20:00:40 haproxy[11387] 192.168.1.231:51902 [10/Sep/2021:20:00:40.526] 0_SNI_frontend SSL_backend/SSL_server 1/0/5 0 -- 1/1/0/0/0 0/0
2021-09-10T20:00:40 haproxy[11387] 192.168.1.231:51902 [10/Sep/2021:20:00:40.527] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 0_SNI_frontend SSL_backend/SSL_server 1/0/39 0 -- 1/1/0/0/0 0/0
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T19:59:26 haproxy[11387] xx.xx.xx.162:25707 [10/Sep/2021:19:59:26.004] 0_SNI_frontend SSL_backend/SSL_server 1/0/35 0 -- 1/1/0/0/0 0/0

MAP_file:
(https://i.ibb.co/wBVxxYG/map.jpg) (https://ibb.co/pbc99nV)

OPENVPN_settings:
(https://i.ibb.co/VDRxbhg/opnvpn-server-conf.jpg) (https://ibb.co/1K1MBpr)

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 12, 2021, 10:42:17 pm

Okay, first you should change the listening interface of the OpenVPN plugin from "any" to "localhost".
Because this is where HAProxy is sending the traffic to since you defined your "OpenVPN real server = 127.0.0.1".

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 13, 2021, 12:34:49 pm

Hi, I changed the interface, the problem remains the same. It looks like the VPN access is not forwarded by the SNI but by the HTTPS server, right? Unfortunately I don't know how to work around the problem.

Code: [Select]

2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 0_SNI_frontend SSL_backend/SSL_server 1/0/39 0 -- 1/1/0/0/0 0/0
2021-09-10T19:59:30 haproxy[11387] xx.xx.xx.162:25819 [10/Sep/2021:19:59:30.212] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-10T19:59:26 haproxy[11387] xx.xx.xx.162:25707 [10/Sep/2021:19:59:26.004] 0_SNI_frontend SSL_backend/SSL_server 1/0/35 0 -- 1/1/0/0/0 0/0

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 13, 2021, 06:38:48 pm

Is your OpenVPN client config correct?
- mode tcp
- port 443
- server vpn.domain.com

Some of this must be wrong.
Otherwise your SNI_frontend wouldn't pass the traffic to the configured default backend (SSL_backend) which is what it is currently doing.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 13, 2021, 07:43:52 pm

here is the extract of my configuration. I think that is all correct.

Code: [Select]

dev tun
persist-tun
persist-key
proto tcp-client
cipher AES-256-CBC
auth SHA256
client
resolv-retry infinite
remote vpn.xxxxx.dedyn.io 443 tcp
lport 0

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nullex on September 13, 2021, 10:05:21 pm

Hello,

I'm having trouble with this guide.. Very strange trouble...
I have followed the steps to a T.. except I am using cloudflare for Dynamic DNS and ACME/let's encrypt Certs. Also, I am using localhost/127.0.0.1 instead of the virtual IP option. Although I have tried it both ways with the same results...

The problem is.. weird.. So, like I said, followed the steps.. But only a few backend servers work. I have set up Plex, Sonarr, Radarr, Ombi, Cams(blueIris) and some other stuff. However, only Plex, Cams, and Radarr work. I've been fighting with it for a while, and previously only Plex and Cams were working.. not sure what I changed to make Radarr work.. but everything else just gives me Error 503 Service Unavailable.. Even though the service is up and running and is accessible through LAN.

My opnsense is fully updated, 21.7.2_1 and os-haproxy 3.5

My HAProxy config is as follows:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug
cache opnsense-haproxy-cache
    total-max-size 512
    max-age 60

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on localhost:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_613eabd9cb19a0.51810931 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_613eabd9cb19a0.51810931

# Frontend: 1_HTTPS_frontend (Listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/613eae5151edb0.32207081.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/613eac85c00a60.86291436.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: Radarr_backend ()
backend Radarr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Radarr_server 192.168.1.111:7878

# Backend: Plex_backend ()
backend Plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Plex_server 192.168.1.159:32400

# Backend: Cams_backend ()
backend Cams_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Cams_server 192.168.1.10:81

# Backend: Ombi_backend ()
backend Ombi_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Ombi_server 192.168.1.159:5055

# Backend: Sonarr_backend ()
backend Sonarr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Sonarr_server 192.168.1.111:8989

# Backend: Tautulli_backend ()
backend Tautulli_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Tautulli_server 192.168.1.7:8181
My map File currently looks like:

Code: [Select]

sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
Ombi Ombi_backend
Tautulli Tautulli_backend
Any advice as to what I'm doing wrong? Why would some services work and others don't when they're using the same exact config?

What's also funny is that if I reconfigure the port and IP on the one of the Real Servers that works (For example, Plex_Server has IP 192.168.1.159:32400, Ombi_server has 192.168.1.159:5055, and I replace Plex_Server port to 5055 and go to Plexi.DOMAIN.com, Ombi pops up like it should. I've cloned new Real servers and Backend Pools while updating the PUBLIC_SUBDOMAINS_map  from the working Plex one, but still no go for ALL of my services.

I've also discovered that if I modify HAProxy Rules & Checks > Rules > Public_subdomains_map-rule > Default backend pool... and change it to a service that DOESN'T work with the map file.. when I hit apply I'm able to access that service on ANY rendition of my domain, as well as the root domain.com address... And if I leave the services that DO work in the map file (Plex), plex.domain.com displays plex as it should, while the rest of the domain is showing the service that doesn't work on it's own.. which further doesn't make any sense.. The map file is working for some services but not others?

Thanks in advance, I'm stumped.

Final edit:

LMFAO I think I figured out why it wasn't working... so apparently for the map file to work you have to have the first part all lowercase, cannot use any uppercase...

So when I changed my map file to:

Code: [Select]

sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
ombi Ombi_backend
tautulli Tautulli_backend

everything started working...

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: budimanjojo on September 14, 2021, 08:56:11 am

Thank you so much for this tutorial! But I have a problem and I don't know what should I do.  :'(
Maybe somebody can help me here.

I already have a Traefik reverse proxy running outside OPNSense listening on 192.168.200.244 port 80 and 443. That Traefik reverse proxying all my kubernetes cluster services including certificates handling for 2 domain names.

Now, what I want to is to have HAProxy in OPNSense to be the reverse proxy for my Traefik. For example:
- My domain names are 1stdomain.com and 2nddomain.com.
- Have a rule that: if the client go to opnsense.1stdomain.com, route it to localhost:55443 (OPNSense itself), else if the client go to *.1stdomain.com or *.2nddomain.com, route it to 192.168.200.244:443.

I don't know where to start, I tested using the tuturial but instead of SNI and HTTP frontend, I created 2 SNI frontend services but HAProxy refused to start at all.

There is an easier route for me though, that is to just create a route in Traefik to my OPNSense and be done with it. But I prefer having a firewall level reverse proxy so I can have another layer to let's say block external access like in this tutorial. Thanks before.  :)

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 01:08:53 pm

@TheHellSite

I think the problem is with the SNI frontend. Here the SSL backend is specified as the default backend. He doesn't even look at the MAP file. he forwards everything to the SSL backend. When I set the openvpn backend as default Backend for a test in the SNI frontend, openvpn work but the other things not.

do you have an idea how I can solve this?

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 02:31:50 pm

Quote from: Lip90 on September 14, 2021, 01:08:53 pm

@TheHellSite

I think the problem is with the SNI frontend. Here the SSL backend is specified as the default backend. He doesn't even look at the MAP file. he forwards everything to the SSL backend. When I set the openvpn backend as default Backend for a test in the SNI frontend, openvpn work but the other things not.

do you have an idea how I can solve this?

I was just about to write you exactly this!  ;D
Your reply confirmed my guess.

Looking through the manual pages of HAProxy it seems that the "Default Backend" setting can only be overwritten by a "Use Backend" rule! Which a "Use map file" rule isn't able to.
https://www.haproxy.com/de/blog/the-four-essential-sections-of-an-haproxy-configuration/
But I can't imagine that this is the intended behaviour.

Anyways... you simply need to create a VPN_condition "host starts with vpn" and a "use backend OPENVPN_backend if VPN_condition=true" rule.
Add this rule to the SNI_frontend and set the default backend back to the SSL_backend.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 05:35:30 pm

Quote from: nullex on September 13, 2021, 10:05:21 pm

My map File currently looks like:

Code: [Select]

sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
Ombi Ombi_backend
Tautulli Tautulli_backend
Any advice as to what I'm doing wrong? Why would some services work and others don't when they're using the same exact config?

What's also funny is that if I reconfigure the port and IP on the one of the Real Servers that works (For example, Plex_Server has IP 192.168.1.159:32400, Ombi_server has 192.168.1.159:5055, and I replace Plex_Server port to 5055 and go to Plexi.DOMAIN.com, Ombi pops up like it should. I've cloned new Real servers and Backend Pools while updating the PUBLIC_SUBDOMAINS_map  from the working Plex one, but still no go for ALL of my services.

I've also discovered that if I modify HAProxy Rules & Checks > Rules > Public_subdomains_map-rule > Default backend pool... and change it to a service that DOESN'T work with the map file.. when I hit apply I'm able to access that service on ANY rendition of my domain, as well as the root domain.com address... And if I leave the services that DO work in the map file (Plex), plex.domain.com displays plex as it should, while the rest of the domain is showing the service that doesn't work on it's own.. which further doesn't make any sense.. The map file is working for some services but not others?

Thanks in advance, I'm stumped.

Final edit:

LMFAO I think I figured out why it wasn't working... so apparently for the map file to work you have to have the first part all lowercase, cannot use any uppercase...

So when I changed my map file to:

Code: [Select]

sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
ombi Ombi_backend
tautulli Tautulli_backend

everything started working...

Glad it is working for you know.
My first guess where some misconfigured real servers (ports, ssl, ssl-verify).

BTW: Your map file is exposing your domain name! You should remove it from the forum post.

Also I did a quick scan of your domain using https://dnsdumpster.com (https://dnsdumpster.com). It lists all your subdomains since you created a single "A Record" for each of them. Consider switching to a "Wildcard A Record" in order to hide them!
If an attacker can see what services you are running it makes it easier for them to find an attack surface.

You can then still create individual a records, f.e. www.domain.tld, since the wildcard a record is resolved after all other a records have been resolved.

This is why my tutorial is using a "Wildcard A Record / Subdomain" in the form of "*.domain.tld".

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 08:23:42 pm

Still the same  :(. Any other idea :)
(https://i.ibb.co/BnKQ9sr/rule.jpg) (https://ibb.co/XYkKqJ3)
(https://i.ibb.co/nf1gxv3/condition.jpg) (https://ibb.co/VDCjfZH)
(https://i.ibb.co/37kGNLD/SNI-Frontend.jpg) (https://ibb.co/DtLsCSx)
(https://i.ibb.co/PgLz2SH/Public-Service.jpg) (https://ibb.co/bX0R4f9)
(https://i.ibb.co/FB0pX4G/logfile.jpg) (https://ibb.co/rZc932n)

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 08:39:39 pm

Please remove the default backend but leave the vpn_rule enabled.

I have another guess...

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 08:49:09 pm

does not work too. i don't think he matched the names

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 08:57:24 pm

Invert the vpn condition and name it "NoVPN_condition".
Rename the vpn rule to "NoVPN_rule" and change it to use SSL_backend.

Test it out.
1. Only the NoVPN_rule on the SNI_frontend.
2. NoVPN_rule on the SNI_frontend along with SSL_backend as default.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 09:23:55 pm

Still the Same :)

Code: [Select]

2021-09-14T21:21:05 haproxy[3256] 80.187.80.8:10670 [14/Sep/2021:21:21:05.818] 0_SNI_frontend SSL_backend/SSL_server 1/0/37 0 -- 1/1/0/0/0 0/0
2021-09-14T21:21:05 haproxy[3256] 80.187.80.8:10670 [14/Sep/2021:21:21:05.818] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure
2021-09-14T21:21:01 haproxy[3256] 80.187.80.8:10495 [14/Sep/2021:21:21:01.658] 0_SNI_frontend SSL_backend/SSL_server 1/0/36 0 -- 1/1/0/0/0 0/0
2021-09-14T21:21:01 haproxy[3256] 80.187.80.8:10495 [14/Sep/2021:21:21:01.658] 1_HTTPS_frontend/192.168.64.1:443: SSL handshake failure

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 14, 2021, 09:30:49 pm

For which try?
1st or second?

If you remove the VPN_backend as default backend and place the novpn_rule in the frontend. Can you still access your seafile Server?
It should be accessible. Is it?
I want to test the sni functionality with this.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 09:41:28 pm

Your first question:
For both.

The second:
Yes i can access the Seafile server with any config. The haproxy give every request to the ssl backend.


Gesendet von iPhone mit Tapatalk

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Lip90 on September 14, 2021, 09:43:05 pm

It looks like that sni don’t work by OpenVPN


Gesendet von iPhone mit Tapatalk

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nullex on September 15, 2021, 12:06:38 am

Quote from: TheHellSite on September 14, 2021, 05:35:30 pm

Quote from: nullex on September 13, 2021, 10:05:21 pm

My map File currently looks like:

Code: [Select]

sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
Ombi Ombi_backend
Tautulli Tautulli_backend
Any advice as to what I'm doing wrong? Why would some services work and others don't when they're using the same exact config?

What's also funny is that if I reconfigure the port and IP on the one of the Real Servers that works (For example, Plex_Server has IP 192.168.1.159:32400, Ombi_server has 192.168.1.159:5055, and I replace Plex_Server port to 5055 and go to Plexi.DOMAIN.com, Ombi pops up like it should. I've cloned new Real servers and Backend Pools while updating the PUBLIC_SUBDOMAINS_map  from the working Plex one, but still no go for ALL of my services.

I've also discovered that if I modify HAProxy Rules & Checks > Rules > Public_subdomains_map-rule > Default backend pool... and change it to a service that DOESN'T work with the map file.. when I hit apply I'm able to access that service on ANY rendition of my domain, as well as the root domain.com address... And if I leave the services that DO work in the map file (Plex), plex.domain.com displays plex as it should, while the rest of the domain is showing the service that doesn't work on it's own.. which further doesn't make any sense.. The map file is working for some services but not others?

Thanks in advance, I'm stumped.

Final edit:

LMFAO I think I figured out why it wasn't working... so apparently for the map file to work you have to have the first part all lowercase, cannot use any uppercase...

So when I changed my map file to:

Code: [Select]

sonarr Sonarr_backend
radarr Radarr_backend
plex Plex_backend
cams Cams_backend
ombi Ombi_backend
tautulli Tautulli_backend

everything started working...

Glad it is working for you know.
My first guess where some misconfigured real servers (ports, ssl, ssl-verify).

BTW: Your map file is exposing your domain name! You should remove it from the forum post.

Also I did a quick scan of your domain using https://dnsdumpster.com (https://dnsdumpster.com). It lists all your subdomains since you created a single "A Record" for each of them. Consider switching to a "Wildcard A Record" in order to hide them!
If an attacker can see what services you are running it makes it easier for them to find an attack surface.

You can then still create individual a records, f.e. www.domain.tld, since the wildcard a record is resolved after all other a records have been resolved.

This is why my tutorial is using a "Wildcard A Record / Subdomain" in the form of "*.domain.tld".

Thanks for catching that, thought I removed all PII but apparently not lol

So I think DNSDumpster is showing those domains because I was previously using dynamic DNS on EACH of those domains and using caddy as a reverse proxy.. which all those WERE direct A records, but since following your guide I switched to wildcard domain so hopefully it should be fixed once DNSDumpster's Database is updated.. I'll keep an eye on it. Thanks again!

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Prismatic on October 06, 2021, 09:35:05 pm

Hello. First of all, many thanks for your tutorial.... working like a charm, even for an (enlightened)  newbie like me....

Im using a Synology NAS + Docker  with different services. So they are available on same LAN IP adress, but different Ports....

I would llike to setup the "access from internatl network" as on your part 6. I do understand that with my setup I cannot use the unbound split DNS option, as this doesnt work with ports...

So I'm relying on option A "NAT reflexion", but I'm unable to find it, inside the created "HAproxy rule", in Firewall/WAN section..... I do find this option in Firewall/NAT section rules, but we didnt created nothing there....

Could you please help ? and update your tutorial with a screenshot of this ?

Many thanks !

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 06, 2021, 11:17:36 pm

Quote from: Prismatic on October 06, 2021, 09:35:05 pm

Im using a Synology NAS + Docker  with different services. So they are available on same LAN IP adress, but different Ports....

I would llike to setup the "access from internatl network" as on your part 6. I do understand that with my setup I cannot use the unbound split DNS option, as this doesnt work with ports...

Of course you can use split DNS!
haproxy is handling the port scenario.

Example:
10.1.23.55:4456 = NEXTCLOUD = cloud.yourdomain.tld
10.1.23.55:4457 = PLEX = plex.yourdomain.tld

Read the Split dns part again!
You have to rewrite ALL of your 1st/2nd-level-subdomains with same IP of your OPNsense that HAProxy is listening on, f.e. the lan ip.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Prismatic on October 07, 2021, 12:55:24 am

Thanks for your answer !

Apologies, but I still don't understand something....

So my services IP & port are configured inHAproxy map file. (everything working fine, comming from WAN with certificate)..
.
this server IP for me on LANis 192.168.1.20
only using 1st level subdomains like:

audio.mydomain.tld
photo.mydomain.tld
video.mydomain.tld
etc...

My "LAN" port for opnsense on appliance is 192.168.1.1 and I did used a "virtual IP" as per your tutorial which is 192.168.50.1...

Using "hosts overrides in unbound" (not "domain") should my configuration look like this :

HOST : audio
DOMAIN: mydomain.tld
IP : 192.168.50.1 or 192.168.1.1  or 192.168.1.20 ??

tried all solutions, but I don't know if it works.... a "tracert" shows that I am always hitting my static public IP adress....

sorry for my approximative english and many thanks for your answer.
cheers !


 

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 08, 2021, 08:46:00 am

Quote from: Prismatic on October 07, 2021, 12:55:24 am

So my services IP & port are configured inHAproxy map file. (everything working fine, comming from WAN with certificate)..
.
this server IP for me on LANis 192.168.1.20
only using 1st level subdomains like:

audio.mydomain.tld
photo.mydomain.tld
video.mydomain.tld
etc...

My "LAN" port for opnsense on appliance is 192.168.1.1 and I did used a "virtual IP" as per your tutorial which is 192.168.50.1...

Using "hosts overrides in unbound" (not "domain") should my configuration look like this :

HOST : audio
DOMAIN: mydomain.tld
IP: 192.168.1.1

IP has to be the lan ip of your opnsense.
Which I very well tell to do in my tutorial.
You should probably read it word by word again!

Also post your HAProxy config export in a code box.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 08, 2021, 05:45:47 pm

Hey all,

Is there a way to tie this in with Cloudflare?
I know ACME can use Cloudflare, but can someone add in where I need to do this for wildcards?

Also I installed the plugins and I don't see LetsEncrypt, rather I see ACME Client, and the settings are a bit different.
Any chance you can update the images?

Also I've tried to follow this to the best of my abilities.
I am getting: 503 Service Unavailable
No server is available to handle this request.

If I click on the padlock it is showing my certificate so I think its hitting right.
I'm doing things for:
homeassistant
jellyfin
radarr
sonarr

All of them say 503.

If I set a backend to maintenance I get Safari can't find the server.
So seems to be hitting it correctly.

Ok not sure what changed but now its no dice. Just server cannot be found

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 08, 2021, 11:45:41 pm

Sorry but I can't help with individual domain / dns registrars, this would simply take way to much of my time. You will have to study cloudflares documentation on your own to make it work with them.
Only difference should be the Dynamic DNS and ACME configuration, to get your certificate.
HAProxy settings are the same for every registrar.

The Let's Encrypt plugin has been renamed to ACME Client in one of the recent OPNsense updates. Apart from that nothing else has changed so all pictures are still valid. No need to change them.
I will update the text when I find the time.

Your error: Post your HAProxy export in a code box. Remove sensitive information (Public IP, Domain Name).

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 09, 2021, 10:46:19 am

Thank for taking the time to help.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_Frontend ()
frontend SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTP_Frontend ()
frontend HTTP_Frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: No_SSL_condition
    acl acl_6160768c129757.05678189 req.ssl_ver gt 0

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_6160768c129757.05678189

# Frontend: HTTPS_Frontend ()
frontend HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6160790e8358e2.93807756.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: Public_Subdomain_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61607700a205d2.15896401.txt)]

# Backend: SSL_backend (HAProxy SSL Backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: homeassistant_backend (Home Assistant Backend)
backend homeassistant_backend
    # health check: homeassistant_tcp_check
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server homeassistant 10.0.0.9:8123 check inter 2s port 8123  ssl verify none

# Backend: jellyfin_backend (Jellyfin Backend)
backend jellyfin_backend
    # health check: jellyfin_http_check
    option httpchk
    http-check send meth OPTIONS uri / ver HTTP/1.0
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server jellyfin 10.0.0.10:8090 check inter 2s port 8090  ssl verify none

The only difference I think I made was to add health checks so I can see if service is UP.
It is showing as UP so that looks good.

And my MAP looks like:

Code: [Select]

# public access subdomains
homeassistant homeassistant_backend
jellyfin jellyfin_backend
Oh one other thing to note is I have dual WAN setup.
I had this on pfSense too and HAProxy was working perfectly fine.
I just had all the external connections go through 1 WAN only which I did again here.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 09, 2021, 02:57:49 pm

Please confirm you are accessing your servers internally using:

http://10.0.0.10:8090/
http://10.0.0.9:8123/
So no https?

Even though it shouldn't matter test the access with health checks disabled.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 09, 2021, 03:17:58 pm

Quote from: TheHellSite on October 09, 2021, 02:57:49 pm

Please confirm you are accessing your servers internally using:

http://10.0.0.10:8090/
http://10.0.0.9:8123/
So no https?

Even though it shouldn't matter test the access with health checks disabled.

Correct if I hit those 2 IP's above internally I get the service correctly.
I did have it running with health checks disabled and same result 503 errors

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 09, 2021, 03:39:32 pm

Is opnsense the only router / firewall in your network?
To me it looks like your servers are in a totally different subnet / vlan.

Please give a little info about your local network layout.
I think your opnsense has no access to your server network!

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 09, 2021, 04:07:09 pm

Yes OPNsense is my only router/firewall.

I have dual WAN created a gateway group to double internet speed.
I created the rule on only 1 WAN gateway tho, and my a records point to the same IP.

I have 3 LANs (not VLANS) 10.0.0.1 (Main LAN) - 192.168.107.1 (IoT) - 192.168.200.1 (Guest)
My NAS Server (10.0.0.10) and OPNsense (10.0.0.1) are on 10.0.0.1/24 LAN, so no going through anything different there.

I wonder if with the dual WAN it needs a specific rule?
As with dual WAN I have to change the default gateway on the Allow LAN to any rule to the specific Multi_Gateway...
Like my LAN can no longer talk to my IOT Lan without having a specific rule to say allow...

I've just added the LAN Net to Loopback but still no dice

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 09, 2021, 05:08:43 pm

The error is not related to your dual WAN setup.  ;D
I misread something in your HAProxy config...

Code: [Select]

server jellyfin 10.0.0.10:8090 check inter 2s port 8090  ssl verify noneYou see that "ssl" word in your server config?
SSL and HTTP don't like each other!  ;)

Since your services are NOT using HTTPS but only HTTP locally... you need to UNCHECK the SSL checkbox in the HAProxy real server config for both of your servers.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on October 09, 2021, 05:58:49 pm

Quote from: TheHellSite on October 09, 2021, 05:08:43 pm

The error is not related to your dual WAN setup.  ;D
I misread something in your HAProxy config...

Code: [Select]

server jellyfin 10.0.0.10:8090 check inter 2s port 8090  ssl verify noneYou see that "ssl" word in your server config?
SSL and HTTP don't like each other!  ;)

Since your services are NOT using HTTPS but only HTTP locally... you need to UNCHECK the SSL checkbox in the HAProxy real server config for both of your servers.

Oh my days!!!
That worked! I spent all night trying to figure it out.

Thank you so very much!

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: blackhand27 on October 15, 2021, 06:06:51 am

Great Guide! Took me a bit since im using Godaddy for my domain name but figured out the cert mess and got my plex server working. Been hoping to get this setup so I can run a Synapse server at home.

I do seem to be having issues with my bitwarden site, I have the proper port set in HAProxy however I have a feeling its something with its configuration still using the internal cert vs the offload. Would anyone have some advice to see what I can look for?

figured it out, I reinstalled the server using self signed certs, for whatever reason it was still using the builtin lets encrypt certs and failing to connect.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on October 15, 2021, 07:25:02 pm

Quote from: TheHellSite on September 14, 2021, 02:31:50 pm

Quote from: Lip90 on September 14, 2021, 01:08:53 pm

@TheHellSite

I think the problem is with the SNI frontend. Here the SSL backend is specified as the default backend. He doesn't even look at the MAP file. he forwards everything to the SSL backend. When I set the openvpn backend as default Backend for a test in the SNI frontend, openvpn work but the other things not.

do you have an idea how I can solve this?

I was just about to write you exactly this!  ;D
Your reply confirmed my guess.

Looking through the manual pages of HAProxy it seems that the "Default Backend" setting can only be overwritten by a "Use Backend" rule! Which a "Use map file" rule isn't able to.
https://www.haproxy.com/de/blog/the-four-essential-sections-of-an-haproxy-configuration/
But I can't imagine that this is the intended behaviour.

Anyways... you simply need to create a VPN_condition "host starts with vpn" and a "use backend OPENVPN_backend if VPN_condition=true" rule.
Add this rule to the SNI_frontend and set the default backend back to the SSL_backend.

Am I confused about this?

It looks like the current set up on page one will not work due to "default backend"??

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 19, 2021, 06:06:55 pm

Quote from: lilsense on October 15, 2021, 07:25:02 pm

Quote from: TheHellSite on September 14, 2021, 02:31:50 pm

I was just about to write you exactly this!  ;D
Your reply confirmed my guess.

Looking through the manual pages of HAProxy it seems that the "Default Backend" setting can only be overwritten by a "Use Backend" rule! Which a "Use map file" rule isn't able to.
https://www.haproxy.com/de/blog/the-four-essential-sections-of-an-haproxy-configuration/
But I can't imagine that this is the intended behaviour.

Anyways... you simply need to create a VPN_condition "host starts with vpn" and a "use backend OPENVPN_backend if VPN_condition=true" rule.
Add this rule to the SNI_frontend and set the default backend back to the SSL_backend.

Am I confused about this?

It looks like the current set up on page one will not work due to "default backend"??

I am not sure what you mean?
The setup still works as described. I also got OpenVPN over TCP working but haven't had the time yet to add it to my tutorial.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: thegriffen on October 27, 2021, 12:37:59 am

Hi everyone

First of all.. Awesome guide and even so, that you update it with new stuff as it comes along.

Iv been kinda hitting my head for a day or two now and gotta throw in the towel, and put my troubles in here.

im running a small test setup, where im gonna have a couple of web services running, but the thing is here, they are running traefik with there own LE ssl validation and so. but i cant get HAproxy to work propper.

My config looking like this:

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend ()
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:8888 name 0.0.0.0:8888
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options
    option tcplog

    # ACTION: Public_Domain_Map_Rule_2
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6178606dd3b431.48344137.txt)]
    # ACTION: Public_Domain_Map_Rule_3
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6178704b1d59f6.07557436.txt)]

# Frontend: 1_Http_Frontend ()
frontend 1_Http_Frontend
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: NoSSL_Condition
    acl acl_61781026415d99.64392811 req.ssl_ver gt 0

    # ACTION: HttpToHttps_Rule
    http-request redirect scheme https code 301 if !acl_61781026415d99.64392811

# Frontend: 1_Https_Frontend ()
frontend 1_Https_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    option httplog

    # ACTION: Public_Domain_Map_Rule_1
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/617812015049f9.00974692.txt)]

# Backend: SSL_Backend (Loopback)
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: Central_OpenSense_FW_Backend (OpenSense Firewal)
backend Central_OpenSense_FW_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Central_OpenSense_FW_SRV 10.10.5.2:55443 ssl verify none

# Backend: FRCD_Backend (domain1.com)
backend FRCD_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server DMZ_domain1_SRV 10.10.20.40:443 ssl alpn h2,http/1.1,http/1.0 verify none

# Backend: FRCD_2_Backend (domain2:8888)
backend FRCD_2_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server DMZ_domain2_SRV 10.10.20.41:8888

Best regrads

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 31, 2021, 10:44:49 am

Quote from: thegriffen on October 27, 2021, 12:37:59 am

Hi everyone

First of all.. Awesome guide and even so, that you update it with new stuff as it comes along.

Iv been kinda hitting my head for a day or two now and gotta throw in the towel, and put my troubles in here.

im running a small test setup, where im gonna have a couple of web services running, but the thing is here, they are running traefik with there own LE ssl validation and so. but i cant get HAproxy to work propper.

I don't really understand what you need haproxy for if you are already using traefik?
Can you please elaborate further what...

1. You are trying to achieve by using HAProxy?

2. What issue you are facing with HAProxy right now?

3. Explain your local network layout further!

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bringha on November 04, 2021, 10:46:42 am

Hi there,

first of all a big big thank you for this awesome and comprehensive tutorial. Very helpfull and a great contribution.

I have an additional question and I am not sure whether I suffer from a big misuderstanding.

I configured my Dyndns as suggested with dedyn.io and have now a domain.dedyn.io properly working. Your tutorial now assumes to create wildcard certificates for the *.domain.dedyn.io (in my case)

I have a main domain registered with a poster somewhere else which is domain.com. Historically I reach my dyndns based subdomains via CNAME DNS entries at my main domain provider's DNS systems, eg home.domain.com points then to home.domain.dedyn.io.

It is now possible to let the acme client generate wildcard certificates also for *.domain.com accordingly in addition/replacing the wildcard certs for *.domain.dedyn.io?

Looking forward to your reply.

Br br

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 04, 2021, 07:15:20 pm

Quote from: bringha on November 04, 2021, 10:46:42 am

I configured my Dyndns as suggested with dedyn.io and have now a domain.dedyn.io properly working. Your tutorial now assumes to create wildcard certificates for the *.domain.dedyn.io (in my case)

I have a main domain registered with a poster somewhere else which is domain.com. Historically I reach my dyndns based subdomains via CNAME DNS entries at my main domain provider's DNS systems, eg home.domain.com points then to home.domain.dedyn.io.

It is now possible to let the acme client generate wildcard certificates also for *.domain.com accordingly in addition/replacing the wildcard certs for *.domain.dedyn.io?

Basically there are two options to solve this

1. You replace "deSEC" as your DynDNS provider and use the DynDNS of your main domain registrar. If he supports this...
--> What is your domain registrar? (Where your real "domain.com" is registered...)

2. You transfer the DNS zone from your main "domain.com" to deSEC. Again, if your current domain registrar supports this.
This way your domain is still hosted at your current domain registrar, but deSEC will be managing all the DNS related stuff. (A records, ...)

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bringha on November 04, 2021, 07:44:54 pm

Thank you for your answer!

My domain Registrar is Ionos. Ionos has also a dyndns service and a dns management api and is contained in the opnsense dyndns list. I would like to leave the services of the main Domain there (mail, Webhosting) and have only the subdomains with my local dyndns based site.

Br br

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 04, 2021, 07:59:11 pm

IONOS works exactly like deSEC, just adopt the deSEC stuff to the IONOS settings.
get a dns API key, set up dyn dns and get your certs.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on November 05, 2021, 09:29:43 pm

Hey @TheHellSite, Do you know a way I can add Basic Auth to one of the sub domains?

I have a dashboard which just runs without login and I would really like it secured behind just basic auth if possible.
I tried creating a condition and a rule and applying it to the sub domain but it doesnt seem to work.

I also saw on the backend there is an option on the domain for basic auth checkbox, so I tried that but no dice either.

Any chance you can add a section to the guide about securing singular domains with basic auth?

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 06, 2021, 11:00:20 am

Quote from: N0_Klu3 on November 05, 2021, 09:29:43 pm

Hey @TheHellSite, Do you know a way I can add Basic Auth to one of the sub domains?

I have a dashboard which just runs without login and I would really like it secured behind just basic auth if possible.
I tried creating a condition and a rule and applying it to the sub domain but it doesnt seem to work.

I also saw on the backend there is an option on the domain for basic auth checkbox, so I tried that but no dice either.

Any chance you can add a section to the guide about securing singular domains with basic auth?

Definitely will be looking into this. You can expect this in the coming weeks, hopefully.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on November 06, 2021, 11:29:59 am

AMAZING! Thank you so much

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bringha on November 06, 2021, 10:15:27 pm

Hi there,

I tried to activate DDNS via the API with ionos as suggested but miserably failed yet.  ??? Might be somewhat unusual but I only want to activate DDNS for two subdomains, not my entire domain ‚domain.com‘ (only ‚sub.domain.com‘).

 I come up to the step that I have created my update URL, the server responds with 200 properly on the Ionos API page. This should then have activated DDNS for the requested subdomains according to the doc. However there is the no entry for these subdomain on the DNS page.

When I  apply Update URL  in the opnsense dyndns config and press ‚save and enforce update‘, my public IP is properly shown in the opnsense dyndns config page.  But still the subdomain is not shown as a DynDNS subdomain at ionos ..
as to be feared there is also no DNS with my public IP for the subdomain distributed …

Do i miss a step ?? I am looking forward to any idea.

Br br

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: N0_Klu3 on November 08, 2021, 10:13:16 am

FYI I got it working the Basic Auth.

So 21.7.4 has an issue with HAProxy where changes to config are not being saved: https://forum.opnsense.org/index.php?topic=25480.0
After I applied the patch it is working.

For Basic Auth all I did was create a user under User Management
(There does seem to be some restrictions around password length or complexity, I didnt spend a load of time testing)
My 20+ Char passwords would not work until I dumbed it down a bit.

So once you have a user go to your Backend Pools, choose your desired service, scroll down to Basic Authentication -> tick the box and add the username you just created.

Boom it works.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 10, 2021, 03:33:49 pm

Quote from: bringha on November 06, 2021, 10:15:27 pm

Hi there,

I tried to activate DDNS via the API with ionos as suggested but miserably failed yet.  ??? Might be somewhat unusual but I only want to activate DDNS for two subdomains, not my entire domain ‚domain.com‘ (only ‚sub.domain.com‘).

 I come up to the step that I have created my update URL, the server responds with 200 properly on the Ionos API page. This should then have activated DDNS for the requested subdomains according to the doc. However there is the no entry for these subdomain on the DNS page.

When I  apply Update URL  in the opnsense dyndns config and press ‚save and enforce update‘, my public IP is properly shown in the opnsense dyndns config page.  But still the subdomain is not shown as a DynDNS subdomain at ionos ..
as to be feared there is also no DNS with my public IP for the subdomain distributed …

Do i miss a step ?? I am looking forward to any idea.

Br br

Your issue is related to DynDNS I think you should fix that first and ask for help at the right place.
Maybe use google "how to set up a DynDNS subdomain at IONOS".
https://www.ionos.de/hilfe/domains/ip-adresse-konfigurieren/dynamisches-dns-ddns-einrichten-bei-company-name/

I am sorry, but since I am not using IONOS you will have to figure out their specifics on your own.
The only differences to my guide will be the DynDNS setup and the DNS challenge specifics for the ACME Client. Everything else is the same.

---

Quote from: N0_Klu3 on November 08, 2021, 10:13:16 am

FYI I got it working the Basic Auth.

So 21.7.4 has an issue with HAProxy where changes to config are not being saved: https://forum.opnsense.org/index.php?topic=25480.0
After I applied the patch it is working.

For Basic Auth all I did was create a user under User Management
(There does seem to be some restrictions around password length or complexity, I didnt spend a load of time testing)
My 20+ Char passwords would not work until I dumbed it down a bit.

So once you have a user go to your Backend Pools, choose your desired service, scroll down to Basic Authentication -> tick the box and add the username you just created.

Boom it works.

Just tested it out myself. Basic Auth is so easy to set up that I am not really willing to cover it in this guide.
First create the user(s) in HAProxy. Then in the relevant backends activate basic auth and select the user(s).

The issue you are facing with your password is the way you included them in your config!
You will need to SHOULD use a password hash not the plain text password in the user management box.
Otherwise your passwords are visible in plain text in the config export.

Generate password hash using OpenSSL binary
1. (WINDOWS ONLY) Download the openssl binary, f.e. from here: http://wiki.overbyte.eu/wiki/index.php/ICS_Download#Download_OpenSSL_Binaries_.28required_for_SSL-enabled_components.29
2. (Windows) Open CMD in the directory of the binary file / (Linux) open the shell.
3. Run "openssl passwd -6" and enter your password.
4. Enter the generated password hash in the password field of the user in HAProxy Settings.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 12, 2021, 04:09:12 pm

I am having issues with modifying config which will not take effect.

as an example when attempting to modify default-dh-param from 2048 to 4096, the save and apply works but the config export is not updated.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 15, 2021, 04:41:55 pm

It was fixed after the latest update, however, I am still having connections issues... I am getting handshake failures.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin expose-fd listeners
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend (listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_Frontend (Listening on 192.168.1.50:80)
frontend 1_HTTP_Frontend
    bind 192.168.1.50:80 name 192.168.1.50:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NOSSL_Condition
    acl acl_60f9d6d0118252.11362730 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_Rule
    http-request redirect scheme https code 301 if !acl_60f9d6d0118252.11362730

# Frontend: 1_HTTPS_Frontend (listening on 192.168.1.50:443)
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.1.50:443 name 192.168.1.50:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60f9db5421ce96.24863488.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: Public_Sub_MapRule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61698448328ff6.66158166.txt)]

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server HAP_VIP 192.168.1.50 send-proxy-v2 check-send-proxy

# Backend: TruePlex_Backend ()
backend TruePlex_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server TruePlex 192.168.1.12:32400 ssl verify none


Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 16, 2021, 10:01:22 am

Quote from: lilsense on November 15, 2021, 04:41:55 pm

It was fixed after the latest update, however, I am still having connections issues... I am getting handshake failures.

From all client devices or just your smart TV(s)?
Does web browser access work?

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 16, 2021, 11:41:05 am

From all clients, nothing works.

I can connect to the plex with the local IP without an issue.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 16, 2021, 12:44:25 pm

Quote from: lilsense on November 16, 2021, 11:41:05 am

From all clients, nothing works.

I can connect to the plex with the local IP without an issue.

1. Post a screenshot of your entire SSL Labs result. But black out any personal information from it, i.e. domain name, public IP,... You can also send me a personal message with the result link.
https://www.ssllabs.com/ssltest/


2. Go to "OPNsense --> System --> Trust --> Authorities" and post a screenshot of that page. If you want, you can black out all lines that don't have "Let's Encrypt" the name.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 16, 2021, 01:47:20 pm

There's an unknown cert and Cloudflare. I'll sent it in a bit.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 16, 2021, 02:26:15 pm

I thought you were asking about an HAProxy + Let's Encrypt issue...

Since you are using cloudflare certificates I am unable to help you. You are better off asking for help in the HAProxy forums or the cloudflare support regarding your issues.

Let me finish by giving you these informations:
1. The SSL Labs test pictures you sent me indicate that your certificate content (cn + alt name) seems to be wrong.
2. Your HAProxy HTTPS frontend settings do not match the ones I provide in my guide.


You could however just follow my tutorial step by step and would end up with a working setup. ;)

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on November 17, 2021, 03:04:01 pm

Thanks TheHellSite for the time taken to share the setup and help people.
This is my first foray into HA because I wanted to open traffic to a webserver on my lan for the first time. First I was going to just forward the port on OPN but then I saw your post and realised we can get so much more functionality from HA and your setup suggestion.

I have setup my webserver (nginix) to listen on custom port say 8082, and there is where I got the letsencrypt & dedyn.io setup first, as a kid of test before moving to OPN.
I've now done the cert as per your guide, essentially moved cert infrastrucure from the real server to OPN. All is good.

I've done the "pre-requisites" in my mind. I'm getting ready to string it together and then my question if I may.

Part 4, step 1. It says to change OPN Admin interface to another port(s) for http and https, usual 80, 443 because HA would expect traffic there.
Is it obligatory or could I keep my real server on 8082 and setup HA to exect that and keep OPN as is on 80 & 443?
I expect to use https with a self signed cert on the real server -instead of the current cert now in OPN-, or maybe only http on that port at the beginning.
Thanks.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: lilsense on November 17, 2021, 07:16:00 pm

Quote from: TheHellSite on November 16, 2021, 02:26:15 pm

I thought you were asking about an HAProxy + Let's Encrypt issue...

Since you are using cloudflare certificates I am unable to help you. You are better off asking for help in the HAProxy forums or the cloudflare support regarding your issues.

Let me finish by giving you these informations:
1. The SSL Labs test pictures you sent me indicate that your certificate content (cn + alt name) seems to be wrong.
2. Your HAProxy HTTPS frontend settings do not match the ones I provide in my guide.


You could however just follow my tutorial step by step and would end up with a working setup. ;)

I am not sure what I am missing... I went over the HA HTTPS Frontend a dozen times and I am not seeing what's not matching... :(

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 18, 2021, 11:08:11 am

Quote from: cookiemonster on November 17, 2021, 03:04:01 pm

Part 4, step 1. It says to change OPN Admin interface to another port(s) for http and https, usual 80, 443 because HA would expect traffic there.
Is it obligatory or could I keep my real server on 8082 and setup HA to exect that and keep OPN as is on 80 & 443?
I expect to use https with a self signed cert on the real server -instead of the current cert now in OPN-, or maybe only http on that port at the beginning.
Thanks.

You can run your real servers (f.e. nginx web server) on whatever port you like (80, 443, 50443, ......) that doesn't matter at all.
The reason why you have to change the OPNsense web server port is quite simple.

OPNsense is obviously running a built in webserver by default (LAN INTERFACE / PORT 443), otherwise you wouldn't be able to access the web interface at all.
HAProxy is listening on ALL INTERFACES by default.

Web Server = Interface = IP:Port
HAProxy = ALL = 0.0.0.0:0 or 0.0.0.0:80+443
OPNsense = LAN = 192.168.1.1:443

This is the conflict! You can't have two or more services listening on the same "Interface + IP:Port". How would they know which traffic belongs to whom? As soon as one of them grabs the traffic the other one will never get it.

TL;DR:
1. Yes, you can run your real servers on any port (f.e. 8082).
2. No you can't change the OPNsense back to port 443 because you wouldn't be able to reach the OPNsense web interface anymore and or HAProxy will refuse to start.
3. I strongly advise you to also run your real server(s) with a self-signed SSL certificate to increase security. It is however not necessary.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 18, 2021, 11:20:00 am

Quote from: lilsense on November 17, 2021, 07:16:00 pm

I am not sure what I am missing... I went over the HA HTTPS Frontend a dozen times and I am not seeing what's not matching... :(

As I already said, it has something to do with your cloudflare domain config!

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on November 18, 2021, 12:46:35 pm

Quote from: TheHellSite on November 18, 2021, 11:08:11 am

Web Server = Interface = IP:Port
HAProxy = ALL = 0.0.0.0:0 or 0.0.0.0:80+443
OPNsense = LAN = 192.168.1.1:443

This is the conflict! You can't have two or more services listening on the same "Interface + IP:Port". How would they know which traffic belongs to whom? As soon as one of them grabs the traffic the other one will never get it.

TL;DR:
1. Yes, you can run your real servers on any port (f.e. 8082).
2. No you can't change the OPNsense back to port 443 because you wouldn't be able to reach the OPNsense web interface anymore and or HAProxy will refuse to start.
3. I strongly advise you to also run your real server(s) with a self-signed SSL certificate to increase security. It is however not necessary.

Thank you for taking the time to answer. The problem with conflicting ip+ports is one I understand but I failed to articulate my question to explain what I meant.
I have been following the guide step by step, back and forth to understand each part. I made a little diagram for me to understand it better. I hope you don't mind if I spoil your thread with it. It just works for me to understand and better ask a question.
(https://i.ibb.co/WNXCxbb/HAP-understanding.png)
Essentially my question was if I could bind the SNI front end internally to a custom port instead of the usual 80,443. I probably should think again if what I'm asking is a sane question.
I've used this diagram and read&re-read your guide and I think I have a better grasp on it.
The nginx in my picture is me trying to figure out where/how I need to do the setup for my use. I hope I have not misrepresented it.
Let me come back with a better question when I've tested things. Much obliged.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 18, 2021, 04:00:00 pm

Quote from: cookiemonster on November 18, 2021, 12:46:35 pm

(https://i.ibb.co/WNXCxbb/HAP-understanding.png)
Essentially my question was if I could bind the SNI front end internally to a custom port instead of the usual 80,443.

First off all: B in your picture. The HTTP frontend shouldn't have the SSL_backend as the backend!!! Instead you only place the HTTPtoHTTPS redirect rule on it and if you need it, the HTTP services that don't require SSL.
You are also missing the PLEX_backend containing the PLEX_server.


Short Answer: No, because then you would have to access your services using that custom port in addition to having a subdomain for them. Which is pretty pointless, imho.

Example with default http/https port = http(s)://plex.yourdomain.tld/
Example with custom Port = 1234 = http(s)://plex.yourdomain.tld:1234/


Long Answer: It might be possible if you create a port forward in your OPNsense that forwards any WAN traffic that hits on TCP 80/443 to 127.0.0.1:1234.



My question to you: Why do you want't to do this in the first place? It makes this setup even more complex, for no reason...

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on November 18, 2021, 08:26:40 pm

Thank you for the clarifications. They help. I'm revising it now.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 11:05:05 am

Quote from: TheHellSite on May 31, 2021, 01:06:11 pm

Hello,

when I started implementing HAProxy in my network I couldn't find any complete and well written guide out there. I had to puzzle everything together from various websites.
So I thought I would save many of you a lot of time and provide my ultimate HAProxy on OPNsense guide.  :)

This tutorial will show you how to configure HAProxy as a reverse proxy on OPNsense using wildcard certificates from Let's Encrypt.
It is going to be a step-by-step guide with images on how to set things up while also explaining why we set things up in a certain way.
I will try to make this as complete and detailed as possible.
If you think that there is anything wrong or missing, feel free to tell me about it and I will consider changing it.

If this guide was helpful to you then please leave me a thanks down below as it took me several days to write this down.

Kind Regards
TheHellSite

This guide was a life-saver. And while implementing, I finally had the chance to understand a little more how HAProxy works. Thank you so much for writing it.

Everything worked as documented, and in a couple of hours I was up and running, the only problem was getting 100% on the ssl quality test. The Cipher Strength was always 90%, so I got A+, with a 90% overall score.

After a couple of hours of tinkering with the ciphers, I figured out that the test doesn't like 128bit ciphers. So I removed those from the Cipher List (TLSv1.2) and Cipher Suites (TLSv1.3) of the HTTPS_frontend, and I finally got the A+ 100% score.

In case anyone faces the same issue, these are the cipher settings for HTTPS_frontend:

Quote

Cipher List: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305

Cipher Suites: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

Thanks again,

Alessandro

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 23, 2021, 11:32:25 am

Quote from: alexdelprete on November 23, 2021, 11:05:05 am

After a couple of hours of tinkering with the ciphers, I figured out that the test doesn't like 128bit ciphers. So I removed those from the Cipher List (TLSv1.2) and Cipher Suites (TLSv1.3) of the HTTPS_frontend, and I finally got the A+ 100% score.

In case anyone faces the same issue, these are the cipher settings for HTTPS_frontend:

Quote

Cipher List: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305

Cipher Suites: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

Thanks again,

Alessandro

Happy to hear it is working for you.
I think you missed the part of my tutorial where I am giving needed ciphers and cipher suites to get an 100% A+ rating.

Quote

My Tutorial
Cipher List: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384
Cipher Suites: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

yours
Cipher List: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305
Cipher Suites: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

The only difference between my guide and your list is the "DHE-RSA-AES256-GCM-SHA384" cipher.
Both will be scoring 100 % A+ while mine offer even more client compatibility.

Seems like I forgot to mention it again during tutorial at the part where the HTTPS_frontend is created. I now put a reference in there pointing to the beginning of my post where I provide the current best cipher list and cipher suits.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 11:54:24 am

Quote from: TheHellSite on November 23, 2021, 11:32:25 am

Happy to hear it is working for you.
I think you missed the part of my tutorial where I am giving needed ciphers and cipher suites to get an 100% A+ rating.

Indeed I missed it, even though I followed the guide with great attention. Problem is that it was not in the tutorial, but at the beginning of the post. Sorry about that, I paid attention mainly to the tutorial. Would have saved a lot of time tinkering with the ciphers. :)

Quote

The only difference between my guide and your list is the "DHE-RSA-AES256-GCM-SHA384" cipher.
Both will be scoring 100 % A+ while mine offer even more client compatibility.

Yes I noticed that single difference. I started from the default ciphers of HAProxy, and that one wasn't in there. I simply removed the AES128 ones from the defaults. The link you put to that  Mozilla Config is fantastic, never even thought something like that existed. Thank you again.

Quote

Seems like I forgot to mention it again during tutorial at the part where the HTTPS_frontend is created. I now put a reference in there pointing to the beginning of my post where I provide the current best cipher list and cipher suits.

That's good, because I missed it completely, would've saved me quite some time. My bad. :)

One thing I wanted to ask you: I followed your naming conventions and I noticed you had the 1_ prefix both for the HTTP and HTTPS frontend. I renamed the HTTPS to 2_HTTPS_frontend, don't know if it was intentional or not, but I interpreted it as a progressive number so that one was a 2.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 23, 2021, 11:57:42 am

Quote from: alexdelprete on November 23, 2021, 11:54:24 am

One thing I wanted to ask you: I followed your naming conventions and I noticed you had the 1_ prefix both for the HTTP and HTTPS frontend. I renamed the HTTPS to 2_HTTPS_frontend, don't know if it was intentional or not, but I interpreted it as a progressive number so that one was a 2.

I did that on purpose to express the "level" of reverse proxying.

Level 1 - SNI traffic
Level 2 - HTTP + HTTPS traffic

However you can name it as you like. It doesn't matter in terms of functionality.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 12:01:57 pm

Quote from: TheHellSite on November 23, 2021, 11:57:42 am

I did that on purpose to express the "level" of reverse proxying.

Level 1 - SNI traffic
Level 2 - HTTP + HTTPS traffic

However you can name it as you like. It doesn't matter in terms of functionality.

Thanks for the explanation.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 12:30:11 pm

Quote from: TheHellSite on November 23, 2021, 11:32:25 am

Seems like I forgot to mention it again during tutorial at the part where the HTTPS_frontend is created. I now put a reference in there pointing to the beginning of my post where I provide the current best cipher list and cipher suits.

I moved the frontend cipher settings and bind options to the Global Parameters -> SSL Default Settings section, emptied the relevant fields in HTTPS_frontend, it works fine and I think it's simpler:

(https://i.imgur.com/6sqazDa.png)

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 23, 2021, 12:43:27 pm

Arguable! ;)
If you have many HTTPS frontends (on different ports) that might need different SSL settings then my way is better.

Otherwise it doesn't really matter where you put the settings. Just note that the SSL default settings get overwritten once you set anything in the associated boxes on the HTTPS frontends.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on November 23, 2021, 12:49:39 pm

Quote from: TheHellSite on November 23, 2021, 12:43:27 pm

Arguable! ;)
If you have many HTTPS frontends (on different ports) that might need different SSL settings then my way is better.

Otherwise it doesn't really matter where you put the settings. Just note that the SSL default settings get overwritten once you set anything in the associated boxes on the HTTPS frontends.

Your second point is the answer to your first observation: Global settings are a "global default". If you need a custom config for a frontend, you can customize its config and it will supersede the global one. ;)

That's why I like it, it's the classic enterprise approach to organizing things, more intuitive for me (global...and custom as an exception).

But like you said...it's all very subjective obviously. Individual preferences.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: dima1002 on November 28, 2021, 09:01:48 am

Hello,

what is wrong here?

In Firefox I get the following error: PR_END_OF_FILE_ERROR

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log audit debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): LetsEncrypt_443 ()

# Frontend (DISABLED): LetsEncrypt_80 ()

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_61a24897421141.86617043 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_61a24897421141.86617043

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61a24a78aa9cc4.11915455.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61a249350142e3.01879320.txt)]

# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Backend: Mail ()
backend Mail
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Main 192.168.111.2:443 ssl verify none

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend: Nextcloud ()
backend Nextcloud
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Nextcloud 192.168.111.3:443 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: Bitwarden_Backend ()
backend Bitwarden_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Vaultwarden80 192.168.111.77:80


Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 28, 2021, 11:16:09 am

Quote from: dima1002 on November 28, 2021, 09:01:48 am

Hello,

what is wrong here?

In Firefox I get the following error: PR_END_OF_FILE_ERROR

Are you only getting this error when using Firefox?
If so, which version of Firefox are you running?

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: SOUK on December 04, 2021, 03:56:42 pm

Delete

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: evathesalmon on December 12, 2021, 11:41:33 pm

Quote from: dima1002 on November 28, 2021, 09:01:48 am

In Firefox I get the following error: PR_END_OF_FILE_ERROR

Hello,

I have same issue in firefox and chrome. In logs it is said "Received something which does not look like a PROXY protocol header"

(https://i.imgur.com/sjyyqey.png)

I thought at first that it is a proxy problem and double checked your 20210613 update but my bind option pass-through set as accept-proxy

(https://i.imgur.com/tkMNypC.png)

(https://i.imgur.com/HrWTwSa.png)

My current HAProxy version is 3.7 and opnsense is 21.7.6

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on December 14, 2021, 09:27:56 am

In Part 6, NAT Reflection: it applies to port forwarding rules, but in the guide you switched to a simple filter rule.

So there's only one option remaining: split DNS.

HAProxy has been rock solid, thanks again for your guide. I'm having a hard time only for Uptime Kuma, it uses websockets, and it's the only service that doesn't work behind HAProxy. The dev published a guide for the configuration behind several reverse proxies, unfortunately the only one missing is HAProxy: https://github.com/louislam/uptime-kuma/wiki/Reverse-Proxy#nginx

I'm sure there's a way to make it work but I can't find it...

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 14, 2021, 10:35:36 am

Quote from: evathesalmon on December 12, 2021, 11:41:33 pm

Quote from: dima1002 on November 28, 2021, 09:01:48 am

In Firefox I get the following error: PR_END_OF_FILE_ERROR

Hello,

I have same issue in firefox and chrome. In logs it is said "Received something which does not look like a PROXY protocol header"

I thought at first that it is a proxy problem and double checked your 20210613 update but my bind option pass-through set as accept-proxy

My current HAProxy version is 3.7 and opnsense is 21.7.6

I am running the setup with the latest OPNsense updates just fine, using Firefox (mainly) but Chrome or Edge work fine either.
It seems you did misconfigure something!

Post your haproxy export in a code box. (redact any sensitive information, but leave in the local IPs!)

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 14, 2021, 11:05:13 am

Quote from: alexdelprete on December 14, 2021, 09:27:56 am

In Part 6, NAT Reflection: it applies to port forwarding rules, but in the guide you switched to a simple filter rule.

So there's only one option remaining: split DNS.

Thank you for pointing this out! I changed the guide some time ago and forgot to update that part.
Just keep in mind that there is nothing wrong with Split DNS, it is even the preferred way of doing it!

Quote from: alexdelprete on December 14, 2021, 09:27:56 am

HAProxy has been rock solid, thanks again for your guide. I'm having a hard time only for Uptime Kuma, it uses websockets, and it's the only service that doesn't work behind HAProxy. The dev published a guide for the configuration behind several reverse proxies, unfortunately the only one missing is HAProxy: https://github.com/louislam/uptime-kuma/wiki/Reverse-Proxy#nginx

I'm sure there's a way to make it work but I can't find it...

Isn't that just a simple web service displaying uptime of servers?
How are you accessing it on your local network? f.e. http://192.168.2.55:3001/ or https://192.168.2.55:3001/
If it is http then you will need to DISABLE SSL in the real server settings for uptime kuma.
If it is https then you will need to ENABLE SSL but DISABLE SSL verification in the real server settings for uptime kuma.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on December 14, 2021, 11:16:25 am

Quote from: TheHellSite on December 14, 2021, 11:05:13 am

Thank you for pointing this out! I changed the guide some time ago and forgot to update that part.
Just keep in mind that there is nothing wrong with Split DNS, it is even the preferred way of doing it!

I know, it's my preferred way too. I love DNS. Was just pointing out that there's only one option. :)

Quote

Isn't that just a simple web service displaying uptime of servers?
How are you accessing it on your local network? f.e. http://192.168.2.55:3001/ or https://192.168.2.55:3001/
If it is http then you will need to DISABLE SSL in the real server settings for uptime kuma.
If it is https then you will need to ENABLE SSL but DISABLE SSL verification in the real server settings for uptime kuma.

I've already reverse-proxied a lot of services, I know how that works. The problem is how Uptime Kuma works: it uses ws:// (websockets) connections in addition to HTTP, so you connect in http first to auth, then it starts communicating through WS, through a sort of tunnel. If you check that link I provided, you will see that for many proxies there's some custom configs to support that. The only proxy that does one-line config magic is caddy...it's tempting me a lot...everybody told me that caddy is the simplest one and it simply works, without doing any hard config work. But I already have HAProxy in place, and would like to stick to it.

UPDATE: I found this article https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/#advanced-configuration

Now I have to understand where to put those things in the UI on OPNsense. Is there a way to edit the config files directly?

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 14, 2021, 11:36:16 am

Quote from: alexdelprete on December 14, 2021, 11:16:25 am

I've already reverse-proxied a lot of services, I know how that works. The problem is how Uptime Kuma works: it uses ws:// (websockets) connections in addition to HTTP, so you connect in http first to auth, then it starts communicating through WS, through a sort of tunnel. If you check that link I provided, you will see that for many proxies there's some custom configs to support that. The only proxy that does one-line config magic is caddy...it's tempting me a lot...everybody told me that caddy is the simplest one and it simply works, without doing any hard config work. But I already have HAProxy in place, and would like to stick to it.

Did you try something like this?
https://stackoverflow.com/a/22735431/17193869
or this
https://discourse.haproxy.org/t/using-reverse-proxy-with-secured-web-sockets-wss/2917

Otherwise your best bet is to ask in the haproxy discourse forum including your config export.
Sadly, I can't help everyone to get specific services running.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on December 14, 2021, 01:14:45 pm

Quote from: TheHellSite on December 14, 2021, 11:36:16 am

Did you try something like this?
https://stackoverflow.com/a/22735431/17193869
or this
https://discourse.haproxy.org/t/using-reverse-proxy-with-secured-web-sockets-wss/2917

I found this article from HAProxy guys:
https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/#simple-configuration

That's why I wanted to know if I can configure it via shell, working on the files directly. Do you know if that's possible?

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 14, 2021, 01:36:46 pm

Quote from: alexdelprete on December 14, 2021, 01:14:45 pm

I found this article from HAProxy guys:
https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/#simple-configuration

That's why I wanted to know if I can configure it via shell, working on the files directly. Do you know if that's possible?

Oh very well, of course there is!
You just need to place these settings (one per line) in the advanced settings of the frontend/backend.

Frontends --> yourfrontend --> top left, enable advanced mode --> Advanced settings --> Option pass-through

As far as I am aware settings these options on the frontend will apply them to ALL services that are going through it.
But you should also be able to set them on the corresponding backend so that it will only apply to the specific service.

Code: [Select]

## routing based on websocket protocol header
  acl hdr_connection_upgrade hdr(Connection)  -i upgrade
  acl hdr_upgrade_websocket  hdr(Upgrade)     -i websocket
  use_backend bk_ws if hdr_connection_upgrade hdr_upgrade_websocket
  default_backend bk_web
This however is also just a combination of acl=condition and rule=use_backend.
It is basically saying: IF HTTP_header=Connection+Upgrade THEN USE_BACKEND AKUMA_ws_backend

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: alexdelprete on December 14, 2021, 02:46:51 pm

Quote from: TheHellSite on December 14, 2021, 01:36:46 pm

As far as I am aware settings these options on the frontend will apply them to ALL services that are going through it. But you should also be able to set them on the corresponding backend so that it will only apply to the specific service.

Not working, I don't think it's a config issue...

Seems like there were issues with websockets with older versions, newest v2.5 fixed several things regarding websockets support: https://www.haproxy.com/blog/announcing-haproxy-2-5/

Latest plugin has HAProxy v2.2.18, will have to wait for the upgrade...unless I can upgrade it manually...

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: evathesalmon on December 15, 2021, 12:07:26 am

Quote from: TheHellSite on December 14, 2021, 10:35:36 am

Post your haproxy export in a code box. (redact any sensitive information, but leave in the local IPs!)

Thank you for reply.

I only have map for internal network because I don't pass-through external traffic. And I also set host binding via unbound overrides all to 192.168.64.1.

Code: [Select]


#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 127.0.0.1:80, 127.0.0.1:443)
frontend 0_SNI_frontend
    bind 127.0.0.1:443 name 127.0.0.1:443
    bind 127.0.0.1:80 name 127.0.0.1:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_61a3f4fdabe975.79392880 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_61a3f4fdabe975.79392880

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61a3f8200d98a7.85727127.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61a3f5af2446f3.11405534.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: unraid_backend ()
backend unraid_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server unraid_server 192.168.0.20:443 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: opnsense_backend ()
backend opnsense_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opnsense_server 192.168.0.1:1443 ssl verify none


(https://i.imgur.com/gD1BC97.png)

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 15, 2021, 01:02:57 am

Quote from: evathesalmon on December 15, 2021, 12:07:26 am

Thank you for reply.

I only have map for internal network because I don't pass-through external traffic. And I also set host binding via unbound overrides all to 192.168.64.1.

Check the cipher list and cipher suites on your HTTPS frontend! They don't match the ones I provide in my first post!

That one thing wonders me and might be your issue... Why did you set the SNI_frontend to listen on the localhost address? This makes no sense at all!
You are allowing inbound traffic on your WAN address 80+443 with our firewall rule but your SNI_frontend is only listening on the localhost address. This way it will never catch any traffic at all!

I never said to do so in my guide! Apart from the SSL_server address which can safely use the localhost address.

Title: Re: Tutorial 2021/11: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: evathesalmon on December 15, 2021, 09:11:15 pm

Quote from: TheHellSite on December 15, 2021, 01:02:57 am

You are allowing inbound traffic on your WAN address 80+443 with our firewall rule but your SNI_frontend is only listening on the localhost address. This way it will never catch any traffic at all!

UPD: nevermind, it was my local cache and unbound cache. After purging everything it works just fine without any specific binding. Thank you for your help. 

I have amended my config and compared again with the guide but I must be missing something.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_61a3f4fdabe975.79392880 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_61a3f4fdabe975.79392880

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61a3f8200d98a7.85727127.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNET_condition
    acl acl_61a3f9b4ed7092.44798843 src 192.168.0.0/24

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61a3f5af2446f3.11405534.txt)] if acl_61a3f9b4ed7092.44798843

# Backend: unraid_backend ()
backend unraid_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server unraid_server 192.168.0.20:443 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: opnsense_backend ()
backend opnsense_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opnsense_server 192.168.0.1:1443 ssl verify none

# Backend: qbittorrent_backend ()
backend qbittorrent_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server qbittorrent_server 192.168.0.20:8080

The setup is working only if I bind my host->ip in unbound overrides the way they are physically, in example unraid is binded to 192.168.0.20, opnsense is binded to 192.168.0.1 and other (see the pic). But if the redirects are set to 192.168.54.1 just like in the guide it just unable to connect.

In other words seems like 0_SNI_frontend doesn't listen to all addresses even if it is configured so.

Also a question: should I set any WAN rules even if I only routing the LAN traffic, cause now I didn't set any.

(https://i.imgur.com/by1yDJ0.png)

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: boredpanda on January 03, 2022, 01:22:51 pm

Quote from: TheHellSite on May 31, 2021, 01:06:11 pm

Hello,

when I started implementing HAProxy in my network I couldn't find any complete and well written guide out there. I had to puzzle everything together from various websites.
So I thought I would save many of you a lot of time and provide my ultimate HAProxy on OPNsense guide.  :)

Hello! And thank you very much for your well-written guide. It made my switch from pfSense to OPNsense far smoother! HAProxy in pfSense looks quite different from HAProxy in OPNsense.

I self-host a bunch of services on a local server, and all the services are in dockers, meaning they all have the same IP but different ports. Most of the services worked just fine following your guide, but I'm noticing a few odd things. For instance, on my Firefly III installation, I get the error in the attachment and I'm unable to log in. If I visit the IP:PORT address, it works fine. Any suggestions on changes I could maybe make to my configuration?

None of my services are publicly exposed and I'm using a FQDN just to not having to remember port numbers.

My config is below. Thanks! :)

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_61d029838380d8.68540995 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_61d029838380d8.68540995

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61d02d156c0846.98881851.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    # ACL: EXTERNAL_conditions
    acl acl_61d066f2cc9639.62892989 src 193.138.218.219
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_61d05461151001.71548589 src 192.168.1.0/24 192.168.70.0/24

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61d053aef188f4.27343600.txt)] if acl_61d066f2cc9639.62892989 || acl_61d05461151001.71548589
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/61d029eb5a9da6.54806678.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: nextcloud_backend ()
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nextcloud_server 192.168.1.12:444 ssl verify none

# Backend: miniflux_backend ()
backend miniflux_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server miniflux_server 192.168.1.12:5600

# Backend: joplin_backend ()
backend joplin_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server joplin_server 192.168.1.12:22300

# Backend: calibre_backend ()
backend calibre_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server calibre_server 192.168.1.12:8083

# Backend: emby_backend ()
backend emby_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server emby_server 192.168.1.12:8096

# Backend: grocy_backend ()
backend grocy_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server grocy_server 192.168.1.12:9283

# Backend: hydra_backend ()
backend hydra_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hydra_server 192.168.1.12:5076

# Backend: piwigo_backend ()
backend piwigo_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server piwigo_server 192.168.1.12:8099

# Backend: collabora_backend ()
backend collabora_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server collabora_server 192.168.1.12:9980

# Backend: freshrss_backend ()
backend freshrss_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server freshrss_server 192.168.1.12:8066

# Backend: wallabag_backend ()
backend wallabag_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server wallabag_server 192.168.1.12:6500

# Backend: wikijs_backend ()
backend wikijs_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server wikijs_server 192.168.1.12:3000

# Backend: heimdall_backend ()
backend heimdall_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server heimdall_server 192.168.1.12:8538

# Backend: monica_backend ()
backend monica_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server monica_server 192.168.1.12:8956

# Backend: firefly_backend ()
backend firefly_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server firefly_server 192.168.1.12:8088

# Backend: paperless_1_backend ()
backend paperless_1_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server paperless_1_server 192.168.1.12:8016

# Backend: paperless_2_backend ()
backend paperless_2_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server paperless_2_server 192.168.1.12:8006


Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 04, 2022, 01:16:56 am

First of all please use the correct cipher list and suites, see the beginning of my OP.
You are still using AES128 ciphers indicating you didn't read my tutorial correctly.

Next thing would be to clear your browser cache.
Is it only firefly that is not working or are others also affected?

Are you using a trusted lets encrypt cert or a selfsigned one?

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: boredpanda on January 05, 2022, 06:04:45 pm

Quote from: TheHellSite on January 04, 2022, 01:16:56 am

First of all please use the correct cipher list and suites, see the beginning of my OP.
You are still using AES128 ciphers indicating you didn't read my tutorial correctly.

Next thing would be to clear your browser cache.
Is it only firefly that is not working or are others also affected?

Are you using a trusted lets encrypt cert or a selfsigned one?

That's fair enough! I have updated my configuration with the correct cipher list and suites. My browser cache has been cleared and I've tried multiple browsers on different computers. The issue still remains. I am using a trusted let's encrypt cert (wildcard domain managed by ACME on OPNsense).

Two of my services aren't working as expected at the moment, and that's Firefly III and Grocy. Everything else is working.

For Firefly, I came across this thread (https://github.com/firefly-iii/firefly-iii/discussions/5118) on Github discussing my exact issue. It was apparently fixed in this comment (https://github.com/firefly-iii/firefly-iii/discussions/5118#discussioncomment-1398790). My Docker env TRUSTED_PROXIES is set to ** already. Do I need to edit 1_HTTPS_frontend or 1_HTTP_frontend? I see that both of those have the option X-Forwarded-For header enabled in my HAProxy.

For Grocy, I'm having the issue described here (https://github.com/linuxserver/docker-grocy/issues/18), on Github. Someone using nginx mentioned they solved the issue by adding proxy_set_header X-Forwarded-Proto https; to their nginx config.

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 05, 2022, 06:34:51 pm

Quote from: boredpanda on January 05, 2022, 06:04:45 pm

Two of my services aren't working as expected at the moment, and that's Firefly III and Grocy. Everything else is working.

For Firefly, I came across this thread (https://github.com/firefly-iii/firefly-iii/discussions/5118) on Github discussing my exact issue. It was apparently fixed in this comment (https://github.com/firefly-iii/firefly-iii/discussions/5118#discussioncomment-1398790). My Docker env TRUSTED_PROXIES is set to ** already. Do I need to edit 1_HTTPS_frontend or 1_HTTP_frontend? I see that both of those have the option X-Forwarded-For header enabled in my HAProxy.

For Grocy, I'm having the issue described here (https://github.com/linuxserver/docker-grocy/issues/18), on Github. Someone using nginx mentioned they solved the issue by adding proxy_set_header X-Forwarded-Proto https; to their nginx config.

You should (but don't have to) leave the x-forwarded-for header enabled. It is very useful since the real servers (firefly, grocy, ...) will get to know the original IP of the client trying to access it.

The links you posted both imply that your issues COULD be resolved by adding the following to your "HTTPS_frontend".

HAProxy --> Virtual Services --> Public Services --> 1_HTTPS_frontend --> Edit --> Enable "advanced mode" (top left corner) --> Scroll down to "Advanced settings" --> Option pass-through --> insert the below code --> Save --> Apply

Code: [Select]

http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Real-IP %[src]
You might not need both lines so play around until you find the necessary line(s) and please post the final solution!

If both of your services are working now then I suggest to remove that setting from your HTTPS_frontend and set it on the firefly and grocy backends instead!

HAProxy --> Virtual Services --> Public Services --> firefly/grocy backend --> Edit --> Enable "advanced mode" (top left corner) --> Scroll down to "Tuning options" --> Option pass-through --> insert the above code --> Save --> Apply

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: boredpanda on January 05, 2022, 07:00:36 pm

Quote from: TheHellSite on January 05, 2022, 06:34:51 pm

The links you posted both imply that your issues COULD be resolved by adding the following to your "HTTPS_frontend".

HAProxy --> Virtual Services --> Public Services --> 1_HTTPS_frontend --> Edit --> Enable "advanced mode" (top left corner) --> Scroll down to "Advanced settings" --> Option pass-through --> insert the below code --> Save --> Apply

Code: [Select]

http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Real-IP %[src]
You might not need both lines so play around until you find the necessary line(s) and please post the final solution!

The second line didn't work, but http-request set-header X-Forwarded-Proto https if { ssl_fc } solved the issue with both services, so I have added the line to both of the back ends. Thank you so much! ;)

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on January 10, 2022, 07:24:43 pm

I wonder if I could ask this question here. I've followed the instruction to the letter, all is good.
I have an additional requirement however, to proxy TCP mode to the real server after TLS offloading at HA.
Say the real server is listening on port 8053 and we can live without tls between HA and it to begin with, is in the internal LAN. The public port for the service however is different, say 853. So HA needs to receive in one and sent to another.
I understand we can't mix tcp and http modes and the tutorial explains it in the relevant part.

My current thinking is that since 0_SNI_frontend and SSL_backend are already in TCP mode, the first part is adding the port, ending with 0.0.0.0:80, 0.0.0.0:443: 0.0.0.0:851
- Since the TLS is being terminated at the 1_HTTPS_frontend but is not TCP, that is not the right place to go next.
I thought then I needed to create a new frontend in TCP mode. I duplicated 1_HTTPS_frontend and changed name to 1_TCP_frontend. Set it to listen on the vip:851. I left the map rule on it and disabled SSL offload.

Testing gives me error:
2022-01-10T17:11:39   haproxy[81388] externalip:port [10/Jan/2022:17:11:39.859] 1_TCP_frontend 1_TCP_frontend/<NOSRV> -1/-1/0 0 SC 2/1/0/0/0 0/0   
2022-01-10T17:11:39   haproxy[81388] Connect from externalip:port to wanip:851 (0_SNI_frontend/TCP)


Makes me think I'm not too far.
My question to you knowledgeable people is if the setup of this tutorial can be expanded to do this, or I'm trying to do something totally that'll never work, and some pointers to what I need to do would be wonderful please.

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 10, 2022, 10:10:57 pm

You have to create a condition and rule for that service. And then place that rule on the 0_SNI_frontend.

However you CAN NOT do url conditions, f.e. service.domain.com ....


OR

You create a new TCP frontend that listens on that port.
Then create server + backend for the service and set this backend on the new tcp frontend as default backend.

OR

you simply use a port forwarding.....

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on January 11, 2022, 12:07:21 am

Option 2 was what I was hoping was possible. It is working and what I found was that I needed to add the port to the 0_SNI_frontend too, which I wasn't sure about. I had it from earlier attempts but once working, my new proxy would not work without it.
So in summary, added the port to 0_SNI_frontend; added the new default backend to the new front end I had created (that I had missing), as well as has the map file rule. I haven't tested without this map yet, that's next.
I'm going to read and re-read the config that is working against the documentation to understand how. I can't follow the new path at present due to my lack of knowledge on the HA options used.
Thank you TheHellSite.
Looks like this and allows me to put my subdomain in my android phone to get DNS over TLS on my own infra that filters domains with Adguard before going out on TLS to the open internet. That was the goal.
For anyone wondering is here:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    1
    hard-stop-after             60s
    maxconn                     10
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 10
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening on 0.0.0.0:80, 0.0.0.0:443, 0.0.0.0:853)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:853 name 0.0.0.0:853
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (listening on 192.168.5.100:80 i.e. http only)
frontend 1_HTTP_frontend
    bind 192.168.5.100:80 name 192.168.5.100:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_619439805021f2.97978352 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_619439805021f2.97978352

# Frontend: 1_HTTPS_frontend (Listening on 192.168.5.100:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.5.100:443 name 192.168.5.100:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61952b9d47d700.25962675.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Frontend: 1_TCP_frontend (Listening on 192.168.5.100:853)
frontend 1_TCP_frontend
    bind 192.168.5.100:853 name 192.168.5.100:853 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61dc51606078d9.11258474.certlist
    mode tcp
    default_backend nginx_backend-tcp
    # tuning options
    timeout client 15m

    # logging options
    option tcplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.5.100 send-proxy-v2 check-send-proxy

# Backend: nginx_backend-tcp ()
backend nginx_backend-tcp
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server nginx_1 192.168.5.152:8053

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Tattoofreak on January 24, 2022, 02:18:58 pm

Thank you for this tutorial.
Unfortunately my requirements are a bit different.
Basically, what I want to do, is:

I want to set up HAProxy just for routing traffic based on URLs (https://xyz.domain.com goes to server 1 and https://abc.domain.com goes to server 2, etc...).
All SSL stuff for the destination web servers is being handled by a separate Linux certificate server and the web servers themselfes, independent from OPNsense/HAProxy. HAProxy is really only needed for routing traffic based on URLs, nothing more, nothing less.

The websites are all working when I use NAT rules, but I was not able yet to make it run with HAProxy in between.

For testing it with just one webserver, I have configured the following:

OPNsense settings:

System --> Settings --> Administration:
TCP port: 4443

Firewall --> Rules --> WAN:
Protocol: IPv4/TCP | Source: any | Port: Any | Destination: This Firewall | Port: 80 (HTTP) | Rule: Pass
Protocol: IPv4/TCP | Source: any | Port: Any | Destination: This Firewall | Port: 443 (HTTPS) | Rule: Pass

HAProxy settings (everthing not mentioned I left on default settings):

Services --> HAProxy --> Settings --> Settings:
Service:
Enable HAProxy: Checked

Services --> HAProxy --> Settings --> Real Servers:
Real Servers:
Name or Prefix: web01_http | FQDN/IP: 192.168.1.10 | Port: 80
Name or Prefix: web01_https | FQDN/IP: 192.168.1.10 | Port: 443

Services --> HAProxy --> Settings --> Virtual Services:
Backend Pools:
Name: web01_backendpool01 | Mode: TCP (Layer4) | Servers: web01_http, web01_https

Services --> HAProxy --> Settings --> Rules & Checks:
Conditions:
Name: myWebsite01_condition01 | Condition type: Host starts with | Host prefix: <mysubdomain>.<mydomain>.com

Services --> HAProxy --> Settings --> Rules & Checks:
Rules:
Name: myWebsite01_rule01 | Select conditions: myWebsite01_condition01 | Execute function: Use specific Backend Pool | Use backend pool: web01_backendpool01

Services --> HAProxy --> Settings --> Virtual Services:
Public Services:
Name: web01_publicService01 | Listen Addresses: <myPublicIP>:80, <myPublicIP>:443 | Type: TCP | Rules: myWebsite01_rule01


Any help would be very appreciated, thank you in advance!

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Tattoofreak on January 27, 2022, 01:26:31 pm

Anyone?

I'm wondering if it is even possible to catch URL based packets when Public Services Type is set to TCP? Would this work?

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 29, 2022, 09:37:27 am

You have to set your backends and frontends to HTTP Mode.
Also disable SSL offloading on the frontends.

But I can't guarantee for sure that it will work.
TCP Mode will never (with a few exceptions) work because there is no header in the packets that would tell HAProxy which service to send the traffic to.

HTTP Mode could work, but you might need to create some "http header contains..." conditions.

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: afall on January 31, 2022, 05:42:25 pm

First of all thank you for the Guide it was great and helped me out a lot.

I just have a question as I have to use a vpn adapter call for work called Zscaler and it try to make a tunnel using port 443. How do i tell HAproxy as a default to only route only thing in the map file and dont intercept anything else. So the vpn client can create a tunnel. 

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: vladnik on February 01, 2022, 04:16:24 pm

Hi @TheHellSite,

just wanted to say thank you for this excellent guide, worked like a charm and thaught me much about how HAProxy works. Thank you for a job well done!  :)

Cheers!

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 03, 2022, 01:37:56 pm

I have two Server with several vhosts behind a OPNsense Router/Firewall

Is this also possible to have two server which need certs? Or only one?

When if it’s possible, how I have to do it?

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Tattoofreak on February 10, 2022, 03:40:27 pm

@TheHellSite
Maybe you would also like on how to enable Websockets on your frontend(s) if your incoming clients are looking for such one(s). Websockets are basically used for example for streaming services over web. I was looking for so long on how to resolve my problem for making HAProxy work with Synology's DS Cam Android app which tries to connect from remote to the Synology Surveillance Station NAS behind HAProxy and I finally found out. You have to insert the following on your frontend (where you have to replace <myBackend(Pool)> with your according backend, of course):

Code: [Select]

acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend <myBackend(Pool)> if is_websocket
Here's the link to my original and solved issue:
 https://www.synoforum.com/threads/connecting-synology-ds-cam-android-app-to-synology-surveillance-station-through-opnsense-haproxy-plugin.7969/ (https://www.synoforum.com/threads/connecting-synology-ds-cam-android-app-to-synology-surveillance-station-through-opnsense-haproxy-plugin.7969/)

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 12, 2022, 09:13:03 pm

I got this error

Quote

[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 02:38:21 pm

Quote from: afall on January 31, 2022, 05:42:25 pm

First of all thank you for the Guide it was great and helped me out a lot.

I just have a question as I have to use a vpn adapter call for work called Zscaler and it try to make a tunnel using port 443. How do i tell HAproxy as a default to only route only thing in the map file and dont intercept anything else. So the vpn client can create a tunnel.

Since your provided little to no info of the setup I am not really able to help you.
Also my guide is just to show beginners how things are done.

However if I understood your issue correctly, you can try the below. If that doesn't work you are better of asking in the HAProxy forums!

Add the following in the "Option pass-through" field of your 0_SNI_frontend.
Don't forget to create the backend along with the server for Zscaler. My example uses OpenVPN.

Code: [Select]

tcp-request inspect-delay 5s
use_backend OPENVPN_backend if req_ssl_hello_type 1
tcp-request content accept if !{ req_ssl_hello_type 1 }


Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 02:46:13 pm

Quote from: Morta on February 03, 2022, 01:37:56 pm

I have two Server with several vhosts behind a OPNsense Router/Firewall

Is this also possible to have two server which need certs? Or only one?

When if it’s possible, how I have to do it?

Quote from: Morta on February 12, 2022, 09:13:03 pm

I got this error

Quote

[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

Never had this issue!

1. Go through my guide again. If you follow it step by step, you WILL HAVE a working setup. Otherwise you did something wrong.

2. Where is your config export? Nobody will be able to help you without a HAProxy config export!

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 15, 2022, 03:01:33 pm

I have following configuration:

NAS (192.168.1.118) → Apache → Nextcloud/DAV
/
ROUTER/Firewall/HAproxy (192.168.1.1)
\
Server (192.168.1.100) → Apache with severals vhost

I want to bring up one vhost as test with following configuration

Quote

<VirtualHost *:80>
     ServerAdmin mail@xxx.com
     ServerName xxx.ch
     ServerAlias www.xxxx.ch
     DocumentRoot /usr/share/webapps/blog/
     DirectoryIndex index.php
     RemoteIPProxyProtocol On
     <Directory /usr/share/webapps/blog>
        Options +Indexes +FollowSymLinks +MultiViews
        AllowOverride All
        Order allow,deny
        allow from all

    <FilesMatch \.php$>
      # For Apache version 2.4.10 and above, use SetHandler to run PHP as a fastCGI process server
      SetHandler "proxy:unix:/run/php-fpm/php-fpm.sock|fcgi://localhost"
    </FilesMatch>
    <Files "*.php">
      MultiviewsMatch Any
    </Files>
    </Directory>
     ErrorLog /var/log/httpd/blog_error.log
     CustomLog /var/log/httpd/blog_access.log combined
</VirtualHost>

My HAproxy.conf file looks like this

cat haproxy.conf

Quote

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: O_SNI_fronted (Listening 0.0.0.0:80 0.0.0.0:443)
frontend O_SNI_fronted
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NO_SSL_Rule
    acl acl_620808a860e296.91534155 req.ssl_ver gt 0

    # ACTION: HTTP_TO_HTTPS_RULE
    http-request redirect scheme https code 301 if !acl_620808a860e296.91534155

# Frontend: 1_HTTPS_frontend (Listening 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6208140971a7a3.08696099.certlist
    mode http
    option http-keep-alive
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_MAP_RULE
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/620809e036a6d1.87483247.txt)]

# Backend: SSL_backend (SSL  backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: 5erver_backend (Server backend)
backend 5erver_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80
# ERROR: server data not found (0b989d9b-eb50-4dff-8a2f-6bc56245fd74)

# Backend: NAS_backend (NAS backend)
backend NAS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nas_Server_80 192.168.1.118:80
# ERROR: server data not found (36c63574-bd94-43f7-836e-cd78c8edc6c0)

My map files looks like this

Quote

#public subdomains mapping
flood 5erver_backend
frank 5erver_backend
www 5erver_backend
torrent 5erver_backend
grafana 5erver_backend
nas 5erver_backend
kvm 5erver_backend
monitoring 5erver_backend
speedtest 5erver_backend
sync 5erver_backend
tracker 5erver_backend
cloud NAS_backend
dav NAS_backend

How I can fix this error
503 error?

Thanks for reply

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 03:25:08 pm

FIRST: You should remove your personal info from your post.

SECOND: Another issue from not properly reading my guide.

Your solution is in Part 5 - Step 6.

Quote

Now we create the backend that belongs to an actual service. You will need one backend for each service.
If you have multiple servers serving the exact same content than you will want to add all servers into a single backend so HAProxy can actually balance the load between the servers.

YOU NEED: ... one backend for each service.

YOU DID: ... one backend for each server hosting individual services.

Just think about it... How should HAProxy even be able to talk to one of your services when you are only pointing him to the IP:Port of the server virtually hosting the service!? This makes no sense...
It is like telling someone "Meet me in New York in a bar." without telling him which bar.

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 15, 2022, 04:17:31 pm

Quote from: TheHellSite on February 15, 2022, 03:25:08 pm

FIRST: You should remove your personal info from your post.

SECOND: Another issue from not properly reading my guide.

Your solution is in Part 5 - Step 6.

Quote

Now we create the backend that belongs to an actual service. You will need one backend for each service.
If you have multiple servers serving the exact same content than you will want to add all servers into a single backend so HAProxy can actually balance the load between the servers.

YOU NEED: ... one backend for each service.

YOU DID: ... one backend for each server hosting individual services.

Just think about it... How should HAProxy even be able to talk to one of your services when you are only pointing him to the IP:Port of the server virtually hosting the service!? This makes no sense...
It is like telling someone "Meet me in New York in a bar." without telling him which bar.

Ok, thanks for finding the error.

English isn't my mother language so can you little bit clarify the steps.

So I have to copy the backend and adjust it for every virtual host of Apache even if they are on the same port?

I have one apache on both SERVER(192.168.1.100 and 192.168.1.118) on port 80 an only one webserver with several vhosts. I thought the map file will do the split of the to the vhosts...

For me is Apache one service on which is on both SERVER and so It's has one backend each for this two server?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 15, 2022, 04:29:47 pm

(https://abload.de/img/haproxydgj5s.png)

So I have to do this for every single vhost?

So it's looking so?

(https://abload.de/img/haproxy_backendlgk4r.png)

So the maps file is also wrong I have to change nas_backend to nas_server_80 and 5erver_backend to 5erver_server_80 ?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 05:37:21 pm

This is not a question of your mother language not being english.  ;D
This is a question of you not understanding the layout of HAProxy.

It is really hard for me to help here since you don't seem to understand the basics of a reverse proxy.
A reverse proxy is not a web server to host your sites! It is meant to route traffic to a web server.

Rule 1: You need at least ONE server for each backend.
With server I mean server created in HAProxy. A server in HAProxy represents the service (f.e. nextcloud, plex, ...) running on one of your physical servers.

Rule 2: Each backend will then present your individual services to your frontend.

HTTPS Frontend --> Mapfile --> PLEX_backend --> PLEX_server --> 192.168.2.30:32400
HTTPS Frontend --> Mapfile --> BLOG_backend --> BLOG_server --> 192.168.2.30:80
HTTPS Frontend --> Mapfile --> CLOUD_backend --> CLOUD_server --> 192.168.2.40:43569

Rule 3: A mapfile is simply mapping an access URL, f.e. www.yourdomain.com, against a backend.


To be fair it seems I misunderstood your setup/goal. But since you still didn't explain exactly what you are trying to achieve I am assuming you are trying to something like this.

Apache is your webserver. (www.yourdomain.com)
These vhost seem to provide subdirectories or individual websites. (f.e. service.yourdomain.com/subdir/)

Code: [Select]

#public subdomains mapping
flood 5erver_backend
frank 5erver_backend
www 5erver_backend
torrent 5erver_backend
grafana 5erver_backend
nas 5erver_backend
kvm 5erver_backend
monitoring 5erver_backend
speedtest 5erver_backend
sync 5erver_backend
tracker 5erver_backend
cloud NAS_backend
dav NAS_backend
Let me further explain to you the map file. I mean actually my first post does it, but you don't seem to understand or didn't read...

Code: [Select]

nas 5erver_backendnas.yourdomain.com --> NAS_backend

Code: [Select]

www 5erver_backendwww.yourdomain.com --> 5erver_backend

Code: [Select]

grafana 5erver_backendgrafana.yourdomain.com --> 5erver_backend

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 15, 2022, 06:00:06 pm

Thanks for your help. I appreciated it!

Quote

cat haproxy.conf
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: O_SNI_fronted (Listening MYIP:80 MYIP:443)
frontend O_SNI_fronted
    bind MY IP:80 name MYIP:80
    bind MY IP:443 name MY IP:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening 192.168.1.1:80)
frontend 1_HTTP_frontend
    bind 192.168.1.1:80 name 192.168.1.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NO_SSL_Rule
    acl acl_620808a860e296.91534155 req.ssl_ver gt 0

    # ACTION: HTTP_TO_HTTPS_RULE
    http-request redirect scheme https code 301 if !acl_620808a860e296.91534155

# Frontend: 1_HTTPS_frontend (Listening 192.168.1.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.1.1:443 name 192.168.1.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6208140971a7a3.08696099.certlist
    mode http
    option http-keep-alive
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_MAP_RULE
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/620809e036a6d1.87483247.txt)]

# Backend: SSL_backend (SSL  backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 192.168.1.1 send-proxy-v2 check-send-proxy

# Backend: blog_server_backend (Server backend blog)
backend blog_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: cloud_nas_backend (cloud  backend NAS)
backend cloud_nas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nas_Server_80 192.168.1.118:80 send-proxy-v2 check-send-proxy

# Backend: dav_nas_backend (dav  backend NAS)
backend dav_nas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nas_Server_80 192.168.1.118:80 send-proxy-v2 check-send-proxy

# Backend: frank_server_backend (Server backend frank)
backend frank_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: flood_server_backend (Server backend flood)
backend flood_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: sync_server_backend (Server backend sync)
backend sync_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: monitoring_server_backend (Server backend monitoring)
backend monitoring_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: kvm_server_backend (Server backend kvm)
backend kvm_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: nas_server_backend (Server backend nas)
backend nas_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

# Backend: tracker_server_backend (Server backend tracker)
backend tracker_server_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server 5erver_Server_80 192.168.1.100:80 send-proxy-v2 check-send-proxy

and my mapfile

Quote

#public subdomains mapping
flood flood_server_backend
frank frank_server_backend
grafana grafana_server_backend
nas  nas_server_backend
kvm kvm_server_backend
monitoring monitoring_server_backend
sync sync_server_backend
tracker tracker_server_backend
cloud cloud_nas_backend
dav dav_nas_backend

You see anymore errors?

EDIT1: Do you recommend to change to nginx who can handle as web and reverse proxy
EDIT2: After studying your latest post I got it.
EDIT3: Why can’t Haproxy multiple services on same port?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 15, 2022, 06:27:48 pm

You still don't understand.  ???
Your first mapfile was correct.

Just do this and tell me if it works.

1. create a mapfile with the below content.

Code: [Select]

#public subdomains mapping
flood WEBSERVER_backend
frank WEBSERVER_backend
www WEBSERVER_backend
torrent WEBSERVER_backend
grafana WEBSERVER_backend
nas WEBSERVER_backend
kvm WEBSERVER_backend
monitoring WEBSERVER_backend
speedtest WEBSERVER_backend
sync WEBSERVER_backend
tracker WEBSERVER_backend
cloud NAS_backend
dav NAS_backend
2. create the following backends and servers

Code: [Select]

WEBSERVER_backend --> contains --> WEBSERVER_server

WEBSERVER_server=192.168.1.100:80


NAS_backend --> contains --> NAS_server

NAS_server=192.168.1.118:80
3. Try to access your webserver and nas.

Code: [Select]

# WEBSERVER_backend
flood.yourdomain.com
frank.yourdomain.com
www.yourdomain.com
torrent.yourdomain.com
grafana.yourdomain.com
nas.yourdomain.com
kvm.yourdomain.com
monitoring.yourdomain.com
speedtest.yourdomain.com
sync.yourdomain.com
tracker.yourdomain.com


# NAS_backend
cloud.yourdomain.com
dav.yourdomain.com


If it still doesn't work then I can only offer you paid support. I hope you understand. It is my free time and I can only help for free up to a certain point.

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opns_neuling on February 19, 2022, 01:17:00 pm

Quote from: Morta on February 12, 2022, 09:13:03 pm

I got this error

Quote

[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

https://forum.opnsense.org/index.php?topic=27065.msg131206#msg131206

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 20, 2022, 05:25:18 pm

I have just tried TCP mode with map file, there is a few more steps to achieve the goal instead of placing the map rule directly to 0_SNI
(I checked the package and found SNI inside, however, haproxy doesn't recognize it in TCP mode, that's why we need to force it to recognize SNI)

1. Create a "Condition" to request client hello
Name: SSL_Hello
Condition type: Custom condition (option pass-through)
Option pass-through: req_ssl_hello_type 1
(https://i.postimg.cc/rD89fwvk/cond-hello.jpg) (https://postimg.cc/rD89fwvk)

2. Create a "Rule" to wait accept SSL hello
Name: tcp_request_content_accept_ssl
Select conditions: SSL_Hello
Execute function: tcp-request-content-accept
(https://i.postimg.cc/mcnNZNhL/rule-ssl.jpg) (https://postimg.cc/mcnNZNhL)

3. Create a "Rule" to wait for inspect
Name: tcp_request_inspect_delay
Optional condition: none
Execute function: tcp-request-inspect-delay
TCP inspection delay: 5s
(https://i.postimg.cc/gX6yxX6v/rule-delay.jpg) (https://postimg.cc/gX6yxX6v)

4. Place the Rule to 0_SNI_frontend in following order
tcp_request_inspect_delay
tcp_request_content_accept_ssl
map
(https://i.postimg.cc/sBx4R0jH/rule-order.jpg) (https://postimg.cc/sBx4R0jH)
(hmdir_ru is my map rule)


Update according to findings in #183 (https://forum.opnsense.org/index.php?topic=23339.msg131582#msg131582)

5. Change the no_SSL condition to Traffic is SSL (locally deciphered)
(https://i.postimg.cc/Cng6Mdtn/nossl.jpg) (https://postimg.cc/Cng6Mdtn)

*Remark
It is advised to use another map file for 1_HTTPS_frontend if necessary
If you really don't want to create another map file, use "SNI TLS extension matches (locally deciphered)" instead

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 20, 2022, 06:04:58 pm

Quote from: opns_neuling on February 19, 2022, 01:17:00 pm

Quote from: Morta on February 12, 2022, 09:13:03 pm

I got this error

Quote

[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

https://forum.opnsense.org/index.php?topic=27065.msg131206#msg131206

Have you Disable web GUI redirect rule in Part 4-1?
If you haven't disable it, your web GUI will keep listening to port 80.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 20, 2022, 07:31:40 pm

I have http webui with port 4444 impossible that listen on 80

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 20, 2022, 09:03:34 pm

Quote from: Morta on February 20, 2022, 07:31:40 pm

I have http webui with port 4444 impossible that listen on 80

(https://i.postimg.cc/YGBYddZD/Screenshot-20220221-035236-01.jpg) (https://postimg.cc/YGBYddZD)
That's what I get when I try to add another frontend with 192.168.6.1:80, which already exists in my running system.
You must have another process running which listening to port 80 when you get this error

That's why I asked rather you disabled the GUI redirect.
As this redirect process is listening to port 80 of your router IP, and help you redirect any port 80 to the HTTPS port you typed.
If you already disabled, you need to figure out what other plugin you have installed and using port 80.

Guess: GUI redirect, nginx, firewall port forward

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 21, 2022, 08:41:30 am

Just take a look at the config file, I feel strange for some part of it
1. You don't need to type WAN IP in 0_SNI_frontend
instead, it should be 0.0.0.0:80 and 0.0.0.0:443
0.0.0.0 means any IP that points to your router.

2. What is your router IP?
If your router is 192.168.1.1, then 1_HTTP_frontend and 1_HTTPS_frontend will obviously conflict with 0_SNI_fronted
Since 0_SNI_fronted is already listening to 80 and 443 port of your router, you won't able to listen it with 192.168.1.1
Please follow Part 4-2 to create Virtual IP, and set 1_HTTP_frontend and 1_HTTPS_frontend to the virtual IP

If you don't want to create any Virtual IP, please remove 0_SNI_frontend
set 1_HTTP_frontend with 0.0.0.0:80 and 1_HTTPS_frontend 0.0.0.0:443 instead
Since all of your servers are running in http mode, it should work for having no SNI frontend

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 21, 2022, 01:13:24 pm

Quote from: bunchtam on February 20, 2022, 05:25:18 pm

I have just tried TCP mode with map file, there is a few more steps to achieve the goal instead of placing the map rule directly to 0_SNI
(I checked the package and found SNI inside, however, haproxy doesn't recognize it in TCP mode, that's why we need to force it to recognize SNI)

As I said in the tutorial I am not running any services that would require me to load balance / reverse proxy plain TCP traffic. This is why I never tested it and only could provide theoretical assumptions on how it could work.
So thank you very much for providing a good guide on this.

If you don't mind I will add this to the tutorial (credits to your post) when I find the time!

Just a quick question: Which service are you running in tcp mode? Does the service send the SNI header?
Or does you solution work with any TCP based service? Becuase as far as I am aware the service needs to add the SNI header otherwise the access URL is not beeing sent to HAProxy.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 21, 2022, 01:21:52 pm

Quote from: bunchtam on February 21, 2022, 08:41:30 am

Just take a look at the config file, I feel strange for some part of it
1. You don't need to type WAN IP in 0_SNI_frontend
instead, it should be 0.0.0.0:80 and 0.0.0.0:443
0.0.0.0 means any IP that points to your router.

2. What is your router IP?
If your router is 192.168.1.1, then 1_HTTP_frontend and 1_HTTPS_frontend will obviously conflict with 0_SNI_fronted
Since 0_SNI_fronted is already listening to 80 and 443 port of your router, you won't able to listen it with 192.168.1.1
Please follow Part 4-2 to create Virtual IP, and set 1_HTTP_frontend and 1_HTTPS_frontend to the virtual IP

If you don't want to create any Virtual IP, please remove 0_SNI_frontend
set 1_HTTP_frontend with 0.0.0.0:80 and 1_HTTPS_frontend 0.0.0.0:443 instead
Since all of your servers are running in http mode, it should work for having no SNI frontend

I am already on him with the fix.
Also explained to him that the HTTP(s)_frontend IPs could cause issues.
His Apache is also misconfigured badly.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 21, 2022, 03:27:41 pm

Sure, just add it to your tutorial if you like.
I have 2 TCP servers running. OpenVPN and v2ray
(both of them have SNI header with it)

I'm sure not all of the TCP services can use haproxy, for example minecraft server without additional tools.
(One of the ways is to add one more rule to redirect other SSL connections to SSL_backend, and set minecraft server as default backend of 0_SNI, as no conditions or rules in haproxy can catch connections that doesn't have SNI header).

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 21, 2022, 03:29:31 pm

Quote from: bunchtam on February 21, 2022, 03:27:41 pm

Sure, just add it to your tutorial if you like.
I have 2 TCP servers running. OpenVPN and v2ray
(both of them have SNI header with it)

I'm sure not all of the TCP services can use haproxy, for example minecraft server without additional tools.
(One of the ways is to add one more rule to redirect other SSL connections to SSL_backend, and set minecraft server as default backend of 0_SNI, as no conditions or rules in haproxy can catch connections that doesn't have SNI header).

Thank you for confirming!

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 22, 2022, 08:30:30 am

Just checked the L6 warning a bit.
This is the code from haproxy github (https://github.com/haproxy/haproxy/blob/master/src/cfgparse.c)

Code: [Select]

int warnif_tcp_http_cond(const struct proxy *px, const struct acl_cond *cond)
{
if (!cond || px->mode != PR_MODE_HTTP)
return 0;

if (cond->use & (SMP_USE_L6REQ|SMP_USE_L6RES)) {
ha_warning("Proxy '%s': L6 sample fetches ignored on HTTP proxies (declared at %s:%d).\n",
   px->id, cond->file, cond->line);
return ERR_WARN;
}
return 0;
}
The condition: req.ssl_ver (Traffic is SSL....)
This condition might get error code, as http doesn't have tls packages.
Although negating this condition will still redirect packages to 443, haproxy will still run.
But we can simply use "Traffic is HTTP" here.
(Then, I don't know why I get such error when I create 2 port 80 frontend) ???

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 22, 2022, 09:29:18 am

Please repost your current config.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 22, 2022, 11:19:30 am

You mean my working config?

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if xray)
frontend tcp_front
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_6212326a7c07e4.28981163 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_6212326a7c07e4.28981163
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6214a3ae639096.17472719.txt)]

# Frontend: http_front (Listen to VIP:80 and redirect to 443)
frontend http_front
    bind 192.168.6.1:80 name 192.168.6.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: http
    acl acl_62123bbee27260.60165685 ssl_fc

    # ACTION: http_to_https
    http-request redirect scheme https code 301 if !acl_62123bbee27260.60165685

# Frontend: ssl_front (Listen to VIP 443, SSL offload cert)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.6.1:443 name 192.168.6.1:443 accept-proxy ssl no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621244f0de5919.36753000.certlist
    mode http
    option http-keep-alive
    default_backend hkbn_back
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: local
    acl acl_62123a1cebe813.09309501 src 192.168.3.0/24 192.168.5.0/24

    # ACTION: local_ru
    use_backend opn_back if acl_62123a1cebe813.09309501

# Backend: hmdir_back (Backend of v2ray)
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.3.3:443

# Backend: hkbn_back (Backend of HKBN)
backend hkbn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hkbn 192.168.4.2:443 ssl verify none

# Backend: SSL_back (Backend to redirect SSL servers)
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server ssl 192.168.6.1 send-proxy-v2 check-send-proxy

# Backend: opn_back (Backend of opnsense with SSL)
backend opn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opn 192.168.3.1:8443 ssl verify none

# Backend: open_back (Backend of OpenVPN)
backend open_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server open 192.168.3.1:10443

BTW, I know why I will get such warning after reading the code.
Since my TCP layer already received TLS packages and inspected it already.
For HTTP and HTTPS frontend, any more "request" for SSL/TLS will probably get such warning.

Thus, for anyone using map file in 0_SNI_frontend.
They not only need to create "tcp_request_inspect_delay", "tcp_request_content_accept_ssl" and "SSL_Hello" condition
but also need to change the non-HTTP traffic condition to either
Negate of Traffic is SSL (locally deciphered) (ssl_fc), which will not request another TLS package
or
Traffic is HTTP (req.proto_http)
but not
Traffic is SSL (TCP request content inspection) (req.ssl_ver)

If Someone already used map file in 0_SNI_frontend
And they have multiple HTTP server need to offload in HTTPS_frontend, they should either use another map file
or
SNI TLS extension matches (locally deciphered) (ssl_fc_sni)
but not
SNI TLS extension ... (TCP request content inspection) (req.ssl_sni)

*For Morta's case, this might also due to multiple inspection of TLS or another port 80 listener

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 22, 2022, 06:28:24 pm

I have looked wir netsocket but no other app is on port 80 and is working so far

I have also problem that the cert redirect automatically from www.xx.ch to xx.ch

When I insert xx.ch in the map file all in existing sub domains goes to xx.ch as examples buff.xx.ch goes to xx.ch

How can I map at least www.xx.ch to a service without redirect or better how can I map xx.ch and www.xx.ch without redirecting all subdomain to xx.ch vhost

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 22, 2022, 07:14:45 pm

Just set your Webserver as default backend on the https frontend.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 22, 2022, 07:32:33 pm

I have two Webserver. So I can took only one.
Which one? WEBSERVER_backend or NAS_backend?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 22, 2022, 08:35:47 pm

Obviously your Webserver and not your NAS.......  ::)

Also you just exposed your Domain again  :-\

Please don't get me wrong but I really hope you know what you are doing by self-hosting and exposing any services from your private internet connection!
There are many risks involved and you don't seem to me like someone that is aware of everything he is doing and why it needs to be done.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on February 23, 2022, 08:07:18 pm

Would be possible to adjust the tutorial with ipv6 support or a hint how to add ipv6 support to a existing configuration?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on February 24, 2022, 01:22:48 am

Quote from: Morta on February 23, 2022, 08:07:18 pm

Would be possible to adjust the tutorial with ipv6 support or a hint how to add ipv6 support to a existing configuration?

For the frontend that using 0.0.0.0:80, "add" [::]:80
For the frontend that using 0.0.0.0:443, "add" [::]:443

In firewall rule, add one more rule to "allow" "ipv6" "TCP" to "this firewall" in "port 80 and 443"

Note: your DDNS need to update ipv6 too
ddclient 3.9.1 should be able to update ipv6 . However, I don't know rather OPNsense had implemented to update ipv6 automatically.

P.S. dyndns plugin is too old, it doesn't support ipv6

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: figadore on February 25, 2022, 07:27:07 am

When creating the virtual IP, there's a new option not included in the picture. Is there a correct setting for "Allow service binding"? Does it make a difference?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 27, 2022, 11:30:42 am

I mean it is self-explanatory.

To answer your question: You will want this enabled.

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opns_neuling on March 13, 2022, 09:42:06 pm

Quote from: Morta on February 12, 2022, 09:13:03 pm

I got this error

Quote

[WARNING] (20353) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:70).
Warnings were found.
Configuration file is valid

What is wrong?

on Step:
Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
Here we will only create a "NoSSL_condition", which is necessary in order to identify non-HTTP traffic.


Please replace "req.ssl_ver gt 0" with "ssl_fc" also, select "custom" and on custom field enter "ssl_fc"

https://redmine.pfsense.org/issues/9261

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 14, 2022, 01:23:40 pm

Can you make a screenshot? I don’t understand you instructions

I have no req.ssl_ver 0 in my configuration!

So?

(https://abload.de/img/auswahl_024oaj7m.png)

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 14, 2022, 08:22:32 pm

Mentioned in #183
Traffic is SSL (locally deciphered) is ssl_fc

And the condition need to be replaced is Part 5.7.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 15, 2022, 06:55:39 pm

Thanks for detailed instructions, I've follow step by step to make a web hosting running nginx with https support.

My HAProxy Config:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_622eebaf197419.36314953 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_622eebaf197419.36314953

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/622eef9a9d7268.16491040.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Backend: backend_pool_abc ()
backend backend_pool_meet_huuich_vn
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server PLEX_server 192.168.82.11:32400 ssl verify none

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy
My Map File content:

Code: [Select]

# public access subdomains
plex PLEX_backend
My nginx website config

Code: [Select]

server {
        listen 32400;
        listen [::]:32400;

        root /var/www/mywebsite.com/html;
        index index.html index.htm index.nginx-debian.html;

        server_name mywebsite.com;

        location / {
                try_files $uri $uri/ =404;
        }
}
I can go my website on internal http://192.168.82.11:32400 is ok but when I access http://mywebsite.com browser go to https://mywebsite.com and show error

Code: [Select]

503 Service Unavailable
No server is available to handle this request.
How can I fix this and show my website https://mywebsite.com ok? Thanks!

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 15, 2022, 07:15:48 pm

Quote from: huuich on March 15, 2022, 06:55:39 pm

I can go my website on internal http://192.168.82.11:32400 is ok but when I access http://mywebsite.com browser go to https://mywebsite.com and show error

Code: [Select]

503 Service Unavailable
No server is available to handle this request.
How can I fix this and show my website https://mywebsite.com ok? Thanks!

First.
The entry "plex PLEX_backend" in the mapfile means that you will have to access it using the "plex" subdomain. --> In your case "plex.mywebsite.com"!

Alternatively just set the PLEX_backend as default backend on your HTTPS_frontend.

Second.
http will always get redirected to https. This is intended and you will most probably want this! This is configured using the HTTPtoHTTPS_rule and NoSSL_condition.

Third.
Apart from the above your config looks good. (just took a very short look at it)

Fourth.
You might have to disable the SSL checkbox in the PLEX_server settings.
But only if you are REALLY accessing it locally using http://IP:32400 and the service is NOT redirecting you to HTTPS. But I highly doubt this since Plex is running on a self-signed SSL cert by default...

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 15, 2022, 11:00:02 pm

Quote from: TheHellSite on March 15, 2022, 07:15:48 pm

First.
The entry "plex PLEX_backend" in the mapfile means that you will have to access it using the "plex" subdomain. --> In your case "plex.mywebsite.com"!

Alternatively just set the PLEX_backend as default backend on your HTTPS_frontend.

Second.
http will always get redirected to https. This is intended and you will most probably want this! This is configured using the HTTPtoHTTPS_rule and NoSSL_condition.

Third.
Apart from the above your config looks good. (just took a very short look at it)

Fourth.
You might have to disable the SSL checkbox in the PLEX_server settings.
But only if you are REALLY accessing it locally using http://IP:32400 and the service is NOT redirecting you to HTTPS. But I highly doubt this since Plex is running on a self-signed SSL cert by default...

You are right, mysubdomain.mywebsite.com, and follow your guide I've edit my Map File to "mysubdomain PLEX_backend" and "disable the SSL checkbox in the PLEX_server settings" and voila my subdomain website run https:// ok with A+ score, Thank you so much for quick and detailed reply!

P/s: I've follow your guide with cloudflare instead. (almost every settings are the same like your guide)

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 16, 2022, 02:47:59 am

Quote from: Tattoofreak on January 24, 2022, 02:18:58 pm

I want to set up HAProxy just for routing traffic based on URLs (https://xyz.domain.com goes to server 1 and https://abc.domain.com goes to server 2, etc...).
All SSL stuff for the destination web servers is being handled by a separate Linux certificate server and the web servers themselfes, independent from OPNsense/HAProxy. HAProxy is really only needed for routing traffic based on URLs, nothing more, nothing less.I want to set up HAProxy just for routing traffic based on URLs (https://xyz.domain.com goes to server 1 and https://abc.domain.com goes to server 2, etc...).
All SSL stuff for the destination web servers is being handled by a separate Linux certificate server and the web servers themselfes, independent from OPNsense/HAProxy. HAProxy is really only needed for routing traffic based on URLs, nothing more, nothing less.

Did you find solution for your purpose? I wanna do same like you.

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 16, 2022, 03:52:23 am

Quote from: TheHellSite on January 29, 2022, 09:37:27 am

You have to set your backends and frontends to HTTP Mode.
Also disable SSL offloading on the frontends.

But I can't guarantee for sure that it will work.
TCP Mode will never (with a few exceptions) work because there is no header in the packets that would tell HAProxy which service to send the traffic to.

HTTP Mode could work, but you might need to create some "http header contains..." conditions.

I can do that using SSL passthrough follow step by step this guide (https://forum.opnsense.org/index.php?topic=18538.msg84958#msg84958) from @alh

Every works ok except my Nginx log can not receive real ip from client, it's only show ip  local of my OPNSense server in file /var/log/nginx/access.log

Do you think can mix SSL passthrough with your guide (edit 0_SNI_frontend with rules from SSL passthrough and Default Backend Pool) or any solution to receive real ip from client with SSL passthrough? Thanks!

Code: [Select]

This should work for any TCP-based SSL/TLS encrypted service in passthrough (HAProxy: TCP) mode... It does NOT work for STARTTLS!

In this example I use TCP port 443.

HAProxy plugin: Create "Real Server" (enter name, IP/FQDN and port number if different from 443, the rest can be left at default)
HAProxy plugin: Create "Backend Pool" (enter name, set mode to TCP and select the real server from step 1)
HAProxy plugin: Create "Condition" (enter name ["traffic_ssl"], condition type is "custom condition (option pass-through)" with value "req_ssl_hello_type 1")
HAProxy plugin: Create "condition" (enter name ["myservice_sni"], condition type is "SNI TLS extension matches (TCP request content inspection)" with value "myservice.example.com" or whatever your FQDN is)
HAProxy plugin: Create "Rule" (enter name ["request_inspect_delay"], select no condition, function is "tcp-request inspect delay" with value "5s" or whatever suits you)
HAProxy plugin: Create "Rule" (enter name ["request_content_accept_ssl"], select condition of 3 ["traffic_ssl"], function is "tcp-request content accept")
HAProxy plugin: Create "Rule" (enter name ["myservice_sni"], select condition of 4 ["myservice_sni"], function is "Use specific backend pool" with your pool from 2)
HAProxy plugin: Create "Public service" (enter name ["https_passthrough"], choose a listen address [":443" for all], type is "TCP" and select the 3 rules created earlier)
HAProxy plugin: Enable plugin or test/apply
Firewall: allow incoming traffic to WAN (address) or whatever for TCP port 443.

That works at least for me. If you have double NAT you would need to disable port randomization for the proxied port...

Does that help you?

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 16, 2022, 11:51:30 am

If all SSL certificate are handled by webserver themselves.
Follow #176 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354), step 1-4 will allow 0_SNI_FRONTEND to recognize TLS package and redirect under TCP mode

Note: All backend redirect from 0_SNI_FRONTEND should be in TCP mode
HAProxy only work with server that using TLS package. For those server that doesn't contain TLS package (for example, game servers), HAProxy won't work. Although you can set default backend server to game server, but there's only one default backend server.

BTW, for backend server getting source IP, enabling X-Forwarded-For header for all frontend should work
However, this one only apply to HTTP.
reference (https://www.haproxy.com/fr/blog/preserve-source-ip-address-despite-reverse-proxies/)
 reference 2 (https://www.haproxy.com/blog/layer-7-load-balancing-transparent-proxy-mode/)

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 16, 2022, 09:20:26 pm

Quote from: Bunch on March 16, 2022, 11:51:30 am

If all SSL certificate are handled by webserver themselves.
Follow #176 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354), step 1-4 will allow 0_SNI_FRONTEND to recognize TLS package and redirect under TCP mode

Note: All backend redirect from 0_SNI_FRONTEND should be in TCP mode
HAProxy only work with server that using TLS package. For those server that doesn't contain TLS package (for example, game servers), HAProxy won't work. Although you can set default backend server to game server, but there's only one default backend server.

BTW, for backend server getting source IP, enabling X-Forwarded-For header for all frontend should work
However, this one only apply to HTTP.
reference (https://www.haproxy.com/fr/blog/preserve-source-ip-address-despite-reverse-proxies/)
 reference 2 (https://www.haproxy.com/blog/layer-7-load-balancing-transparent-proxy-mode/)

I've follow your guide but though my backend server ctl_backend has code

Code: [Select]

http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Real-IP %[src]
But it can not get source ip, here is my full config, could you show me how to get source ip? Or I have to have 2 OPNSense, one for SSL passthrough and one use Let's Encrypt Wildcard Certificates like this topic guide? Could 1  OPNSense sever run both nginx reverse proxy and HAProxy?

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    # tuning options
    timeout client 30s

    # logging options
    # ACL: traffic_ssl
    acl acl_601a842f14cee3.17646593 req_ssl_hello_type 1

    # ACTION: request_content_accept_ssl
    tcp-request content accept if acl_601a842f14cee3.17646593
    # ACTION: request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Frontend (DISABLED): 1_HTTP_frontend (Listening on 192.168.64.1:80)

# Frontend (DISABLED): 1_HTTPS_frontend (Listening on 192.168.64.1:443)

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Real-IP %[src]
    http-reuse safe
    server PLEX_server 192.168.82.11:32400

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend (DISABLED): SSL_backend ()

# Backend: ctl_backend ()
backend ctl_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Real-IP %[src]
    http-reuse safe
    server ctl_server 192.168.82.11:32401

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 17, 2022, 04:15:01 am

Notice that you are using TCP mode frontend only.
X-forwarded-for will not simply work under any case that using TCP mode, as X-forwarded-for will only work under HTTP mode and need to enable in all layers (TCP mode will not forward any header)

For TCP mode, you need to set all backend that wants to get real client IP and redirected from this frontend to use Proxy Protocol (either version 1 or 2, while verion 1 sending ASCII data and version 2 sending binary)
You will find "send-proxy" under that backend, which means it will send another package via proxy protocol to your real server.

Your real server need to modify config to read proxy protocol package too.
 Nginx guide to set it accept, reading and logging proxy protocol address (https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/)
(Not 2 lines, tones should be modified whenever there is a layer of TCP mode)

P.S. Theoretically, setting like OP having (SNI frontend->send-proxy backend->HTTPS frontend with X-forwarded-for) should also forward proxy protocol package to real server too.

Example: my v2ray server is living in TCP mode (although I don't care about the IP log)
I will get router IP in log if I don't user proxy protocol.
I cannot reach v2ray server if I use proxy protocol in haproxy but not changing any config of v2ray server (probably rejected by the server itself)
I will get correct client IP if I use proxy protocol in haproxy and changing the config of v2ray to accept proxy protocol

My HAProxy config is as follow:
1. open_back is OPENVPN live in TCP mode, and will not get any real client IP (Always 192.168.3.1)
2. hmdir_back is v2ray (another proxy server) live in TCP mode, and will get real client IP (I need to modify v2ray config to make it read and log proxyprotocol
3. opn_back is opnsense page live in HTTP mode, will not get real client IP as I don't care.
4. unifi_back is unifi controller page live in HTTP mode, I don't care rather it get real IP too
5. hkbn_back is ISP router page, which has no access right to my LAN, just a dummy to fake ISP that I'm using their device, all unwanted access will reditect to this page (Their device always open 443 to public access)
5. opn_back and unifi_back can only be accessed with source IP as 192.168.3.0/24 and 192.168.5.0/24 and I have tested no one can access to these pages except my LAN clients. Thus, real client IP should be passed to SSL_frontend successfully too.
Just need to add backend with http-request set-header X-Real-IP %[src] (Although there will be warning, you can do it in frontend, as HAProxy config manual (http://cbonte.github.io/haproxy-dconv/2.0/configuration.html)mentioned both ways will work) and config webserver read and log X-Real-IP

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if v2ray)
frontend tcp_front
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_6212326a7c07e4.28981163 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_6212326a7c07e4.28981163
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6214a3ae639096.17472719.txt)]

# Frontend: http_front (Listen to VIP:80 and redirect to 443)
frontend http_front
    bind 192.168.6.1:80 name 192.168.6.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: http
    acl acl_62123bbee27260.60165685 ssl_fc

    # ACTION: http_to_https
    http-request redirect scheme https code 301 if !acl_62123bbee27260.60165685

# Frontend: ssl_front (Listen to VIP 443, SSL offload cert)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.6.1:443 name 192.168.6.1:443 accept-proxy ssl no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621244f0de5919.36753000.certlist
    mode http
    option http-keep-alive
    default_backend hkbn_back
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: local
    acl acl_62123a1cebe813.09309501 src 192.168.3.0/24 192.168.5.0/24

    # ACTION: local_map_ru
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623230d7bffd04.94702836.txt)] if acl_62123a1cebe813.09309501

# Backend: hmdir_back (Backend of v2ray)
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.3.3:443 send-proxy-v2 check-send-proxy

# Backend: hkbn_back (Backend of HKBN)
backend hkbn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hkbn 192.168.4.2:443 ssl verify none

# Backend: SSL_back (Backend to redirect SSL servers)
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server ssl 192.168.6.1 send-proxy-v2 check-send-proxy

# Backend: opn_back (Backend of opnsense with SSL)
backend opn_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server opn 192.168.3.1:8443 ssl verify none

# Backend: unifi_back (Backend of unifi with SSL)
backend unifi_back
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server unifi 192.168.3.4:8443 ssl verify none

# Backend: open_back (Backend of OpenVPN)
backend open_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server open 192.168.3.1:10443

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 17, 2022, 09:38:06 am

@Bunch Thank you for detailed guide and attached config. I've tried to simulate like your server but now my server not working

This is my config

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if v2ray))
frontend tcp_front
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: traffic_ssl
    acl acl_601a842f14cee3.17646593 req_ssl_hello_type 1

    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_601a842f14cee3.17646593
    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Frontend: http_front (Listening on 192.168.64.1:80)
frontend http_front
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_622eebaf197419.36314953 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_622eebaf197419.36314953

# Frontend: ssl_front (Listening on 192.168.64.1:443)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/622eef9a9d7268.16491040.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Backend: hmdir_back ()
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.82.11:32401 send-proxy-v2 check-send-proxy

# Backend (DISABLED): PLEX_backend ()

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend: SSL_back ()
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend (DISABLED): ctl_backend ()
My hmdir_ru is link to PUBLIC_SUBDOMAINS_map map file with content:

Code: [Select]

# public access subdomains
c hmdir_back
Could you share content of map file in local_map_ru in this config

Code: [Select]

# ACTION: local_map_ru
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623230d7bffd04.94702836.txt)] if acl_62123a1cebe813.09309501
And ACL http in this config

Code: [Select]

# logging options
    # ACL: http
    acl acl_62123bbee27260.60165685 ssl_fc
Thank you!

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 17, 2022, 10:07:30 am

My local_map_rule is:
Test type= if
Selection condition= local
Execute function= Map domains to backend pools using a map file
Map_file= local_map

Condition local:
Condition type= Source IP matches specified IP
Source IP=192.168.3.0/24 192.168.5.0/24

Local_map:

Code: [Select]

opn opn_back
unifi unifi_back
For ssl_fc, it is mentioned in #174 Step 5 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354)
And #183 (https://forum.opnsense.org/index.php?topic=23339.msg131582#msg131582)

For testing why it won't work in your case,
You can try not using Proxy Protocol first. If you can access your backend, that means your webserver config is not accepting proxy protocol (It won't work both way at the same time)
You need to use Proxy Protocol and modify webserver config ( for example, /etc/nginx/sites-enabled/default) at the same time

Similar test case mentioned in previous post

Quote

Example: my v2ray server is living in TCP mode (although I don't care about the IP log)
I will get router IP in log if I don't user proxy protocol.
I cannot reach v2ray server if I use proxy protocol in haproxy but not changing any config of v2ray server (probably rejected by the server itself)
I will get correct client IP if I use proxy protocol in haproxy and changing the config of v2ray to accept proxy protocol

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 17, 2022, 10:15:45 am

Quote from: Bunch on March 17, 2022, 10:07:30 am

For testing why it won't work in your case,
You can try not using Proxy Protocol first. If you can access your backend, that means your webserver config is not accepting proxy protocol (It won't work both way at the same time)
You need to use Proxy Protocol and modify webserver config ( for example, /etc/nginx/sites-enabled/default) at the same time

Similar test case mentioned in previous post

Quote

Example: my v2ray server is living in TCP mode (although I don't care about the IP log)
I will get router IP in log if I don't user proxy protocol.
I cannot reach v2ray server if I use proxy protocol in haproxy but not changing any config of v2ray server (probably rejected by the server itself)
I will get correct client IP if I use proxy protocol in haproxy and changing the config of v2ray to accept proxy protocol

My /etc/nginx/sites-enabled is

Code: [Select]

server {
listen 32401;
listen [::]:32401;

server_name c.mywebsite.com;

root /var/www/html;

location / {
try_files $uri $uri/ /yourls-loader.php$is_args$args;
}

location ~ \.php$ {
include fastcgi.conf;

fastcgi_index index.php;

fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}

How could use Proxy Protocol and modify webserver config with my HAProxy config above?
Sorry for basic question. Thank you for detailed explaination.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 17, 2022, 11:04:16 am

That's not basic question IMO.  :D
The modification method is mentioned in Nginx guide (https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/)

Should be modified as below (haven't tested it)

Code: [Select]

server {
listen 32401 proxy_protocol;
listen [::]:32401 proxy_protocol;

server_name c.mywebsite.com;

root /var/www/html;

location / {
try_files $uri $uri/ /yourls-loader.php$is_args$args;
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr
}

location ~ \.php$ {
include fastcgi.conf;

fastcgi_index index.php;

fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}

For the log config, I forgot rather it is /etc/nginx/nginx.conf by default
You need to change the log format inside http{} to something like

Code: [Select]

log_format combined '$proxy_protocol_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent"';

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 18, 2022, 10:05:44 am

Quote from: Bunch on March 17, 2022, 11:04:16 am

The modification method is mentioned in Nginx guide (https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/)

Should be modified as below (haven't tested it)

Your code "haven't tested but working for me with http website, but when I change to https with certbot

Code: [Select]

certbot --nginx -d c.mywebsite.com
My https website can not access though I've change my server port to 443, this is my nginx website config

Code: [Select]

server {
listen 32401 proxy_protocol;
listen [::]:32401 proxy_protocol;

listen [::]:443 ssl ipv6only=on proxy_protocol; # managed by Certbot
    listen 443 ssl proxy_protocol; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/c.mywebsite.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/c.mywebsite.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

server_name c.mywebsite.com;

root /var/www/html;

location / {
try_files $uri $uri/ /yourls-loader.php$is_args$args;
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr;
}

location ~ \.php$ {
include fastcgi.conf;

fastcgi_index index.php;

fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}
}
and my current HAProxy configure

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    6
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): https_passthrough ()

# Frontend: tcp_front (Listen to 0.0.0.0:443, TCP SNI handler, redirect if v2ray))
frontend tcp_front
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_back
    # tuning options
    timeout client 30s

    # logging options
    # ACL: traffic_ssl
    acl acl_601a842f14cee3.17646593 req_ssl_hello_type 1

    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_601a842f14cee3.17646593
    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Frontend: http_front (Listening on 192.168.64.1:80)
frontend http_front
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_622eebaf197419.36314953 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_622eebaf197419.36314953

# Frontend: ssl_front (Listening on 192.168.64.1:443)
frontend ssl_front
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/622eef9a9d7268.16491040.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: hmdir_ru
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/622eeaa3044ba7.74145133.txt)]

# Backend: hmdir_back ()
backend hmdir_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server hmdir 192.168.82.11:443 ssl verify none send-proxy-v2 check-send-proxy

# Backend (DISABLED): PLEX_backend ()

# Backend (DISABLED): acme_challenge_backend (Added by Let's Encrypt plugin)

# Backend: SSL_back ()
backend SSL_back
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend (DISABLED): ctl_backend ()
Could you take a look and guide me how to run https on my website?

Quote from: Bunch on March 17, 2022, 11:04:16 am

For the log config, I forgot rather it is /etc/nginx/nginx.conf by default
You need to change the log format inside http{} to something like

Code: [Select]

log_format combined '$proxy_protocol_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent"';

Fire I use above code has a error, so I google and found a solution for working log, I post here for anyone need

Code: [Select]

log_format my_log '$proxy_protocol_addr - $remote_user [$time_local] '
                       '"$request" $status $body_bytes_sent '
                       '"$http_referer" "$http_user_agent"';

access_log /var/log/nginx/access.log my_log;

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 18, 2022, 12:59:38 pm

Better check /etc/letsencrypt/options-ssl-nginx.conf
Check rather the SSL protocol is too old (is it having something like TLSv1 TLSv1.1)
If the settings are too old, try to change those settings in that file according to this link (https://ssl-config.mozilla.org/#server=nginx&version=1.17.7&config=intermediate&openssl=1.1.1d&guideline=5.6)

Guess below lines need to be modified if case exists.

Code: [Select]

ssl_session_timeout 1d;
    ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
    ssl_session_tickets off;

    # intermediate configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    # HSTS (ngx_http_headers_module is required) (63072000 seconds)
    add_header Strict-Transport-Security "max-age=63072000" always;

    # OCSP stapling
    ssl_stapling on;
    ssl_stapling_verify on;
For real server in HAProxy, I have tried turning SSL on or off doesn't affect my servers under TCP mode, don't know rather it is the same in your case, please try it too.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on March 18, 2022, 01:06:28 pm

I'm probably out of place saying this, as is not my thread, but should't this discussion go to another thread and leave this one for it's original purpose?
It has branched off now to "how can I enable TLS on my website", from "how can I log the client ip not the proxy ip on the backend webserver" and "how do I use proxy_protocol".
What do you think?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 18, 2022, 01:13:51 pm

Yes, I think it is out of HAProxy scope too.
When send-proxy or X-forwarded-for part is done in HAProxy, nothing can be done in HAProxy afterwards.
All other settings are related to Nginx or some other server settings.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: huuich on March 18, 2022, 02:14:35 pm

@cookiemonster Thank for your comment about back to original purpose, I'll create a new topic relate my questions. Thanks.

@Bunch: I'm very thankful to you for your help. Best Regards!

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: The_Dave on March 18, 2022, 03:10:11 pm

I have a very serious problem with this haproxy config since I updated to 22.1.3. Suddenly haproxy didn't start anymore. On further investigation trying to start haproxy through the commandline showed that suddenly the ipadresses for the frontend cannot be bound anymore:

Code: [Select]

root@OPNsense:/home/David # /usr/local/etc/rc.d/haproxy start
Starting haproxy.
[ALERT]    (21351) : Starting frontend 1_HTTP_frontend: cannot bind socket (Can't assign requested address) [192.168.64.1:80]
[ALERT]    (21351) : Starting frontend 1_HTTPS_frontend: cannot bind socket (Can't assign requested address) [192.168.64.1:443]
[ALERT]    (21351) : [/usr/local/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.
/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy
Any help on how to fix this would be really appreciated

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 18, 2022, 06:12:04 pm

It seems that it is the same issue as This thread (https://forum.opnsense.org/index.php?topic=27547.msg133659#msg133659)
I have the same issue after update and reboot.
For temporary fix, edit the VIP, save without any changes, then apply.
You will able to start HAProxy again.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: The_Dave on March 18, 2022, 08:52:27 pm

Quote from: Bunch on March 18, 2022, 06:12:04 pm

For temporary fix, edit the VIP, save without any changes, then apply.

Thank you this actually works for now, I hope there will be a proper solution or fix for this soon though

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: balrog on March 20, 2022, 10:49:35 am

I had the problem after updating to OPNsense 22.1.3 that the HAProxy service did not start anymore. I was able to solve the problem by editing the Virtual IP and saving it again without adjusting it. After that the service could be started again.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 20, 2022, 02:13:15 pm

Code: [Select]

2022/03/20 15:00:53 [error] 1124599#1124599: *22208 upstream prematurely closed connection while reading response header from upstream, client: 192.168.1.1, server: sync.xxx.ch, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/run/uwsgi/mozilla-firefox-sync-server.sock:", host: "sync.xxx.ch"

I got a 502 Gateway error on this site with nginx.

Where is the option of keep-alive for the backends? Or anybody knows how to fix this error?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 22, 2022, 08:19:47 am

Check and change Services: HAProxy: Settings: Default Parameters
Check Tuning Options of your frontends rather you overwrite the setting too.
If it doesn't fix the issue, then it is Nginx setting problem
Check this link (https://stackoverflow.com/questions/36488688/nginx-upstream-prematurely-closed-connection-while-reading-response-header-from) too for Nginx setting

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 22, 2022, 09:37:57 am

Thanks for reply bunch!

The link is for the settings with a reverse proxy with nginx.

I will look at the tunables of Haproxy.

I found this link

https://serveanswer.com/questions/upstream-prematurely-closed-connection-while-reading-upstream-large-files (https://serveanswer.com/questions/upstream-prematurely-closed-connection-while-reading-upstream-large-files)

Also possible that is a connection error to the SQLite database.

I will give a try at home.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 22, 2022, 02:30:22 pm

I try to add full IPv6 support to my Haproxy configuration.
Now I have a problem with IPv6 localhost or loopback address

I did a real server ssl_server_ipv6 with IPv6 ::1

(https://abload.de/img/e083e5c0-101f-494d-984zjpo.png)
(https://abload.de/img/52010dc8-40b7-4474-abm6jvq.png)

I got following error when I add ssl_server_ipv6 to SSL_backend
(https://abload.de/img/ea69a72d-3153-4847-bb8fjuf.png)

How I can fix it or do a workaround?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 22, 2022, 03:41:30 pm

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 1000s
    timeout connect 1000s
    timeout server 1000s
    retries 3
    default-server init-addr libc,last
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend (Listening o)
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind :::80 name :::80
    bind :::443 name :::443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 1000s

    # logging options

# Frontend: HTTP_frontend (Listening 127.0.0.1:80)
frontend HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    bind [::1]:80 name [::1]:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 1000s

    # logging options
    # ACL: NoSSL_condition
    acl acl_621d0b77c74989.24704837 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_621d0b77c74989.24704837

# Frontend: HTTPS_frontend (Listinging on 127.0.0.1:443)
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621d11c7cad951.61400293.certlist
    bind [::1]:443 name [::1]:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621d11c7cad951.61400293.certlist
    mode http
    option http-keep-alive
    default_backend WEBSERVER_backend
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/621d0c7054ddb7.46420139.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: WEBSERVER_backend ()
backend WEBSERVER_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server WEBSERVER_server 192.168.1.100:80 send-proxy-v2 check-send-proxy
    server WEBSERVER_server_ipv6 2a02:XXX:XXX::2000:80 send-proxy-v2 check-send-proxy

# Backend: NAS_backend ()
backend NAS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server NAS_server 192.168.1.118:80
    server NAS_server_ipv6 2a02:XXX:XXX::1000:80

# Backend: WEBSERVER_SSL_backend ()
backend WEBSERVER_SSL_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server WEBSERVER_server_ssl 192.168.1.100:443
    server WEBSERVER_server_ssl_ipv6 2a02:XXX:XXX::2000:443

It's this a correct and possible configuration?

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 22, 2022, 04:01:34 pm

I don't think you need to create another ipv6 real server, as long as it is the same sever in ipv4
You only need to add :::443 and :::80 to frontend listener (in frontend, [::]:80 is the same as :::80, in case you confused with the syntax)
That will be ipv6 to 4 setup.

If you add 2 real server to the same backend, you are load balancing them.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on March 22, 2022, 05:35:53 pm

Quote from: Bunch on March 22, 2022, 04:01:34 pm

I don't think you need to create another ipv6 real server, as long as it is the same sever in ipv4
You only need to add :::443 and :::80 to frontend listener (in frontend, [::]:80 is the same as :::80, in case you confused with the syntax)
That will be ipv6 to 4 setup.

If you add 2 real server to the same backend, you are load balancing them.

Thanks for the Input.
Real IPv6 or Dual Stack support of HAproxy would be nice but so I have a fallback opportunity if IPv4 or IPv6 are out of service by me or the ISP! 

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: os_admin on March 27, 2022, 08:59:47 pm

Quote from: The_Dave on March 18, 2022, 03:10:11 pm

I have a very serious problem with this haproxy config since I updated to 22.1.3. Suddenly haproxy didn't start anymore. On further investigation trying to start haproxy through the commandline showed that suddenly the ipadresses for the frontend cannot be bound anymore:

This behaviour seems to be gone since 22.1.4_1. No guarantee.

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: os_admin on March 27, 2022, 09:43:26 pm

Quote from: TheHellSite on November 10, 2021, 03:33:49 pm

Just tested it out myself. Basic Auth is so easy to set up that I am not really willing to cover it in this guide.
First create the user(s) in HAProxy. Then in the relevant backends activate basic auth and select the user(s).

Thanks for this tutorial. It saved my ass. I learned a lot about OPNsense and HAProxy. At last I enabled basic auth. on one of my backends. Anything was fine before, but after activating it I can't no longer login into the service web frontend itself. If I access the frontend browser asked for the basic auth, after that I see the login screen of the service, but after put in the service credentials the FE refresh and shows the login screen again.
If I disable basic auth, the service FE works as expected.

What do I miss? Maybe someone out there has a hint...

Title: Re: Tutorial 2021/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 31, 2022, 06:58:19 am

Strange,  have just tested one of my backend
Create user->Enable Basic Auth and select Allowed users in backend
It works as expected.

Edit: Just tested a bit deeper, for some pages like unifi controller, it will always redirect to wrong page
For some pages like opnsense web UI, in chrome(PC), it will keep prompting for auth, but in firefox(PC & mobile), everything works normally

BTW, nothing more can be done in haproxy too, as some site in some browser works normally. Thus, the problem is due to webserver and browser.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 01, 2022, 05:58:07 pm

Quote from: cookiemonster on March 18, 2022, 01:06:28 pm

I'm probably out of place saying this, as is not my thread, but should't this discussion go to another thread and leave this one for it's original purpose?
It has branched off now to "how can I enable TLS on my website", from "how can I log the client ip not the proxy ip on the backend webserver" and "how do I use proxy_protocol".
What do you think?

Absolutely true!  ;D

I don't want to sound like an asshole here, but this tutorial was intended to get the basics working for new users.
This is also why I stopped answering questions about issues like "my service_abc has the requirement_xyz how work????".

If there are any questions in that regard then people should consider posting them...
here: https://forum.opnsense.org/index.php?board=28.0
or here: https://discourse.haproxy.org/

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 01, 2022, 05:58:31 pm

Quote from: Bunch on March 18, 2022, 06:12:04 pm

It seems that it is the same issue as This thread (https://forum.opnsense.org/index.php?topic=27547.msg133659#msg133659)
I have the same issue after update and reboot.
For temporary fix, edit the VIP, save without any changes, then apply.
You will able to start HAProxy again.

Thank you for posting the workaround!

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 11, 2022, 09:23:22 pm

Just a quick notifcation for everyone following the thread.
It seems like Let's Encrypt changed something regarding wildcard certificates.

I updated the picture in Part 3 - Step 6 to reflect the changes necessary in order to obtain a certificate.

You will have to remove the alt name "*.yourdomain.tld" and change the common name to "*.yourdomain.tld".

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: ElliotAlderson on April 12, 2022, 03:54:12 am

Code: [Select]

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind website.com:443 name website.com:443
    bind website.com:80 name website.com:80
    mode tcp
    default_backend SSL_backend
    timeout client 30s

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 10.10.10.1:80 name 10.10.10.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 30s

    # ACL: NoSSL_cond
    acl acl_62548efaf067e6.21908045 req.ssl_ver gt 0
    # ACTION: HTTPupgrade_rule
    http-request redirect scheme https if !acl_62548efaf067e6.21908045

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 10.10.10.1:443 name 10.10.10.1:443 accept-proxy ssl ssl-min-ver TLSv1.3 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62549082216928.65241361.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 30s

    # ACTION: PUBLIC_SUBDOMAINS_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62548f2d97ef05.80304462.txt)]

# Backend: club_backend ()
backend club_backend
website.com    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server club_host 10.0.0.94:3000 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
website.com    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    timeout connect 30s
    timeout server 30s
    server SSL_host 10.10.10.1 send-proxy-v2 check-send-proxy

I switched over from pfSense to OPNSense months ago and I had to set my side projects to the side because I simply could not replicate my HAProxy setup from before. I'm thankful for this tutorial since it's seems like the closest to what I used to have.

I'm extremely lost here. I have gone through this tutorial many times to double check my steps, I have tried changing things on my own.
At the moment, the HTTP->HTTPS redirect doesn't seem to work at all (empty response, no redirected) and the https site gives a blank response as well.
I have checked it with tcpdump directly on the OPNSense shell and could see that packets do get exchanged between my host and the virtual IP.  (TCP handshake, TLSv1 Client Hello, End connection )
The site itself is definitely working correctly internally at the host specified in the config file.

In the config I posted, I used website.com:443 in the SNI frontend.
I have tried it with 0.0.0.0:443 and my public IP with no success.

10.0.0.94 is in my LAN.
10.10.10.1 is the virtual interface I created.

My wildcard certificate seems to be working correctly.

I would really appreciate some help  :'(


By the way, what would the process be for getting another domain and wildcard cert to work added to this setup?

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 12, 2022, 07:06:19 pm

You are missing "code 301" in HTTPupgrade_rule
(Part 5-9)

Explanation of the code can be found here (https://www.haproxy.com/blog/redirect-http-to-https-with-haproxy/)

Quote

This technique will only work when using mode http because it redirects at the HTTP layer using a 302 Found HTTP response status, which is known as a temporary redirect. Once you’re fully committed to using HTTPS and have tested it thoroughly on your website, you may wish to instruct the browser to cache the redirect, which will save one round trip between the browser and HAProxy, speeding up page load times. Set the code parameter to 301 to send a 301 Moved Permanently status back, which browsers can cache:

Although it should work with 302, just give it a try

BTW, I don't know why there is website.com in the code of both backend. There shouldn't be something like this.

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: ElliotAlderson on April 12, 2022, 09:01:22 pm

Oh, yeah. I actually did have the code 301 there originally. It's still the same effect with that though. That's just something I forgot to change back when I was trying new things. (Saw an older forum post that didn't use the "code 301" part.)

Regarding about the website.com,
In the config I posted, I used website.com:443 in the SNI frontend.
I have tried it with 0.0.0.0:443 and my public IP as well with no success.

I just can't get it to work so I'm trying different things.

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 13, 2022, 02:16:04 am

I mean

Quote

# Backend: SSL_backend ()
backend SSL_backend
website.com    mode tcp

And

Quote

# Backend: club_backend ()
backend club_backend
website.com    mode http

Usually the wouldn't have SNI or domain names hard code in backends

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 13, 2022, 10:36:48 am

This week i have moved away from pfSense, I had acme, cloudflare & HAProxy working prior to the switch. Installed opnsense while slowly getting my services back online I came across this well written tutorial which seems more in-depth than my old setup but run into issues while accessing the hosted web service, it is failing to load with a 522 error, the connection if timing out before a response I think?

I have not got any further in the guide than part 5, step 10. Accessing from outside of my network as this is not possible so far.

I have a static WAN IP.. in cloudflare a have [A record *.example.com > Static IP]

I have double checked all the settings in this tutorial and after some googling i came across a reddit post, suggesting they fixed the 522 error in opnsense because HAProxy wasn't listening on port 80 during the HTTPtoHTTPS redirect. Is there a way I can diagnose this issue and trace the route somehow.

Lastly

ACME do not show any error in the log files. 

Code: [Select]

2022-04-13T18:53:42 php AcmeClient: running automation (configd): Restart HAProxy
2022-04-13T18:53:42 php AcmeClient: running automations for certificate: *.example.com
2022-04-13T18:53:42 opnsense AcmeClient: updated ACME X.509 certificate: *.example.com
2022-04-13T18:53:42 opnsense AcmeClient: successfully issued/renewed certificate: *.example.com
2022-04-13T18:51:27 opnsense AcmeClient: using challenge type: CloudFlare_DNS-01
2022-04-13T18:51:27 opnsense AcmeClient: account is registered: example.com
2022-04-13T18:51:27 opnsense AcmeClient: using CA: letsencrypt
2022-04-13T18:51:27 opnsense AcmeClient: issue certificate: *.example.com

HAProxy has no errors in the log file either

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62565b172acae6.05588153 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62565b172acae6.05588153

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62565eb5d0ff12.02152772.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62565c00b116b3.27816426.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: MineOS_backend ()
backend MineOS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server MineOS_server 192.168.1.103:8443 ssl verify none


Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: hopey on April 13, 2022, 03:34:46 pm

@TheHellSite: Thanks for your tutorial, it helped me to understand things a bit better!

I just had this problem: https://forum.opnsense.org/index.php?topic=27903.new#new. That's why I was reading your thread.

 What was strange to me, that you need to change the port of your opensense web interface! That's the point were I thought there might be a better solution 😬 I was reading and found this tutorial: https://schulnetzkonzept.de/opnsense. This guy just used a virtual IP, like you did. But instead of pointing to the lookup adress he just introduced a new adress and let haproxy listen on. And so did I - and it worked like charm! (detailes in the upper linked problem)

Maybe you wanna add it to your tutorial too?!

Have a nice day!

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: pottproll on April 13, 2022, 04:47:13 pm

Thanks for the tutorial! Its working great so far! I have one question: in my ACME Client log it says after renewal: php[2613]   AcmeClient: automation not supported: restart_haproxy
Is this just me or maybe because i didnt check HA-Proxy Integration?

Edit: never mind, worked perfectly the next renewal.

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 14, 2022, 04:22:13 am

I usually trace the session in Services: HAProxy: Statistics, Counters
When saving HAProxy setting, it will reset the session stat.
So, we can check the session stop at which server if we try to access immediately after a reset.
For example, I expect a session goes through
1. Frontend: TCP_front
2. Frontend: SSL_front
3. Backend: opn_back (usually don't have problem if you type correctly)
4. Server: opn

If all of them have session counts, but you still failed to access the website.
That means, it should be problem between browser and server, but you can access to the page when you type the server IP and port directly. Then, there might be issue about your ciphers, certs, OCSP settings, etc. You might find handshake error in server log too.

If it stop at 2. That means, your haproxy is not recognizing SNI correctly. Check your map file, or you can try to create condition with "SNI TLS extension matches (locally deciphered)" and your full SNI (the.domain.com), then create a rule to "Use specified Backend Pool" when condition matches.
Removing all rules and set default backend to test server first can also be a choice. (At least you will know rather it fails only in SNI part or more parts suffers)

If it stop at 1. There might be issue with the VIP again. (The bug similar to this (https://forum.opnsense.org/index.php?topic=27547.msg133659#msg133659))

If no session count. The listener is not working. TCP_frontend have wrong Listen Addresses. If you try it from WAN, check your firewall rules too, or maybe DNS record issue.

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 14, 2022, 05:03:24 am

Okay i can now access my webservices but in doing so a missed out the Virtual IP step.
After thinking about my issue some more I am listening HTTP & HTTPS traffic on 192.168.64.1 which I think is when the time out happens.

So I;
Service > HAProxy > Settings > Real Servers > SSL_server: changed FQDN or IP, from 192.168.64.1 to 192.168.1.1

Service > HAProxy > Settings > Virtual Services > 1_HTTPS_frontend: changed Listen Addresses, from 192.168.64.1:443 to 192.168.1.1:443

Service > HAProxy > Settings > Virtual Services > 1_HTTP_frontend: changed Listen Addresses, from 192.168.64.1:80 to 192.168.1.1:80

Now it is all working, What did I do wrong in setting up the Virtual IP I wonder.

0_SNI_frontend > Listen Addresses:0.0.0.0:80, 0.0.0.0:443
should this need to be the Virtual IP as opnsense runs on 192.168.1.1

^^fyi thankyou for the tips on tracing

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 14, 2022, 05:42:11 am

Quote

0_SNI_frontend > Listen Addresses:0.0.0.0:80, 0.0.0.0:443
should this need to be the Virtual IP as opnsense runs on 192.168.1.1

^^fyi thankyou for the tips on tracing

For this question, lets clear the package path that OP wants to do first.
Assume the following IP config:
WAN IP: 1.2.3.4
Firewall IP: 192.168.1.1
VIP: 192.168.64.1
Server IP:port and SNI: 192.168.1.2:80, the.website.com

A browser try to access https://the.website.com from internet.
It asked system to resolve from DNS server: the.website.com. DNS server replies it is 1.2.3.4
Browser try to access 1.2.3.4 with port 443, sending TLS package with SNI=the.website.com

Since haproxy SNI_frontend is listening to 0.0.0.0:443, that means it is listening to port 443 that all IP can represent the firewall. In this case, it is 1.2.3.4:443 and 192.168.1.1:443

SNI_frontend catches the TLS package in 1.2.3.4:443 and passes to SSL_backend(VIP) without changing port

Since SSL_frontend is listening to 192.168.64.1:443, it takes the TLS package and knows that it try to access 192.168.1.2:80. SSL_frontend communicate to the browser, exchanging the SSL cert and keys according to ciphers.

Session to webserver_backend start, SSL_frontend redirect remaining packages to webserver_backend.

I don't know why your VIP won't work, maybe missing opnsense update, misconfig of VIP or another new bug. (You can try to ping the VIP in LAN to check rather it reply. If the VIP is working normally, it should reply)
And SNI frontend wont use VIP unless you use hopey's method (He still need to add NAT rule to redirect packages to VIP as I mentioned in another thread)

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 14, 2022, 07:21:31 am

You guys are awesome, really appreciate you explaining the process. It is easy to follow along to a guide but to understand what is happening makes it that much easier down the line.

After pinging the VIP 192.168.64.1, it was timing out.

Checked the setting and all is correct to the tutorial.
Decided to changed the submask from 32 to 24.
Then I was able to ping the VIP and access my web services.
Reset the submask back to 32 and i am still able to ping the VIP and web services working with HTTP & HTTPS listening on the VIP. Very strange but it seems resolved.

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 14, 2022, 07:43:34 am

The issue is really similar to this one (https://forum.opnsense.org/index.php?topic=27547.msg133659)
I guess the issue should be solved in 22.1.4.
I run the patch in 22.1.3, so I don't know rather it is really fixed. But I don't have such problem anymore (currently 22.1.5)

If your are in 22.1.4 or 22.1.5, having similar issue. But general log don't have something like

Code: [Select]

/firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '192.168.64.1' -alias' failed to executePlease create another thread under 22.1 Production Series (https://forum.opnsense.org/index.php?board=41.0)

If you have exactly the same log and you are in 22.1.4 or 22.1.5. Please try to reply to the thread (https://forum.opnsense.org/index.php?topic=27547.msg133659)

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 14, 2022, 08:05:12 am

I will make my way over to the other thread you linked then as it is a similar issue.

I am on 22.1.5 and after checking the general log I also have the below

Code: [Select]

2022-04-14T16:42:58 Error opnsense /firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '192.168.64.1' -alias' failed to execute


--EDIT--
After reading the linked thread, a patch was applied in 22.1.4 but you also need to untick "Allow service binding".
I can confirm this has fixed the issue. Thanks again for point me in the right direction.

---EDIT 2---
I am now interested in the last part of the tutorial, I did not have anything like this setup on my previous pfsense setup so it will be a great addition. It would make for remembering local ip/ ports easier when navigating to locally hosted services.
I have followed the steps to make these subdomains accessible only from my local network but getting a "503 service unavailable".

I think its not reading the local map file correctly. I followed the previious steps for tracing and can see the counters go up when I try to access local.website.com but nothing is being passed to the prism_backend.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62565b172acae6.05588153 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62565b172acae6.05588153

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62565eb5d0ff12.02152772.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6257dfacde7e16.43417850 src_is_local

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6257d684d34507.32920094.txt)] if acl_6257dfacde7e16.43417850
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62565c00b116b3.27816426.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: MineOS_backend ()
backend MineOS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server MineOS_server 192.168.1.103:8443 ssl verify none

# Backend: Prism_backend ()
backend Prism_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Prism_server 192.168.1.103:2342

PUBLIC_SUBDOMAINS_map

Code: [Select]

# public access subdomains
mineos MineOS_backend
LOCAL_SUBDOMAINS_map

Code: [Select]

# local access subdomains
prism Prism_backend

# public access subdomains
mineos MineOS_backend

just to confirm;
mineos.website.com > works locally and externally
prism.website.com > 503 error locally and externally

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 14, 2022, 03:41:11 pm

Quote

mineos.website.com > works locally and externally
prism.website.com > 503 error locally and externally

Sorry, I haven't read the error 503.

Try not to use "src_is_local"
I remember that there is issue with this function, especially under proxy protocol. (It's hyproxy issue since 1.7?)
Try to use "Source IP matches a specific IP" instead

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 15, 2022, 12:22:50 am

Quote from: Bunch on April 14, 2022, 03:41:11 pm

Try to use "Source IP matches a specific IP" instead

I actually set this first time around which gave the same output. So I then decided to try a broader range with "IP is local".

I tried again "Source IP matches a specific IP" of the specific subnet again with no luck. have even tried isolating it to the IP of the client I am currently using.

the counter still is not being passed on the final stage of the route. After checking the HAProxy log file, as I navigate to local.website.com i am receiving a external request from cloudflare. I suspect the issue lies within the browser redirecting the request out or HAproxy not grabbing it locally before it goes out??

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on April 17, 2022, 01:03:33 pm

Strange, there shouldn't be something related to cloudflare
Maybe try to create condition: "SNI TLS extension matches (locally deciphered)" with your full sni, prism.website.com
Create another rule that
Use specified Backend Pool: Prism_backend
When
Source IP matches a specific IP...
And
SNI TLS extension matches (locally deciphered)

Use this rule to replace the one with local map file first

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bothson on April 19, 2022, 05:28:18 pm

Hello and thank you for this tutorial.
It helped me alot.

My services are available from the outside.
But from inside they are not accessible.

In the logs i see:

Code: [Select]

Informational haproxy 10.10.10.206:63264 [19/Apr/2022:17:26:27.483] 1_HTTPS_frontend/10.12.0.1:443: Received something which does not look like a PROXY protocol header
I already checked the ciphers. They seem to be ok.
Does anybody know where i can search for the problem?

My config:

Code: [Select]

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 10.12.0.1:80 name 10.12.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_60d1a0c1b278f7.63252237 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_60d1a0c1b278f7.63252237

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 10.12.0.1:443 name 10.12.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6256591773a972.14047672.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/625655d89e4274.43878203.txt)]

# Backend: bitwarden_backend ()
backend bitwarden_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server bitwarden_host 10.10.10.11:8080

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 10.12.0.1 send-proxy-v2 check-send-proxy
Best
Mathias

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 19, 2022, 06:26:15 pm

Quote from: Bothson on April 19, 2022, 05:28:18 pm

My services are available from the outside.
But from inside they are not accessible.

Well, there you got the point of error.
You probably configured the wrong IP in your DNS overwrites.

What is your OPNsense LAN IP, what is the DNS Overwrite IP and what is the virtual IP of the "HAProxy SSL Server"?

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bothson on April 19, 2022, 08:50:05 pm

Quote

Well, there you got the point of error.
You probably configured the wrong IP in your DNS overwrites.

What is your OPNsense LAN IP, what is the DNS Overwrite IP and what is the virtual IP of the "HAProxy SSL Server"?

My LAN IP is 10.10.10.1/24
My DNS Override points to    10.12.0.1
Which is also my Virtual IP (Loopback).

I do not get the point.

Edit: Ok, changed the Override IP to my LAN IP (10.10.10.1). Now it works.
But to be honest, i do not understand why.

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 19, 2022, 08:59:40 pm

Quote from: Bothson on April 19, 2022, 08:50:05 pm

Edit: Ok, changed the Override IP to my LAN IP (10.10.10.1). Now it works.
But to be honest, i do not understand why.

You error explains why!

Code: [Select]

Informational haproxy 10.10.10.206:63264 [19/Apr/2022:17:26:27.483] 1_HTTPS_frontend/10.12.0.1:443: Received something which does not look like a PROXY protocol header
The HTTPS_frontend expects that all data sent to it has the "proxy protocol header".
Since you pointed your internal requests directly to your HTTPS_frontend (HAProxy_VIP) instead of your SNI_frontend (any of the real local IPs of your OPNsense) the data didn't get the PROXY protocol header attached by the SSL_backend.

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bothson on April 20, 2022, 11:31:06 am

Quote from: TheHellSite on April 19, 2022, 08:59:40 pm

Quote from: Bothson on April 19, 2022, 08:50:05 pm

Edit: Ok, changed the Override IP to my LAN IP (10.10.10.1). Now it works.
But to be honest, i do not understand why.

You error explains why!

Code: [Select]

Informational haproxy 10.10.10.206:63264 [19/Apr/2022:17:26:27.483] 1_HTTPS_frontend/10.12.0.1:443: Received something which does not look like a PROXY protocol header
The HTTPS_frontend expects that all data sent to it has the "proxy protocol header".
Since you pointed your internal requests directly to your HTTPS_frontend (HAProxy_VIP) instead of your SNI_frontend (any of the real local IPs of your OPNsense) the data didn't get the PROXY protocol header attached by the SSL_backend.

Ah ok, i get the point.
Thank you very much for the explanation.

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 24, 2022, 05:45:11 am

I am not sure if this is the correct way to achieve multiple domains pointing to different backends but it seems to be working for me.
At first I ran into a issue were all domains could access the same subdomain, this is when I realized I just needed some extra conditions.

Here are the steps to achieve; service.example.com & service1.example1.com

Services --> ACME Client --> Certificates
Add the certificate for your extra domains and forcefully issue your certificate

Services --> HAProxy --> Settings --> Advanced --> Map Files
Here we will create a new map file for each domain "PUBLIC_SUBDOMAINS_map-example" & "PUBLIC_SUBDOMAINS_map-example1"

Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
Add a new condition for each domain that you have added.
Name = "example1_condition"
Description = "Traffic matches example1.com"
Condition type = "host contains"
Host Contains = "example1.com"

Services --> HAProxy --> Settings --> Rules & Checks --> Rules
Add a map rule for each domain while also selecting our newly created "example1_condition"
Name = "PUBLIC_SUBDOMAINS_map-rule-example1"
Select conditions = "example1_condition"
Map file = "PUBLIC_SUBDOMAINS_map-example1"

Services --> HAProxy --> Settings --> Virtual Services --> Public Services
Finally we edit our "1_HTTPS_frontend"
Add all extra domains in the "Certificates" input.
Scroll down and add each map-rule-example1 in the "Select Rules" input

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 24, 2022, 11:51:04 am

Quote from: keyboardDabbler on April 24, 2022, 05:45:11 am

I am not sure if this is the correct way to achieve multiple domains pointing to different backends but it seems to be working for me.
At first I ran into a issue were all domains could access the same subdomain, this is when I realized I just needed some extra conditions.

Here are the steps to achieve; service.example.com & service1.example1.com

Services --> ACME Client --> Certificates
Add the certificate for your extra domains and forcefully issue your certificate

Services --> HAProxy --> Settings --> Virtual Services --> Public Services
Finally we edit our "1_HTTPS_frontend"
Add all extra domains in the "Certificates" input.

Just the steps above are necessary and the following step.

Then edit the "PUBLIC_SUBDOMAINS_map" and change the subdomain entries to full FQDNs.

Code: [Select]

old
===
nas NAS_backend

new
===
nas.domain1.com NAS_1_backend
nas.domain2.com NAS_2_backend
Of course you could also create dedicated map-files and rules for each domain, but in terms of functionality it is not necessary.

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on April 25, 2022, 04:45:20 am

Although it has the same outcome, The steps you provided using only 1 map file is a lot cleaner and easier to follow.

Thanks again for pointing this out.

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: christian_domes on April 29, 2022, 09:19:57 am

@theHellSite
Hello
When I overide the dns Server will the haproxy still be used or not?

Title: Re: Tutorial 2022/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on May 11, 2022, 05:11:28 am

Quote from: christian_domes on April 29, 2022, 09:19:57 am

@theHellSite
Hello
When I overide the dns Server will the haproxy still be used or not?

It depends on how you override the dns record.
Assume that you set your SNI frontend with 0.0.0.0:80 and 0.0.0.0:443

For example, you added a DNS record in Cloudflare "abc.domain.com" pointing to your WAN IP, and your tested it and found HAProxy working both locally and externally.
Then you removed the DNS record from Cloudflare, and add one in unbounded "abc.domain.com" pointing to your OpnSense IP (either LAN or WAN, doesn't metter)

Then your HAProxy should work locally, but failed externally.
*If you want to override DNS record in unbound, always point to SNI frontend. If you set it to either HTTP frontend or HTTPS frontend, it will fail.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 04, 2022, 03:42:34 pm

I just updated the tutorial with a very important change to the DynDNS part. It therefore only affects users with a dynamic WAN IP.

• 20220604
  
  • Updated the DynDNS part to use the newer "Dynamic DNS Client" (os-ddclient) plugin.
    The previous "Dynamic DNS Support" (os-dyndns) plugin will be removed with the release of OPNsense 22.7.x.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: schnerring on June 10, 2022, 12:57:32 am

I've been following this excellent guide to a tee, but I get the following warning:

Quote

[WARNING] (51339) : Proxy '1_HTTP_frontend': L6 sample fetches ignored on HTTP proxies (declared at /usr/local/etc/haproxy.conf.staging:69).
Warnings were found.
Configuration file is valid

This is the affected part of the HAProxy config (the last line is 69):

Code: [Select]

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_629f48c6073c95.86527303 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_629f48c6073c95.86527303

For reference, other people also encountered this issue:

• Reply #191 and after in this thread (https://forum.opnsense.org/index.php?topic=23339.msg133309#msg133309)
• HAProxy warnung nach update auf 22.1 - L6 sample fetches ignored on HTTP proxies (https://forum.opnsense.org/index.php?topic=27065)
• https://redmine.pfsense.org/issues/9261 (https://redmine.pfsense.org/issues/9261)

The issue seems to be the NoSSL_condition. The suggested solution is to replace the condition type from Traffic is SSL (TCP request content inspection)  to Traffic is SSL (locally deciphered):

Code: [Select]

     # logging options
     # ACL: NoSSL_condition
-    acl acl_629f48c6073c95.86527303 req.ssl_ver gt 0
+    acl acl_629f48c6073c95.86527303 ssl_fc
 
     # ACTION: HTTPtoHTTPS_rule
     http-request redirect scheme https code 301 if !acl_629f48c6073c95.86527303

When doing so, the warning is gone. However, this is the first time I'm using HAProxy and I don't really know what I'm doing, so I wanted to check-in with you guys to ensure my solution is correct?

If so, I'd appreciate if you updated the screenshot in step 5.7 @TheHellSite

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 10, 2022, 02:14:12 pm

Quote from: schnerring on June 10, 2022, 12:57:32 am

The issue seems to be the NoSSL_condition. The suggested solution is to replace the condition type from Traffic is SSL (TCP request content inspection)  to Traffic is SSL (locally deciphered):

Code: [Select]

     # logging options
     # ACL: NoSSL_condition
-    acl acl_629f48c6073c95.86527303 req.ssl_ver gt 0
+    acl acl_629f48c6073c95.86527303 ssl_fc
 
     # ACTION: HTTPtoHTTPS_rule
     http-request redirect scheme https code 301 if !acl_629f48c6073c95.86527303

When doing so, the warning is gone. However, this is the first time I'm using HAProxy and I don't really know what I'm doing, so I wanted to check-in with you guys to ensure my solution is correct?

If so, I'd appreciate if you updated the screenshot in step 5.7 @TheHellSite

Thanks for the detailed explanation!
I already had this on my to do list, but then couldn't find the time for it and in the end simply forgot about it.

The HAProxy docs are also stating to use the "ssl_fc" condition.
https://www.haproxy.com/blog/redirect-http-to-https-with-haproxy/

I will change it in the next couple of days!

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 11, 2022, 03:20:57 pm

• 20220611
  
  • Changed the IP address of the "SSL_server" to one that belongs to the localhost subnet.
  • Updated the "NoSSL_condition" based on the HAProxy docs.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: PaulePils on June 14, 2022, 11:41:40 am

Hello,

I just recently upgraded my home network with an opnsense and want to reconfigure some stuff in the same run.
Luckily I found this tutorial which was really easy to follow through, especally because of the screenshots :-)
Sadly I must have done something wrong but I can't put my finger on it.
When I try to access from internal LAN (IP address or Name), I get the following error:

• Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
• Edge:ERR_SSL_PROTOCOL_ERROR

From external (via mobile data):
503 Service Unavailable

If I enter http://IP_ADRESS then it opens the Nextcloud Login page but after entering the login data, it switches to https:// and the error appears.

HAProxy config

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62a76f360f0732.68695084 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62a76f360f0732.68695084

# Frontend: 2_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 2_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets no-tlsv12 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62a772caaae308.49400660.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62a76fbf29df39.71162057.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Nextcloud_backend ()
backend Nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Nextcloud 192.168.10.40:443 ssl verify none


Map files

Code: [Select]

#public access subdomains
nextcloud NEXTCLOUD_backend
Log files

Code: [Select]

Informational haproxy 192.168.10.101:4054 [14/Jun/2022:11:11:05.082] 2_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
My Network:
ISP --> Modem --> OPNSense --> Proxmox --> Server 1 Nextcloud

Maybe you can push me in the right direction :-)

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on June 15, 2022, 05:01:20 am

Quote from: PaulePils on June 14, 2022, 11:41:40 am

Hello,

I just recently upgraded my home network with an opnsense and want to reconfigure some stuff in the same run.
Luckily I found this tutorial which was really easy to follow through, especally because of the screenshots :-)
Sadly I must have done something wrong but I can't put my finger on it.
When I try to access from internal LAN (IP address or Name), I get the following error:

• Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
• Edge:ERR_SSL_PROTOCOL_ERROR

From external (via mobile data):
503 Service Unavailable

If I enter http://IP_ADRESS then it opens the Nextcloud Login page but after entering the login data, it switches to https:// and the error appears.

HAProxy config

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62a76f360f0732.68695084 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62a76f360f0732.68695084

# Frontend: 2_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 2_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets no-tlsv12 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62a772caaae308.49400660.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62a76fbf29df39.71162057.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Nextcloud_backend ()
backend Nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Nextcloud 192.168.10.40:443 ssl verify none


Map files

Code: [Select]

#public access subdomains
nextcloud NEXTCLOUD_backend
Log files

Code: [Select]

Informational haproxy 192.168.10.101:4054 [14/Jun/2022:11:11:05.082] 2_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
My Network:
ISP --> Modem --> OPNSense --> Proxmox --> Server 1 Nextcloud

Maybe you can push me in the right direction :-)

Since you are forcing HAProxy to use TLS 1.3. (As you use no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets no-tlsv12)
There might be issue taking out TLS_AES_128_GCM_SHA256.

According to RFC 8446, S9.1 (https://www.rfc-editor.org/rfc/rfc8446#section-9.1)

Quote

A TLS-compliant application MUST implement the TLS_AES_128_GCM_SHA256 [GCM] cipher suite and SHOULD implement the TLS_AES_256_GCM_SHA384 [GCM] and TLS_CHACHA20_POLY1305_SHA256 [RFC8439] cipher suites (see Appendix B.4).

Although it shouldn't have any effect by implementing a Cipher List, but Cipher List should be removed under TLS 1.3.



From external (via mobile data):
503 Service Unavailable

one of the reasons can be: it cannot read the map file, so it doesn't redirect to target backend.
Some device is hard to load map files during boot time. (One of my setup under Esxi is having such issue)
Apply the HAProxy one more time after all services finished loading can make it read the map file again. Or you can implement condition and rule to ignore such issue
example:
Condition name: nextcloud_cond, condition type: host contains, Host Contains: nextcloud
Rule name: nextcloud_ru, Test type: IF, Select conditions: nextcloud_cond, Execute function: Use specified backend Pool, Use backend pool: NEXTCLOUD_backend

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: PaulePils on June 15, 2022, 11:26:53 am

Thanks for your input. I changed the cipher accordingly to https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6 (https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6) and also applied your rule but still no success.
I am just curios because in the main post the value TLS_AES_128_GCM_SHA256 is also left out  ???

Out of curiosity I tried do enter the only the main domain DOMAIN.dedyn.io and not SUBDOMAIN.DOMAIN.dedyn.io. Then I get a different error

Quote

Firefox vertraut dieser Website nicht, weil das von der Website verwendete Zertifikat nicht für DOMAIN.dedyn.io gilt. Das Zertifikat ist nur gültig für *.DOMAIN.dedyn.io.
SSL_ERROR_BAD_CERT_DOMAIN

I can accept the risk but this is something I need to do on all devices...

Could the problem be that I already had DynDNS by another provider and LE certificate? If yes how can I revoke it?

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 15, 2022, 03:15:28 pm

Quote from: PaulePils on June 14, 2022, 11:41:40 am

When I try to access from internal LAN (IP address or Name), I get the following error:

• Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
• Edge:ERR_SSL_PROTOCOL_ERROR

This error usually means that you tried to access a service using HTTPS that only supports HTTP.

Quote

internal LAN (IP address or Name)

What do you mean by that? You should use the FQDN and not the local hostname / IP in order to use the reverse proxy (HAProxy).

Quote from: PaulePils on June 14, 2022, 11:41:40 am

If I enter http://IP_ADRESS then it opens the Nextcloud Login page but after entering the login data, it switches to https:// and the error appears.

Again, which IP are you entering? The nextcloud local IP or your public IP?
Your public IP should ALWAYS forward HTTP to HTTPS and not display any webpages via HTTP whatsoever! (HTTPtoHTTPS_rule)

Quote from: PaulePils on June 14, 2022, 11:41:40 am

My Network:
ISP --> Modem --> OPNSense --> Proxmox --> Server 1 Nextcloud

Is the firewall in your modem disabled or is it still doing NAT?

Quote from: PaulePils on June 15, 2022, 11:26:53 am

Thanks for your input. I changed the cipher accordingly to https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6 (https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6) and also applied your rule but still no success.
I am just curios because in the main post the value TLS_AES_128_GCM_SHA256 is also left out  ???

You won't need any 128 bit ciphers unless you are using very very old devices to access your services which I highly doubt since you only want to enable TLS v1.3 anyway.
So you can safely use the cipher suites in my first post which are identical to the ones from the Mozilla SSL configurator but have the "insecure / weak" 128 bit ciphers removed.

Quote from: PaulePils on June 15, 2022, 11:26:53 am

Out of curiosity I tried do enter the only the main domain DOMAIN.dedyn.io and not SUBDOMAIN.DOMAIN.dedyn.io. Then I get a different error

Quote

Firefox vertraut dieser Website nicht, weil das von der Website verwendete Zertifikat nicht für DOMAIN.dedyn.io gilt. Das Zertifikat ist nur gültig für *.DOMAIN.dedyn.io.
SSL_ERROR_BAD_CERT_DOMAIN

I can accept the risk but this is something I need to do on all devices...

Could the problem be that I already had DynDNS by another provider and LE certificate? If yes how can I revoke it?

Well, the error is pretty much self-explanatory isn't it?
In my tutorial the wildcard certificate is only valid for the 2nd-level subdomains "*.DOMAIN.dedyn.io" but not for the 1st-level subdomain "DOMAIN.dedyn.io" itself.

If you want the certificate to also cover for "domain.dedyn.io" then you will have to change the certificate in the ACME client to match that. See Part 3 - Step 6.
You will however only need this if you are serving a website in the domain root without "www" infront of it.

Code: [Select]

# currently configured
Common Name = *.DOMAIN.dedyn.io

# what you MIGHT want
Common Name = DOMAIN.dedyn.io
Alt Names = *.DOMAIN.dedyn.io

This will cover the 1st-level subdomain including all 2nd-level subdomains.
Don't forget to reissue the certificate.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 15, 2022, 03:25:18 pm

Quote from: Bunch on June 15, 2022, 05:01:20 am

Since you are forcing HAProxy to use TLS 1.3. (As you use no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets no-tlsv12)
There might be issue taking out TLS_AES_128_GCM_SHA256.

According to RFC 8446, S9.1 (https://www.rfc-editor.org/rfc/rfc8446#section-9.1)

Quote

A TLS-compliant application MUST implement the TLS_AES_128_GCM_SHA256 [GCM] cipher suite and SHOULD implement the TLS_AES_256_GCM_SHA384 [GCM] and TLS_CHACHA20_POLY1305_SHA256 [RFC8439] cipher suites (see Appendix B.4).

Well, in order to get an 100% A+ rating at SSL Labs it is mandatory to remove any ciphers below or equal to 128 bit. This is why I haven't included them in my cipher list and cipher suites.
So unless you have really old devices with outdated browsers I would suggest to only use ciphers with a strength of 256 bit or above.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: PaulePils on June 15, 2022, 05:28:26 pm

Quote from: TheHellSite on June 15, 2022, 03:15:28 pm

Quote from: PaulePils on June 14, 2022, 11:41:40 am

When I try to access from internal LAN (IP address or Name), I get the following error:

• Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
• Edge:ERR_SSL_PROTOCOL_ERROR

This error usually means that you tried to access a service using HTTPS that only supports HTTP.

Does this mean I need an additional vhost config for the Nextcloud? One for :80 and one for :443? But the certificate lies on a different machine (opnsense). How can I point it there? Or am I missing something?
By deleting the "overwirte" codes in the nextcloud config I at least got access in my internal LAN. I assume the HAProxy config is correct and I need to make corrections in the nextcloud config.

Quote

Quote

internal LAN (IP address or Name)

What do you mean by that? You should use the FQDN and not the local hostname / IP in order to use the reverse proxy (HAProxy).

It was the IP address of the Nextcloud machine. But if it doesn't work that way that is new for me but thanks for pointing it out.

Quote

Quote from: PaulePils on June 14, 2022, 11:41:40 am

If I enter http://IP_ADRESS then it opens the Nextcloud Login page but after entering the login data, it switches to https:// and the error appears.

Again, which IP are you entering? The nextcloud local IP or your public IP?
Your public IP should ALWAYS forward HTTP to HTTPS and not display any webpages via HTTP whatsoever! (HTTPtoHTTPS_rule)

I entered the local IP address of the nextcloud machine. But this also happens when I enter the FQDN.

Quote

Quote from: PaulePils on June 14, 2022, 11:41:40 am

My Network:
ISP --> Modem --> OPNSense --> Proxmox --> Server 1 Nextcloud

Is the firewall in your modem disabled or is it still doing NAT?

It is dumb modem (TC-4400) therefor it should not have NAT. It only provides the connection to my ISP (Vodafone Cable).

Quote

Quote from: PaulePils on June 15, 2022, 11:26:53 am

Thanks for your input. I changed the cipher accordingly to https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6 (https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=modern&openssl=1.1.1d&guideline=5.6) and also applied your rule but still no success.
I am just curios because in the main post the value TLS_AES_128_GCM_SHA256 is also left out  ???

Quote

You won't need any 128 bit ciphers unless you are using very very old devices to access your services which I highly doubt since you only want to enable TLS v1.3 anyway.
So you can safely use the cipher suites in my first post which are identical to the ones from the Mozilla SSL configurator but have the "insecure / weak" 128 bit ciphers removed.

OK

Quote

Quote from: PaulePils on June 15, 2022, 11:26:53 am

Out of curiosity I tried do enter the only the main domain DOMAIN.dedyn.io and not SUBDOMAIN.DOMAIN.dedyn.io. Then I get a different error

Quote

Firefox vertraut dieser Website nicht, weil das von der Website verwendete Zertifikat nicht für DOMAIN.dedyn.io gilt. Das Zertifikat ist nur gültig für *.DOMAIN.dedyn.io.
SSL_ERROR_BAD_CERT_DOMAIN

I can accept the risk but this is something I need to do on all devices...

Could the problem be that I already had DynDNS by another provider and LE certificate? If yes how can I revoke it?

Well, the error is pretty much self-explanatory isn't it?
In my tutorial the wildcard certificate is only valid for the 2nd-level subdomains "*.DOMAIN.dedyn.io" but not for the 1st-level subdomain "DOMAIN.dedyn.io" itself.

If you want the certificate to also cover for "domain.dedyn.io" then you will have to change the certificate in the ACME client to match that. See Part 3 - Step 6.
You will however only need this if you are serving a website in the domain root without "www" infront of it.

Code: [Select]

# currently configured
Common Name = *.DOMAIN.dedyn.io

# what you MIGHT want
Common Name = DOMAIN.dedyn.io
Alt Names = *.DOMAIN.dedyn.io

This will cover the 1st-level subdomain including all 2nd-level subdomains.
Don't forget to reissue the certificate.

I don't need the coverage of "domain.dedyn.io". So this should be fine.[/quote]

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 15, 2022, 10:23:09 pm

Quote from: PaulePils on June 15, 2022, 05:28:26 pm

Quote from: TheHellSite on June 15, 2022, 03:15:28 pm

Quote from: PaulePils on June 14, 2022, 11:41:40 am

When I try to access from internal LAN (IP address or Name), I get the following error:

• Firefox:SSL_ERROR_RX_RECORD_TOO_LONG
• Edge:ERR_SSL_PROTOCOL_ERROR

This error usually means that you tried to access a service using HTTPS that only supports HTTP.

Does this mean I need an additional vhost config for the Nextcloud? One for :80 and one for :443? But the certificate lies on a different machine (opnsense). How can I point it there? Or am I missing something?
By deleting the "overwirte" codes in the nextcloud config I at least got access in my internal LAN. I assume the HAProxy config is correct and I need to make corrections in the nextcloud config.

Code: [Select]

    server Nextcloud 192.168.10.40:443 ssl verify noneThis line in your HAProxy config indicates that you are accessing your nextcloud webinterface from your local network using "https://192.168.10.40".
If the above is not true and you are accessing it using "http://192.168.10.40" instead then you should have figured your error after reading this sentence.

IF you can already access nextcloud from within your local using its Local_IP:Port then you won't have to change any settings at nextcloud.

Also... Please do yourself a favour and read about what a reverse proxy is, what it does and how it works.
No offense but I think you don't really know what you are doing here.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brynjolm on June 17, 2022, 09:07:30 pm

Hi! First off, i want to thank you for the detailed guide you posted. I am new to HAProxy and have some questions regarding some configs, do you still reply to this post? or should i ask or open another thread somewhere? Thanks!

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 17, 2022, 10:03:40 pm

Quote from: brynjolm on June 17, 2022, 09:07:30 pm

Hi! First off, i want to thank you for the detailed guide you posted. I am new to HAProxy and have some questions regarding some configs, do you still reply to this post? or should i ask or open another thread somewhere? Thanks!

As long as the question is related to THIS tutorial then feel free to ask!

Otherwise please ask here: https://forum.opnsense.org/index.php?board=28.0

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brynjolm on June 17, 2022, 11:39:15 pm

Thanks! So, i set up HAP as per the guide, and am wondering if its a normal consequence to have 503 error on my truenas webui. Internally i can acces it no problems, outside of the home network i am unable to. Sidenote, i do not plan to expose the web ui or planned to. I was just using it as a sort of test run to see if it will resolve to my domain.truenas.com. I also did expose plex under port 32400. But seem to encounter some TLS handshake errors. It would be nice if you could take a look at my config and point me in the right direction. I did only a A and AAAA record for wildcard hostname i.e *.mydomain.com. I dont know if thats enough.
My general goal is to only expose plex and some other services like nextcloud, but thats for another section of the forums. Also my network is kinda flat. All services run through a single VLAN.
I created two Servers and two backends. One for Truenas and plex respectively.
i have been trying to make a stable connection to plex but it just seems to either time out or lose connection after a while.

Heres and output of my config file, i have some disabled stuff in the config file since i turn on and off stuff for testing. Also renamed some stuff, truenas is just the Physical server with the IP. (myloopbackip) is the virutal address.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

#
# NOTE: HAProxy is currently DISABLED
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr libc,last
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend (Listening on ip:80 / ip:443)
frontend SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind :::80 name :::80
    bind :::443 name :::443
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTP_frontend (Listening on 127.10.20.5)
frontend HTTP_frontend
    bind (myloopbackip):80 name (myloopbackip):80 accept-proxy
    bind :::80 name :::80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_62aa8dcf894a87.42381056 ssl_fc

    # ACTION: HTTPtoHTTPS
    http-request redirect scheme https code 301 if !acl_62aa8dcf894a87.42381056

# Frontend: HTTPS_frontend (Listening on (myloopbackip))
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind (myloopbackip):443 name (myloopbackip):443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62aad04a028639.71957640.certlist
    bind :::443 name :::443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62aad04a028639.71957640.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: Public_subdomains_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62aa8e31993357.88056717.txt)]
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server (myloopbackip) send-proxy-v2 check-send-proxy

# Backend (DISABLED): router_Backend (router Backend)

# Backend: truenas_Backend (truenas Backend)
backend truenas_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server truenas truenasip:443 ssl sni str(truenas) verify none send-proxy-v2 check-send-proxy

# Backend: plex_backend (plex Backend)
backend plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Plex truenasip:32400 send-proxy-v2 check-send-proxy


This is my map file:

Code: [Select]

# public access domains
truenas truenas_backend
plex plex_backend


Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: droren on June 18, 2022, 06:17:16 am

I've followed the article and able to setup few subdomain to internal machines in my network.

I have dumb questions... when I created new real server and backend server for one my Synology packages, I initially used HTTPS port and received 400 bad request error The plain HTTP request was sent to HTTPS port

When I changed the port in real server settings to HTTP request, everything worked fine and I am able to access the internal server with SSL certificate.

Why?

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 21, 2022, 10:49:24 am

Quote from: droren on June 18, 2022, 06:17:16 am

I've followed the article and able to setup few subdomain to internal machines in my network.

I have dumb questions... when I created new real server and backend server for one my Synology packages, I initially used HTTPS port and received 400 bad request error The plain HTTP request was sent to HTTPS port

When I changed the port in real server settings to HTTP request, everything worked fine and I am able to access the internal server with SSL certificate.

Why?

FAQ - Do I need to enable "SSL" in the Real Server configuration of a service?

Thanks for NOT reading my first post properly.  :(
I explained everything there very well.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 21, 2022, 10:58:08 am

Quote from: brynjolm on June 17, 2022, 11:39:15 pm

Heres and output of my config file, i have some disabled stuff in the config file since i turn on and off stuff for testing. Also renamed some stuff, truenas is just the Physical server with the IP. (myloopbackip) is the virutal address.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

#
# NOTE: HAProxy is currently DISABLED
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr libc,last
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend (Listening on ip:80 / ip:443)
frontend SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind :::80 name :::80
    bind :::443 name :::443
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTP_frontend (Listening on 127.10.20.5)
frontend HTTP_frontend
    bind (myloopbackip):80 name (myloopbackip):80 accept-proxy
    bind :::80 name :::80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_62aa8dcf894a87.42381056 ssl_fc

    # ACTION: HTTPtoHTTPS
    http-request redirect scheme https code 301 if !acl_62aa8dcf894a87.42381056

# Frontend: HTTPS_frontend (Listening on (myloopbackip))
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind (myloopbackip):443 name (myloopbackip):443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62aad04a028639.71957640.certlist
    bind :::443 name :::443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62aad04a028639.71957640.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: Public_subdomains_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62aa8e31993357.88056717.txt)]
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server (myloopbackip) send-proxy-v2 check-send-proxy

# Backend (DISABLED): router_Backend (router Backend)

# Backend: truenas_Backend (truenas Backend)
backend truenas_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server truenas truenasip:443 ssl sni str(truenas) verify none send-proxy-v2 check-send-proxy

# Backend: plex_backend (plex Backend)
backend plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Plex truenasip:32400 send-proxy-v2 check-send-proxy


This is my map file:

Code: [Select]

# public access domains
truenas truenas_backend
plex plex_backend


1. The map file is case sensitive. Fix it.

2. Remove the "send-proxy-v2 check-send-proxy" directives from the backends of your actual services. These two options are only necessary on the "SSL_backend".

3. Why do you have the sni setting configured in your truenas real server? that shouldn't be necessary at all. Also remove the "send-proxy-v2 check-send-proxy" directives from the backend (see 1.).

Code: [Select]

server truenas truenasip:443 ssl sni str(truenas) verify none send-proxy-v2 check-send-proxy
4. Your "HTTP_frontend" and "HTTPS_frontend" should ONLY be listening to your SSL_server IP address. Not to "0.0.0.0:0" or "::::0". Just think about it and take a look at my diagram in the first post... You should quickly figure that this doesn't make any sense.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brynjolm on June 25, 2022, 12:16:30 am

Quote

1. The map file is case sensitive. Fix it.

2. Remove the "send-proxy-v2 check-send-proxy" directives from the backends of your actual services. These two options are only necessary on the "SSL_backend".
4. Your "HTTP_frontend" and "HTTPS_frontend" should ONLY be listening to your SSL_server IP address. Not to "0.0.0.0:0" or "::::0". Just think about it and take a look at my diagram in the first post... You should quickly figure that this doesn't make any sense.

Thanks! That solved a lot of the problems i had.

I also had to edit some settings on my cloudflare setup in order for it to work. I now have full access to everything aside from nextcloud and some other stuff. while still being proxied behind cloudflare. I dont know if it will be useful for other cloudflare users. But for me at least since i wanted to go cloudflare for everything. I ended up also using the cloudflare certs and uploading it to HAP while still being onf Full(Strict) mode. i dont know if its and intended sideffect but i didnt have to use split dns for this. I could just call up everything by subdomain.domain.com locally.
Also with regards to the map file for the local subdomanins. The format should look like this?

Quote

local loca_backend

plex plex_backend

Should there be a space in between?
Last questions. Is it possible to use the generated Cert for both truenas and opnsense instead of the self signed one? would it break the config? Also is there a specifi HAP sub, or should i just post in the General section?

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 27, 2022, 04:08:58 pm

Quote from: brynjolm on June 25, 2022, 12:16:30 am

I could just call up everything by subdomain.domain.com locally.

This is because your domains resolve to cloudflares IP and not your own public IP.

Quote from: brynjolm on June 25, 2022, 12:16:30 am

Also with regards to the map file for the local subdomanins. The format should look like this?

Quote

local loca_backend

plex plex_backend

Should there be a space in between?

https://www.haproxy.com/documentation/hapee/latest/configuration/map-files/syntax/

Quote from: brynjolm on June 25, 2022, 12:16:30 am

Last questions. Is it possible to use the generated Cert for both truenas and opnsense instead of the self signed one? would it break the config? Also is there a specifi HAP sub, or should i just post in the General section?

Shouldn't break anything but isn't neccessary.
https://discourse.haproxy.org/
or
https://forum.opnsense.org/index.php?board=28.0

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: xxavarlonxx on June 29, 2022, 01:04:47 pm

Thnak you very much. Thats the best guide about HAProxy - Lets Encrypt I've ever found. Everything works and its not a mickey mouse setup, its a good setup for production..

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on July 04, 2022, 10:25:37 pm

I am fighting with this setup - what is most troubling is that it worked a few days ago, but for the life of me I cannot find what went wrong.

I now have a stripped-down version of the setup where I just want to use a map-based setup with SSL offloading for some internal servers (now my setup contains only one).

This is my setup:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend (Listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_Frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_Frontend (Listening on 127.0.0.1:80)
frontend 1_HTTP_Frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: No_SSL_Condition
    acl acl_629b7d353dc6e8.95969175 ssl_fc

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_629b7d353dc6e8.95969175

# Frontend: 1_HTTPS_Frontend (Listening on 127.0.0.1:443)
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/629b82033c9ac6.13569566.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_Subdomains_Map_Rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/629b7dc0816c90.87321785.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend (DISABLED): PLEX_backend (PLEX Backend)

# Backend: BLOB_backend (BLOB Webserver Backend)
backend BLOB_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server BLOB_server 192.168.10.3 ssl sni str(ssl.xyz.de) verify none resolve-prefer ipv4

The corresponding map file would be:

Code: [Select]

ssl.xyz.de BLOB_backend

Symptoms are: I can see the certificate containing ssl.xyz.de (and others) when I connect to my OpnSense via 'openssl s_client'.

If I try 'wget -O- https://ssl.xyz.de', the connection hangs after initial TLS handshake - when I use one of the other domains like 'abc.xyz.de' which resolve to the same IP, I get the expected 503. I can even see the HTTP->HTTPS redirection when I use http instead of https.

So, I conclude that the certificate is O.K., I get through to 1_HTTP_Frontend (otherwise there would be no 503 when I use an unmapped URL). I seems like most of the time, the frontend does not connect to the backend (server) - it does sometimes, however.

I am at a loss what causes this because I also verified that OpnSense can reach BLOB server at 192.168.10.3. It also has a Let's Encrypt certificate on ssl.xyz.de, because before, I just port-forwarded port 443. I can get data via 'curl --connect-to 192.168.10.3:443 https://ssl.xyz.de' from it.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 04, 2022, 11:02:05 pm

Quote from: meyergru on July 04, 2022, 10:25:37 pm

So, I conclude that the certificate is O.K., I get through to 1_HTTP_Frontend (otherwise there would be no 503 when I use an unmapped URL). I seems like most of the time, the frontend does not connect to the backend (server) - it does sometimes, however.

I am at a loss what causes this because I also verified that OpnSense can reach BLOB server at 192.168.10.3. It also has a Let's Encrypt certificate on ssl.xyz.de, because before, I just port-forwarded port 443. I can get data via 'curl --connect-to 192.168.10.3:443 https://ssl.xyz.de' from it.

Code: [Select]

server BLOB_server 192.168.10.3 ssl sni str(ssl.xyz.de) verify none resolve-prefer ipv4
1. You have to include the server port in the config of the server, obviously...
2. Remove the SNI string in the server config, not neccessary at all. I also DID NOT use it in my tutorial, so why are you using it?
3. Remove the resolve-prefer directive. Again, I did not use it in my tutorial...

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on July 05, 2022, 12:33:32 am

1. I thought that the server port would be 443 per default, but I now set it and it did not change anything.
2. I need the SNI string because the server reacts to different names but I cannot use the "external" name since I want to contact the LAN IP. However, I set an alternative name so that I can leave out the SNI for testing and - no dice.
3. Disabling resolve-prefer did nothing as well.

4. I even tried without SSL, same result. So it must be something before that.

New config:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend (Listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_Frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_Frontend (Listening on 127.0.0.1:80)
frontend 1_HTTP_Frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: No_SSL_Condition
    acl acl_629b7d353dc6e8.95969175 ssl_fc

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_629b7d353dc6e8.95969175

# Frontend: 1_HTTPS_Frontend (Listening on 127.0.0.1:443)
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/629b82033c9ac6.13569566.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_Subdomains_Map_Rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/629b7dc0816c90.87321785.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend (DISABLED): PLEX_backend (PLEX Backend)

# Backend: BLOB_backend (BLOB Webserver Backend)
backend BLOB_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server BLOB_server blob.xyz:80



Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on July 05, 2022, 03:02:16 am

Why you use 127.0.0.1 as SSL_backend IP?
Please setup a VIP

For a server, 0.0.0.0 is ANY IP that can represent the server itself, including localhost
Thus, 0_SNI_frontend conflict with SSL_backend obviously.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on July 05, 2022, 08:16:57 am

I already tried using 127.4.4.3 for SSL_server, HTTP and HTTPS frontends as well with no change in behaviour. The tutorial explicitely states that this is optional (see FAQ #4) and besides, if you are right, there is no difference, because 0.0.0.0/0 would also overlap 127.4.4.3/32.

Thanks for any input, though. I tried several hours and as I said, this already worked...

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 05, 2022, 08:56:40 am

Your haproxy is configured correctly. VIP is indeed optional.

Your real server IP:Port (SSL enabled yes or no / SSL verify yes or no) is the only issue left that could cause it from the haproxy side. Otherwise your web server is misconfigured.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on July 05, 2022, 09:29:53 am

As I said, I triple-checked everything and I also cannot see what goes wrong (even sometimes, it works).

I also tried a non-SSL backend, including another server to eliminate problems on the backend.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 05, 2022, 03:24:45 pm

Here is my reference config.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends


# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60a6828680bca8.63910725.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/60bdf8931a97c9.33132019.txt)]

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_607ae66cdeaed1.61504267 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_607ae66cdeaed1.61504267

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 1h
    http-reuse safe
    server PLEX_server 192.168.215.60:32400 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy


If you compare it with yours. You will see that your real server is the problem!

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on July 05, 2022, 08:23:36 pm

Wow. This was a tough one...

I also believed that my real server is the problem - or to put it another way: ALL my real servers seemed to be the problem.

I replaced the real server by 127.0.0.1 (i.e. OpnSense itself) and guess what? It works.
Then with the LAN IP of my OpnSense - works. I replaced it with www.google.com - works!!!

None of my LAN servers work, however. First, I re-did all of the HAProxy configuration from scratch - with the same result.
I found out that curl has problems as well (most of the time) - so in effect, OpnSense cannot make (reliably) any outgoing TCP connections (while pings work!).

So I asked myself: Is that a firewall or NAT problem? I disabled all firewall rules, but no dice.
After that, I did a tcpdump on the receiving side, only to find out that packets were sent from the OpnSense, and answers also went out but seem to get eaten, although there were no firewall log entries.

Before that, I already had tried to disable all hardware offloading, but only in the global settings.
Looking into that again, I noticed a hint that you have to reconfigure the interfaces in order to take over the defaults, so I disabled all offloading on the LAN interface.

After that - hey, presto! Everything works!

Matter-of-fact, this is a very subtle error: On the AMD 10 GBe (ax) interfaces, the hardware CRC offloading is obviously broken, but you will notice this only in the direction which is rarely used.
I had no problems with "through" traffic. The problem must have surfaced when I trusted my hardware too much, which explains why the whole HAProxy setup once worked.

Thanks for all hints!

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: shuvitcrew on July 11, 2022, 03:39:32 pm

Hello,

first I have to say thank you for this perfect tutorial. I have setup my haproxy for my webservers and everything works fine for internal and external use. Now I've tried to implement OpenVPN on Port 443 in TCP mode. I added the configuration parts as mentioned in Reply #171. The config of haproxy seems to be corrrect, but I can't connect via vpn. I've tried to setup a second vpn-server on port 1194 with upd and i works staight away. Only the vpn in tcp-mode on port 443 refuses to work. Here is my haproxy config:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 40s
    timeout server 40s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Hört auf 0.0.0.0:80 und 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_62c874b4f2fdc4.23213917 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_62c874b4f2fdc4.23213917
    # ACTION: VPN_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62c87ba1538c16.11776198.txt)]

# Frontend: 1_HTTP_frontend (Hört auf 192.168.161.1:80)
frontend 1_HTTP_frontend
    bind 192.168.161.1:80 name 192.168.161.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62360185bf9055.41837138 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62360185bf9055.41837138

# Frontend: 1_HTTPS_frontend (Hört auf 192.168.161.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.161.1:443 name 192.168.161.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62360bcec06250.52672470.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_FQDN_condition
    acl acl_62361ba046b312.42897137 src darkstar.example.xyz
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_62361a89a23796.93721092 src 192.168.110.0/24

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623619b6c7da11.06632077.txt)] if acl_62361ba046b312.42897137 || acl_62361a89a23796.93721092
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623607a0728a46.68273508.txt)]

# Backend: nextcloud_backend (Nextcloud Backend)
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    # ACL: nextcloud-caldav
    acl acl_6236326bbeed09.73911658 path_beg -i /.well-known/caldav
    # ACL: nextcloud-carddav
    acl acl_6236329a31b372.83647612 path_beg -i /.well-known/carddav

    # ACTION: nextcloud-caldav-carddav
    http-request set-path /remote.php/dav if acl_6236326bbeed09.73911658 || acl_6236329a31b372.83647612
    http-reuse safe
    server nextcloud_host 192.168.160.10:443 ssl verify none

# Backend: SSL_backend (SSL Backend TCP)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    server SSL_Server 192.168.161.1 send-proxy-v2 check-send-proxy

# Backend: bitwarden_backend (Bitwarden Backend)
backend bitwarden_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    # ACL: bitwarden-admin
    acl acl_62362f799a0826.60491269 path_beg -i /admin

    # ACTION: bitwarden-admin_block
    http-request deny if acl_62362f799a0826.60491269
    http-reuse safe
    server bitwarden_host 192.168.160.20:80

# Backend: zyxel-1_backend (Zyxel-1 Backend)
backend zyxel-1_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server zyxel-1 192.168.150.8:443 ssl verify none

# Backend: zyxel-2_backend (Zyxel-2 Backend)
backend zyxel-2_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server zyxel-2 192.168.150.9:443 ssl verify none

# Backend: checkmk_backend (CheckMK Backend)
backend checkmk_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server checkmk_host 192.168.150.21:8080

# Backend: ampache_backend (Ampache Backend)
backend ampache_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server ampache_host 192.168.160.15:443 ssl verify none

# Backend: jellyfin_backend (Jellyfin Backend)
backend jellyfin_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server jellyfin_host 192.168.160.16:8096

# Backend: guacamole_backend (Guacamole Backend)
backend guacamole_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server guacamole_host 192.168.150.22:8080

# Backend: vpn_backend (OpenVPN Backend)
backend vpn_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    server vpn_host 192.168.110.1:1195
Here the OpenVPN config:

Code: [Select]

dev ovpns1
verb 1
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_server1.pid
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto tcp-server
cipher AES-256-CBC
auth SHA512
up /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkup
down /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkdown
local 192.168.110.1
client-disconnect "/usr/local/etc/inc/plugins.inc.d/openvpn/attributes.sh server1"
tls-server
server 10.10.1.0 255.255.255.0
client-config-dir /var/etc/openvpn-csc/1
username-as-common-name
auth-user-pass-verify "/usr/local/etc/inc/plugins.inc.d/openvpn/ovpn_auth_verify user 'Local Database' 'false' 'server1'" via-env
tls-verify "/usr/local/etc/inc/plugins.inc.d/openvpn/ovpn_auth_verify tls 'vpn.shuvit.space' 1"
lport 1195
management /var/etc/openvpn/server1.sock unix
max-clients 5
push "route 192.168.110.0 255.255.255.0"
ca /var/etc/openvpn/server1.ca
cert /var/etc/openvpn/server1.cert
key /var/etc/openvpn/server1.key
dh /usr/local/etc/dh-parameters.4096.sample
tls-auth /var/etc/openvpn/server1.tls-auth 0
persist-remote-ip
float
I hope, that someone could help! Thanks in advance.

Title: Re: Tutorial 2022/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 11, 2022, 04:24:06 pm

Quote from: shuvitcrew on July 11, 2022, 03:39:32 pm

Hello,

first I have to say thank you for this perfect tutorial. I have setup my haproxy for my webservers and everything works fine for internal and external use. Now I've tried to implement OpenVPN on Port 443 in TCP mode. I added the configuration parts as mentioned in Reply #171. The config of haproxy seems to be corrrect, but I can't connect via vpn. I've tried to setup a second vpn-server on port 1194 with upd and i works staight away. Only the vpn in tcp-mode on port 443 refuses to work. Here is my haproxy config:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 40s
    timeout server 40s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Hört auf 0.0.0.0:80 und 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_62c874b4f2fdc4.23213917 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_62c874b4f2fdc4.23213917
    # ACTION: VPN_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62c87ba1538c16.11776198.txt)]

# Frontend: 1_HTTP_frontend (Hört auf 192.168.161.1:80)
frontend 1_HTTP_frontend
    bind 192.168.161.1:80 name 192.168.161.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62360185bf9055.41837138 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62360185bf9055.41837138

# Frontend: 1_HTTPS_frontend (Hört auf 192.168.161.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.161.1:443 name 192.168.161.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62360bcec06250.52672470.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_FQDN_condition
    acl acl_62361ba046b312.42897137 src darkstar.example.xyz
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_62361a89a23796.93721092 src 192.168.110.0/24

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623619b6c7da11.06632077.txt)] if acl_62361ba046b312.42897137 || acl_62361a89a23796.93721092
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/623607a0728a46.68273508.txt)]

# Backend: nextcloud_backend (Nextcloud Backend)
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    # ACL: nextcloud-caldav
    acl acl_6236326bbeed09.73911658 path_beg -i /.well-known/caldav
    # ACL: nextcloud-carddav
    acl acl_6236329a31b372.83647612 path_beg -i /.well-known/carddav

    # ACTION: nextcloud-caldav-carddav
    http-request set-path /remote.php/dav if acl_6236326bbeed09.73911658 || acl_6236329a31b372.83647612
    http-reuse safe
    server nextcloud_host 192.168.160.10:443 ssl verify none

# Backend: SSL_backend (SSL Backend TCP)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    server SSL_Server 192.168.161.1 send-proxy-v2 check-send-proxy

# Backend: bitwarden_backend (Bitwarden Backend)
backend bitwarden_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    # ACL: bitwarden-admin
    acl acl_62362f799a0826.60491269 path_beg -i /admin

    # ACTION: bitwarden-admin_block
    http-request deny if acl_62362f799a0826.60491269
    http-reuse safe
    server bitwarden_host 192.168.160.20:80

# Backend: zyxel-1_backend (Zyxel-1 Backend)
backend zyxel-1_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server zyxel-1 192.168.150.8:443 ssl verify none

# Backend: zyxel-2_backend (Zyxel-2 Backend)
backend zyxel-2_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server zyxel-2 192.168.150.9:443 ssl verify none

# Backend: checkmk_backend (CheckMK Backend)
backend checkmk_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server checkmk_host 192.168.150.21:8080

# Backend: ampache_backend (Ampache Backend)
backend ampache_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server ampache_host 192.168.160.15:443 ssl verify none

# Backend: jellyfin_backend (Jellyfin Backend)
backend jellyfin_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server jellyfin_host 192.168.160.16:8096

# Backend: guacamole_backend (Guacamole Backend)
backend guacamole_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    http-reuse safe
    server guacamole_host 192.168.150.22:8080

# Backend: vpn_backend (OpenVPN Backend)
backend vpn_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 40s
    timeout server 40s
    server vpn_host 192.168.110.1:1195
Here the OpenVPN config:

Code: [Select]

dev ovpns1
verb 1
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_server1.pid
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto tcp-server
cipher AES-256-CBC
auth SHA512
up /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkup
down /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkdown
local 192.168.110.1
client-disconnect "/usr/local/etc/inc/plugins.inc.d/openvpn/attributes.sh server1"
tls-server
server 10.10.1.0 255.255.255.0
client-config-dir /var/etc/openvpn-csc/1
username-as-common-name
auth-user-pass-verify "/usr/local/etc/inc/plugins.inc.d/openvpn/ovpn_auth_verify user 'Local Database' 'false' 'server1'" via-env
tls-verify "/usr/local/etc/inc/plugins.inc.d/openvpn/ovpn_auth_verify tls 'vpn.shuvit.space' 1"
lport 1195
management /var/etc/openvpn/server1.sock unix
max-clients 5
push "route 192.168.110.0 255.255.255.0"
ca /var/etc/openvpn/server1.ca
cert /var/etc/openvpn/server1.cert
key /var/etc/openvpn/server1.key
dh /usr/local/etc/dh-parameters.4096.sample
tls-auth /var/etc/openvpn/server1.tls-auth 0
persist-remote-ip
float
I hope, that someone could help! Thanks in advance.

Quote from: Bunch on February 20, 2022, 05:25:18 pm

I have just tried TCP mode with map file, there is a few more steps to achieve the goal instead of placing the map rule directly to 0_SNI
(I checked the package and found SNI inside, however, haproxy doesn't recognize it in TCP mode, that's why we need to force it to recognize SNI)

1. Create a "Condition" to request client hello
Name: SSL_Hello
Condition type: Custom condition (option pass-through)
Option pass-through: req_ssl_hello_type 1
(https://i.postimg.cc/rD89fwvk/cond-hello.jpg) (https://postimg.cc/rD89fwvk)

2. Create a "Rule" to wait accept SSL hello
Name: tcp_request_content_accept_ssl
Select conditions: SSL_Hello
Execute function: tcp-request-content-accept
(https://i.postimg.cc/mcnNZNhL/rule-ssl.jpg) (https://postimg.cc/mcnNZNhL)

3. Create a "Rule" to wait for inspect
Name: tcp_request_inspect_delay
Optional condition: none
Execute function: tcp-request-inspect-delay
TCP inspection delay: 5s
(https://i.postimg.cc/gX6yxX6v/rule-delay.jpg) (https://postimg.cc/gX6yxX6v)

4. Place the Rule to 0_SNI_frontend in following order
tcp_request_inspect_delay
tcp_request_content_accept_ssl
map
(https://i.postimg.cc/sBx4R0jH/rule-order.jpg) (https://postimg.cc/sBx4R0jH)
(hmdir_ru is my map rule)


Update according to findings in #183 (https://forum.opnsense.org/index.php?topic=23339.msg131582#msg131582)

5. Change the no_SSL condition to Traffic is SSL (locally deciphered)
(https://i.postimg.cc/Cng6Mdtn/nossl.jpg) (https://postimg.cc/Cng6Mdtn)

*Remark
It is advised to use another map file for 1_HTTPS_frontend if necessary
If you really don't want to create another map file, use "SNI TLS extension matches (locally deciphered)" instead

Might want to ask @Bunch for help, since I am still not using any non SSL services.

I can only give you the below config. I figured that one out a while back with another user (@Lip90 (https://forum.opnsense.org/index.php?action=profile;u=23531)).
As far as I can remeber it worked fine but it was neccessary to establish the connection twice because the first time (always) failed.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 debug

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    option log-separate-errors
    option tcplog
    # ACL: OPENVPN_condition
    acl acl_6143a3ff7e6bf2.30491250 req_ssl_hello_type 1

    # ACTION: OPENVPN_rule
    use_backend OPENVPN_backend if !acl_6143a3ff7e6bf2.30491250
    # WARNING: pass through options below this line
    tcp-request inspect-delay 5s
    tcp-request content accept if !{ req_ssl_hello_type 1 }

# Frontend: 1_HTTP_frontend (Listening on 0.0.0.0:80)
frontend 1_HTTP_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_6138b110159553.96461818 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6138b110159553.96461818

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6138b32401a006.77997133.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options
    option httplog
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6141ef8f0a8841.88130105 src 192.168.0.0/16

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6141ef3275d630.55285385.txt)] if acl_6141ef8f0a8841.88130105
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6138b15d48a964.28077676.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: OPENVPN_backend ()
backend OPENVPN_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server OPENVPN_server 127.0.0.1:1194


Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: xkpx on July 23, 2022, 06:28:37 pm

Lovely , Thanks for hard work !
Question: is it possbile to cover somehow  multi domain wildcard (for www.firewall.network.com ) -

I got problem with this settings it covers the subdomains but not www.
Common Name: *.network.com
Multidomain name: network.com

Any idea how to issue one cert for all services with subdomains and 1st level domain and www.
Or what is the right way to do this , or maybe to redirect www -> *.network.com without it?

** So far i issued new cert and added in HaProxy and its working so i guess this is the way
www.dev.network.com

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 27, 2022, 12:11:08 am

Quote from: xkpx on July 23, 2022, 06:28:37 pm

Lovely , Thanks for hard work !
Question: is it possbile to cover somehow  multi domain wildcard (for www.firewall.network.com ) -

I got problem with this settings it covers the subdomains but not www.
Common Name: *.network.com
Multidomain name: network.com

Any idea how to issue one cert for all services with subdomains and 1st level domain and www.
Or what is the right way to do this , or maybe to redirect www -> *.network.com without it?

** So far i issued new cert and added in HaProxy and its working so i guess this is the way
www.dev.network.com

If you want to cover also the base domain and not only the subdomains of it, then you will have to change the certificate settings to:

Common Name: yourdomain.com
Alt names: *.yourdomain.com

You will also have to create a rule in HAProxy to respond to requests on your base domain (yourdomain.com). Alternatively just set the desired backend for your basedomain (i.e. WWW_backend) as default backend on the 1_HTTPS_frontend.

In the end you should have a working certificate and HAProxy redirection for all your subdomains (i.e. cloud.yourdomain.com) and your base domain (yourdomain.com).

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on July 28, 2022, 11:50:22 am

Hello,

Thank you so much for writing this guide.

I were able to get this working and got the A+ authentication as well as access my server from outside network. I tested using VPN and everything work including the lock on https.

However, I'm running in to problem with the very last part which is accessing my server using https within my network.

Attatched is the screenshot of my setting in unbound dns. I also made sure that unbound dns service is running as well.

Thank you again.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 28, 2022, 01:00:42 pm

Check that the client devices in you LAN are actually using unbound dns resolver.

Edit: You have to put the OPNsense LAN IP in the DNS overide. Not the IP of the service. I explicitly say this in the tutorial.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: schnerring on July 28, 2022, 07:04:43 pm

I upgraded to 22.7. HAProxy spits out some deprecation warnings (https://forum.opnsense.org/index.php?topic=29515.msg142527#msg142527), but my config seems to be working fine.

edit: a PR with a fix has already been merged, so we just have to wait for a new haproxy plugin release

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on August 03, 2022, 09:02:59 am

Hello,

Sorry for bothering you again. I have to restart my opnsense because my piHole were messing it up so I did a clean install. However, this time I am getting stuck at the last step verifying SSL to get the A+ score.

I received an error "Assessment failed: No secure protocols supported" I've tried to went back and double check my setting and I couldn't find any error. Checked my ACME and registered, my cert is verified okay. The ip of my domain is updated automatically on the server.

My goal is to aim to get this certified so I could use my domain to add to my adguard to implement DNS over HTTPS to block ad that route as well.

I'm also included the attachments you have requested.

THank you so much for your time.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on August 05, 2022, 05:21:10 am

Hello,

I've tried to fresh re-installed Opnsense and followed your step again and finally i got A rating. not A plus but i think it worked.

Thank you so much.

This time I read your comment and got to accessed the local as well. However, for some reason I'm still getting blocked by my ISP router.

If I connected from external network I received 503 Service Unavailble. I think this make sense since I didn't allow external IP to connect my server yet (which is one of your last step)

When I connected from my lan network, my ISP router log-in page keep popping up. I've tried to put my Opnsense router to the DMZ port and tried to port forward 80 and 443 of my router internal LAN ipaddress but still didn't work. I couldn't get pass my ISP router's log in page.

Thank you again for taking your time and write this.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on August 05, 2022, 06:02:03 am

Sorry, I forgot to add my config log

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on August 05, 2022, 09:48:57 am

Hello,

I think I figured it out.

My mistake were at the very last step where you now have everything setup and wildcard which is *.zzzz.com. I copied the screenshot without understanding what I'm doing so I remove my host and kept it as zzzz where my domain is .com and I were able to access my TrueNas.

Then I understand about the public and local domain if I put my map at my local domain then i can only access it via local network. if I put it at the public map files then I can  access it at the external network and local network. Do I have it understand correctly?

However I have this one last problem I hope you can help me point out. I have adguard installed on the same IP as my opnsense. I changed my port https of my opnsense according to your guide and the port adguard's web UI listening is also different. However, when I add them in the Real server according to their port which they currently listening to. I cannot get them working. It still happening where my external network connect to it, I have the 503 error which make sense since I am only allow local. But when I access them locally I hit the ISP main router log-in page.

I hope what i wrote make sense. I'm so close, I hope someone can guide me to the right direction.

Thank you all.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: 8dgrpsu on August 06, 2022, 12:28:31 am

Thanks for this guide saved me after 2 days, the next bit is passing remote desktop through, i saw this Reddit post but I am not sure how i add to your setup or do I need to create new?

https://www.reddit.com/r/OPNsenseFirewall/comments/l2usx5/opnsense_haproxy_remote_desktop_gateway/

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 06, 2022, 07:12:14 pm

Quote from: phamd4 on August 05, 2022, 09:48:57 am

Then I understand about the public and local domain if I put my map at my local domain then i can only access it via local network. if I put it at the public map files then I can  access it at the external network and local network. Do I have it understand correctly?

Yes.

Quote from: phamd4 on August 05, 2022, 09:48:57 am

However I have this one last problem I hope you can help me point out. I have adguard installed on the same IP as my opnsense. I changed my port https of my opnsense according to your guide and the port adguard's web UI listening is also different. However, when I add them in the Real server according to their port which they currently listening to. I cannot get them working. It still happening where my external network connect to it, I have the 503 error which make sense since I am only allow local. But when I access them locally I hit the ISP main router log-in page.

Well I can't help you there... If your other services are working then you probably have your internal network misconfigured, given that you have another router in front of your opnsense.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 06, 2022, 07:13:23 pm

Quote from: 8dgrpsu on August 06, 2022, 12:28:31 am

Thanks for this guide saved me after 2 days, the next bit is passing remote desktop through, i saw this Reddit post but I am not sure how i add to your setup or do I need to create new?

https://www.reddit.com/r/OPNsenseFirewall/comments/l2usx5/opnsense_haproxy_remote_desktop_gateway/

1. You can easily add this to my/your current setup. Just follow the guide in the reddit thread.

2. Not related to my tutorial so I won't be helping here.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on August 07, 2022, 01:51:38 am

Hello,

I finally got it.

I think my problem were that my firefox browser keep pusing the connection to http thus my ISP router log-in page keep pop up. However, when I use edge or chrome the https connection pushed through and I were able to access the service.

May I ask how would I fix this problem? I tried to delete the certificate from firefox and tried to re-install the firefox but when I tried to access my service it keep asking me this connection is not secured and forced me to use http.

Thank you again.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 08, 2022, 08:37:48 pm

Delete all of the firefox history (cache, cookies, website settings ...).
if that doesn't work, it is your network, not your browser.

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Aphid667 on August 08, 2022, 09:25:21 pm

First of all, thank you for taking the time and effort to write this impressive guide. Despite this guide I still run into problems  :-[

I have a few web servers running that each have their own subdomain name. I am now trying to make the switch from pfsense to opnsense and have followed your guide to set up haproxy. Currently there is no service running on the domain name. However, when I now try to access my web server via both lan and wan I kept getting error 503 service not available. These web servers are all visualized on a proxmox server.

A second question I have, single post above you talk about "You have to put the OPNsense LAN IP in the DNS overide. Not the IP of the service." I am confused about this piece, is it possible to explain a little more about this.

Thanks in advance for feedback

Title: Re: Tutorial 2022/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 08, 2022, 10:21:09 pm

Quote from: Aphid667 on August 08, 2022, 09:25:21 pm

Currently there is no service running on the domain name. However, when I now try to access my web server via both lan and wan I kept getting error 503 service not available.

Well, if there is no service running, then HAProxy will spit out the 503 message because it can not reach the service.  ???

Code: [Select]

# Backend: cloud_backend ()
backend cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server cloud_server 192.168.7.5:80 ssl verify required ca-file /etc/ssl/cert.pem

Code: [Select]

    server cloud_server 192.168.7.5:80 ssl verify required ca-file /etc/ssl/cert.pemAre you really uploading the self-signed cert of the service to the OPNsense and checking it with HAProxy?
Otherwise that line is wrong.

Quote from: Aphid667 on August 08, 2022, 09:25:21 pm

A second question I have, single post above you talk about "You have to put the OPNsense LAN IP in the DNS overide. Not the IP of the service." I am confused about this piece, is it possible to explain a little more about this.

It is very well explained in part 6 of the post. Nothing I could really add here.
I also provided documentation links that explain the solutions.
But to maybe explain your question.
If you put the service IP in the DNS override, the client will connect to the service and therefore WON'T be able to use the Let's Encrypt - HAProxy cert.
If you put the OPNsense LAN IP in the DNS override, the client will first connect to HAProxy using a valid cert and then HAProxy talks to the client using either no or the self-signed cert of the service.

I hope you understand that I don't have the time to teach everyone how something works.
Thanks to the internet and search machines you should be able to find that out yourself.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: underwriter on August 10, 2022, 12:44:55 pm

Thank you much for this amazing tutorial. I have referred a few people from reddit to this.

Questions I hope someone can help me with: If I have xdomain.com, xczxdomain.com and ltsdomain.com;

• does this support multi domain usage?
  can I use this tutorial to assign a particular domain for a given service?
  do I need to recreate the whole entries for each or at which point do I make the adjustment?

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 10, 2022, 02:26:45 pm

Quote from: underwriter on August 10, 2022, 12:44:55 pm

Thank you much for this amazing tutorial. I have referred a few people from reddit to this.

Questions I hope someone can help me with: If I have xdomain.com, xczxdomain.com and ltsdomain.com;

• does this support multi domain usage?
  can I use this tutorial to assign a particular domain for a given service?
  do I need to recreate the whole entries for each or at which point do I make the adjustment?

Of course it does! Just make sure the domains point to any public IP of your OPNsense.

You will have to make three changes to the setup.

1. Let's Encrypt: Here you will have to add one certificate for each domain.
If their DNS Zones are managed at different domain registrars you will also have to create the corresponding DNS-01 challenges for each registrar or move their DNS Zones to deSEC (Managed DNS).

2. HAProxy HTTPS Frontend: Add the newly created certificates for each individual domain.

3. HAProxy Public Subdomain Map File: Change the map file content from f.e. "plex PLEX_backend" to "plex.xdomain.com PLEX_backend", "cloud.xczxdomain.com CLOUD_backend" and so on. This way HAProxy can map each subdomain to the correct domain and backend.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: underwriter on August 11, 2022, 12:04:53 am

Thank you so much. I appreciate your support.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on August 19, 2022, 04:02:03 pm

I was going to open a separate thread to ask for help but I think I might try and see if it is a legitimate variation to the tutorial.
First things first. My setup is exactly as per the tutorial.
I am now investigating the use of an application that uses http transport on port 80 or a custom port but without TLS. The encryption instead is done on the app layers on top. It's called rport.
I'm trying custom port 5000. Looking for the last 10 or so pages of the thread I can only see something similar in Bunch's input "Reply #171 on: February 20, 2022, 05:25:18 pm ". Not quite the same.
I've tried creating conditions matching on the hostname i.e. my subdomain part of mysubdomain.mydomain.dedyn.io and matching on http traffic. Then tried the rule on the SNI frontend, the http frontend and  https frontends. Essentially all frontends trying to make the exception there but in all cases after the sni, the http to https rule gets evaluated first, defeating any exception I've tried.

If it's not too much a deviation, could I have a suggestion on how to approach it? In sum, I'm looking for a way to route my http custom port to a back end as an exception in this Tutorial setup.

My config (you'll notice my port 5000 already in the SNI and the real server listening on that port).

Code: [Select]

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    1
    hard-stop-after             60s
    no strict-limits
    maxconn                     10
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 10
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listens on 80, 443, 853, 5000)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:853 name 0.0.0.0:853
    bind 0.0.0.0:5000 name 0.0.0.0:5000
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (listening on 192.168.5.100:80 i.e. http only)
frontend 1_HTTP_frontend
    bind 192.168.5.100:80 name 192.168.5.100:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_619439805021f2.97978352 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_619439805021f2.97978352

# Frontend: 1_HTTPS_frontend (Listening on 192.168.5.100:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.5.100:443 name 192.168.5.100:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61952b9d47d700.25962675.certlist
    bind 192.168.5.100:5000 name 192.168.5.100:5000 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61952b9d47d700.25962675.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Frontend: 1_TCP_frontend (Listening on 192.168.5.100:853)
frontend 1_TCP_frontend
    bind 192.168.5.100:853 name 192.168.5.100:853 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/61dc51606078d9.11258474.certlist
    mode tcp
    default_backend nginx_backend-tcp
    # tuning options
    timeout client 15m

    # logging options
    option tcplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/619521e7265391.88020289.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.5.100 send-proxy-v2 check-send-proxy

# Backend: nginx_backend-tcp ()
backend nginx_backend-tcp
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server nginx_2 192.168.5.1:8054 resolve-prefer ipv4 send-proxy check-send-proxy

# Backend: bastion_backend (bastion_backend)
backend bastion_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server bastion-1 192.168.5.157:5000

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: authelia on August 30, 2022, 06:55:39 am

I am trying to follow the instructions to enable HAProxy for internal domains. However, I can't seem to get the frontend listener for the virtual ip to work. Service binding is disabled for the virtual ip.

When the frontend listener for the virtual ip is enabled:

1. haproxy cannot start (when webgui is running).
2. webgui cannot start (when haproxy is running).

I have tried various things such as assigning the virtual ip from a brand new subnet etc. However the frontend listener for virtual ip seems to conflict with lighttpd no matter what I do. The only way I can get both services to start is to remove the virtual ip from /var/etc/lighty-webConfigurator.conf.

Virtual IP in LAN Subnet (192.168.1.0/24)

Code: [Select]

root@OPNsense:~ # sockstat -4 -l | grep lighttpd
root     lighttpd   28364 6  tcp4   192.168.1.65:443      *:*
root     lighttpd   28364 8  tcp4   192.168.1.1:443       *:*
root     lighttpd   28364 10 tcp4   192.168.1.65:80       *:*
root     lighttpd   28364 12 tcp4   192.168.1.1:80        *:*
root     sshd       84263 5  tcp4   192.168.1.1:22        *:*

root@OPNsense:~ # /usr/local/etc/rc.d/haproxy start
Starting haproxy.
[ALERT]    (2036) : Starting frontend 1_HTTP_frontend: cannot bind socket (Can't assign requested address) [192.168.1.65:80]
[ALERT]    (2036) : Starting frontend 1_HTTPS_frontend: cannot bind socket (Can't assign requested address) [192.168.1.65:443]
[ALERT]    (2036) : [/usr/local/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.
/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy

Virtual IP in Brand New Subnet (192.168.10.0/32)

Code: [Select]

root@OPNsense:~ # /usr/local/etc/rc.restart_webgui
Starting web GUI...done.
Generating RRD graphs...done.

root@OPNsense:~ # sockstat -4 -l | grep lighttpd
root     lighttpd   64654 6  tcp4   192.168.10.65:443     *:*
root     lighttpd   64654 8  tcp4   192.168.1.1:443       *:*
root     lighttpd   64654 10 tcp4   192.168.10.65:80      *:*
root     lighttpd   64654 12 tcp4   192.168.1.1:80        *:*
root     sshd       84263 5  tcp4   192.168.1.1:22        *:*

root@OPNsense:~ # /usr/local/etc/rc.d/haproxy start
Starting haproxy.
[ALERT]    (18033) : Starting frontend 1_HTTP_frontend: cannot bind socket (Address already in use) [192.168.10.65:80]
[ALERT]    (18033) : Starting frontend 1_HTTPS_frontend: cannot bind socket (Address already in use) [192.168.10.65:443]
[ALERT]    (18033) : [/usr/local/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 30, 2022, 01:37:07 pm

Quote from: authelia on August 30, 2022, 06:55:39 am

I am trying to follow the instructions to enable HAProxy for internal domains. However, I can't seem to get the frontend listener for the virtual ip to work. Service binding is disabled for the virtual ip.

When the frontend listener for the virtual ip is enabled:

1. haproxy cannot start (when webgui is running).
2. webgui cannot start (when haproxy is running).

Part 4 - Step 1.

If you would have followed the tutorial STEP BY STEP you wouldn't have any issues... Just stick to the tutorial and don't skip a single step.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 30, 2022, 01:41:06 pm

Quote from: cookiemonster on August 19, 2022, 04:02:03 pm

Then tried the rule on the SNI frontend, the http frontend and  https frontends. Essentially all frontends trying to make the exception there but in all cases after the sni, the http to https rule gets evaluated first, defeating any exception I've tried.

If it's not too much a deviation, could I have a suggestion on how to approach it? In sum, I'm looking for a way to route my http custom port to a back end as an exception in this Tutorial setup.

The order of the rules is important! Make sure that all "http-redirect-to-backend" rules are placed BEFORE the HTTPtoHTTPS rule on the HTTP_frontend.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: boredpanda on August 30, 2022, 03:29:34 pm

Hello! I've used this guide a while back and my self-hosted services are running rock solid. There hasn't been any problems at all with HAProxy - so thank you very much for this guide! I haven't exposed any of my services publicly and mainly use HAProxy to be able to use FQDN at home, and when I VPN in using my phone or personal laptop.

However, there is an instance where it would be very nice to be able to white-list one (or a couple) of specific IPs, so that I could access my services at home from my office. I am not able to install software at the office, and there are other restrictions preventing me from using a VPN.

How would I go about white-listing a single IP, allowing access to some of my internal services? Please let me know if my question is out-of-scope for this tutorial and I'll ask elsewhere. ;)

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on August 30, 2022, 03:39:32 pm

Quote from: boredpanda on August 30, 2022, 03:29:34 pm

However, there is an instance where it would be very nice to be able to white-list one (or a couple) of specific IPs, so that I could access my services at home from my office. I am not able to install software at the office, and there are other restrictions preventing me from using a VPN.

This is a very easy task, given that the IPs are static!

I don't know how you restricted local access but if you followed my tutorial you will just have to do this.
Take a look at part 7 of the tutorial.

Create the public subdomains map file, create a condition containing all the whitelist public IPs and create the corresponding redirect rule just as I did with the local access subdomains map file.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheEPOCH on August 30, 2022, 11:18:52 pm

Hello,

my previous question is canceled. I build up my whole OPNsense from scratch and now the Tutorial worked very fine.

Thanks for writing this!

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on August 30, 2022, 11:52:21 pm

Quote from: TheHellSite on August 30, 2022, 01:41:06 pm

Quote from: cookiemonster on August 19, 2022, 04:02:03 pm

Then tried the rule on the SNI frontend, the http frontend and  https frontends. Essentially all frontends trying to make the exception there but in all cases after the sni, the http to https rule gets evaluated first, defeating any exception I've tried.

If it's not too much a deviation, could I have a suggestion on how to approach it? In sum, I'm looking for a way to route my http custom port to a back end as an exception in this Tutorial setup.

The order of the rules is important! Make sure that all "http-redirect-to-backend" rules are placed BEFORE the HTTPtoHTTPS rule on the HTTP_frontend.

Thank you. Unfortunately I haven't been able to do this. The exact warning is:

Code: [Select]

[WARNING] (96704) : parsing [/usr/local/etc/haproxy.conf.staging:74] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.
Warnings were found.
Configuration file is validAny other ideas are welcome :)

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on September 01, 2022, 12:39:57 pm

Quote from: cookiemonster on August 30, 2022, 11:52:21 pm

Quote from: TheHellSite on August 30, 2022, 01:41:06 pm

Quote from: cookiemonster on August 19, 2022, 04:02:03 pm

Then tried the rule on the SNI frontend, the http frontend and  https frontends. Essentially all frontends trying to make the exception there but in all cases after the sni, the http to https rule gets evaluated first, defeating any exception I've tried.

If it's not too much a deviation, could I have a suggestion on how to approach it? In sum, I'm looking for a way to route my http custom port to a back end as an exception in this Tutorial setup.

The order of the rules is important! Make sure that all "http-redirect-to-backend" rules are placed BEFORE the HTTPtoHTTPS rule on the HTTP_frontend.

Thank you. Unfortunately I haven't been able to do this. The exact warning is:

Code: [Select]

[WARNING] (96704) : parsing [/usr/local/etc/haproxy.conf.staging:74] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.
Warnings were found.
Configuration file is validAny other ideas are welcome :)

I've found what I think is a workaround with the service in question, leaving the haproxy setup still as per this tutorial's. Thanks for the earlier suggestion.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brooklynmind on September 03, 2022, 01:35:04 pm

I've been following this wonderfully crafted tutorial, so "THANK YOU" to the op for this.
Question (I know this might outside the scope of this tutorial):

• If I want HAProxy to handle *.my1stdomain.xyz which would be for specific services (already have this working flawlessly),
  but I would like to forward *.my2nddomain.xyz to nginx proxy manager running on docker so that nginx proxy manager will be used to manage that.

Is that something this setting can help to implement?

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 06, 2022, 09:16:08 am

Quote from: brooklynmind on September 03, 2022, 01:35:04 pm

I've been following this wonderfully crafted tutorial, so "THANK YOU" to the op for this.
Question (I know this might outside the scope of this tutorial):

• If I want HAProxy to handle *.my1stdomain.xyz which would be for specific services (already have this working flawlessly),
  but I would like to forward *.my2nddomain.xyz to nginx proxy manager running on docker so that nginx proxy manager will be used to manage that.

Is that something this setting can help to implement?

This has been answered 12 messages back.
https://forum.opnsense.org/index.php?topic=23339.msg143886#msg143886

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brooklynmind on September 08, 2022, 02:46:39 am

This has been answered 12 messages back.
https://forum.opnsense.org/index.php?topic=23339.msg143886#msg143886
[/quote]

Not really, as that person's question was different from mine.

• I'm looking to forward everything related to domain *.my1stdomain.xyz to nginx proxy manager (I needed to do this as npm works well with Authentik which I use for SSO for some of my apps)
  Whereas HAProxy handles everything related to *.my1stdomain.xyz (I already have this working without issues.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on September 08, 2022, 04:52:40 am

Quote from: brooklynmind on September 03, 2022, 01:35:04 pm

I've been following this wonderfully crafted tutorial, so "THANK YOU" to the op for this.
Question (I know this might outside the scope of this tutorial):

• If I want HAProxy to handle *.my1stdomain.xyz which would be for specific services (already have this working flawlessly),
  but I would like to forward *.my2nddomain.xyz to nginx proxy manager running on docker so that nginx proxy manager will be used to manage that.

Is that something this setting can help to implement?

If you want nginx to handle TLS of *.my2nddomain.xyz, then the backend of *.my2nddomain.xyz will be in TCP mode.
And you need to set SNI_FRONTEND to something like #176 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354)

If you want keeping HAPROXY to handle TLS, you need to change config of nginx to accept proxy protocol, which is really out of scope

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brooklynmind on September 08, 2022, 12:44:01 pm

If you want nginx to handle TLS of *.my2nddomain.xyz, then the backend of *.my2nddomain.xyz will be in TCP mode.
And you need to set SNI_FRONTEND to something like #176 (https://forum.opnsense.org/index.php?topic=23339.msg131354#msg131354)

If you want keeping HAPROXY to handle TLS, you need to change config of nginx to accept proxy protocol, which is really out of scope
[/quote]

Thanks a lot for your input. I'll play around with that and look at #176 as you pointed out.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stevoni on September 09, 2022, 05:15:16 pm

Great tutorial!

I'm running into a problem accessing the sites within the network after following this tutorial and enabling Cloudflare proxy. Without the Cloudflare proxy I can access the sites both externally and internally but when I enable the Cloudflare proxy I'm unable to access the sites from the internal network.

This post, https://vitobotta.com/2019/12/23/real-ip-haproxy-ingress-behind-cloudflare/ (https://vitobotta.com/2019/12/23/real-ip-haproxy-ingress-behind-cloudflare/), explains how to get the correct IP but I'm not clear on how to implement that in the OPNsense HAProxy implementation. I found a similar question on the forums, https://forum.opnsense.org/index.php?topic=26419.msg127542#msg127542 (https://forum.opnsense.org/index.php?topic=26419.msg127542#msg127542), but there wasn't any answer.

I created a condition with the Cloudflare IPs but I don't know where to go from there, any suggestions?

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 20, 2022, 08:47:40 pm

Quote from: Stevoni on September 09, 2022, 05:15:16 pm

Great tutorial!

I'm running into a problem accessing the sites within the network after following this tutorial and enabling Cloudflare proxy. Without the Cloudflare proxy I can access the sites both externally and internally but when I enable the Cloudflare proxy I'm unable to access the sites from the internal network.

This post, https://vitobotta.com/2019/12/23/real-ip-haproxy-ingress-behind-cloudflare/ (https://vitobotta.com/2019/12/23/real-ip-haproxy-ingress-behind-cloudflare/), explains how to get the correct IP but I'm not clear on how to implement that in the OPNsense HAProxy implementation. I found a similar question on the forums, https://forum.opnsense.org/index.php?topic=26419.msg127542#msg127542 (https://forum.opnsense.org/index.php?topic=26419.msg127542#msg127542), but there wasn't any answer.

I created a condition with the Cloudflare IPs but I don't know where to go from there, any suggestions?

Sorry, but this is out of scope of this tutorial.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mozart on September 23, 2022, 01:18:11 pm

I followed the tutorial and added the necessary config to enable OpenVPN on port 443 but somehow it always keeps sending everything to the SSL_Backend.

My config file:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_632625326b34a3.00256787 req_ssl_hello_type 1

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_632625326b34a3.00256787
    # ACTION: Openvpn_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6329deb37cfb97.45093681.txt)] if acl_632625326b34a3.00256787

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_632463bc8a4e03.38927091 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_632463bc8a4e03.38927091

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 crt-list /tmp/haproxy/ssl/632498ac5e6503.54058036.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6324642dba2f56.47189800.txt)]

# Backend: wiki ()
backend wiki
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server apache03 192.168.254.4:80

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Zoneminder_backend ()
backend Zoneminder_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server ZoneMinder 192.168.254.22:80 source 192.168.254.2

# Backend: Nextcloud_Backend ()
backend Nextcloud_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server owncloud 192.168.254.23:443 ssl verify none

# Backend: KH_backend ()
backend KH_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server kh 192.168.10.50:80

# Backend: HA_backend ()
backend HA_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server ha 192.168.0.51:80

# Backend: HASS_backend ()
backend HASS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server hass 192.168.254.13:8123

# Backend: BITWARDEN_backend ()
backend BITWARDEN_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server bitwarden 192.168.254.4:81

# Backend: Webmin_Backend ()
backend Webmin_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server sshgateway 192.168.254.34:10000 ssl verify none

# Backend: OPENVPN_backend ()
backend OPENVPN_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server OPENVPN_server 127.4.4.3:10194

When I try to connect to the OpenVPN server on port 443 I get this in the haproxy logs:

Code: [Select]

2022-09-23T13:16:23 Informational haproxy 1.2.3.4:43265 [23/Sep/2022:13:16:23.981] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
2022-09-23T13:16:23 Informational haproxy Connect from 1.2.3.4:43265 to 4.3.2.1:443 (0_SNI_frontend/TCP)
2022-09-23T13:16:19 Informational haproxy 1.2.3.4:43264 [23/Sep/2022:13:16:19.866] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
2022-09-23T13:16:19 Informational haproxy Connect from 1.2.3.4:43264 to 4.3.2.1:443 (0_SNI_frontend/TCP)

Any ideas?

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 23, 2022, 02:26:19 pm

Quote from: mozart on September 23, 2022, 01:18:11 pm

I followed the tutorial and added the necessary config to enable OpenVPN on port 443 but somehow it always keeps sending everything to the SSL_Backend.

Any ideas?

Configure your SNI_frontend like below and it should work.

Code: [Select]

frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    option log-separate-errors
    option tcplog
    # ACL: OPENVPN_condition
    acl acl_6143a3ff7e6bf2.30491250 req_ssl_hello_type 1

    # ACTION: OPENVPN_rule
    use_backend OPENVPN_backend if !acl_6143a3ff7e6bf2.30491250
    # WARNING: pass through options below this line
    tcp-request inspect-delay 5s
    tcp-request content accept if !{ req_ssl_hello_type 1 }


Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mozart on September 23, 2022, 02:56:22 pm

Thanks, I updated my frontend config:

Code: [Select]

# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options
    # ACL: SSL_hello
    acl acl_632625326b34a3.00256787 req_ssl_hello_type 1

    # ACTION: Openvpn_map-rule
    use_backend OPENVPN_backend if acl_632625326b34a3.00256787
    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request content accept if !{ req_ssl_hello_type 1 }


I think it matches your example but I still get errors in the haproxy log:

Code: [Select]

2022-09-23T14:52:32 Informational haproxy 1.2.3.4:42250 [23/Sep/2022:14:52:32.904] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
2022-09-23T14:52:32 Informational haproxy Connect from 1.2.3.4:42250 to 4.3.2.1:443 (0_SNI_frontend/TCP)
2022-09-23T14:52:31 Informational haproxy Connect from 192.168.254.13:52460 to 192.168.254.1:443 (0_SNI_frontend/TCP)

I looked a bit further and now I do see entries appearing in the OpenVPN logs so I will check those out first.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on September 24, 2022, 04:02:14 am

hi guys,

I am trying to setup a SSO using keycloak, openLDAP and other providers. I have it all working locally but now wanting to communicate to the LDAP server via ldap://ldap.mydomain.com instead of ldap://192.168.1.104:1389.

The only difference that i have tried is changing the mode on the backend server from HTTP to TCP. I just recieve a timeout error when testing.

Code: [Select]

2022-09-24 13:40:25,284 ERROR [org.keycloak.services] (executor-thread-39) KC-SERVICES0055: Error when connecting to LDAP: ldap.mydomain.com:389: javax.naming.CommunicationException: ldap.mydomain.com:389 [Root exception is java.net.SocketTimeoutException: connect timed out]

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     100000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62bbec3b1189e7.31090598 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62bbec3b1189e7.31090598

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62bbef8e4ab6b5.77631912.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62bbecc24b7a71.66647551.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: PRISM_backend ()
backend PRISM_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server PRISM_server 192.168.1.103:2342

# Backend: REQUEST_backend ()
backend REQUEST_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server REQUEST_server 192.168.1.104:5055

# Backend: LDAP_backend ()
backend LDAP_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server LDAP_server 192.168.1.104:1636 ssl verify none


Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bunchofreeds on September 27, 2022, 01:22:56 am

Thanks @TheHellSite for an awesome tutorial and support to us all with OPNsense & HAproxy
I have a perfectly working HAproxy thanks to you with dynamic DNS, LetsEncrypt and multiple reverse proxied sites.

However, I have not changed the default port for OPNsense away from 443

I have three externally accessible sites all via my single public IP, which are directed to three different backend servers using HAproxy
These public urls are specified in the 'listening addresses' of my public service using their ww.example.com:443 format. I have not specified a broad 0.0.0.0:443 for example

What's your view on this setup and its possible conflict with the OPNsesne webserver. Am I correct to specify the url's and therefore HAproxy is only listening for these and not the url or IP for OPNsense?

Thanks for any advice

Title: Thanks very much for this post - I have a quick question if you don't mind
Post by: phib3r on September 28, 2022, 04:30:17 pm

My config is working great thanks to this post ! I do have a quick question that you may be able to expand on - requiring client certificates from a specific sub domain. I have tried out a few things, but all that seems to happen is that all the sub domains request client certs. I have tried to create a new public https frontend and have that look up from a different mapping file - but all that seems to happen is that the original https front end just now requires client certs. I would post my config - but it is currently working and to the letter what you have posted with just some more sub domains added. Any help or pointing in the right direction would be fantastic.

M

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phib3r on September 28, 2022, 10:53:16 pm

To the LDAP question above might be that LDAP is using UDP and not tcp

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phib3r on September 28, 2022, 10:53:50 pm

Might be - sorry

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on October 06, 2022, 07:53:44 pm

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 1000s
    timeout connect 1000s
    timeout server 1000s
    retries 3
    default-server init-addr libc,last
    default-server maxconn 5000

# autogenerated entries for ACLs

# userlists generated from groups
userlist Allowedusers
    user joel insecure-password XXX
    user mopidy insecure-password XXX
    # NOTE: UserlistAddUsers called with empty group data


# autogenerated entries for config in backends/frontends
userlist list_6245eeb66d3ab2.08976803
    # Origin: MOPIDY_backend
    user mopidy insecure-password XXX
    user joel insecure-password XXX
    # WARNING: skipping duplicate username (mopidy)


# autogenerated entries for stats




# Frontend: SNI_frontend (Listening on http&https)
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind :::80 name :::80
    bind :::443 name :::443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 1000s

    # logging options

# Frontend: HTTP_frontend (Listening 127.0.0.1:80)
frontend HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    bind [::1]:80 name [::1]:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 1000s

    # logging options
    # ACL: NoSSL_condition
    acl acl_621d0b77c74989.24704837 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_621d0b77c74989.24704837

# Frontend: HTTPS_frontend (Listinging on 127.0.0.1:443)
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621d11c7cad951.61400293.certlist
    bind [::1]:443 name [::1]:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621d11c7cad951.61400293.certlist
    mode http
    option http-keep-alive
    default_backend WEBSERVER_backend
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/621d0c7054ddb7.46420139.txt)]
    # WARNING: pass through options below this line
      # Matrix client traffic
      acl matrix-host hdr(host) -i chat.XXX.ch chat.XXX.ch:443
      acl matrix-path path_beg /_matrix
      acl matrix-path path_beg /_synapse/client
   
      use_backend MATRIX_backend if matrix-host matrix-path

# Frontend: MATRIX_frontend (Listining * Port 8448)
frontend MATRIX_frontend
    bind *:8448 name *:8448 alpn h2,http/1.1 ssl  crt-list /tmp/haproxy/ssl/6256daae2378c2.17892750.certlist
    bind [::]:8448 name [::]:8448 alpn h2,http/1.1 ssl  crt-list /tmp/haproxy/ssl/6256daae2378c2.17892750.certlist
    mode http
    option http-keep-alive
    default_backend MATRIX_backend
    # tuning options
    timeout client 1000s

    # logging options
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
      http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
      http-request set-header X-Forwarded-For %[src]

# Frontend: SSH_frontend (Listining * Port 22)
frontend SSH_frontend
    bind *:22 name *:22 alpn h2,http/1.1
    bind [::]:22 name [::]:22 alpn h2,http/1.1
    mode tcp
    # tuning options
    timeout client 1000s

    # logging options

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: WEBSERVER_backend ()
backend WEBSERVER_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    acl restricted_page path_beg /wp-admin
    acl auth_ok http_auth(Allowedusers)
    http-request auth if restricted_page !auth_ok
   
    http-reuse safe
    server WEBSERVER_server 192.168.1.100:80 send-proxy-v2 check-send-proxy
    server WEBSERVER_server_ipv6 XXX:168:a774::2000:80 send-proxy-v2 check-send-proxy

# Backend: NAS_backend ()
backend NAS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server NAS_server 192.168.1.118:80
    server NAS_server_ipv6 XXX:168:a774::1000:80

# Backend: WEBSERVER_SSL_backend ()
backend WEBSERVER_SSL_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
   
    http-reuse safe
    server WEBSERVER_server_ssl 192.168.1.100:443
    server WEBSERVER_server_ssl_ipv6 XXX:168:a774::2000:443

# Backend: MOPIDY_backend ()
backend MOPIDY_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    acl auth_ok http_auth(list_6245eeb66d3ab2.08976803)
    http-request auth if !auth_ok
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    acl is_root path -i /
    redirect code 301 location /iris if is_root
    http-reuse safe
    server MOPIDY_server 192.168.1.100:6680

# Backend: MATRIX_backend ()
backend MATRIX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
      http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
      http-request set-header X-Forwarded-For %[src]
    http-reuse safe
    server MATRIX_server 192.168.1.100:8008
    server MATRIX_server_ipv6 XXX:168:a774::2000:8008

# Backend: KVM_backend ()
backend KVM_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server KVM_server 192.168.1.105:80

# Backend: SYNC_backend ()
backend SYNC_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
      http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
      http-request set-header X-Forwarded-For %[src]
    http-reuse safe
    server SYNC_server 192.168.1.100:5050

# Backend: ROUTER_SSH_backend ()
backend ROUTER_SSH_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server ROUTER_SSH_Server 192.168.1.1:22
    server ROUTER_SSH_Server_ipv6 XXX:168:a774::1000:22

# Backend: NAS_SSH_backend ()
backend NAS_SSH_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server NAS_server_ipv6 XXX:168:a774::1000:80
    server NAS_SSH_server 192.168.1.118:22

# Backend: KVM_SSH_backend ()
backend KVM_SSH_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server KVM_SSH_server 192.168.1.105:22

# Backend: SERVER_SSH_backend ()
backend SERVER_SSH_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server SERVER_SSH_server 192.168.1.100:22
    server SERVER_SSH_server_ipv6 XXX:168:a774::2000:22

Could someone say why my SSH service points everytime to the Router and doesn't split to ssh, ssh.kvm, ssh.server, ssh.nas?

My map file looks like

Code: [Select]

#public access subdomains
flood WEBSERVER_backend
kvm KVM_backend
nas WEBSERVER_backend
grafana WEBSERVER_backend
phpmyadmin WEBSERVER_backend
speedtestserver WEBERSERVER_backend
cloud NAS_backend
dav NAS_backend
stefan NAS_backend
mopidy MOPIDY_backend
git WEBSERVER_backend
chat MATRIX_backend
admin WEBSERVER_backend
sync SYNC_backend
ssh.nas NAS_SSH_backend
ssh.server SERVER_SSH_backend
ssh ROUTER_SSH_backend
ssh.kvm KVM_SSH_backend

Is a frontend for port 22 necessary?
Thanks for advices.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on October 10, 2022, 06:01:41 pm

Just think a little bit deeper:
I'm afraid SSH_frontend will never able to bind if your router is 192.168.1.1
As it always using port 22, you will never able to take port 22 with HAProxy.
Thus, everything related to SSH_frontend will never work until you change SSH of your router to different port

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Morta on October 25, 2022, 07:10:49 pm

Ok. I changed OPNsene Port to 222 and SSH_frontend to 22.

Now I can access all my clients over haproxy with ssh,ipv4 and port 22.

Thanks for the Input.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meelokun on November 08, 2022, 09:02:33 am

deleted post

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: techsolo12 on November 10, 2022, 01:46:14 pm

Hello Guys!

Today its my first post here at this forum. At first @TheHellSite THANK YOU for your tutorial it helps my a lot! Before i used nginx proxy manager which was a lot easier than haproxy :)

I had one for my big problem and need the help from you all, please. I want to configure vaultwarden with websocket support in haproxy. The normal redirect to vaultwarden is no problem, but to add websocket support is still driven my crazy!

https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples (https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples)
Here are some examples how the proxy setup should, but i dont understand were my problem is.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_636976fd9d4d71.97561865 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_636976fd9d4d71.97561865

# Frontend: 1_HTTPS_frontend (Listening on 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/636aad8d3cbe18.58884679.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: nc_carddav
    acl acl_636ba4e5b6aa82.28881573 path_end -i /.well-known/carddav
    # ACL: nc_caldav
    acl acl_636ba2d9f14933.27250118 path_end -i /.well-known/caldav
    # ACL: vw_ws_acl01_condition
    acl acl_636c2f2b5accd9.55827620 path_beg -i /notifications/hub
    # ACL: vw_ws_acl02_condition
    acl acl_636cc909734817.72974823 path_beg -i /notifications/hub/negotiate
    # ACL: vw_ws_acl03_condition
    acl acl_636ccac64fcd74.27409543 path_beg -i /notifications/hub
    # ACL: vw_ws_acl04_condition
    acl acl_636ccae443ca48.73072029 path_beg -i /notifications/hub/negotiate

    # ACTION: nc_carddav_rule
    http-request redirect code 301 location /remote.php/dav if acl_636ba4e5b6aa82.28881573
    # ACTION: nc_caldav_rule
    http-request redirect code 301 location /remote.php/dav if acl_636ba2d9f14933.27250118
    # ACTION: PUBLIC_SUBDOMAINS-map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63653d33935cd3.47503593.txt)]
    # ACTION: vw_ws_acl01_rule
    use_backend vw_backend if !acl_636c2f2b5accd9.55827620
    # ACTION: vw_ws_acl02_rule
    use_backend vw_backend if acl_636cc909734817.72974823
    # ACTION: vw_ws_acl03_rule
    use_backend vw_ws_backend if acl_636ccac64fcd74.27409543
    # ACTION: vw_ws_acl04_rule
    use_backend vw_ws_backend if !acl_636ccae443ca48.73072029

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: office_backend (Onlyoffice)
backend office_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server office_server 10.10.20.8:80

# Backend: vw_backend (Vaultwarden)
backend vw_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server vw_server 10.10.20.7:80

# Backend: mc_backend (Minecraft Server)
backend mc_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server mc_server 10.10.40.4:80

# Backend: cloud_backend (Nextcloud01)
backend cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server cloud_server 10.10.20.5:80

# Backend: demo_backend (Nextcloud02)
backend demo_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server demo_server 10.10.20.6:80

# Backend: kunden_backend (Nextcloud03)
backend kunden_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server kunden_server 10.10.20.11:80

# Backend: vw_ws_backend (Vaultwarden Websocket)
backend vw_ws_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server vw_ws_server 10.10.20.7:3012



# statistics are DISABLED

Code: [Select]

2022-11-10T13:33:48 Informational haproxy Connect from 10.10.10.239:54010 to PUBLICIP:443 (1_HTTPS_frontend/HTTP)
2022-11-10T13:33:48 Informational haproxy Connect from 10.10.10.239:54010 to PUBLICIP:443 (1_HTTPS_frontend/HTTP)
2022-11-10T13:33:48 Informational haproxy Connect from 10.10.10.239:54010 to PUBLICIP:443 (1_HTTPS_frontend/HTTP)
2022-11-10T13:33:48 Informational haproxy Connect from 10.10.10.239:54010 to PUBLICIP:443 (0_SNI_frontend/TCP)
With best regard,
techsolo12

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 10, 2022, 02:36:49 pm

Quote from: techsolo12 on November 10, 2022, 01:46:14 pm

Hello Guys!

Today its my first post here at this forum. At first @TheHellSite THANK YOU for your tutorial it helps my a lot! Before i used nginx proxy manager which was a lot easier than haproxy :)

I had one for my big problem and need the help from you all, please. I want to configure vaultwarden with websocket support in haproxy. The normal redirect to vaultwarden is no problem, but to add websocket support is still driven my crazy!

Sorry, but out of scope of this tutorial. Please ask in the official HAProxy forum.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meelokun on November 11, 2022, 02:52:45 am

Alright! Im back with a clean baseline. Lets try this again... And apologies if anything below sounds dumb, im still faily new to understanding Certs, and how wildcard domains work. (although i had it working in NGINX with another domain of mine, i want to upgrade from a B score to A+ Score, and part of that was moving to HAProxy) - also i have no interest in utilizing 2nd-level-subdomains. I will only be using 1st-level-subdomains for all my services.

Heres where im getting stuck (Part 3, Step 6): Cert Validation - fails
Since i use Cloudflare, I tried my best to adapt your DynDNS setup to Cloudflare (DynDNS confirmed working)


Cloudflare account DNS Management
(https://i.imgur.com/5Fi4xnT.png)

Dynamic DNS Settings
(https://i.imgur.com/iW3UIgY.png)

ACME Settings
(https://i.imgur.com/Sf7cWdM.png)
(https://i.imgur.com/kWNSVLy.png)
(https://i.imgur.com/cB2MevV.png)

What am i doing wrong?

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: robbie11r1 on November 11, 2022, 04:33:04 am

Good Evening,

Thank you for the excellent tutorial! I have setup HAProxy + Wildcard Certificates following this tutorial, but am experiencing a 503 error when trying to access jellyfin.example.com using the setup here. I do not have any of these services accessible from outside my network (I.E. no WAN 443/80 ports open) and am only interested in being able to access "jellyfin.example.com" --> 192.168.5.88:8096 with a valid Let's Encrypt cert from inside my network. Right now when I access jellyfin.example.com, my browser shows a valid cert but throws a 503 error. Double and triple checking the tutorial has left me without any further options to explore.

Does anyone have any suggestions where I can look? Thank you very much in advance!

Unbound Host Override: *.example.com --> 192.168.5.1 (OpnSense LAN IP)

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: HTTPS_Frontend (Listening on 127.4.4.3:443)
frontend HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63687fb14df779.98297035.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: Local_Only_Subnet
    acl acl_63687bc7cf9331.77802781 src 192.168.5.0/24

    # ACTION: Local_subdomain_map
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63687b6d92a544.19528694.txt)] if acl_63687bc7cf9331.77802781

# Frontend: SNI_Frontend (Listening to 0.0.0.0:80; 0.0.0.0:443)
frontend SNI_Frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_Backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTP_Frontend (Listening on 127.4.4.3:80)
frontend HTTP_Frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_Condition
    acl acl_63687974def2f4.69235454 ssl_fc

    # ACTION: HTTPtoHTTPS
    http-request redirect scheme https code 301 if !acl_63687974def2f4.69235454

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Jellyfin_Backend ()
backend Jellyfin_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Jellyfin 192.168.5.88:8096 ssl verify none

# Backend: TPLink_Backend ()
backend TPLink_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server TPLinkSwitch 192.168.5.5:80

# Backend: Opnsense_Backend ()
backend Opnsense_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Opnsense 192.168.5.1:8100 ssl verify none

# Backend: Proxmox_Backend ()
backend Proxmox_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Proxmox 192.168.5.201:8006 ssl verify none



# statistics are DISABLED
Map File:

Code: [Select]

jellyfin Jellyfin_Backend
tplink TPLink_Backend
opnsense Opnsense_Backend
proxmox Proxmox_Backend


Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 11, 2022, 11:11:04 am

Quote from: meelokun on November 11, 2022, 02:52:45 am

Alright! Im back with a clean baseline. Lets try this again... And apologies if anything below sounds dumb, im still faily new to understanding Certs, and how wildcard domains work. (although i had it working in NGINX with another domain of mine, i want to upgrade from a B score to A+ Score, and part of that was moving to HAProxy) - also i have no interest in utilizing 2nd-level-subdomains. I will only be using 1st-level-subdomains for all my services.

Heres where im getting stuck (Part 3, Step 6): Cert Validation - fails
Since i use Cloudflare, I tried my best to adapt your DynDNS setup to Cloudflare (DynDNS confirmed working)

Out of scope of this tutorial!
As a hint: Your certificate configuration looks fine. Probably wrong DNS-01 settings. Please search on Google for OPNsense Cloudflare ACME guide.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 11, 2022, 11:15:15 am

Quote from: robbie11r1 on November 11, 2022, 04:33:04 am

Good Evening,

Thank you for the excellent tutorial! I have setup HAProxy + Wildcard Certificates following this tutorial, but am experiencing a 503 error when trying to access jellyfin.example.com using the setup here. I do not have any of these services accessible from outside my network (I.E. no WAN 443/80 ports open) and am only interested in being able to access "jellyfin.example.com" --> 192.168.5.88:8096 with a valid Let's Encrypt cert from inside my network. Right now when I access jellyfin.example.com, my browser shows a valid cert but throws a 503 error. Double and triple checking the tutorial has left me without any further options to explore.

Does anyone have any suggestions where I can look? Thank you very much in advance!

Unbound Host Override: *.example.com --> 192.168.5.1 (OpnSense LAN IP)

Map File:

Code: [Select]

jellyfin Jellyfin_Backend
tplink TPLink_Backend
opnsense Opnsense_Backend
proxmox Proxmox_Backend


Since you get a 503 with the right certificate your overall setups seems to be working.

Is only Jellyfin not working or does it affect all services mentioned in the map file?
If it only affects Jellyfin: You might want to check the server settings for it in regards of SSL (yes or no). By default Jellyfin comes WITHOUT a self-signed SSL certificate so you will want to disable SSL for this server in HAProxy. Also from what I know the Jellyfin port 8096 is HTTP so no SSL. But you have SSL enabled in the server settings.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: robbie11r1 on November 11, 2022, 01:01:32 pm

@TheHellSite, I saw some other posts over thr last year with similar problems to me (503 error) and solved it by de-selecting SSL in the Real Server section. I tried that with Jellyfin and nothing changed for me (still same 503 with valid cert).

I get the same error for every service, no matter if SSL is selected or not. Is there anything else I should be changing aside from just de-selecting SSL?

Thanj you again!

EDIT: As of this morning (and changing nothing), everything started to work. Not sure what did it, but awesome tutorial and thanks for the feedback/troubleshooting with everyone, helped immensely.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meelokun on November 12, 2022, 11:19:38 am

Im up and running (Everything works!) I Figured out my Cloudflare DNS challenge issue (you have to make sure to get the Token that is displayed AFTER you create a Cloudflare API Token for the Zone.DNS read and Zone.DNS Edit... I mistakenly thought it was the Global ID Key..)

Now the only thing i cant get to work is the Unbound DNS Override.

You stated

Quote

If you are running all of your services on your 1st level subdomain "your_subdomain.dedyn.io" than you will just need to override this one.

Since im utilizing a wildcard, i figured it should work this way, so that any subdomain i enter, will be redirected to HAProxy's SNI_Frontend. And since its listening on 0.0.0.0, i figured the virtual IP should work - i also tried the Firewalls IP address with no luck.

(https://i.imgur.com/tU7X9xH.png)

And yes - the virtual ip is set to loopback.

Also - the Unbound DNS Overrides section looks different, now theres 2 tabs (Host Overrides), a main entry and an aliases entry,
(https://i.imgur.com/nhr5gHq.png)
then a (Domain Overrides) tab.
(https://i.imgur.com/1TmbDeX.png)

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 12, 2022, 12:04:40 pm

Quote from: meelokun on November 12, 2022, 11:19:38 am

You stated

Quote

If you are running all of your services on your 1st level subdomain "your_subdomain.dedyn.io" than you will just need to override this one.

Since im utilizing a wildcard, i figured it should work this way, so that any subdomain i enter, will be redirected to HAProxy's SNI_Frontend. And since its listening on 0.0.0.0, i figured the virtual IP should work - i also tried the Firewalls IP address with no luck.

(https://i.imgur.com/tU7X9xH.png)

My tutorial clearly states that you have to use the OPNsense LAN IP in the DNS override.

How on earth would the lan devices be able to talk to a virtual IP created on the loopback device of the OPNsense. You should Google what the localhost is.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: techsolo12 on November 13, 2022, 04:08:24 pm

Quote from: TheHellSite on November 10, 2022, 02:36:49 pm

Quote from: techsolo12 on November 10, 2022, 01:46:14 pm

Hello Guys!

Today its my first post here at this forum. At first @TheHellSite THANK YOU for your tutorial it helps my a lot! Before i used nginx proxy manager which was a lot easier than haproxy :)

I had one for my big problem and need the help from you all, please. I want to configure vaultwarden with websocket support in haproxy. The normal redirect to vaultwarden is no problem, but to add websocket support is still driven my crazy!

Sorry, but out of scope of this tutorial. Please ask in the official HAProxy forum.

Hello Guys!

Unfortunally nobody in the other forums can help me with this situation. Anybody in vaultwarden or haproxy forum. Is here nobody who had vaultwarden getting worked? :(

With best regards;
techsolo12

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 14, 2022, 12:53:41 am

Quote from: techsolo12 on November 13, 2022, 04:08:24 pm

Hello Guys!

Unfortunally nobody in the other forums can help me with this situation. Anybody in vaultwarden or haproxy forum. Is here nobody who had vaultwarden getting worked? :(

With best regards;
techsolo12

As I already explained a dozen times: This tutorial is about getting HAProxy up and running with a LE cert. I won't be helping here with service specific settings, issues and what not!

That beeing said... I am also running Vaultwarden and it is working fine. However I don't use its websockets as I simply have no need for the additional features.

So please find this one out by yourself and feel free to post the solution here. :)

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meelokun on November 14, 2022, 06:50:25 am

apologies, my tired eyes at 4am missed this part

Quote from: TheHellSite on May 31, 2021, 01:06:11 pm

• But if you would like to do it my way then you will need to create a virtual IP that is in a different subnet than any of your other networks. Preferably you would chose an IP that belongs to the localhost subnet in order to avoid IP conflicts in your local network.

i also misunderstand that setting a virtual IP that has a "loopback" does not automatically mean that it serves to "Loopback" to the firewall. that one was on me.

I updated the virtual ip to be 127.4.4.3 (mirroring your setup)

and updated SSL_Server to 127.4.4.3

HTTPS and HTTP Front Ends to listen on 127.4.4.3.

Updated unbound overide IP value to 10.0.1.1 (Firewall/OPNSense IP)

Restarted HAProxy - and its still not working. i wonder what i did wrong...

Update: Rebooted the firewall and that fixed it...

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 02, 2022, 06:55:54 am

Okay, I've been through the instructions at least 3 times and cannot find why it's not working. Can someone please take a look? Other than it being currently disabled, obviously.

Firewall rule is:
IPv4 TCP Src* Port* Dest WAN address Port AliasforHTTP/HTTPS Gateway* Schedule*

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

#
# NOTE: HAProxy is currently DISABLED
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch 1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend ()
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 15m

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_63859d8c6a7b81.10799804 ssl_fc

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_63859d8c6a7b81.10799804

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6385a4c7e68d06.81674833.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63859df5259306.89264162.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: homeassistant_backend ()
backend homeassistant_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server HomeAssistant 192.168.0.3:8123 check inter 30s port 8123

# Backend: web_backend ()
backend web_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server web 192.168.0.4:80

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: factorio_backend ()
backend factorio_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server factorio 192.168.0.17:80

# Backend: jira_backend ()
backend jira_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server jira 192.168.0.20:80

# Backend: meshcentral_backend ()
backend meshcentral_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server meshcentral 192.168.0.41:443

# Backend: nextcloud_backend ()
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nextcloud 192.168.0.38:443 ssl alpn h2,http/1.1 verify none


# statistics are DISABLED

And the mapping file, which I have tried with the full FQDN and without the periods as well.

Code: [Select]

# public access subdomains
hass. homeassistant_backend
factorio. factorio_backend
jira. jira_backend
mesh. meshcentral_backend
nextcloud. nextcloud_backend
web_backend

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 03, 2022, 09:27:22 pm

Quote from: Stingily7770 on December 02, 2022, 06:55:54 am

Okay, I've been through the instructions at least 3 times and cannot find why it's not working. Can someone please take a look? Other than it being currently disabled, obviously.

Firewall rule is:
IPv4 TCP Src* Port* Dest WAN address Port AliasforHTTP/HTTPS Gateway* Schedule*

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

#
# NOTE: HAProxy is currently DISABLED
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch 1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend ()
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 15m

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_63859d8c6a7b81.10799804 ssl_fc

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_63859d8c6a7b81.10799804

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6385a4c7e68d06.81674833.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63859df5259306.89264162.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: homeassistant_backend ()
backend homeassistant_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server HomeAssistant 192.168.0.3:8123 check inter 30s port 8123

# Backend: web_backend ()
backend web_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server web 192.168.0.4:80

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: factorio_backend ()
backend factorio_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server factorio 192.168.0.17:80

# Backend: jira_backend ()
backend jira_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server jira 192.168.0.20:80

# Backend: meshcentral_backend ()
backend meshcentral_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server meshcentral 192.168.0.41:443

# Backend: nextcloud_backend ()
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nextcloud 192.168.0.38:443 ssl alpn h2,http/1.1 verify none


# statistics are DISABLED

And the mapping file, which I have tried with the full FQDN and without the periods as well.

Code: [Select]

# public access subdomains
hass. homeassistant_backend
factorio. factorio_backend
jira. jira_backend
mesh. meshcentral_backend
nextcloud. nextcloud_backend
web_backend

1. No real error description. What is the error? What is not working?

2. Your map file seems off. Remove the dots after each service, I didn't have them in my example config so why do you have them in yours?

3. At the end of your mapfile, you have "web_backend" without any matching scheme before it.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 03, 2022, 10:44:19 pm

The dots were just an attempt to see if the default backend was an issue by listing my default in the map, but I wanted to be sure nothing else would conflict with the same 'startswith'. web_backend had my domain and I missed that it was left empty when I cleaned up PII. I have went through the config again, removing the periods and defining the default on the mapping rule again.

Rejected connections every time, but I know the traffic is making it because my port forward to an internal server I am using with Nginx Proxy Manager (and am seeking to replace with this setup) works when I disable HAProxy and put back its port forward. I've completely removed the port forward and rebooted the router to ensure that it isn't interfering. My Opnsense WebGUI port was already changed to 81.

If I attempt to browse to my IP from outside my network, http shows ERR_EMPTY_RESPONSE in Chrome, https shows ERR_CONNECTION_CLOSED.

One thing I find really odd is I'm not getting anything in the log file until I disable the service, then I see the stopping messages. I setup a health check that's working and writes to the log, but that was just more troubleshooting to be sure it was able to see my internal service from the router.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch 1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend ()
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 accept-proxy
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 15m

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_63859d8c6a7b81.10799804 ssl_fc

    # ACTION: HTTP_to_HTTPS
    http-request redirect scheme https code 301 if !acl_63859d8c6a7b81.10799804

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6385a4c7e68d06.81674833.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63859df5259306.89264162.txt,web_backend)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: homeassistant_backend ()
backend homeassistant_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server HomeAssistant 192.168.0.3:8123

# Backend: web_backend ()
backend web_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server web 192.168.0.4:80

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: factorio_backend ()
backend factorio_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server factorio 192.168.0.17:80

# Backend: jira_backend ()
backend jira_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server jira 192.168.0.20:80

# Backend: meshcentral_backend ()
backend meshcentral_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server meshcentral 192.168.0.41:443

# Backend: nextcloud_backend ()
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server nextcloud 192.168.0.38:443 ssl alpn h2,http/1.1 verify none

# Backend: gallery_backend ()
backend gallery_backend
    # health checking is DISABLED
    mode http
    balance source

    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server gallery 192.168.0.12:80



# statistics are DISABLED

SNI_Mapping

Code: [Select]

# public access subdomains
hass homeassistant_backend
factorio factorio_backend
jira jira_backend
mesh meshcentral_backend
nextcloud nextcloud_backend
gallery gallery_backend

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 04, 2022, 01:32:38 am

Quote from: Stingily7770 on December 03, 2022, 10:44:19 pm

If I attempt to browse to my IP from outside my network, http shows ERR_EMPTY_RESPONSE in Chrome, https shows ERR_CONNECTION_CLOSED.

If you don't even get any 503s with a blank white Page and the HAProxy Log is not indicating any traffic, then your firewall rule is configured wrong.

Also if you are not willing to share the HAProxy log then I am unable to help. You have to set it to "Informational" in the top right corner!

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 04, 2022, 01:52:45 am

Quote from: TheHellSite on December 04, 2022, 01:32:38 am

If you don't even get any 503s with a blank white Page and the HAProxy Log is not indicating any traffic, then your firewall rule is configured wrong.

Also if you are not willing to share the HAProxy log then I am unable to help. You have to set it to "Informational" in the top right corner!

Shouldn't the HAProxy log show startups as well? This is fresh after a reboot (empty log), restarting the HAProxy service from System, Diagnostics, Services (still empty), then unchecking Enable HAProxy and applying (Stop messages appear), then rechecking Enable HAProxy and applying (nothing new added).

Confirmed haproxy is listening. I've also tried the DNS redirection to my opnsense internal IP with the same results from inside the network.

Code: [Select]

root@OPNsense:~ # sockstat -l | grep '443\|80'
www      haproxy    3539  4  tcp4   *:443                 *:*
www      haproxy    3539  5  tcp4   *:80                  *:*
www      haproxy    3539  6  tcp4   127.4.4.3:80          *:*
www      haproxy    3539  7  tcp4   127.4.4.3:443         *:*

I've also compared the firewall policy once again, and it is exactly like https://postimg.cc/VS3DKGPg other than I named my alias HTTP_HTTPS.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 04, 2022, 03:26:45 pm

Please just post the overview of YOUR WAN rules page and YOUR wan rule. Troubleshooting with pictures of my tutorial won't get you any further...
If you don't expose your WAN IP or public domain name then you have nothing to worry about.

Again since your HAProxy log is empty this means that there is NO traffic reaching HAProxy and maybe not even your firewall. You would have to check the firewall logs for this though.

Also is there another firewall / router placed before your OPNsense (double NAT)?

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 04, 2022, 05:19:18 pm

Ignore the HTTP_HTTPS Port Forward. I've been removing it and rebooting to ensure a clean test, but didn't want to mess with it this morning. The port forwards all work though, so the firewall is definitely receiving the traffic. There's an ATT Gateway in front of my router in passthrough mode.

Also grabbed a packet capture of the traffic, after disabling the port forward and confirming haproxy is bound to 80/443. Phone was off the network and packet capture was filtered to its public ip.

Code: [Select]

10:15:09.123024 IP PHONE.25700 > ROUTER.443: Flags [SEW], seq 642215500, win 65535, options [mss 1460,nop,wscale 12,sackOK,TS val 2478600365 ecr 0], length 0
10:15:09.123057 IP ROUTER.443 > PHONE.25700: Flags [S.E], seq 639098840, ack 642215501, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 3059434307 ecr 2478600365], length 0
10:15:09.141384 IP PHONE.25700 > ROUTER.443: Flags [.], ack 1, win 256, options [nop,nop,TS val 2478600384 ecr 3059434307], length 0
10:15:09.142758 IP PHONE.25700 > ROUTER.443: Flags [P.], seq 1:518, ack 1, win 256, options [nop,nop,TS val 2478600384 ecr 3059434307], length 517
10:15:09.142790 IP ROUTER.443 > PHONE.25700: Flags [.], ack 518, win 510, options [nop,nop,TS val 3059434328 ecr 2478600384], length 0
10:15:09.142818 IP ROUTER.443 > PHONE.25700: Flags [F.], seq 1, ack 518, win 514, options [nop,nop,TS val 3059434328 ecr 2478600384], length 0
10:15:09.161122 IP PHONE.25700 > ROUTER.443: Flags [.], ack 2, win 256, options [nop,nop,TS val 2478600404 ecr 3059434328], length 0
10:15:09.161149 IP PHONE.25700 > ROUTER.443: Flags [F.], seq 518, ack 2, win 256, options [nop,nop,TS val 2478600404 ecr 3059434328], length 0
10:15:09.161163 IP ROUTER.443 > PHONE.25700: Flags [.], ack 519, win 513, options [nop,nop,TS val 3059434346 ecr 2478600404], length 0
10:15:15.141831 IP PHONE.26438 > ROUTER.443: Flags [SEW], seq 3285634286, win 65535, options [mss 1460,nop,wscale 12,sackOK,TS val 4185299120 ecr 0], length 0
10:15:15.141883 IP ROUTER.443 > PHONE.26438: Flags [S.E], seq 4283526657, ack 3285634287, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 95186048 ecr 4185299120], length 0
10:15:15.160570 IP PHONE.26438 > ROUTER.443: Flags [.], ack 1, win 256, options [nop,nop,TS val 4185299139 ecr 95186048], length 0
10:15:15.161943 IP PHONE.26438 > ROUTER.443: Flags [P.], seq 1:518, ack 1, win 256, options [nop,nop,TS val 4185299139 ecr 95186048], length 517
10:15:15.161977 IP ROUTER.443 > PHONE.26438: Flags [.], ack 518, win 510, options [nop,nop,TS val 95186067 ecr 4185299139], length 0
10:15:15.162008 IP ROUTER.443 > PHONE.26438: Flags [F.], seq 1, ack 518, win 514, options [nop,nop,TS val 95186067 ecr 4185299139], length 0
10:15:15.181057 IP PHONE.26438 > ROUTER.443: Flags [.], ack 2, win 256, options [nop,nop,TS val 4185299159 ecr 95186067], length 0
10:15:15.181181 IP PHONE.26438 > ROUTER.443: Flags [F.], seq 518, ack 2, win 256, options [nop,nop,TS val 4185299159 ecr 95186067], length 0
10:15:15.181199 IP ROUTER.443 > PHONE.26438: Flags [.], ack 519, win 513, options [nop,nop,TS val 95186086 ecr 4185299159], length 0


Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 04, 2022, 10:20:24 pm

Quote from: Stingily7770 on December 04, 2022, 01:52:45 am

Shouldn't the HAProxy log show startups as well? This is fresh after a reboot (empty log), restarting the HAProxy service from System, Diagnostics, Services (still empty), then unchecking Enable HAProxy and applying (Stop messages appear), then rechecking Enable HAProxy and applying (nothing new added).

HAProxy shouldn't even print a stop message in the haproxy log at all. Only if there are errors, f.e. misconfiguration of your firewall.

After enabling HAProxy and hitting "Apply" then waiting for 5sec and reloading the HAProxy settings page. Is there a green Play icon in the top right corner when you are on the HAProxy Settings page?

If not, then you have something misconfigured or another service is listening on the same ip:port.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 05, 2022, 12:21:54 am

Yes, the green play button is there.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 05, 2022, 01:16:17 am

Please reboot the firewall, then post an updated haproxy config export and haproxy log export (after trying to access your services using an FQDN) .

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 05, 2022, 01:29:26 am

Log is empty. Green play button is present.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 05, 2022, 01:40:36 am

I strongly believe it is not empty... AGAIN You have to set it to informational.
Post a screenshot of your PUBLIC_SUBDOMAINS_map-rule.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 05, 2022, 02:26:20 am

Here's my log page and settings and the map-rule.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 05, 2022, 03:55:45 pm

Based on this the traffic is clearly NOT reaching HAProxy. Why? I can't tell you.
You will have to troubleshoot here on your own. Your HAProxy config and wan firewall rule looks fine.

I can only imagine that you have some leftover port forwards or whatever that are intercepting the traffic.
If you enable the logging on the WAN "HAProxy" rule and go to Firewall --> Log Files --> Live View and filter for the WAN rule... You should see the traffic beeing green.
However this does not guarantee that any old port forward rules are interfering.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Stingily7770 on December 05, 2022, 05:07:01 pm

Alright. Thanks for the attempt.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 06, 2022, 12:32:58 am

Well there has to be something wrong with either you ATT modem or you overall OPNsense settings.

Since the SNI_frontend is listening on all IPs, interfaces and 80+443 port (0.0.0.0:80+443) it should at least spit out something to the HAProxy logs when there is a connection to your OPNsense WAN IP on any of the two ports. Even if the SSL handshake or anything after it fails, the SNI_frontend will always make log entries.

If it doens't then there is something wrong with your OPNsense in general or some other network device in your WAN facing setup.

Take a look at my HAProxy log example. The first hit always goes to the SNI_frontend, from there to the SSL_server and is then catched by the HTTP(S)_frontend.

Code: [Select]

2022-12-06T00:23:39 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:34677 to OPNSENSE_WAN_IP:443 (1_HTTPS_frontend/HTTP)
2022-12-06T00:23:39 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:34677 to OPNSENSE_WAN_IP:443 (0_SNI_frontend/TCP)
2022-12-06T00:23:39 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:9659 to OPNSENSE_WAN_IP:443 (1_HTTPS_frontend/HTTP)
2022-12-06T00:23:39 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:9659 to OPNSENSE_WAN_IP:443 (0_SNI_frontend/TCP)
2022-12-06T00:23:03 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:62798 to OPNSENSE_WAN_IP:443 (1_HTTPS_frontend/HTTP)
2022-12-06T00:23:03 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:62798 to OPNSENSE_WAN_IP:443 (0_SNI_frontend/TCP)
2022-12-06T00:23:01 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:62797 to OPNSENSE_WAN_IP:443 (1_HTTPS_frontend/HTTP)
2022-12-06T00:23:01 Informational haproxy Connect from REMOTE_CLIENT_PUBLIC_IP:62797 to OPNSENSE_WAN_IP:443 (0_SNI_frontend/TCP)

You really should consider testing if your HTTP(S) WAN rule is working properly (see my previous reply) and or if there is any other service on your OPNsense or network device in general messing with the data stream.

I am sadly unable to help here since the traffic is not hitting your firewall and or haproxy.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: vladnik on December 15, 2022, 01:33:46 pm

Hey all,

first of all, thank you for the amazing guide @TheHellSite, I have set it up over a year ago and everything works like a charm.

Today, I have a small problem. I have a Mikrotik switch that can only be accessed via Port 80 and requires HTTP basic auth.
I have setup my backend and map file like always, the site is reachable fine, however I cannot login. I enter my credentials and press enter, and the dialog for entering my credentials just shows back up infinitely. The credentials are correct, if I go to the switch via IP-address I can log in normally.

(https://i.ibb.co/5WTzqPb/haproxy-mikrotik.png)

I have tried adding a user and password to 'User management' & ticking the box for Basic auth in the backend and selecting my user, no luck sadly. I'm pretty sure I just have to add an option somewhere to pass the auth header, however, I can't figure it out. Googling for the problem just leads to info on how to setup HAproxy to do basic auth, which I don't need... The logs also don't seem to show anything useful.

Anyone has any ideas? Thanks in advance.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 15, 2022, 02:09:56 pm

Quote from: vladnik on December 15, 2022, 01:33:46 pm

I have tried adding a user and password to 'User management' & ticking the box for Basic auth in the backend and selecting my user, no luck sadly. I'm pretty sure I just have to add an option somewhere to pass the auth header, however, I can't figure it out. Googling for the problem just leads to info on how to setup HAproxy to do basic auth, which I don't need... The logs also don't seem to show anything useful.

Anyone has any ideas? Thanks in advance.

The user management in HAProxy has nothing to do at all with any login forms of services that are behind HAProxy! You can use this to add a login form that pops up before the client can even connect to the service that is behind HAProxy. So unrelated to your issue.

Apart from that please ask in the official HAProxy forums about your issue since it is not related to my tutorial.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: vladnik on December 15, 2022, 02:21:49 pm

Quote from: TheHellSite on December 15, 2022, 02:09:56 pm

The user management in HAProxy has nothing to do at all with any login forms of services that are behind HAProxy! You can use this to add a login form that pops up before the client can even connect to the service that is behind HAProxy. So unrelated to your issue.

Figured as much. Was worth a try though.

Quote from: TheHellSite on December 15, 2022, 02:09:56 pm

Apart from that please ask in the official HAProxy forums about your issue since it is not related to my tutorial.

Alright, will do. Thanks anyways!

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: EuleMitKeule on December 25, 2022, 03:34:26 pm

I know this is not really part of the (great) tutorial, but I wanted to ask if it is possible to change the SSL certificate that is being used based on whether the traffic is local or public. I already have setup the rule for subdomains that are only accessible from local IPs.

The reason for this is that I want to enable Full (Strict) mode in Cloudflare. That means I have to use the Cloudflare Origin Server Certificate for public access to my HAProxy. I already uploaded the certificate to OPNsense and selected it along with the Let's Encrypt certificate for the HTTPS frontend. However it seems only the LE certificate is being used, so public access via Cloudflare fails. I looked for an HAProxy function that chooses a specific certificate, but it does not seem to exist.

Can anyone point me in the right direction?

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on December 25, 2022, 07:44:32 pm

Use different set of frontends to handle it.
For example, my current setup is
WAN_TCP_frontend->WAN_SSL_frontend, with 192.168.5.1, 192.168.6.1 as listening IP
LAN_TCP_frontend->LAN_SSL_frontend, with 192.168.7.1 and 192.168.8.1 as listening IP

WAN NAT port forward to 192.169.5.1
While Unbound DNS overwrite domain name to 192.168.7.1

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DeWilde on December 28, 2022, 01:50:44 pm

Hello,

and thank you for this great tutorial!
I managed to get the HAProxy running and I am able to access the desired services from the internet side.
But when i want to access them from the internal netwerk i am unable to reach them.
i tried the DNS override as explained but this does not work. I suppose i am doing something wrong here.
As far as I understood, i must create an "override" for each host i also want to reach internally and assign it the IP adres of the LAN interface of the OPNsense. Is that correct?
My default LAN interface has "192.168.10.1/24" so i created a host override f.e. "mynas.mydomain.com" pointing to 192.168.10.1.
I assume the HAProxy is also listening on the LAN interface?

thank you for your help!

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on December 28, 2022, 10:48:41 pm

Quote from: DeWilde on December 28, 2022, 01:50:44 pm

I assume the HAProxy is also listening on the LAN interface?

If configured correctly, yes. However no way to know since...

Quote from: DeWilde on December 28, 2022, 01:50:44 pm

thank you for your help!

No logs, no haproxy config export, no other details...
Until provided, no help.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DeWilde on December 29, 2022, 12:16:31 pm

Hi, attached you can find my haproxy.conf and map file.
I replicated your tutorial 1:1

The OPNsense firewall LAN IP is 192.168.10.1
for the 2 items in the map file i created a DNS override for those FQDN's pointing to 192.168.10.1

access from the internet works fine. The wildcard ssl cert is being used and the port redirection works and is not visible. (ex :55443 for the firewall interface and :5000 for the NAS)

internally the browser says "ERR_CONNECTION_TIMED_OUT" and nothing is displayed.

If you need more info or details i'll be happy to provide them to you.
thank you for your help!

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: guest36292 on December 30, 2022, 12:47:08 am

Hey y'all,

First of all, truly appreciate the energy you put into this thorough guide @TheHellSite! Managed to make everything work, thank you!

There's one thing I can't sort out and your guide doesn't touch the topic either. HAProxy does not forward client IP to the server as expected. The server in this case is a Synology NAS which is only aware of the OPNSense IP if the connection is made via the reverse proxy.

Is this an expected behavior or could I have some settings wrong? My settings follow your guide entirely and the "X-Forwarded-For header" checkboxes are checked in both HTTP and HTTPS Frontends under the Public Services tab. Is it possible that the SNI Frontend doesn't forward the client IP "downstream" so the other two can't properly forward it?

I have found some forum posts that explain how this should be working in the HAProxy config file but I can't really translate that into the user interface as I lack the appropriate level of knowledge here.
https://access.redhat.com/solutions/3552581 (https://access.redhat.com/solutions/3552581)
https://serverfault.com/questions/722151/haproxy-how-to-append-client-ip-in-x-client-ip-and-x-forwarded-for-headers (https://serverfault.com/questions/722151/haproxy-how-to-append-client-ip-in-x-client-ip-and-x-forwarded-for-headers)
https://community.synology.com/enu/forum/1/post/150860?reply=488269 (https://community.synology.com/enu/forum/1/post/150860?reply=488269)
https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/haproxy_pass_clientip_to_webserver (https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/haproxy_pass_clientip_to_webserver)

I was hoping you guys could help me find a solution to this that can be achieved via the OPNSense UI.

Thank you in advance!
------------------

Update:
So anyone bumping into this issue on a Synology NAS, know it can be easily resolved. Apparently the server should be aware that a reverse proxy is forwarding the connections so it can resolve the original client IP.
The solution to my problem was posted here: https://www.reddit.com/r/synology/comments/mmubnv/tip_for_using_dsm_reverse_proxy_logging_correct/ (https://www.reddit.com/r/synology/comments/mmubnv/tip_for_using_dsm_reverse_proxy_logging_correct/)

Quote

In the DSM control panel, go to security, at the bottom is "trusted proxies". Add the dsm ip address HAProxy IP address, and boom! The correct external IP address is logged at a connection attempt and you will get notified about new login behavior if you have that turned on, and ip address blocking should now work if you have that turned on.

------------------

Update 2:
Apparently this above solution comes with a caveat, which may not make it a satisfying solution at all.
In order to establish connection between the client and the Synology the proxy must be allowed in the Synology's firewall. And as it appears even though the client IP is passed on the connection itself is made through the proxy's IP address. Even if the client's IP is specifically blocked, through the the proxy it is allowed to connect to every service the proxy is allowed to.

This effectively renders the the Synology Firewall useless.

I'm going to open a new topic for this issue for better visibility.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mintchipmadness on January 07, 2023, 11:49:57 pm

Hello and thank you very much for this guide. I am definitely learning a lot as I am setting up HAProxy on my opensense for public and local access. I have hopefully a quick question. Does this setup work to access the opnsense GUI or is that a special item that I need to setup? I am still troubleshooting but I'd thought I'd ask just in case. Thank you for your help.

Edit: Addition domain name structure information for my setup and some additional troubleshooting

Public Services: service.example.com
Internal Services: service.internal.example.com

I have been troubleshooting my setup and had a small breakthrough with my unbound settings. Under the general settings of unbound I needed to check "Do not register system A/AAAA records" to get the overrides to work when I ping. Before all my interfaces were getting registered under the firewall domain name causing a random interface IP to get pulled when I pinged opnsense.local.example.com of the firewall. Only certain IPs are allowed to access the GUI from my lan. Now my override works appropriately.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mike0000 on January 08, 2023, 01:06:23 am

TheHellSite, thanks a lot for all the work that you've put into this tutorial. I have followed every step of it and almost everything is working well.

One issue I am facing is that when I ping a local domain (e.g. opnsense.mydomain.com (router/fw box), nas.mydomain.com (qnap nas)) the IP gets resolved as my external WAN IP address.

When I direct my browser at one of my internal domains I'm not getting the same result as when I simply go to the corresponding IP.

I am not sure whether there is a misconfiguration in the HAProxy setup, or whether it is in fact unrelated to your tutorial. Below are the outputs I see from accessing two different subdomains from both my phone on cell service, as well as from internal wifi:

When accessing my OPNSense webui on opnsense.mydomain.com:55443 (internal IP 192.168.5.1):

• Accessing via browser (Firefox/Chrome/Safari (iPhone) I'm getting the following error: "The connection has timed out. An error occurred during a connection to opnsense.mydomain.com:55443" (Firefox); "This site can’t be reached. opnsense.mydomain.com took too long to respond." (Chrome); "Safari could not open the page because the server stopped responding." (Safari iPhone)

When accessing my QNAP NAS on nas.mydomain.com (internal IP 192.168.5.60):

• Accessing via browser (Firefox/Chrome/Safari (iPhone) I'm getting the following error: "503 Service Unavailable. No server is available to handle this request." This suggests to me the http server is actually responding to the request but not serving an SSL page, even though https://192.168.5.60 returns the NAS login page.

• Testing the server response from shell on the nas [edit1]:

Code: [Select]

user@OPNsense:~ $ wget --save-headers http://nas.mydomain.com:8080
--2023-01-08 12:47:22--  http://nas.mydomain.com:8080/
Resolving nas.mydomain.com (nas.mydomain.com)... 192.168.5.60
Connecting to nas.mydomain.com (nas.mydomain.com)|192.168.5.60|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 580 [text/html]
Saving to: 'index.html'
index.html 100%[=====================================================================================>] 580  --.-KB/s    in 0s
2023-01-08 12:47:22 (139 MB/s) - 'index.html' saved [580/580]
When checking the response on https request an error comes (with the '--no-check-certificate' parameter yields the same result/output as the http request):

Code: [Select]

user@OPNsense:~ $ wget --save-headers https://nas.mydomain.com
--2023-01-08 13:01:21--  https://nas.mydomain.com/
Resolving nas.mydomain.com (nas.mydomain.com)... 192.168.5.60
Connecting to nas.mydomain.com (nas.mydomain.com)|192.168.5.60|:443... connected.
ERROR: cannot verify nas.mydomain.com's certificate, issued by 'CN=R3,O=Let\'s Encrypt,C=US':
  Unable to locally verify the issuer's authority.
To connect to nas.mydomain.com insecurely, use `--no-check-certificate'.

There seem to be two issues here: 1) there is a certificate error (I imported the acme/LE wildcard .crt and .key into the NAS).

But 2) more generally as observed with the pings thrown at the WAN IP rather than the correct internal IP the request from the browser is also being forwarded to the WAN (mydomain.com) - this is shown by wget on mydomain.com which also returns a 503, the same that the browser does:

Code: [Select]

user@OPNsense:~ $ wget --save-headers https://mydomain.com
--2023-01-08 13:13:59--  https://mydomain.com/
Resolving mydomain.com (mydomain.com)... 185.176.xxx.xxx [WAN IP]
Connecting to mydomain.com (mydomain.com)|185.176.xxx.xxx [WAN IP]|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2023-01-08 13:13:59 ERROR 503: Service Unavailable.
Any ideas or can you recommend any tools to do further troubleshooting or does anyone spot what the issue is?

The purpose of my setup is that all subdomains should only be accessible from the LAN or through VPN (this is set up correctly, I can VPN in via OpenVPN).

-------------------------

edit1: inserted code view of wget header response for nas.mydomain.com access - seems to be working in shell but not from browser

-------------------------
Many thanks
Michael

Attached the config files requested
- HAProxy Config Export
- HAProxy errors and/or log entries
- Details about setup: above, but happy to elaborate further if unclear

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: dirkscheck on January 08, 2023, 04:17:49 pm

@TheHellSide
First thank you for this wonderful guide. I learned a lot of and it helps to understand everything a little bit better.
At the moment I play around with these features to understand it even better.

My setup is a bit different so I need to play around with it and see what happens.

At the moment lets encrypt is working and HAProxy is configured. And here I have some questions in general:

1. Why do I have to open port 80 and 443 when using DNS-01 challenge ? I thought that is NOT needed and that was the reason why I choose DNS01
2. I DIDNT make the internal procedure BUT I can access the configured backend internally without any problems via the name !!! BUT
a. in Safari i get an ssl certificate (it is a self signed ssl certificate from my router)
b. in firefox it is still unsecured

Why is that ? If I have to send to you some information please let me know but I think that are general questions and no files / screenshots are needed or ?

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 09, 2023, 12:24:34 am

Quote from: DeWilde on December 29, 2022, 12:16:31 pm

My default LAN interface has "192.168.10.1/24" so i created a host override f.e. "mynas.mydomain.com" pointing to 192.168.10.1.
I assume the HAProxy is also listening on the LAN interface?

Yes, your OPNsense LAN IP is the correct DNS Override target, as explained in the tutorial.
Yes, HAProxy is also listening on that interface since the SNI_frontend is listening on ALL IPs:Ports (0.0.0.0:0).

Quote from: DeWilde on December 29, 2022, 12:16:31 pm

Hi, attached you can find my haproxy.conf and map file.
I replicated your tutorial 1:1

The OPNsense firewall LAN IP is 192.168.10.1
for the 2 items in the map file i created a DNS override for those FQDN's pointing to 192.168.10.1

access from the internet works fine. The wildcard ssl cert is being used and the port redirection works and is not visible. (ex :55443 for the firewall interface and :5000 for the NAS)

internally the browser says "ERR_CONNECTION_TIMED_OUT" and nothing is displayed.

If you need more info or details i'll be happy to provide them to you.
thank you for your help!

If access from external network is working fine, then your DNS override isn't taking effect.
Check this using nslookup/ping from any PC within your network.
Also check if you are actually using Unbound as your DNS resolver.

Then there is your HAProxy config...
Your SFINX and MYNAS both have the exact same IP:Port in the server settings, double check this.
BTW: I would never expose my OPNsense Web UI externally, but this is up to you.
Apart from that the HAProxy config looks fine.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 09, 2023, 12:33:54 am

Quote from: mike0000 on January 08, 2023, 01:06:23 am

TheHellSite, thanks a lot for all the work that you've put into this tutorial. I have followed every step of it and almost everything is working well.

One issue I am facing is that when I ping a local domain (e.g. opnsense.mydomain.com (router/fw box), nas.mydomain.com (qnap nas)) the IP gets resolved as my external WAN IP address.

If you also followed "Part 6 - Access from internal networks" of my guide this should be working.
If not, then check if the DNS overrides are working using nslookup/ping.

Oh and your HAProxy config looks wrong / incomplete.
There are no actual services configured (server+backend).

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 09, 2023, 12:46:21 am

Quote from: dirkscheck on January 08, 2023, 04:17:49 pm

@TheHellSide
First thank you for this wonderful guide. I learned a lot of and it helps to understand everything a little bit better.
At the moment I play around with these features to understand it even better.

My setup is a bit different so I need to play around with it and see what happens.

At the moment lets encrypt is working and HAProxy is configured. And here I have some questions in general:

1. Why do I have to open port 80 and 443 when using DNS-01 challenge ? I thought that is NOT needed and that was the reason why I choose DNS01
2. I DIDNT make the internal procedure BUT I can access the configured backend internally without any problems via the name !!! BUT
a. in Safari i get an ssl certificate (it is a self signed ssl certificate from my router)
b. in firefox it is still unsecured

Why is that ? If I have to send to you some information please let me know but I think that are general questions and no files / screenshots are needed or ?

1. Maybe because the internet works almost entirely over TCP ports 80(HTTP) and 443(HTTPS)?
So if we don't open WAN port 80+443 you will never be able to access your services over your WAN IP.
If you would have understood what you are doing, which you actually should have since I explained it pretty detailed in the tutorial, you would know that 80 is only open so any unencrypted traffic hitting port 80 gets redirected to the encryption required port 443.

2. I am assuming there is another router/firewall in front of your OPNsense that handles NAT, DNS and DHCP.
"My setup is a bit different so I need to play around with it and see what happens."
See my answer to your 3rd question.

3. "Why is that ? If I have to send to you some information please let me know but I think that are general questions and no files / screenshots are needed or ?"
WRONG.
You are stating that your setup is a bit different and then ask a question about some issues because you didn't follow the tutorial which is why you are probably having these issues in the first place.
I am pretty confused.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DeWilde on January 10, 2023, 09:46:16 am

@TheHellSide

MYNAS vs SFINX was for testing purpose because i was doubting if i configured something wrong. So i created MYNAS.
About OPNsense GUI being accessible from the internet, i completely understand your concern.
I was looking for a webinterface to test the config with, but indeed, not realy a good thing to do.

Some extra troubleshooting.
Interfaces: Diagnostics: DNS Lookup

Code: [Select]

host: feniks.domain.net
server: 192.168.10.1
response: A feniks.domain.net. 3600 IN A 192.168.10.200 192.168.10.1 0 msec

Interfaces: Diagnostics: Trace Route

Code: [Select]

# /usr/sbin/traceroute -w 2 -n  -m '18'  'feniks.domain.net'
traceroute to feniks.domain.net (192.168.10.200), 18 hops max, 40 byte packets
 1  192.168.10.200  0.787 ms  0.462 ms  0.475 ms

on network client:

Code: [Select]

Pinging feniks.domain.net [192.168.10.200] with 32 bytes of data:
Reply from 192.168.10.200: bytes=32 time<1ms TTL=64


Code: [Select]

Tracing route to feniks.domain.net [192.168.10.200]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  FENIKS.domain.net [192.168.10.200]

Trace complete.

so now i'm lost  ???
i'll try to find/set-up another internal website to test with.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 10, 2023, 08:34:29 pm

Quote from: DeWilde on January 10, 2023, 09:46:16 am

Some extra troubleshooting.
Interfaces: Diagnostics: DNS Lookup
host: feniks.domain.net
server: 192.168.10.1
response: A   feniks.domain.net. 3600 IN A 192.168.10.200   192.168.10.1   0 msec

so now i'm lost  ???
i'll try to find/set-up another internal website to test with.

Please use codeboxes when posting such results. This makes it a lot easier to read them!

Your issue is mostlikely related to misconfigured DNS overwrites or another DNS resolver that is controling the DNS replies in your local network.

Question 1: Is Unbound your only DNS resolver in your network or are you running something like piHole?

Question 2: Are your client devices (f.e. iPhone, Notebook, ...) using your OPNsense as their DNS resolver inside your network or are they configured to use something like Google DNS, Cloudflare DNS, AdGuard, ...?

Also please post/attach screenshots of your configured DNS overwrites.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DeWilde on January 20, 2023, 10:50:19 am

Hi,
as far as i am aware of, Unbound is my primary DNS resolver.
I do have Zenarmor installed on the OPNsense. But this is only a web filter, not a DNS resolver.

i checked my smartphone, laptop, kids computer, ... all of them are using OPNsense as there DNS resolver.

I attached some screenshots for you.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DeWilde on January 20, 2023, 10:50:51 am

and some more  ;)

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 20, 2023, 11:32:02 am

Actually I shouldn't offer this free support since you are clearly requesting it for business use.
That being said...

If I where you, I would remove "opnsense_04.jpg" asap from your post!
I just got direct access to your "opnsense.yourdomain.com" and was presented with the webinterface. Seriously don't expose it via HAProxy. Use WireGuard for this!!!

However, this screenshot also might point out your issue. Please try and write the hostnames in the host overrides in lowercase letters only.
Your "opnsense" override (lowercase) is working, but none of the others (all uppercase).

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: AlexisM on January 20, 2023, 07:48:33 pm

Many thank's for your tutorial. It real help me.

I'd need to throughout for my Synology Nas so I use the informations found in your topic , https://forum.opnsense.org/index.php?topic=18538.msg84958#msg84958 and https://forum.opnsense.org/index.php?topic=22630.msg118934#msg118934

I don't need to throughout admin console of my Nas but the services with port 433 (exemple https://drive.xxxx.synology.me, https://video.xxxx.synology.me etc.)

That I'm doing in completion of your tutorial (in order):

• HAProxy plugin: Create real server "nas_synology" with is local ip and port 443
• HAProxy plugin: Create backend "nas_synology_backend" with "nas_synology" with TCP (Layer 4)
• HAProxy plugin: Create "Condition" (enter name ["traffic_ssl"], condition type is "custom condition (option pass-through)" with value "req_ssl_hello_type 1")
• HAProxy plugin: Create Condition "sni_synology_me", condition "SNI TLS extension ends with (TCP request content inspection), suffixe SNI ".synologe.me"
• HAProxy plugin: Create "Rule" (enter name ["request_inspect_delay"], select no condition, function is "tcp-request inspect delay" with value "5s" or whatever suits you)
• HAProxy plugin: Create "Rule" (enter name ["request_content_accept_ssl"], select condition of 3 ["traffic_ssl"], function is "tcp-request content accept")
• HAProxy plugin: Create "Rule" (enter name ["sni_synology_me-rule"], select condition "SNI_synology_me", execute function : "use backend", Backend :"Synology_backend"
• HAProxy plugin: Modify in "Public service" (service named ["0_SNI_frontend"], select the 3 rules "request_inspect_delay", "request_content_accept_ssl" and "throughout_ssl_map_domain" and you can choose Nothing for Backend Pool by Default

Then, when I'm going with my mobile device to "plex.mydomain.com", it use backend with SSL from OpnSense
And when I use "drive.xxxx.synology.me", it throughout the ssl and use SSL from my Synology NAS

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 20, 2023, 07:51:21 pm

Please provide haproxy config export

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: AlexisM on January 20, 2023, 10:56:09 pm

Oops. In fact, 'Create map file "throughout_ssl_map_domain" with content :' doesn't work (because SNI work on TCP).

Replace : "HAProxy plugin: Create map file "throughout_ssl_map_domain" with content : ..."
By Create Condition "SNI_synology_me", condition "SNI TLS extension ends with (TCP request content inspection), suffixe SNI ".synologe.me"

Change : HAProxy plugin: Create "Rule" (enter name ["sni_throughout_ssl-rule"], select condition "SNI_synology_me", execute function : "use backend", Backend :"Synology_backend"

*** haproxy config export :

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 500

# autogenerated entries for ACLs

# autogenerated entries for config in backends/frontends

# autogenerated entries for stats

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    # tuning options
    timeout client 30s

    # logging options
    option tcplog
    # ACL: traffic_ssl
    acl acl_63c840bdd3f440.07842774 req_ssl_hello_type 1
    # ACL: SNI_synology_me
    acl acl_63c826ed0527a7.29957165 req.ssl_sni -m end -i .synology.me

    # ACTION: request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5
    # ACTION: request_content_accept_ssl
    tcp-request content accept if acl_63c840bdd3f440.07842774
    # ACTION: sni_throughout_ssl-rule
    use_backend Synology_backend if acl_63c826ed0527a7.29957165

# Frontend: 1_HTTP_frontend (Listening on 127.74.0.0:80)
frontend 1_HTTP_frontend
    bind 127.74.0.0:80 name 127.74.0.0:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_63c813f73e3ac8.56482289 ssl_fc

    # ACTION: HTTP_to_HTTPS-rule
    http-request redirect scheme https code 301 if !acl_63c813f73e3ac8.56482289

# Frontend: 1_HTTPS_frontend (Listening on 127.74.0.0:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 127.74.0.0:443 name 127.74.0.0:443 accept-proxy ssl prefer-client-ciphers ssl-min-ver TLSv1.2 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63c817f31748b0.16739019.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63c814d7b1ebe0.58772734.txt)]

# Backend: OpnSense_backend (OpnSense Pool)
backend OpnSense_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Opnsense 192.168.74.1:444 ssl verify required ca-file /etc/ssl/cert.pem

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.74.0.0 send-proxy-v2 check-send-proxy

# Backend: Synology_backend ()
backend Synology_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server Synology 192.168.74.4:443 ##

# statistics are DISABLED

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 20, 2023, 11:44:15 pm

I am unable to help here. Please ask the people that already did the things you mentioned.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on January 21, 2023, 08:01:28 pm

Quote from: TheHellSite on January 20, 2023, 11:44:15 pm

I am unable to help here. Please ask the people that already did the things you mentioned.

Well, I guess he is not asking question, but to update how he manage redirecting package to NAS in TCP mode by adding conditions and rules for recognizing SNI

(I have read his config and compare with mine one, and guess his config should be working flawlessly)

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: AlexisM on January 24, 2023, 12:12:52 pm

yes  ;)

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brynjolm on February 01, 2023, 07:07:39 am

Hello again Mr.Hellsite the guide you provided has been rock solid for a year now. no hiccups or whatsoever. Im writing back in this post because i wanted to know exactly what you meant on NR.6 on the faq page as im interested in managing a traefik instance behind haproxy

Quote

How can we load balance TCP traffic that we don't want to get SSL offloaded, f.e. OpenVPN over TCP?
In my tutorial I only explain how to "redirect+load balance SSL offloaded traffic".
This is because I myself don't have (yet) the need to actually load balance any non SSL traffic.
However balancing non SSL traffic is pretty much the same as balancing SSL traffic.
You only have to make sure that your "NOSSLservice_rule" or "NOSSLservices_map-file_rule" is placed on the "SNI_frontend" instead of the "HTTPS_frontend" and that the backend that belongs to your

Would this kind of setup be applicable to do traefik behind haproxy? also what do you exactly mean by NOSSL_service_rule NOSSL_services_map_file_rule?

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 03, 2023, 05:06:38 pm

Please refer to this post about it. Be warned I can not provide help for this since I am not using such a setup.

https://forum.opnsense.org/index.php?topic=18538.msg84958#msg84958

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: tomdh76 on February 04, 2023, 08:19:08 pm

Thx you @TheHellSite for this tutorial. Unfortunately I cannot get it to work. I always get a "503 service unavaible status"

Here is my config

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 1_HTTP_Frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_Frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_63de5470175f22.54470191 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_63de5470175f22.54470191

# Frontend: 0_SNI_Frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_Backend

    # logging options

# Frontend: 1_HTTPS_Frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63de597c094f01.72503480.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63de5520a92049.75714996.txt)]

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: BITWARDEN_backend ()
backend BITWARDEN_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server BITWARDEN 192.168.2.55:80 ssl verify none

# Backend: CALIBRE_backend ()
backend CALIBRE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server CALIBRE 192.168.2.40:8083 ssl verify none



# statistics are DISABLED
If I use the internal addresses of the websites (192.168.2.40:8083 for example) I can login. But using the website.domain.com I get a 503 error. In the logs I see that the client is accessing the public Ip on port 443.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 04, 2023, 09:48:07 pm

Quote from: tomdh76 on February 04, 2023, 08:19:08 pm

Thx you @TheHellSite for this tutorial. Unfortunately I cannot get it to work. I always get a "503 service unavaible status"

If I use the internal addresses of the websites (192.168.2.40:8083 for example) I can login. But using the website.domain.com I get a 503 error. In the logs I see that the client is accessing the public Ip on port 443.

Please give further details on what is and whet it is not working.

Are you able to access your services via their domain name from a device outside of your local network?

Did you configure the DNS overrides for the local clients?

Also your Bitwarden server seems to be misconfigured are you sure it is serving SSL on the HTTP port? Also verify this for your other service.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: tomdh76 on February 04, 2023, 11:11:11 pm

Quote from: TheHellSite on February 04, 2023, 09:48:07 pm

Quote from: tomdh76 on February 04, 2023, 08:19:08 pm

Thx you @TheHellSite for this tutorial. Unfortunately I cannot get it to work. I always get a "503 service unavaible status"

If I use the internal addresses of the websites (192.168.2.40:8083 for example) I can login. But using the website.domain.com I get a 503 error. In the logs I see that the client is accessing the public Ip on port 443.

Please give further details on what is and whet it is not working.

Are you able to access your services via their domain name from a device outside of your local network?

Did you configure the DNS overrides for the local clients?

Also your Bitwarden server seems to be misconfigured are you sure it is serving SSL on the HTTP port? Also verify this for your other service.

Well, nothing is working, both not from within the local network and also not from outside.

I did configure the DNS override, but I first try to access the services from my mobile device.

I do not understand what you mean by serving SSL on the HTTP port. I think I followed your tutorial to the letter (except for using a Let's encrypt certificate by using cloudflare API from my domain)

Edit: I found it, I needed to uncheck the SSL tickbox in the real server settings. In your tutorial you have it checked and I saw in this forum someone else who had the same problem...

Thx alot!!

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 04, 2023, 11:30:25 pm

Quote from: tomdh76 on February 04, 2023, 11:11:11 pm

I do not understand what you mean by serving SSL on the HTTP port. I think I followed your tutorial to the letter (except for using a Let's encrypt certificate by using cloudflare API from my domain)

It is dangerous to do things like exposing services to the internet when you don't even understand this simple question from me!  :-\

Read step 9 of my FAQ. You should also really read the explanation of the "SSL checkbox" in the server setup page!
I bet you are not accessing your services by their local ip using HTTPS you are likely accessing them using HTTP.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: tomdh76 on February 04, 2023, 11:32:41 pm

Quote from: TheHellSite on February 04, 2023, 11:30:25 pm

Quote from: tomdh76 on February 04, 2023, 11:11:11 pm

I do not understand what you mean by serving SSL on the HTTP port. I think I followed your tutorial to the letter (except for using a Let's encrypt certificate by using cloudflare API from my domain)

It is dangerous to do things like exposing services to the internet when you don't even understand this simple question from me!  :-\

Read step 9 of my FAQ. You should also really read the explanation of the "SSL checkbox" in the server setup page!
I bet you are not accessing your services by their local ip using HTTPS you are likely accessing them using HTTP.

Yes that was the problem...

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: xkpx on February 05, 2023, 09:54:51 am

Strange but 503 error appear to me as well.
I tested with apache,nodejs,wamp nothing worked. They i try to redirect to my switch to see if my windows is not the problem... but nope.
DynamicDNS is configured and working fine,
All gui redirections disabled and opnsense gui port changed.
Added firewall rule to WAN , and no additional LAN rules added ( it's almost fresh install )
Acme - generated fine cert via dns. ( 2/4/2023, 7:23:39 PM   OK   2/4/2023, 7:23:40 PM )

(https://i.ibb.co/rtPW5S4/rule.png)
Tested from external network via smartphone on cellular data.

One thing is that i am using proxmox to virtualize opnsense as "routerOnStick/Forbidden Router" and i pass two ports from quad NIC on promox-server as LAN/WAN for opnsense , and lan is going to dumb switch that transfer vlans/lan to rest of my house , so far not a single problem with that but maybe just maybe..

Code: [Select]

OPNsense 23.1_6-amd64
FreeBSD 13.1-RELEASE-p5
OpenSSL 1.1.1s 1 Nov 2022

Code: [Select]

HAProxy version 2.6.7-c55bfdb 2022/12/02 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.7.html
Running on: FreeBSD 13.1-RELEASE-p5 FreeBSD 13.1-RELEASE-p5 stable/23.1-n250372-c4ad069e50a SMP amd64

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:443 and 0.0.0.0:80)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option log-separate-errors
    option tcplog

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option log-separate-errors
    option httplog
    # ACL: NoSSL_condition
    acl acl_63dea06740dee5.93056632 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_63dea06740dee5.93056632

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63dea303583a84.37941891.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option log-separate-errors
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63dea0bbafdf17.31648976.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: XKP_backend ()
backend XKP_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server XKP_server 192.168.1.104:80 ssl verify none



# statistics are DISABLED

Code: [Select]

2023-02-05T10:45:38 Error haproxy ********:31073 [05/Feb/2023:10:45:38.668] 1_HTTPS_frontend~ 1_HTTPS_frontend/<NOSRV> -1/-1/-1/-1/0 503 217 - - SC-- 2/1/0/0/0 0/0 "GET https://[********:/favicon.ico HTTP/2.0"

Code: [Select]

root@firewall:~ # haproxy -f /usr/local/etc/haproxy.conf -d
Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.

Available filters :
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace
Using kqueue() as the polling mechanism.
00000000:0_SNI_frontend.accept(0004)=0014 from [********:31207] ALPN=<none>
00000001:1_HTTPS_frontend.accept(0007)=0017 from [********:31207] ALPN=h2
00000001:1_HTTPS_frontend.clireq[0017:ffffffff]: GET https://********/ HTTP/2.0
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: host: ********
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: cache-control: max-age=0
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-mobile: ?1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-platform: "Android"
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: save-data: on
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: dnt: 1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: upgrade-insecure-requests: 1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: user-agent: Mozilla/5.0 (Linux; Android 13; Mi 9T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-site: cross-site
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-mode: navigate
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-user: ?1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-dest: document
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-encoding: gzip, deflate, br
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-language: en-US,en;q=0.9
00000001:1_HTTPS_frontend.clicls[0017:ffff]
00000001:1_HTTPS_frontend.closed[0017:ffff]
00000002:1_HTTPS_frontend.accept(0007)=0017 from [********:31207] ALPN=h2
00000002:1_HTTPS_frontend.clireq[0017:ffffffff]: GET https://********/favicon.ico HTTP/2.0
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: host: ********
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: dnt: 1
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-mobile: ?1
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: save-data: on
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: user-agent: Mozilla/5.0 (Linux; Android 13; Mi 9T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-platform: "Android"
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-site: same-origin
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-mode: no-cors
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-dest: image
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: referer: https://********/
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-encoding: gzip, deflate, br
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-language: en-US,en;q=0.9
00000002:1_HTTPS_frontend.clicls[0017:ffff]
00000002:1_HTTPS_frontend.closed[0017:ffff]
00000000:SSL_backend.srvcls[0014:ffff]
00000000:SSL_backend.clicls[ffff:ffff]
00000000:SSL_backend.closed[ffff:ffff]


Code: [Select]

Interesting is that from opnsense ssh via wget i managed to download from server, and from windows too..
wget --save-headers http://ccc.network.ccc
This was with DNS override , but still not accessible by browser

image - https://i.ibb.co/bL8Wgbj/34.png

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: tomdh76 on February 05, 2023, 10:47:01 am

Quote from: xkpx on February 05, 2023, 09:54:51 am

Strange but 503 error appear to me as well.
I tested with apache,nodejs,wamp nothing worked. They i try to redirect to my switch to see if my windows is not the problem... but nope.
DynamicDNS is configured and working fine,
All gui redirections disabled and opnsense gui port changed.
Added firewall rule to WAN , and no additional LAN rules added ( it's almost fresh install )
Acme - generated fine cert via dns. ( 2/4/2023, 7:23:39 PM   OK   2/4/2023, 7:23:40 PM )

Tested from external network via smartphone on cellular data.

One thing is that i am using proxmox to virtualize opnsense as "routerOnStick/Forbidden Router" and i pass two ports from quad NIC on promox-server as LAN/WAN for opnsense , and lan is going to dumb switch that transfer vlans/lan to rest of my house , so far not a single problem with that but maybe just maybe..

Code: [Select]

OPNsense 23.1_6-amd64
FreeBSD 13.1-RELEASE-p5
OpenSSL 1.1.1s 1 Nov 2022

Code: [Select]

HAProxy version 2.6.7-c55bfdb 2022/12/02 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.7.html
Running on: FreeBSD 13.1-RELEASE-p5 FreeBSD 13.1-RELEASE-p5 stable/23.1-n250372-c4ad069e50a SMP amd64

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:443 and 0.0.0.0:80)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option log-separate-errors
    option tcplog

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option log-separate-errors
    option httplog
    # ACL: NoSSL_condition
    acl acl_63dea06740dee5.93056632 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_63dea06740dee5.93056632

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63dea303583a84.37941891.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option log-separate-errors
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63dea0bbafdf17.31648976.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: XKP_backend ()
backend XKP_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server XKP_server 192.168.1.104:80 ssl verify none



# statistics are DISABLED

Code: [Select]

2023-02-05T10:45:38 Error haproxy ********:31073 [05/Feb/2023:10:45:38.668] 1_HTTPS_frontend~ 1_HTTPS_frontend/<NOSRV> -1/-1/-1/-1/0 503 217 - - SC-- 2/1/0/0/0 0/0 "GET https://[********:/favicon.ico HTTP/2.0"

Code: [Select]

root@firewall:~ # haproxy -f /usr/local/etc/haproxy.conf -d
Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.

Available filters :
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace
Using kqueue() as the polling mechanism.
00000000:0_SNI_frontend.accept(0004)=0014 from [********:31207] ALPN=<none>
00000001:1_HTTPS_frontend.accept(0007)=0017 from [********:31207] ALPN=h2
00000001:1_HTTPS_frontend.clireq[0017:ffffffff]: GET https://********/ HTTP/2.0
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: host: ********
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: cache-control: max-age=0
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-mobile: ?1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-platform: "Android"
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: save-data: on
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: dnt: 1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: upgrade-insecure-requests: 1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: user-agent: Mozilla/5.0 (Linux; Android 13; Mi 9T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-site: cross-site
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-mode: navigate
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-user: ?1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-dest: document
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-encoding: gzip, deflate, br
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-language: en-US,en;q=0.9
00000001:1_HTTPS_frontend.clicls[0017:ffff]
00000001:1_HTTPS_frontend.closed[0017:ffff]
00000002:1_HTTPS_frontend.accept(0007)=0017 from [********:31207] ALPN=h2
00000002:1_HTTPS_frontend.clireq[0017:ffffffff]: GET https://********/favicon.ico HTTP/2.0
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: host: ********
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: dnt: 1
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-mobile: ?1
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: save-data: on
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: user-agent: Mozilla/5.0 (Linux; Android 13; Mi 9T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-platform: "Android"
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-site: same-origin
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-mode: no-cors
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-dest: image
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: referer: https://********/
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-encoding: gzip, deflate, br
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-language: en-US,en;q=0.9
00000002:1_HTTPS_frontend.clicls[0017:ffff]
00000002:1_HTTPS_frontend.closed[0017:ffff]
00000000:SSL_backend.srvcls[0014:ffff]
00000000:SSL_backend.clicls[ffff:ffff]
00000000:SSL_backend.closed[ffff:ffff]


I see also in your backend for XKP SSL is checked. Could you try by unchecking the SSL?

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: xkpx on February 05, 2023, 10:56:55 am

Totally my mistake of course, i somehow manage to confuse myself to put in map file xkp XKP_Server, instead of xkp XKP_backend, damn i did this tutorial 3 times already today to realize it.
Also forgot in ACME , OCSP must staple.
So far everything works.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brynjolm on February 06, 2023, 04:31:29 pm

Quote from: TheHellSite on February 03, 2023, 05:06:38 pm

Please refer to this post about it. Be warned I can not provide help for this since I am not using such a setup.

https://forum.opnsense.org/index.php?topic=18538.msg84958#msg84958

Still thank you for posting a link to a possible soloution. I have had my hunches but at least it narrows down the area at which to look at. Thank you and have a nice day!

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: user78425653 on February 21, 2023, 12:40:16 pm

Just registered to say thank you @TheHellSite
Everything works!

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Dimi3 on February 23, 2023, 03:53:58 pm

Hello,

Decided to post the issue I'm having here, since for the love of god can’t find what’s wrong 😊

I implemented the guide to the letter using virtual IP. I setup 1 public service accessible from internet and few local accessible only ( I separate this on my DNS settings for domain. I only pointed xyz.mydomain.net to my public IP ( static ). Everything is working as expected.

Now I want to open full domain to my public IP, and implement step 7 of this guide to make only public services available over the internet, and limit the local services to LAN access only.

I added the local subdomains rule and map file as described in the guide, but for some reason it doesn’t work...services are always available even if accessed from internet...like the LOCAL_SUBDOMAINS_map_conditions would not apply...but if I remove the PUBLIC_SUBDOMAINS-map-rule form HTTPS_frontend its is working, but only from local lan, as it should since, its limiting access to LAN only....when I put both  rules in my HTTPS_frontend the rule for local lan access only is not sticking, but websites are accessible from lan and internet.

Did I missed something obvious ?, or as it looks to me, like 2 rules are not handled in parallel, since with only one rule (either) its working.

Any help appreciated.

Posting my config;

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    1
    hard-stop-after             60s
    no strict-limits
    maxconn                     128
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 100
    timeout client 30s
    timeout connect 4s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 100

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_Backend

    # logging options
    option tcplog

# Frontend: 1_HTTP_Frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_Frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option httplog
    # ACL: NoSSL_Condition
    acl acl_6241c8286b2146.46286925 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6241c8286b2146.46286925

# Frontend: 1_HTTPS_Frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6241cc05878570.68121182.certlist
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option httplog
    # ACL: LOCAL_SUBDOMAINS_map_conditions
    acl acl_63f758e46145e5.66171870 src 192.168.1.0/26

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63f7583a8314e2.36363887.txt)] if acl_63f758e46145e5.66171870
    # ACTION: PUBLIC_SUBDOMAINS_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6241c892a54f84.31767078.txt)]

# Backend: SSL_Backend (SSL_Backend)
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Unifi_backend (Unifi_Backend)
backend Unifi_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Unifi 172.1.1.2:8443 ssl alpn h2,http/1.1 verify none source 192.168.1.1

# Backend: Homeassistant_backend (Homeassistant_Backend)
backend Homeassistant_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Homeassistant 192.168.1.3:8123

# Backend: Docker_OCI_backend (Docker_OCI_Backend)
backend Docker_OCI_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server docker 172.1.1.2:9443 ssl alpn h2,http/1.1 verify none source 192.168.1.1



# statistics are DISABLED





Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 23, 2023, 04:39:03 pm

Quote from: Dimi3 on February 23, 2023, 03:53:58 pm

Hello,

Decided to post the issue I'm having here, since for the love of god can’t find what’s wrong 😊

I implemented the guide to the letter using virtual IP. I setup 1 public service accessible from internet and few local accessible only ( I separate this on my DNS settings for domain. I only pointed xyz.mydomain.net to my public IP ( static ). Everything is working as expected.

Now I want to open full domain to my public IP, and implement step 7 of this guide to make only public services available over the internet, and limit the local services to LAN access only.

I added the local subdomains rule and map file as described in the guide, but for some reason it doesn’t work...services are always available even if accessed from internet...like the LOCAL_SUBDOMAINS_map_conditions would not apply...but if I remove the PUBLIC_SUBDOMAINS-map-rule form HTTPS_frontend its is working, but only from local lan, as it should since, its limiting access to LAN only....when I put both  rules in my HTTPS_frontend the rule for local lan access only is not sticking, but websites are accessible from lan and internet.

Did I missed something obvious ?, or as it looks to me, like 2 rules are not handled in parallel, since with only one rule (either) its working.

Please post a screenshot of the rule order in the HTTPS_frontend.
Please also post your two map files. (domain name redacted)

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Dimi3 on February 23, 2023, 05:20:30 pm

sure..

screnshoots attached.

thanks.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 23, 2023, 05:32:06 pm

Quote from: Dimi3 on February 23, 2023, 03:53:58 pm

Hello,

Decided to post the issue I'm having here, since for the love of god can’t find what’s wrong 😊

I implemented the guide to the letter using virtual IP. I setup 1 public service accessible from internet and few local accessible only ( I separate this on my DNS settings for domain. I only pointed xyz.mydomain.net to my public IP ( static ). Everything is working as expected.

Now I want to open full domain to my public IP, and implement step 7 of this guide to make only public services available over the internet, and limit the local services to LAN access only.

I added the local subdomains rule and map file as described in the guide, but for some reason it doesn’t work...services are always available even if accessed from internet...like the LOCAL_SUBDOMAINS_map_conditions would not apply...but if I remove the PUBLIC_SUBDOMAINS-map-rule form HTTPS_frontend its is working, but only from local lan, as it should since, its limiting access to LAN only....when I put both  rules in my HTTPS_frontend the rule for local lan access only is not sticking, but websites are accessible from lan and internet.

Did I missed something obvious ?, or as it looks to me, like 2 rules are not handled in parallel, since with only one rule (either) its working.

Part 7 - Step 4 ist your solution.
A typical "I didn't read properly" mistake.

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Dimi3 on February 23, 2023, 05:45:21 pm

call me stupid but i dont see the error in cofig :), my map files, public and local have all backends (subdomains defined internet accessible and local accessible), also the https frontend has the LOCAL_SUBDOMAINS_map-rule in first place and PUBLIC in second place.

will read it a few more times :)

Title: Re: Tutorial 2022/08: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 23, 2023, 06:48:35 pm

Quote from: Dimi3 on February 23, 2023, 05:45:21 pm

call me stupid but i dont see the error in cofig :), my map files, public and local have all backends (subdomains defined internet accessible and local accessible), also the https frontend has the LOCAL_SUBDOMAINS_map-rule in first place and PUBLIC in second place.

will read it a few more times :)

I updated Part 7 to make it (no offense) noob proof.
Please read ALL of it again, slowly.

Title: Re: Tutorial 2023/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: xkpx on March 03, 2023, 08:50:27 pm

I saw in few places that haproxy is only tcp/http proxy, today i tried to setup RustDesk host and everything except that hbbs - RustDesk ID/Rendezvous server ( https://rustdesk.com/docs/en/self-host/install/ ) works so far.
I get that 21116 is UDP and maybe will be not possible to reverse proxy this but i need to ask is there a way to get it working ?

Another question is there a way to make range of port like 21115-21117 because adding some services may make the UI little overcrowded in time

Title: Re: Tutorial 2023/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 03, 2023, 09:18:04 pm

Quote from: xkpx on March 03, 2023, 08:50:27 pm

I get that 21116 is UDP and maybe will be not possible to reverse proxy this but i need to ask is there a way to get it working ?

No, it is not possible to reverse proxy UDP traffic based on the access URL or FQDN. Unlike TCP or HTTP, which use a header with the destination hostname or URL, UDP packets do not contain any information about the intended destination.

UDP is a connectionless protocol, which means that packets are sent from the client to the server without any prior communication or setup. Each packet is independent and can be routed independently. This makes it impossible to perform URL-based routing or reverse proxying for UDP traffic.

HAProxy can still perform load balancing for UDP traffic based on IP address and port, but it cannot route based on the content of the UDP packet or the access URL or FQDN.

Because of that, if you don't run multiple RustDesk servers and therefore have no need of actually load balancing it, you can just use a simple UDP port forward to your rust desk server. You won't have any benefit with setting this up in HAProxy.

Quote from: xkpx on March 03, 2023, 08:50:27 pm

Another question is there a way to make range of port like 21115-21117 because adding some services may make the UI little overcrowded in time

No, because each server has to be unique since a reverse proxy is also a load balancer.
You can however combine them in the same backend, but only IF they server the exact same content. Otherwise packets will end up on the wrong port.

I think you actually can, however the OPNsense HAProxy GUI doesn't support setting it up. It expects a single port (or none) for each server. It also doesn't support setting it up using the option passtrough directive.

Title: Re: Tutorial 2023/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: xkpx on March 03, 2023, 09:36:26 pm

Thanks for the detailed explanation ! + karma

Title: Re: Tutorial 2023/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on March 03, 2023, 11:37:14 pm

I wonder if this has been asked before: is it possible to have HAProxy not expose a certificate in case no correct Host: header has been supplied?

Currently, if somebody scans my IP with just "openssl s_client <myip>:443", the default certificate(s) will be exposed. Going from there, one could enumerate every (sub-)domain that is presented with that certificate in order to find vulnerabilities.

If the certificate was instead withheld, one had to know at least one valid name in order to do that.

I imagine that e.g. with Cloudflare tunnels, a small range of IPs are being used to terminate all tunnel connections, but only after the Host: header has been presented will a specific customer certificate be chosen and presented, so that there is no possibilty for a "scan". I would like to do something similar with HAProxy on my OpnSense.

Title: Re: Tutorial 2023/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: jonf on March 04, 2023, 12:07:38 am

Hi,

I followed this tutorial and my services now work as intended by typing https://[service].[hostname] for each web service I have (in Docker containers). Now I have a Docker called Organizr installed which I want to act as a 'homepage' that displays links to those services, and I want this to be accessible by just typing https://[hostname] and still use the same Let's Encrypt certificate to secure it.

How can I do this within the framework of this setup?

Title: Re: Tutorial 2023/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 04, 2023, 12:48:40 pm

Quote from: meyergru on March 03, 2023, 11:37:14 pm

I wonder if this has been asked before: is it possible to have HAProxy not expose a certificate in case no correct Host: header has been supplied?

Currently, if somebody scans my IP with just "openssl s_client <myip>:443", the default certificate(s) will be exposed. Going from there, one could enumerate every (sub-)domain that is presented with that certificate in order to find vulnerabilities.

If the certificate was instead withheld, one had to know at least one valid name in order to do that.

I imagine that e.g. with Cloudflare tunnels, a small range of IPs are being used to terminate all tunnel connections, but only after the Host: header has been presented will a specific customer certificate be chosen and presented, so that there is no possibilty for a "scan". I would like to do something similar with HAProxy on my OpnSense.

Yes sure this is possible. In fact I am doing this for a long time already. Just didn't bother adding it to the guide.

I will add it as part 8 to the guide. Just give me a little time!

EDIT: Added this as Part 8.

Title: Re: Tutorial 2023/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: jonf on March 04, 2023, 07:15:52 pm

Quote from: jonf on March 04, 2023, 12:07:38 am

Hi,

I followed this tutorial and my services now work as intended by typing https://[service].[hostname] for each web service I have (in Docker containers). Now I have a Docker called Organizr installed which I want to act as a 'homepage' that displays links to those services, and I want this to be accessible by just typing https://[hostname] and still use the same Let's Encrypt certificate to secure it.

How can I do this within the framework of this setup?

OK I tried by adding my full hostname as a new entry to the sub-domains mapping file followed by the corresponding backend for the Organizr service. When I type the hostname without https:// it doesn't load any SSL certificate or try to auto-redirect to HTTPS.  If I manually type https:// with the hostname, it gives an 'ERR_SSL_VERSION_OR_CIPHER_MISMATCH' error in my browser.

Title: Re: Tutorial 2023/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on March 05, 2023, 11:34:03 am

Quote from: TheHellSite on March 04, 2023, 12:48:40 pm

Yes sure this is possible. In fact I am doing this for a long time already. Just didn't bother adding it to the guide.

I will add it as part 8 to the guide. Just give me a little time!

EDIT: Added this as Part 8.

Thank you very much, I just added it - works like a charm!

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: xkpx on March 07, 2023, 07:32:13 pm

Everything works for me but i tried to check some logs and saw something, Is this normal output when enable Detailed logging in 1_HTTPS_frontend, also if this is expected is there a way to keep the logging and disable this output somehow ?

Code: [Select]

2023-03-07T20:39:05 Error haproxy 192.168.1.102:47192 [07/Mar/2023:20:39:05.886] 1_HTTPS_frontend~ ADGUARD_backend/ADGUARD_server 0/-1/-1/-1/0 -1 0 - - CR-- 2/1/0/0/0 0/0 "GET https://adguard.xxx.yyy/assets/favicon.png HTTP/2.0"
2023-03-07T20:35:04 Error haproxy 3.252.130.46:37184 [07/Mar/2023:20:35:04.759] 1_HTTPS_frontend~ 1_HTTPS_frontend/<NOSRV> -1/-1/-1/-1/0 503 217 - - SC-- 4/2/0/0/0 0/0 "GET / HTTP/1.0"
2023-03-07T20:31:32 Error haproxy 192.168.1.102:47192 [07/Mar/2023:20:31:32.228] 1_HTTPS_frontend~ PROXMOX_backend/PROXMOX_server 0/0/0/-1/2 -1 0 - - CD-- 2/1/0/0/0 0/0 "GET https://proxmox.xxx.yyy/pve2/ext6/theme-crisp/resources/images/grid/hd-pop.png HTTP/2.0"
2023-03-07T20:31:03 Error haproxy 192.168.1.102:47192 [07/Mar/2023:20:31:03.640] 1_HTTPS_frontend~ PROXMOX_backend/PROXMOX_server 0/0/0/3/3 501 197 - - ---- 2/1/0/0/0 0/0 "GET https://proxmox.xxx.yyy/images/logo-ceph.png HTTP/2.0"

2023-03-07T20:31:00 Error haproxy 192.168.1.102:47212 [07/Mar/2023:20:31:00.634] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
2023-03-07T20:31:00 Error haproxy 192.168.1.102:47217 [07/Mar/2023:20:31:00.659] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
2023-03-07T20:31:00 Error haproxy 192.168.1.102:47214 [07/Mar/2023:20:31:00.649] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
And can someone post some implementation screenshots of NOSSLservice_rule

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Grenen on March 07, 2023, 08:44:45 pm

First of all, thanks for a awesome guide!

In 5.8 you refer to a "FAQ about Map Files". Could you please link me to this FAQ? Perhaps it could be added as a link in the post so ppl easier can find it.

Second comes my question.
Ive finished the setup. I sorted out a *.-certificate for mydomain.com and added a A-record in the DNS for the domain with a homeassistant.mydomain.com that points to my public IP.
In HAProxy ive added a real server HA_server which points to the IP of my HA-server with port 8123. SSL checked.
I've added a backend pool HA_backend that points to my HA_server
Ive added a host override for host homeassistant domain mydomain.com with the internal IP of my HA-server.

My Local MAP-filehttps://ibb.co/D8xwgH5 (https://ibb.co/D8xwgH5)
My Public MAP-filehttps://ibb.co/D8xwgH5 (https://ibb.co/D8xwgH5)

When browsing to homeassistant.mydomain.com i get a "Unable to connect" message.

What did i do wrong, any tips where i should start looking?

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 07, 2023, 08:50:11 pm

Quote from: xkpx on March 07, 2023, 07:32:13 pm

Everything works for me but i tried to check some logs and saw something, Is this normal output when enable Detailed logging in 1_HTTPS_frontend, also if this is expected is there a way to keep the logging and disable this output somehow ?

Code: [Select]

2023-03-07T20:39:05 Error haproxy 192.168.1.102:47192 [07/Mar/2023:20:39:05.886] 1_HTTPS_frontend~ ADGUARD_backend/ADGUARD_server 0/-1/-1/-1/0 -1 0 - - CR-- 2/1/0/0/0 0/0 "GET https://adguard.xxx.yyy/assets/favicon.png HTTP/2.0"
2023-03-07T20:35:04 Error haproxy 3.252.130.46:37184 [07/Mar/2023:20:35:04.759] 1_HTTPS_frontend~ 1_HTTPS_frontend/<NOSRV> -1/-1/-1/-1/0 503 217 - - SC-- 4/2/0/0/0 0/0 "GET / HTTP/1.0"
2023-03-07T20:31:32 Error haproxy 192.168.1.102:47192 [07/Mar/2023:20:31:32.228] 1_HTTPS_frontend~ PROXMOX_backend/PROXMOX_server 0/0/0/-1/2 -1 0 - - CD-- 2/1/0/0/0 0/0 "GET https://proxmox.xxx.yyy/pve2/ext6/theme-crisp/resources/images/grid/hd-pop.png HTTP/2.0"
2023-03-07T20:31:03 Error haproxy 192.168.1.102:47192 [07/Mar/2023:20:31:03.640] 1_HTTPS_frontend~ PROXMOX_backend/PROXMOX_server 0/0/0/3/3 501 197 - - ---- 2/1/0/0/0 0/0 "GET https://proxmox.xxx.yyy/images/logo-ceph.png HTTP/2.0"

2023-03-07T20:31:00 Error haproxy 192.168.1.102:47212 [07/Mar/2023:20:31:00.634] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
2023-03-07T20:31:00 Error haproxy 192.168.1.102:47217 [07/Mar/2023:20:31:00.659] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
2023-03-07T20:31:00 Error haproxy 192.168.1.102:47214 [07/Mar/2023:20:31:00.649] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure
And can someone post some implementation screenshots of NOSSLservice_rule

This happens when the client doesn't support all the ciphers or uses Part 8 (if configured) to connect.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 07, 2023, 08:53:05 pm

Quote from: Grenen on March 07, 2023, 08:44:45 pm

First of all, thanks for a awesome guide!

In 5.8 you refer to a "FAQ about Map Files". Could you please link me to this FAQ? Perhaps it could be added as a link in the post so ppl easier can find it.

Second comes my question.
Ive finished the setup. I sorted out a *.-certificate for mydomain.com and added a A-record in the DNS for the domain with a homeassistant.mydomain.com that points to my public IP.
In HAProxy ive added a real server HA_server which points to the IP of my HA-server with port 8123. SSL checked.
I've added a backend pool HA_backend that points to my HA_server
Ive added a host override for host homeassistant domain mydomain.com with the internal IP of my HA-server.

My Local MAP-filehttps://ibb.co/D8xwgH5 (https://ibb.co/D8xwgH5)
My Public MAP-filehttps://ibb.co/D8xwgH5 (https://ibb.co/D8xwgH5)

When browsing to homeassistant.mydomain.com i get a "Unable to connect" message.

What did i do wrong, any tips where i should start looking?

You didn't read the tutorial properly. Read it again from the very top to the very bottom. Everything you just asked is answered there. Also if you would have followed the tutorial correctly you wouldn't see any errors now.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Grenen on March 07, 2023, 11:26:41 pm

Quote from: TheHellSite on March 07, 2023, 08:53:05 pm

You didn't read the tutorial properly. Read it again from the very top to the very bottom. Everything you just asked is answered there. Also if you would have followed the tutorial correctly you wouldn't see any errors now.

Ah, missunderstood the text regarding the map FAQ. Now i understand what you refered to.

Regarding my setup. Before i asked for help ive already gone through it twice. Ive now been through it twice again and i cant find anything wrong. I havent used virtual IP and instead using 127.0.0.1 as IP in the cases where you use your virtual IP.

Perhaps its working and Homeassistant is the problem? I found some threads regarding adding trusted proxies there. Anyone had the same problem and know what to configure in HA?

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 08, 2023, 03:34:50 am

Quote from: Grenen on March 07, 2023, 11:26:41 pm

Quote from: TheHellSite on March 07, 2023, 08:53:05 pm

You didn't read the tutorial properly. Read it again from the very top to the very bottom. Everything you just asked is answered there. Also if you would have followed the tutorial correctly you wouldn't see any errors now.

Ah, missunderstood the text regarding the map FAQ. Now i understand what you refered to.

Regarding my setup. Before i asked for help ive already gone through it twice. Ive now been through it twice again and i cant find anything wrong. I havent used virtual IP and instead using 127.0.0.1 as IP in the cases where you use your virtual IP.

Perhaps its working and Homeassistant is the problem? I found some threads regarding adding trusted proxies there. Anyone had the same problem and know what to configure in HA?

If you really did the guide four times now then I am suprised that you still don't know how to ask for help.  ???
Hint: I describe it in the first post.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Grenen on March 08, 2023, 09:52:38 am

Quote from: TheHellSite on March 08, 2023, 03:34:50 am

If you really did the guide four times now then I am suprised that you still don't know how to ask for help.  ???
Hint: I describe it in the first post.

Relevant question. I focused on the guide part, not the text above sorry.

Here is the config export.

Quote

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening to localhost)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_64025b0cc7a716.63164065 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_64025b0cc7a716.63164065

# Frontend: 1_HTTPS_frontend (Listening on 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/64025f85730fc6.50514236.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/64025bea0b8443.12301363.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: BITWARDEN_backend ()
backend BITWARDEN_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server BITWARDEN_server 192.168.100.161:443 ssl verify none

# Backend: HA_backend ()
backend HA_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server HA_server 192.168.1.106:8123



# statistics are DISABLED

In the log file if i try to connect from the outside i get:

Quote

2023-03-08T09:49:15   Informational   haproxy   Connect from -external ip-:8335 to -public ip:443 (1_HTTPS_frontend/HTTP)   
2023-03-08T09:49:13   Informational   haproxy   Connect from -external ip-:8335 to -public ip-:443 (1_HTTPS_frontend/HTTP)   
2023-03-08T09:49:11   Informational   haproxy   Connect from -external ip-8335 to -public ip-:443 (1_HTTPS_frontend/HTTP)   
2023-03-08T09:49:10   Informational   haproxy   Connect from -external ip-:8335 to -public ip-:443 (1_HTTPS_frontend/HTTP)   
2023-03-08T09:49:10   Informational   haproxy   Connect from -external ip-:8335 to -public ip-:443 (0_SNI_frontend/TCP)   
2023-03-08T09:49:10   Informational   haproxy   Connect from -external ip-:8438 to -public ip-:443 (1_HTTPS_frontend/HTTP)   
2023-03-08T09:49:09   Informational   haproxy   Connect from -external ip-:8438 to -public ip8:443 (0_SNI_frontend/TCP)   
2023-03-08T09:49:09   Informational   haproxy   Connect from -external ip-:8430 to -public ip:443 (1_HTTPS_frontend/HTTP)   
2023-03-08T09:49:09   Informational   haproxy   Connect from -external ip-:8430 to -public ip:443 (0_SNI_frontend/TCP)

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 08, 2023, 11:35:46 am

Is the access to your bitwarden working?
If yes, is it working from WAN and LAN via haproxy? Or just WAN?

Please post a screenshot of your Unbound DNS overrides. You can blank out the domain name.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Grenen on March 08, 2023, 12:01:00 pm

Bitwarden only works from internal since I set up HA-Proxy.

Here’s a picture of the override:
 https://ibb.co/ysNPg09 (https://ibb.co/ysNPg09)

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 08, 2023, 12:20:07 pm

Did you read and understand  what I wrote in part 6 - option a - step 3 of my tutorial? Or did you also not bother reading? Please explain to me what I am saying there and then explain what you did there... Maybe you will spot your error.

Also post the content (in a code box) of your public and local subdomains map file.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Grenen on March 08, 2023, 03:44:22 pm

Quote from: TheHellSite on March 08, 2023, 12:20:07 pm

Did you read and understand  what I wrote in part 6 - option a - step 3 of my tutorial? Or did you also not bother reading? Please explain to me what I am saying there and then explain what you did there... Maybe you will spot your error.

Also post the content (in a code box) of your public and local subdomains map file.

interface IP as the IP of the gateway? I understood it as any ip in the range of the vlan. Yes I’ve read through the steps several times, English isn’t my native language so some things might be lost in translation.

Code: [Select]


# public access subdomains
bit BITWARDEN_backend
home HA_backend


Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 08, 2023, 05:08:44 pm

No, it has to be the the interface IP of your OPNsense that is reachable by the clients that want to use haproxy... The guide is VERY clear about that. You can't just use any IP!? Stick to the guide!!   :-\

Please post your interface overview...

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Grenen on March 08, 2023, 07:26:23 pm

Quote from: TheHellSite on March 08, 2023, 05:08:44 pm

No, it has to be the the interface IP of your OPNsense that is reachable by the clients that want to use haproxy... The guide is VERY clear about that. You can't just use any IP!? Stick to the guide!!   :-\

Please post your interface overview...

As i wrote, was misinterpreting from my part. I still get a 503-error when trying to reach the services after changing IP to the gateway of each network.

My HA-server is on my LAN, the Windows server which hosts my Docker Desktop with Bitwarden is on VLAN100.

Pictures of interface overview of LAN,MGMT(vlan100) & WAN:

https://ibb.co/SDCqcrn
https://ibb.co/sKMjnv5
https://ibb.co/p0nCc4y
https://ibb.co/C8jgnf2
https://ibb.co/SxN16PS

Screenshots taken from RDP-session from my phone, so they are a bit cropped.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 08, 2023, 07:54:18 pm

And as I already told you before and in the guide!!!
The gateway of the service does not matter at all!? Why is that so hard to understand?

The client dns request needs to be overwritten, not the dns requests of any service!!! So guess what, if the client is in subnet A and wants to access ANY service in ANY subnet then what IP will the client use to connect to the service? It obviously has to be subnet A gateway address since the client is in subnet A.
If the client is in subnet B you will have to create the same override but with subnet B gateway as target.
And so on.

All of the above is however only relevant for local access from within your network.

Now answer this
Is bitwarden working from external networks (mobile data,...)?
Is bitwarden now working from internal network?

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Grenen on March 09, 2023, 05:38:05 am

Quote from: TheHellSite on March 08, 2023, 07:54:18 pm

And as I already told you before and in the guide!!!
The gateway of the service does not matter at all!? Why is that so hard to understand?

The client dns request needs to be overwritten, not the dns requests of any service!!! So guess what, if the client is in subnet A and wants to access ANY service in ANY subnet then what IP will the client use to connect to the service? It obviously has to be subnet A gateway address since the client is in subnet A.
If the client is in subnet B you will have to create the same override but with subnet B gateway as target.
And so on.

All of the above is however only relevant for local access from within your network.

Now answer this
Is bitwarden working from external networks (mobile data,...)?
Is bitwarden now working from internal network?

No, I can only access it from internal networks. Not from the outside.

Bitwarden works in internal networks.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 09, 2023, 07:23:43 am

Which URL do you use to access them from inside?

How do you access them directly (ip:port) full URL?

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Grenen on March 09, 2023, 08:34:02 am

Quote from: TheHellSite on March 09, 2023, 07:23:43 am

Which URL do you use to access them from inside?

How do you access them directly (ip:port) full URL?

Bitwarden is accesses via Bitwarden.mydomain.com for internal

Quote

(it works) and external (can’t reach server)
HA is accessed internally from 192.168.1.106:8123 (works) and external via homeassistant.mydomain.com (can’t reach server).

There are some settings for HA to configure proxy that I think I need to set up, but since Bitwarden doesn’t work it’s not the complete solution to my problem.


From https://www.home-assistant.io/integrations/http/

Quote

use_x_forwarded_for boolean (optional, default: false)
Enable parsing of the X-Forwarded-For header, passing on the client’s correct IP address in proxied setups. You must also whitelist trusted proxies using the trusted_proxies setting for this to work. Non-whitelisted requests with this header will be considered IP spoofing attacks, and the header will, therefore, be ignored.

trusted_proxies string | list (optional)
List of trusted proxies, consisting of IP addresses or networks, that are allowed to set the X-Forwarded-For header. This is required when using use_x_forwarded_for because all requests to Home Assistant, regardless of source, will arrive from the reverse proxy IP address. Therefore in a reverse proxy scenario, this option should be set with extreme care. If the immediate upstream proxy is not in the list, the request will be rejected. If any other intermediate proxy is not in the list, the first untrusted proxy will be considered the client.

I’m using suricata IPS/IDS and tried disabling these, but no change. Using blocklists in Unbound DNS but that should not interfere.

I’m out of ideas. I will review everything again when I get home later today.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sapphire on March 09, 2023, 10:14:58 am

Hey everyone! Hey TheHellSite!

First of all, I'm really thankful for your effort to write this really good guide and give support to date! That's awesome and should be honoured.

A bit of a backstory before I dive into the problem.
I'm a system engineer working with Sophos Products for years at enterprises. Because of that, I was using Sophos UTM Home Editions for years for my own private  servers as well -  but wanted to make the switch to OPNsense long ago. Some months ago, I finally decided to work on the switch. Your guide supported me so hard to make this switch from Sophos UTM to OPNsense.

Fast forward
I was thinking, my haproxy on my OPNsense was working completely. I had some issues before, where I could render websites from my local network (altough not using Split DNS or similar - just public IPs), but not from the internet (tested this with my 5G connection from my phone). After some tinkering, I got it even working on my phone with 5G and thought, everything should be okay now.

My Setup
I'm using 2 OPNsense, one at home and one in a datacenter on a VPS. I have these 2 connected to eachother via a Wireguard Tunnel (OPNsense plugin) using this guide (sorry, it's German: https://www.busche.org/index.php/2021/03/21/ipv4-ueber-wireguard-von-opnsense-zu-opnsense-routen-cgnat-umgehen/). And I'm using exactly your guide with the only difference using my wireguard interface instead of WAN for firewallrules.

Wireguard is working awesome, leads all traffic via Proxy ARP virtual IP (2nd public IP on VPS) on the VPS OPNsense to my home OPNsense.

Now the problem
More often then not, it seems that my websites aren't reachable from the internet. It's working from my 5G Internet Connection, it's working for a webdeveloper, who was assisting me with a web-project from hungary. But when friends test it or when I test it from my AWS Windows Machine, it's running into a timeout. Same for multiple status checking sites like https://isitdownorjust.me/. https://www.ssllabs.com/ssltest/ on the other hand is able to reach and check my websites.

I checked the haproxy logs and in those cases where it doesn't work, I get [09/Mar/2023:09:50:14.471] 1_HTTPS_frontend/127.0.0.1:443: SSL handshake failure

I checked your tutorial like 100 times. Checked my config side by side. It doesn't make any sense in my mind, that it works in some cases, and in some other cases it doesn't. And it drives me crazy. I was trying to fix this issue myself for weeks before I decided to write a comment and ask for help here.

My Config Export looks like this (deleted most backends and only left 2 for reference):

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends


# autogenerated entries for stats



# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_Condition
    acl acl_6293a5ef2e36e8.09400894 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6293a5ef2e36e8.09400894

# Frontend: 1_HTTPS_frontend (Listening on 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets prefer-client-ciphers ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6293aa32edc294.46241266.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6293a65a1027d4.72742608.txt)]
    # WARNING: pass through options below this line
    # add X-Forwarded-Proto
    http-request set-header X-Forwarded-Proto https if { ssl_fc }

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: nextcloud-mf_backend ()
backend nextcloud-mf_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server nextcloud-mf nextcloud-mf.fuchsbau.local:443 ssl verify none

# Backend: survey_backend ()
backend survey_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server adv-survey 10.0.110.5:80

listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED


I hope someone can help me, because I don't want to use the Sophos UTM as a fallback anymore and I'm feeling like I'm losing my mind.

Thanks in advance!

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 09, 2023, 01:37:49 pm

Quote from: Grenen on March 09, 2023, 08:34:02 am

Quote from: TheHellSite on March 09, 2023, 07:23:43 am

Which URL do you use to access them from inside?

How do you access them directly (ip:port) full URL?

Bitwarden is accesses via Bitwarden.mydomain.com for internal

Quote

(it works) and external (can’t reach server)
HA is accessed internally from 192.168.1.106:8123 (works) and external via homeassistant.mydomain.com (can’t reach server).

There are some settings for HA to configure proxy that I think I need to set up, but since Bitwarden doesn’t work it’s not the complete solution to my problem.


From https://www.home-assistant.io/integrations/http/

Quote

use_x_forwarded_for boolean (optional, default: false)
Enable parsing of the X-Forwarded-For header, passing on the client’s correct IP address in proxied setups. You must also whitelist trusted proxies using the trusted_proxies setting for this to work. Non-whitelisted requests with this header will be considered IP spoofing attacks, and the header will, therefore, be ignored.

trusted_proxies string | list (optional)
List of trusted proxies, consisting of IP addresses or networks, that are allowed to set the X-Forwarded-For header. This is required when using use_x_forwarded_for because all requests to Home Assistant, regardless of source, will arrive from the reverse proxy IP address. Therefore in a reverse proxy scenario, this option should be set with extreme care. If the immediate upstream proxy is not in the list, the request will be rejected. If any other intermediate proxy is not in the list, the first untrusted proxy will be considered the client.

I’m using suricata IPS/IDS and tried disabling these, but no change. Using blocklists in Unbound DNS but that should not interfere.

I’m out of ideas. I will review everything again when I get home later today.

Quote

# public access subdomains
bit BITWARDEN_backend
home HA_backend

From the map file you provided, you should be accessing your Bitwarden with bit.mydomain.com in local network
If you failed with bit.mydomain.com locally, check your Unbound override to see if you have override bit.mydomain.com to the IP that TCP frontend listening to (That means, LAN or VLAN gateways or Loopback IPs)

Until you can access internally with bit.mydomain.com (not any other path), then you check rather you set your WAN port forward correctly.

BTW. I have Bitwarden running in my home, can access from both LAN and WAN
Suricata and Zenarmor and dns blocklist are on for my network. They won't block your Bitwarden.
The following list is related to your issue
1. (Scope: both internal and external)Rather HAProxy setup correctly (If you follow the guide without missing any point, that should be correct)
2. (Scope: Internally) Internal DNS override bit.mydomain.com to your TCP fontend
3. (Scope: Externally) WAN port forward at least TCP 443 to TCP frontend
4. (Scope: Externally) DDNS updated bit.mydomain.com to your WAN address

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 09, 2023, 08:03:28 pm

Quote from: sapphire on March 09, 2023, 10:14:58 am

Hey everyone! Hey TheHellSite!

First of all, I'm really thankful for your effort to write this really good guide and give support to date! That's awesome and should be honoured.

A bit of a backstory before I dive into the problem.
I'm a system engineer working with Sophos Products for years at enterprises. Because of that, I was using Sophos UTM Home Editions for years for my own private  servers as well -  but wanted to make the switch to OPNsense long ago. Some months ago, I finally decided to work on the switch. Your guide supported me so hard to make this switch from Sophos UTM to OPNsense.

Fast forward
I was thinking, my haproxy on my OPNsense was working completely. I had some issues before, where I could render websites from my local network (altough not using Split DNS or similar - just public IPs), but not from the internet (tested this with my 5G connection from my phone). After some tinkering, I got it even working on my phone with 5G and thought, everything should be okay now.

My Setup
I'm using 2 OPNsense, one at home and one in a datacenter on a VPS. I have these 2 connected to eachother via a Wireguard Tunnel (OPNsense plugin) using this guide (sorry, it's German: https://www.busche.org/index.php/2021/03/21/ipv4-ueber-wireguard-von-opnsense-zu-opnsense-routen-cgnat-umgehen/). And I'm using exactly your guide with the only difference using my wireguard interface instead of WAN for firewallrules.

Wireguard is working awesome, leads all traffic via Proxy ARP virtual IP (2nd public IP on VPS) on the VPS OPNsense to my home OPNsense.

Now the problem
More often then not, it seems that my websites aren't reachable from the internet. It's working from my 5G Internet Connection, it's working for a webdeveloper, who was assisting me with a web-project from hungary. But when friends test it or when I test it from my AWS Windows Machine, it's running into a timeout. Same for multiple status checking sites like https://isitdownorjust.me/. https://www.ssllabs.com/ssltest/ on the other hand is able to reach and check my websites.

I checked the haproxy logs and in those cases where it doesn't work, I get [09/Mar/2023:09:50:14.471] 1_HTTPS_frontend/127.0.0.1:443: SSL handshake failure

I checked your tutorial like 100 times. Checked my config side by side. It doesn't make any sense in my mind, that it works in some cases, and in some other cases it doesn't. And it drives me crazy. I was trying to fix this issue myself for weeks before I decided to write a comment and ask for help here.

You'r welcome!

This error usually indicates that the client (os/browser/app) does not support the SSL settings of haproxy (min tls version, cipher list, cipher suites). In your config I can see that you are still using the default cipher list + suites in your https frontend, those include "weak" 128 bit ciphers which are not really necessary anymore today except if you have very ancient devices connecting to your haproxy. So better use the ciphers I provide in the beginning of the tutorial.

In combination with the above it is also important to know that the ciphers are mostly used in a server (here haproxy) preferred order! (if I remember correctly) Sometimes this is enough to let the handshake fail if the client connecting has very strict SSL requirements. Again, you are using 128 bit ciphers!

Are you scoring a 100% A+ in the SSL Labs test? (please post a screenshot of the result diagram)

Another thing worth checking is packet fragmentation between your two OPNsense instances since you have a wireguard tunnel involved. I also had very strange https timeouts/bugs with wireguard until I configured the correct mtu+mss on my wireguard interface to my VPN provider. I am using VDSL though so PPPoE on the WAN interface resulting in a WAN MTU of 1492. Those 8 Bytes are enough to seriously mess with wireguard tunnels.



Here is a good explanation of SSL negotiation:

During the SSL/TLS handshake, the client sends a list of supported ciphers to the server, and the server responds with a list of ciphers it supports and their preferred order. The client then selects the first cipher in its list that is also present in the server's list, and the server and client agree to use that cipher for the remainder of the SSL/TLS session.

If there is no cipher that both the client and server support, the SSL/TLS handshake fails and the connection is terminated.

Therefore, it is important to configure the server's cipher suite list in a way that prioritizes the strongest ciphers, while still ensuring compatibility with a broad range of clients. This can help to ensure that the SSL/TLS connection is using the strongest possible encryption while maintaining maximum compatibility.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sapphire on March 10, 2023, 10:00:11 am

Good Morning HellSite!

thank you for your very detailed reply.

I was using your cipher list and cipher suits, before I tried everything to fix the issue. I got the list and suits from https://ssl-config.mozilla.org/ and used "intermediate" because it made somehow sense to me.

I now switched it back to your list and suits and this is the outcome:
(https://i.ibb.co/QvFwshK/ssllabs02.png) (https://ibb.co/60RC6GW)

I've got a FibreChannel Connection from Deutsche Glasfaser at home (therefore I need the VPS OPNsense to get a static public IP at home). I didn't change the MTU for Wireguard because the mentioned guide used values for VDSL. So I'm running on default 1500 MTU.

With a "wrong" MTU, shouldn't the VPN connection be shaky with every device from every "outside" network? It's working 100% all the time on my mobile internet (and my girlfriend's as well), and it's working 100% all the time for ssllabs but only some "Is it down or just me"-kinda sites. But for at least 2 friends (one using the newest Opera Browser on Windows), there are timeouts while trying to connect to my websites. Same for a Windows VPS hosted on AWS - can't get a handshake there either (using newest Chrome browser).

I got no problems with my mailserver/proxmox mail gateway at home. I got no hickups with SSH via NAT. I got no issues with gaming servers at home (friends can connect to it). So I think, something isn't working correctly with my haproxy, sadly :(

Edit: Home MTU is at the Wireguard default value = 1420. In the VPS it's set to 1340 (nowhere changed though - so it must have decided to use this MTU automatically on the Wireguard interface). And the VPS Wireguard Interace has some error (in/out - 1 / 688).

Changed the VPS MTU to 1420 as well. Errors now 5/3 since tunnel restart. But nothing else changed. I still get "ERR_TIMED_OUT" on my websites from AWS VPS.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: coreyinoz on March 11, 2023, 12:36:39 am

First off, thank you TheHellSite for this amazing tutorial. I highly appreciate your level of attention to detail and the  after post support.

The purpose of my design is to expose my Jellyfin Server for remote access from a 2nd home, but hopefully in a secure fashion. As far as I can tell, I'm trying to match your network diagram from the first post aside from JF vs Plex.

I almost feel bad asking for help today because I sense it's something stone deaf obvious I've missed. I even slept on it and went back through each step from the very start and I can't see any fat finger errors.

So with apologies in advance, I'm hoping you can offer some troubleshooting for instances where the SSL Server Test comes back as T / Certificate name mismatch. I've made it to the end of Step 5.

Protocol Support, Key Exchange, and Cipher Strength are all top marks, but SSL Test is marking me T because of the invalid cert.

Here's a link to my HAProxy Config - https://pastebin.com/P5QtYPUt (https://pastebin.com/P5QtYPUt)

My ACME Client log looks identical to your screenshot in the tutorial. I have redone the issue/renew procedure and the log comes back looking happy again.

I'm happy to post screenshots that would help diagnose. Appreciate any redirect on where to start looking.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on March 11, 2023, 09:57:24 am

Quote from: sapphire on March 10, 2023, 10:00:11 am

With a "wrong" MTU, shouldn't the VPN connection be shaky with every device from every "outside" network? It's working 100% all the time on my mobile internet (and my girlfriend's as well), and it's working 100% all the time for ssllabs but only some "Is it down or just me"-kinda sites. But for at least 2 friends (one using the newest Opera Browser on Windows), there are timeouts while trying to connect to my websites. Same for a Windows VPS hosted on AWS - can't get a handshake there either (using newest Chrome browser).

I got no problems with my mailserver/proxmox mail gateway at home. I got no hickups with SSH via NAT. I got no issues with gaming servers at home (friends can connect to it). So I think, something isn't working correctly with my haproxy, sadly :(

That depends on PMTU discovery, so not every connection has to fail. Try reducing MTU/MSS just for the wireguard interface group like so:

(https://gcdnb.pbrd.co/images/E2FzctSpuJlY.png)

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 11, 2023, 12:14:21 pm

Quote from: coreyinoz on March 11, 2023, 12:36:39 am

First off, thank you TheHellSite for this amazing tutorial. I highly appreciate your level of attention to detail and the  after post support.

The purpose of my design is to expose my Jellyfin Server for remote access from a 2nd home, but hopefully in a secure fashion. As far as I can tell, I'm trying to match your network diagram from the first post aside from JF vs Plex.

I almost feel bad asking for help today because I sense it's something stone deaf obvious I've missed. I even slept on it and went back through each step from the very start and I can't see any fat finger errors.

So with apologies in advance, I'm hoping you can offer some troubleshooting for instances where the SSL Server Test comes back as T / Certificate name mismatch. I've made it to the end of Step 5.

Protocol Support, Key Exchange, and Cipher Strength are all top marks, but SSL Test is marking me T because of the invalid cert.

Here's a link to my HAProxy Config - https://pastebin.com/P5QtYPUt (https://pastebin.com/P5QtYPUt)

My ACME Client log looks identical to your screenshot in the tutorial. I have redone the issue/renew procedure and the log comes back looking happy again.

I'm happy to post screenshots that would help diagnose. Appreciate any redirect on where to start looking.

The error pretty much says it all. I really don't know why you have to ask for help. I wonder if people nowadays are just too lazy to simply google such easy errors themselve?

Actually you shouldn't even need to google it. I might start taking money for giving lazy support...
https://www.globalsign.com/en/blog/what-is-common-name-mismatch-error

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: coreyinoz on March 11, 2023, 06:37:22 pm

Quote from: TheHellSite on March 11, 2023, 12:14:21 pm

Quote from: coreyinoz on March 11, 2023, 12:36:39 am

So with apologies in advance, I'm hoping you can offer some troubleshooting for instances where the SSL Server Test comes back as T / Certificate name mismatch. I've made it to the end of Step 5.

The error pretty much says it all. I really don't know why you have to ask for help. I wonder if people nowadays are just too lazy to simply google such easy errors themselves?

Actually you shouldn't even need to google it. I might start taking money for giving lazy support...
https://www.globalsign.com/en/blog/what-is-common-name-mismatch-error

Yikes. I'm sorry to have put you out. I guess one of the risks of having a clear tutorial is that people like me can get in beyond their expertise.

I didn't realize that inserting 'www' in front of my domain on the SSL test was the difference between a pass or fail. I get an A when testing www.mydomain.dedyn.io

Thanks for the help.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: otherego on March 20, 2023, 08:18:58 pm

Thank you fir this nice guide i followed to replace nginx reverse proxy. i left my
 nexcloud server untouched.
After following the guide, i've got "too many redirects" error.
i've looking around for a possible solution... but i don't understand what is wrong
i'm using cloudflare and this it's apache configuration

Code: [Select]

<VirtualHost *:80>
    DocumentRoot "/usr/local/www/nextcloud"
    ServerName nextcloud.mysite.com
<FilesMatch \.php$>
        SetHandler "proxy:fcgi://127.0.0.1:9000/"
    </FilesMatch>
    DirectoryIndex /index.php index.php
</VirtualHost>

config.php

Code: [Select]


{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "10.0.0.48",
            "nextcloud.mysite.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "25.0.4.1",
        "overwrite.cli.url": "https:\/\/nextcloud.mysite.com",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "overwritehost": "nextcloud.mysite.com",
        "overwriteprotocol": "https",
        "overwritecondaddr": "^10\\.0\\.0\\.1$",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "updater.release.channel": "stable",
        "default_phone_region": "IT",
        "app_install_overwrite": [],
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "data-fingerprint": "d1c023081e0c9b662bc8049cf295c443"
    }
}



 haproxy.conf

Code: [Select]


global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening on 0.0.0.0:80. 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_64188d5dce2390.01132494 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_64188d5dce2390.01132494

# Frontend: 1_HTTPS_frontend (Lisening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/64189270e357f4.63771565.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/64188dd26c8986.37023969.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Nextcloud_backend ()
backend Nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nextcloud_server 10.0.0.48:80 ssl verify none





Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sapphire on March 21, 2023, 11:07:57 am

Quote from: meyergru on March 11, 2023, 09:57:24 am

Quote from: sapphire on March 10, 2023, 10:00:11 am

With a "wrong" MTU, shouldn't the VPN connection be shaky with every device from every "outside" network? It's working 100% all the time on my mobile internet (and my girlfriend's as well), and it's working 100% all the time for ssllabs but only some "Is it down or just me"-kinda sites. But for at least 2 friends (one using the newest Opera Browser on Windows), there are timeouts while trying to connect to my websites. Same for a Windows VPS hosted on AWS - can't get a handshake there either (using newest Chrome browser).

I got no problems with my mailserver/proxmox mail gateway at home. I got no hickups with SSH via NAT. I got no issues with gaming servers at home (friends can connect to it). So I think, something isn't working correctly with my haproxy, sadly :(

That depends on PMTU discovery, so not every connection has to fail. Try reducing MTU/MSS just for the wireguard interface group like so:

(https://gcdnb.pbrd.co/images/E2FzctSpuJlY.png)

OMG, this fixed it! Now I can reach my addresses even with before problematic peers. Thank you so much @thehellsite and @meyergru!

Root Domains
Now I got another question. Did I understand it right, that the tutorial is only working with subdomains, not with root domains? I think, I would have to setup rules to achieve redirects from example.com to www.example.com right?

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 22, 2023, 06:49:12 am

Quote from: sapphire on March 21, 2023, 11:07:57 am

OMG, this fixed it! Now I can reach my addresses even with before problematic peers. Thank you so much @thehellsite and @meyergru!

Root Domains
Now I got another question. Did I understand it right, that the tutorial is only working with subdomains, not with root domains? I think, I would have to setup rules to achieve redirects from example.com to www.example.com right?

You are welcome. Like I (we) said most of the times when there are intermidiate issues with VPN tunnels it is because of MTU/MSS beeing to high resulting in packet fragmentation.

You can also make it work for root domains. Either set the backend you want to be served on the root domain as default backend in the HTTPS_frontend or change your mapfile and write the full FQDN before the backend (so not just the subdomain). The first method works for sure the later I am not entirely sure. But you can also create a redirect condition and rule set and place it before the map file rule.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 22, 2023, 09:33:32 am

Quote from: otherego on March 20, 2023, 08:18:58 pm

Thank you fir this nice guide i followed to replace nginx reverse proxy. i left my
 nexcloud server untouched.
After following the guide, i've got "too many redirects" error.
i've looking around for a possible solution... but i don't understand what is wrong
i'm using cloudflare and this it's apache configuration

Code: [Select]

<VirtualHost *:80>
    DocumentRoot "/usr/local/www/nextcloud"
    ServerName nextcloud.mysite.com
<FilesMatch \.php$>
        SetHandler "proxy:fcgi://127.0.0.1:9000/"
    </FilesMatch>
    DirectoryIndex /index.php index.php
</VirtualHost>

config.php

Code: [Select]


{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "10.0.0.48",
            "nextcloud.mysite.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "25.0.4.1",
        "overwrite.cli.url": "https:\/\/nextcloud.mysite.com",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "overwritehost": "nextcloud.mysite.com",
        "overwriteprotocol": "https",
        "overwritecondaddr": "^10\\.0\\.0\\.1$",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "updater.release.channel": "stable",
        "default_phone_region": "IT",
        "app_install_overwrite": [],
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "data-fingerprint": "d1c023081e0c9b662bc8049cf295c443"
    }
}



 haproxy.conf

Code: [Select]


global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening on 0.0.0.0:80. 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_64188d5dce2390.01132494 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_64188d5dce2390.01132494

# Frontend: 1_HTTPS_frontend (Lisening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/64189270e357f4.63771565.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/64188dd26c8986.37023969.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Nextcloud_backend ()
backend Nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nextcloud_server 10.0.0.48:80 ssl verify none





Your nextcloud server is misconfigured in haproxy, I highly doubt your nextcloud needs SSL ticked. But this might not be the only issue and I won't give support for services I am not using.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: otherego on March 22, 2023, 08:26:40 pm

thank you btw i've tried to remove SSL ticket too.. but unfortunately error still there..
i have no clue where i can get some help. things are mixing so need to be adviced for nextcloud, for haproxy and how to implement into opnsense...

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Bunch on March 23, 2023, 02:14:27 am

Quote from: otherego on March 20, 2023, 08:18:58 pm

Thank you fir this nice guide i followed to replace nginx reverse proxy. i left my
 nexcloud server untouched.
After following the guide, i've got "too many redirects" error.
i've looking around for a possible solution... but i don't understand what is wrong
i'm using cloudflare and this it's apache configuration

Code: [Select]

<VirtualHost *:80>
    DocumentRoot "/usr/local/www/nextcloud"
    ServerName nextcloud.mysite.com
<FilesMatch \.php$>
        SetHandler "proxy:fcgi://127.0.0.1:9000/"
    </FilesMatch>
    DirectoryIndex /index.php index.php
</VirtualHost>

config.php

Code: [Select]


{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "10.0.0.48",
            "nextcloud.mysite.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "25.0.4.1",
        "overwrite.cli.url": "https:\/\/nextcloud.mysite.com",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "overwritehost": "nextcloud.mysite.com",
        "overwriteprotocol": "https",
        "overwritecondaddr": "^10\\.0\\.0\\.1$",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "updater.release.channel": "stable",
        "default_phone_region": "IT",
        "app_install_overwrite": [],
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "data-fingerprint": "d1c023081e0c9b662bc8049cf295c443"
    }
}



 haproxy.conf

Code: [Select]


global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening on 0.0.0.0:80. 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_64188d5dce2390.01132494 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_64188d5dce2390.01132494

# Frontend: 1_HTTPS_frontend (Lisening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/64189270e357f4.63771565.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/64188dd26c8986.37023969.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Nextcloud_backend ()
backend Nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nextcloud_server 10.0.0.48:80 ssl verify none





Have you tried removing the value of the following items?
"overwrite.cli.url"
"overwritehost"
"overwriteprotocol"
"overwritecondaddr"

Since SNI is handled by HAProxy already. I don't think they are necessary.
And the issue might caused by this too.
(HAProxy tried to reach 10.0.0.48:80, nextcloud response to browser it need to "overwrite"(redirect) to https://nextcloud.mysite.com, but browser is already accessing with this path. So the browser don't know what should be redirect (or having dead loop for the worst case) and give you "too many redirects" error.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Grenen on March 23, 2023, 10:07:21 pm

I need some help with ciphers and understanding the part in HTTPS_frontend and what i can change.

Im trying to setup a webhook to integrate SmartThings to my Homeassistant, and get a error when trying to validate it.

According to the troubleshooting-guide at https://www.home-assistant.io/integrations/smartthings/#troubleshooting (https://www.home-assistant.io/integrations/smartthings/#troubleshooting) there are some problems doing this with a reverse proxy, and suggest that the cipher suite is too restricted.

Quote

Some reverse proxy configuration settings can interfere with communication from SmartThings. For example, TLSv1.3 is not supported. Setting the supported cipher suite too restrictly will prevent handshaking. The following NGINX SSL configuration is known to work:

Code: [Select]

# cert.crt also contains intermediate certificates
ssl_certificate /path/to/cert.crt;
ssl_certificate_key /path/to/cert.key;
ssl_dhparam /path/to/dhparam.pem;
ssl_protocols TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_ecdh_curve secp384r1;
ssl_session_timeout  10m;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;

Could someone help me with the current string and how i can edit it with the suggestion the troubleshooting-guide above. Is it enought to add EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH to the current, or do i need to edit something out as well?

Current:
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384


Thanks in advance,

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: otherego on March 23, 2023, 10:10:12 pm

i removed
"overwrite.cli.url"
"overwritehost"
"overwriteprotocol"
"overwritecondaddr"
from my php.conf, but i still have same result

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 24, 2023, 03:58:24 am

Quote from: Grenen on March 23, 2023, 10:07:21 pm

I need some help with ciphers and understanding the part in HTTPS_frontend and what i can change.

Im trying to setup a webhook to integrate SmartThings to my Homeassistant, and get a error when trying to validate it.

According to the troubleshooting-guide at https://www.home-assistant.io/integrations/smartthings/#troubleshooting (https://www.home-assistant.io/integrations/smartthings/#troubleshooting) there are some problems doing this with a reverse proxy, and suggest that the cipher suite is too restricted.

Quote

Some reverse proxy configuration settings can interfere with communication from SmartThings. For example, TLSv1.3 is not supported. Setting the supported cipher suite too restrictly will prevent handshaking. The following NGINX SSL configuration is known to work:

Code: [Select]

# cert.crt also contains intermediate certificates
ssl_certificate /path/to/cert.crt;
ssl_certificate_key /path/to/cert.key;
ssl_dhparam /path/to/dhparam.pem;
ssl_protocols TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_ecdh_curve secp384r1;
ssl_session_timeout  10m;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;

Could someone help me with the current string and how i can edit it with the suggestion the troubleshooting-guide above. Is it enought to add EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH to the current, or do i need to edit something out as well?

Current:
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384


Thanks in advance,

My provided ciphers are fine! Also TLS_v1.2 is available with my config. If TLS_v1.3 is not available on the client side it will (try) to use TLS_v1.2 instead.
Don't weaken the ciphers there is likely another configuration problem on your side.

If there are no errors in the haproxy log upon connection of the SmartThings client then there is nothing wrong with the haproxy cipher settings.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Grenen on March 24, 2023, 07:48:26 am

Quote from: TheHellSite on March 24, 2023, 03:58:24 am

My provided ciphers are fine! Also TLS_v1.2 is available with my config. If TLS_v1.3 is not available on the client side it will (try) to use TLS_v1.2 instead.
Don't weaken the ciphers there is likely another configuration problem on your side.

If there are no errors in the haproxy log upon connection of the SmartThings client then there is nothing wrong with the haproxy cipher settings.

I get a handshake failure in the log when trying to establish the webhook:

Quote

52.213.77.15:56225 [24/Mar/2023:07:34:18.143] 1_HTTPS_frontend/127.0.0.1:443: SSL handshake failure

The error matches the text in the setup guide for HA+SmartThings "Setting the supported cipher suite too restrictly will prevent handshaking."

Though i dont want too weak cipher, id like to test and add the EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+ED, but i dont understand how to build them in which order.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on March 24, 2023, 08:42:03 am

Just use Mozilla SSL Generator.

But the safest solution is to find out which ciphers are supported by SmartThings and then take the one with the highest cryptographic strength and add it to the end of my cipher list. Cipher suites can be left untouched since they only apply to TLS_v1.3 connections, which seem to be unsupported by SmartThings anyway.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: aleksander.lyse@gmail.com on March 24, 2023, 05:00:55 pm

I have followed the guide, and I think to the letter. Everything works on absolutely all services I added, except Portainer. I see on other forums Portainer is a bit choosy when it comes to proxy, but no real solution out there to fix it.

The target of this is to proxy portainer.domain.tld to ip-of-portainer:9000 (portainer docs say to route proxy to 9000, the non ssl port, not 9443, the ssl port), but I have tried both. Same result.

The error is: ERR_HTTP2_SERVER_REFUSED_STREAM - I was thinking it could e the HTTP2 option in 1_HTTPS_frontend, so I tried to uncheck it to try. It then give the error: ERR_EMPTY_RESPONSE

This is the corresponding log in debug mode:

With HTTP2 unchecked:

Code: [Select]

2023-03-24T16:58:08 Informational haproxy 10.0.1.100:65204 [24/Mar/2023:16:58:08.850] 0_SNI_frontend SSL_backend/SSL_server 1/0/11 506 -- 1/1/4/0/0 0/0
2023-03-24T16:58:08 Informational haproxy 10.0.1.100:65204 [24/Mar/2023:16:58:08.856] 1_HTTPS_frontend~ SSL_backend/<NOSRV> -1/-1/-1/-1/0 500 0 - - IR-- 2/1/5/0/0 0/0 "GET / HTTP/1.1"
With HTTP2 checked:

Quote

2023-03-24T16:59:29   Informational   haproxy   10.0.1.100:65272 [24/Mar/2023:16:59:29.608] 1_HTTPS_frontend~ SSL_backend/<NOSRV> -1/-1/-1/-1/0 500 0 - - IR-- 2/1/10/0/0 0/0 "GET https://portainer.domain.tld/ HTTP/2.0"   
2023-03-24T16:59:29   Informational   haproxy   10.0.1.100:65272 [24/Mar/2023:16:59:29.604] 1_HTTPS_frontend~ SSL_backend/<NOSRV> -1/-1/-1/-1/0 500 0 - - IR-- 2/1/9/0/0 0/0 "GET https://portainer.domain.tld/ HTTP/2.0"   
2023-03-24T16:59:29   Informational   haproxy   10.0.1.100:65272 [24/Mar/2023:16:59:29.602] 1_HTTPS_frontend~ SSL_backend/<NOSRV> -1/-1/-1/-1/0 500 0 - - IR-- 2/1/8/0/0 0/0 "GET https://portainer.domain.tld/ HTTP/2.0"

Here is the full config export:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua
cache opnsense-haproxy-cache
    total-max-size 4
    max-age 60
    process-vary off

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option tcplog

# Frontend: 1_HTTP_frontend (Listening on 10.0.1.15:80)
frontend 1_HTTP_frontend
    bind 10.0.1.15:80 name 10.0.1.15:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_6418a05ee27409.36162049 ssl_fc

    # ACTION: HTTP_to_HTTPS_rule
    http-request redirect scheme https code 301 if !acl_6418a05ee27409.36162049

# Frontend: 1_HTTPS_frontend (Listening on 10.0.1.15:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 10.0.1.15:443 name 10.0.1.15:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets strict-sni ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6418a3dac5bd67.76211333.certlist
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option httplog
    # ACL: LOCAL_SUBDOMAINS_SUBNET_conditions
    acl acl_6418cb7541c572.65233357 src 10.0.1.0/24 10.0.3.0/24 10.0.4.0/24

    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6418cadaadbdd1.17042036.txt,SSL_backend)] if acl_6418cb7541c572.65233357
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6418a10a9104c5.33815067.txt,SSL_backend)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 10.0.1.15 send-proxy-v2 check-send-proxy

# Backend: Dashboard_backend ()
backend Dashboard_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Dashboard_server 10.0.3.4:3000

# Backend: Radarr_backend ()
backend Radarr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Radarr_server 10.0.3.6:7878

# Backend: Router_backend ()
backend Router_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Router_server 10.0.1.1:8443

# Backend: Nzbget_backend ()
backend Nzbget_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nzbget_server 10.0.3.7:6789

# Backend: Nas_backend ()
backend Nas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nas_server 10.0.1.10:5000

# Backend: Jackett_backend ()
backend Jackett_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Jackett_server 10.0.3.8:9117

# Backend: Sonarr_backend ()
backend Sonarr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Sonarr_server 10.0.3.5:8989

# Backend: Portainer_backend ()
backend Portainer_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Portainer_server 10.0.3.3:9000 send-proxy-v2 check-send-proxy

# Backend: Plex_backend ()
backend Plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Plex_server 10.0.1.10:32400

# Backend: Overseerr_backend ()
backend Overseerr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Overseerr_server 10.0.3.10:5055

# Backend: Tautulli_backend ()
backend Tautulli_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Tautulli_server 10.0.3.9:8181

# Backend: Download_backend ()
backend Download_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Download_server 10.0.1.10:8000

# Backend: Nzb_backend ()
backend Nzb_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nzb_server 10.0.3.11:8080



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED


Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DoomSalamander on March 28, 2023, 07:58:21 am

Hello,
thanks for this great guide. I am trying to get a connection to a Jellyfin server working and at first glance the proxy works but for whatever reason it loads extremely slow or does not load at all (probably due to timeout). In the firewall logs I can see that the ports are not being blocked and Jellyfin is being accessed by the correct port coming from the interface gateway. Is this correct behaviour or should it come from the proxyserver IP?

Here is my full HAProxy config. I have checked it multiple times and I think it is set up correctly. The only difference is that I haven't checked the SSL mark for the real server Jellyfin since I have not set up a certificate on it yet but maybe I have just overlooked something.

The server is on it's own vlan maybe this causes issues too.

Code: [Select]

#
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_Backend
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.1.0.1:80 name 127.1.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    maxconn 10

    # logging options
    # ACL: NoSSL_condition
    acl acl_641d7e3e6bda64.61444458 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_641d7e3e6bda64.61444458

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    bind 127.1.0.1:443 name 127.1.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/641d813861abd5.14037775.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option log-separate-errors
    option httplog
    option socket-stats

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/641d81c0801b87.43530099.txt)]

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server 127.1.0.1 send-proxy-v2 check-send-proxy

# Backend: Jellyfin_Backend ()
backend Jellyfin_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Jellyfin_Server 10.0.0.2:8096



# statistics are DISABLED


In the logs I see some errors when trying to access Jellyfin:

Code: [Select]

2023-03-28T07:32:59 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.814] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/69396 -1 0 - - CC-- 2/1/0/0/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Resume?Limit=12&Recursive=true&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&EnableTotalRecordCount=false&MediaTypes=Book HTTP/2.0"
2023-03-28T07:32:59 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.823] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/69386 -1 0 - - CC-- 2/1/1/1/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=2c29078a81f6a6f262bb18c85a177434 HTTP/2.0"
2023-03-28T07:32:59 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.823] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/69386 -1 0 - - CC-- 2/1/2/2/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=7a2175bccb1f1a94152cbd2b2bae8f6d HTTP/2.0"
2023-03-28T07:32:59 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.823] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/69386 -1 0 - - CC-- 2/1/3/3/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=c5b5a7b14778a8f0c4eb996823229f65 HTTP/2.0"
2023-03-28T07:32:59 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.823] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/69380 -1 0 - - CC-- 2/1/4/4/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=0c41907140d802bb58430fed7e2cd79e HTTP/2.0"
2023-03-28T07:32:35 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:32:05.583] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30003/4/30007 200 245 - - ---- 2/1/5/5/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/System/Endpoint HTTP/2.0"
2023-03-28T07:32:29 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:29.401] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30003/5/60069 101 392 - - sD-- 2/1/6/6/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/socket?api_key=67299c39e6ab4eb084116da10f7866b9&deviceId=TW96aWxsYS81LjAgKExpbnV4OyBBbmRyb2lkIDEyOyBYUS1BUzUyKSBBcHBsZVdlYktpdC81MzcuMzYgKEtIVE1MLCBsaWtlIEdlY2tvKSBDaHJvbWUvMTEwLjAuNTQ4MS4xOTIgTW9iaWxlIFNhZmFyaS81MzcuMzYgT1BSLzc0LjEuMzkyMi43MTE5OXwxNjc5ODA2ODI4MzI2 HTTP/2.0"
2023-03-28T07:32:19 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.823] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30000/8/30008 200 1154 - - ---- 2/1/7/7/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=3a6e3078ba2b0ad4408f206c9abe65a4 HTTP/2.0"
2023-03-28T07:32:19 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.811] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30011/9/30020 200 256 - - ---- 2/1/8/8/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Resume?Limit=12&Recursive=true&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&EnableTotalRecordCount=false&MediaTypes=Audio HTTP/2.0"
2023-03-28T07:32:19 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.528] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30003/2/30005 200 1095 - - ---- 2/1/9/9/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432 HTTP/2.0"
2023-03-28T07:32:19 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.493] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30011/2/30013 200 1095 - - ---- 2/1/10/10/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432 HTTP/2.0"
2023-03-28T07:31:59 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:29.401] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30003/4/30007 200 1097 - - ---- 2/1/10/10/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432 HTTP/2.0"
2023-03-28T07:30:44 Error haproxy xx.xxx.xx.xx:7837 [28/Mar/2023:07:30:36.432] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/0/4/8566 101 392 - - CD-- 2/1/0/0/0 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/socket?api_key=67299c39e6ab4eb084116da10f7866b9&deviceId=TW96aWxsYS81LjAgKExpbnV4OyBBbmRyb2lkIDEyOyBYUS1BUzUyKSBBcHBsZVdlYktpdC81MzcuMzYgKEtIVE1MLCBsaWtlIEdlY2tvKSBDaHJvbWUvMTEwLjAuNTQ4MS4xOTIgTW9iaWxlIFNhZmFyaS81MzcuMzYgT1BSLzc0LjEuMzkyMi43MTE5OXwxNjc5ODA2ODI4MzI2 HTTP/2.0"
2023-03-28T07:30:44 Error haproxy xx.xxx.xx.xx:7837 [28/Mar/2023:07:30:36.432] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/8566 -1 0 - - CC-- 2/1/1/1/0 0/0 "POST https://jellyfin.MYDOMAIN.dedyn.io/Sessions/Capabilities/Full HTTP/2.0"
2023-03-28T07:30:44 Error haproxy xx.xxx.xx.xx:7837 [28/Mar/2023:07:30:36.432] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/8566 -1 0 - - CC-- 2/1/2/2/0 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432 HTTP/2.0"
2023-03-28T07:30:44 Error haproxy xx.xxx.xx.xx:7837 [28/Mar/2023:07:30:36.432] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/8558 -1 0 - - CC-- 2/1/3/3/0 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/DisplayPreferences/usersettings?userId=0d64d22975ba4cd6a57253d0e138c432&client=emby HTTP/2.0"
2023-03-28T07:30:35 Error haproxy xx.xxx.xx.xx:7837 [28/Mar/2023:07:30:05.079] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30011/3/30014 302 121 - - ---- 2/1/0/0/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/ HTTP/2.0"
2023-03-27T18:30:31 Error haproxy xx.xxx.xx.xx:23349 [27/Mar/2023:18:29:31.309] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/60030 -1 0 - - CC-- 1/1/0/0/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=7a2175bccb1f1a94152cbd2b2bae8f6d HTTP/2.0"
2023-03-27T18:30:31 Error haproxy xx.xxx.xx.xx:23349 [27/Mar/2023:18:29:30.979] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/60361 -1 0 - - CC-- 1/1/1/1/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432 HTTP/2.0"
2023-03-27T18:30:31 Error haproxy xx.xxx.xx.xx:23349 [27/Mar/2023:18:29:31.309] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/60021/-1/60030 -1 1220 - - CD-- 1/1/2/2/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=2c29078a81f6a6f262bb18c85a177434 HTTP/2.0"

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: otherego on March 29, 2023, 01:11:23 pm

Quote from: TheHellSite on March 22, 2023, 09:33:32 am

Quote from: otherego on March 20, 2023, 08:18:58 pm

Thank you fir this nice guide i followed to replace nginx reverse proxy. i left my
 nexcloud server untouched.
After following the guide, i've got "too many redirects" error.
i've looking around for a possible solution... but i don't understand what is wrong
i'm using cloudflare and this it's apache configuration

Code: [Select]

<VirtualHost *:80>
    DocumentRoot "/usr/local/www/nextcloud"
    ServerName nextcloud.mysite.com
<FilesMatch \.php$>
        SetHandler "proxy:fcgi://127.0.0.1:9000/"
    </FilesMatch>
    DirectoryIndex /index.php index.php
</VirtualHost>

config.php

Code: [Select]


{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "10.0.0.48",
            "nextcloud.mysite.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "25.0.4.1",
        "overwrite.cli.url": "https:\/\/nextcloud.mysite.com",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "overwritehost": "nextcloud.mysite.com",
        "overwriteprotocol": "https",
        "overwritecondaddr": "^10\\.0\\.0\\.1$",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "updater.release.channel": "stable",
        "default_phone_region": "IT",
        "app_install_overwrite": [],
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "data-fingerprint": "d1c023081e0c9b662bc8049cf295c443"
    }
}



 haproxy.conf

Code: [Select]


global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening on 0.0.0.0:80. 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_64188d5dce2390.01132494 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_64188d5dce2390.01132494

# Frontend: 1_HTTPS_frontend (Lisening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/64189270e357f4.63771565.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/64188dd26c8986.37023969.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Nextcloud_backend ()
backend Nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nextcloud_server 10.0.0.48:80 ssl verify none





Your nextcloud server is misconfigured in haproxy, I highly doubt your nextcloud needs SSL ticked. But this might not be the only issue and I won't give support for services I am not using.

after some investigations i realized (by using s simple apache server behind haproxy) that is not nextcloud that gives too many redirects error but the trouble is coming because i'm using cloudflare as dns, by switching ssl to Full (strict) instead of Flexible.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: ckocank on April 08, 2023, 06:49:47 pm

Thank you. This works like a charm. I only change dns to cloudflare and uncheck SSL for real server of my docker container. Took me a week to set up everything.  :) :) :)

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 08, 2023, 07:07:08 pm

Quote from: aleksander.lyse@gmail.com on March 24, 2023, 05:00:55 pm

I have followed the guide, and I think to the letter. Everything works on absolutely all services I added, except Portainer. I see on other forums Portainer is a bit choosy when it comes to proxy, but no real solution out there to fix it.

The target of this is to proxy portainer.domain.tld to ip-of-portainer:9000 (portainer docs say to route proxy to 9000, the non ssl port, not 9443, the ssl port), but I have tried both. Same result.

The error is: ERR_HTTP2_SERVER_REFUSED_STREAM - I was thinking it could e the HTTP2 option in 1_HTTPS_frontend, so I tried to uncheck it to try. It then give the error: ERR_EMPTY_RESPONSE

This is the corresponding log in debug mode:

With HTTP2 unchecked:

Code: [Select]

2023-03-24T16:58:08 Informational haproxy 10.0.1.100:65204 [24/Mar/2023:16:58:08.850] 0_SNI_frontend SSL_backend/SSL_server 1/0/11 506 -- 1/1/4/0/0 0/0
2023-03-24T16:58:08 Informational haproxy 10.0.1.100:65204 [24/Mar/2023:16:58:08.856] 1_HTTPS_frontend~ SSL_backend/<NOSRV> -1/-1/-1/-1/0 500 0 - - IR-- 2/1/5/0/0 0/0 "GET / HTTP/1.1"
With HTTP2 checked:

Quote

2023-03-24T16:59:29   Informational   haproxy   10.0.1.100:65272 [24/Mar/2023:16:59:29.608] 1_HTTPS_frontend~ SSL_backend/<NOSRV> -1/-1/-1/-1/0 500 0 - - IR-- 2/1/10/0/0 0/0 "GET https://portainer.domain.tld/ HTTP/2.0"   
2023-03-24T16:59:29   Informational   haproxy   10.0.1.100:65272 [24/Mar/2023:16:59:29.604] 1_HTTPS_frontend~ SSL_backend/<NOSRV> -1/-1/-1/-1/0 500 0 - - IR-- 2/1/9/0/0 0/0 "GET https://portainer.domain.tld/ HTTP/2.0"   
2023-03-24T16:59:29   Informational   haproxy   10.0.1.100:65272 [24/Mar/2023:16:59:29.602] 1_HTTPS_frontend~ SSL_backend/<NOSRV> -1/-1/-1/-1/0 500 0 - - IR-- 2/1/8/0/0 0/0 "GET https://portainer.domain.tld/ HTTP/2.0"

Here is the full config export:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua
cache opnsense-haproxy-cache
    total-max-size 4
    max-age 60
    process-vary off

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option tcplog

# Frontend: 1_HTTP_frontend (Listening on 10.0.1.15:80)
frontend 1_HTTP_frontend
    bind 10.0.1.15:80 name 10.0.1.15:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_6418a05ee27409.36162049 ssl_fc

    # ACTION: HTTP_to_HTTPS_rule
    http-request redirect scheme https code 301 if !acl_6418a05ee27409.36162049

# Frontend: 1_HTTPS_frontend (Listening on 10.0.1.15:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 10.0.1.15:443 name 10.0.1.15:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets strict-sni ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6418a3dac5bd67.76211333.certlist
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option httplog
    # ACL: LOCAL_SUBDOMAINS_SUBNET_conditions
    acl acl_6418cb7541c572.65233357 src 10.0.1.0/24 10.0.3.0/24 10.0.4.0/24

    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6418cadaadbdd1.17042036.txt,SSL_backend)] if acl_6418cb7541c572.65233357
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6418a10a9104c5.33815067.txt,SSL_backend)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 10.0.1.15 send-proxy-v2 check-send-proxy

# Backend: Dashboard_backend ()
backend Dashboard_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Dashboard_server 10.0.3.4:3000

# Backend: Radarr_backend ()
backend Radarr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Radarr_server 10.0.3.6:7878

# Backend: Router_backend ()
backend Router_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Router_server 10.0.1.1:8443

# Backend: Nzbget_backend ()
backend Nzbget_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nzbget_server 10.0.3.7:6789

# Backend: Nas_backend ()
backend Nas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nas_server 10.0.1.10:5000

# Backend: Jackett_backend ()
backend Jackett_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Jackett_server 10.0.3.8:9117

# Backend: Sonarr_backend ()
backend Sonarr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Sonarr_server 10.0.3.5:8989

# Backend: Portainer_backend ()
backend Portainer_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Portainer_server 10.0.3.3:9000 send-proxy-v2 check-send-proxy

# Backend: Plex_backend ()
backend Plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Plex_server 10.0.1.10:32400

# Backend: Overseerr_backend ()
backend Overseerr_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Overseerr_server 10.0.3.10:5055

# Backend: Tautulli_backend ()
backend Tautulli_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Tautulli_server 10.0.3.9:8181

# Backend: Download_backend ()
backend Download_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Download_server 10.0.1.10:8000

# Backend: Nzb_backend ()
backend Nzb_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nzb_server 10.0.3.11:8080



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED


As I already said multiple times my tutorial is meant to get the overall setup running.
Your problem is basically off-topic. Since you are having trouble with a specific service you are better off asking in the respective forum/community. HAProxy isn't the issue here since all your other services are working.

Also, I would never ever expose something like Portainer using a reverse proxy, this is one of the dumbest things you could ever do!
With that beeing said, my guess would be that Portainer is simply blocking connections coming from a proxy (in this case HAProxy). If you are lucky there is an option to set a trusted proxy in Portainer, but I highly doubt this.

That is all you are going to get from me here - no support for service specific issues from my side and in this topic.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 08, 2023, 07:12:29 pm

Quote from: DoomSalamander on March 28, 2023, 07:58:21 am

Hello,
thanks for this great guide. I am trying to get a connection to a Jellyfin server working and at first glance the proxy works but for whatever reason it loads extremely slow or does not load at all (probably due to timeout). In the firewall logs I can see that the ports are not being blocked and Jellyfin is being accessed by the correct port coming from the interface gateway. Is this correct behaviour or should it come from the proxyserver IP?

Here is my full HAProxy config. I have checked it multiple times and I think it is set up correctly. The only difference is that I haven't checked the SSL mark for the real server Jellyfin since I have not set up a certificate on it yet but maybe I have just overlooked something.

The server is on it's own vlan maybe this causes issues too.

Code: [Select]

#
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_Backend
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.1.0.1:80 name 127.1.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    maxconn 10

    # logging options
    # ACL: NoSSL_condition
    acl acl_641d7e3e6bda64.61444458 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_641d7e3e6bda64.61444458

# Frontend: 1_HTTPS_frontend ()
frontend 1_HTTPS_frontend
    bind 127.1.0.1:443 name 127.1.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/641d813861abd5.14037775.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option log-separate-errors
    option httplog
    option socket-stats

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/641d81c0801b87.43530099.txt)]

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server 127.1.0.1 send-proxy-v2 check-send-proxy

# Backend: Jellyfin_Backend ()
backend Jellyfin_Backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Jellyfin_Server 10.0.0.2:8096



# statistics are DISABLED


In the logs I see some errors when trying to access Jellyfin:

Code: [Select]

2023-03-28T07:32:59 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.814] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/69396 -1 0 - - CC-- 2/1/0/0/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Resume?Limit=12&Recursive=true&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&EnableTotalRecordCount=false&MediaTypes=Book HTTP/2.0"
2023-03-28T07:32:59 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.823] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/69386 -1 0 - - CC-- 2/1/1/1/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=2c29078a81f6a6f262bb18c85a177434 HTTP/2.0"
2023-03-28T07:32:59 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.823] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/69386 -1 0 - - CC-- 2/1/2/2/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=7a2175bccb1f1a94152cbd2b2bae8f6d HTTP/2.0"
2023-03-28T07:32:59 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.823] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/69386 -1 0 - - CC-- 2/1/3/3/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=c5b5a7b14778a8f0c4eb996823229f65 HTTP/2.0"
2023-03-28T07:32:59 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.823] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/69380 -1 0 - - CC-- 2/1/4/4/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=0c41907140d802bb58430fed7e2cd79e HTTP/2.0"
2023-03-28T07:32:35 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:32:05.583] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30003/4/30007 200 245 - - ---- 2/1/5/5/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/System/Endpoint HTTP/2.0"
2023-03-28T07:32:29 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:29.401] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30003/5/60069 101 392 - - sD-- 2/1/6/6/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/socket?api_key=67299c39e6ab4eb084116da10f7866b9&deviceId=TW96aWxsYS81LjAgKExpbnV4OyBBbmRyb2lkIDEyOyBYUS1BUzUyKSBBcHBsZVdlYktpdC81MzcuMzYgKEtIVE1MLCBsaWtlIEdlY2tvKSBDaHJvbWUvMTEwLjAuNTQ4MS4xOTIgTW9iaWxlIFNhZmFyaS81MzcuMzYgT1BSLzc0LjEuMzkyMi43MTE5OXwxNjc5ODA2ODI4MzI2 HTTP/2.0"
2023-03-28T07:32:19 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.823] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30000/8/30008 200 1154 - - ---- 2/1/7/7/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=3a6e3078ba2b0ad4408f206c9abe65a4 HTTP/2.0"
2023-03-28T07:32:19 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.811] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30011/9/30020 200 256 - - ---- 2/1/8/8/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Resume?Limit=12&Recursive=true&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&EnableTotalRecordCount=false&MediaTypes=Audio HTTP/2.0"
2023-03-28T07:32:19 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.528] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30003/2/30005 200 1095 - - ---- 2/1/9/9/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432 HTTP/2.0"
2023-03-28T07:32:19 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:49.493] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30011/2/30013 200 1095 - - ---- 2/1/10/10/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432 HTTP/2.0"
2023-03-28T07:31:59 Error haproxy xx.xxx.xx.xx:9644 [28/Mar/2023:07:31:29.401] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30003/4/30007 200 1097 - - ---- 2/1/10/10/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432 HTTP/2.0"
2023-03-28T07:30:44 Error haproxy xx.xxx.xx.xx:7837 [28/Mar/2023:07:30:36.432] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/0/4/8566 101 392 - - CD-- 2/1/0/0/0 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/socket?api_key=67299c39e6ab4eb084116da10f7866b9&deviceId=TW96aWxsYS81LjAgKExpbnV4OyBBbmRyb2lkIDEyOyBYUS1BUzUyKSBBcHBsZVdlYktpdC81MzcuMzYgKEtIVE1MLCBsaWtlIEdlY2tvKSBDaHJvbWUvMTEwLjAuNTQ4MS4xOTIgTW9iaWxlIFNhZmFyaS81MzcuMzYgT1BSLzc0LjEuMzkyMi43MTE5OXwxNjc5ODA2ODI4MzI2 HTTP/2.0"
2023-03-28T07:30:44 Error haproxy xx.xxx.xx.xx:7837 [28/Mar/2023:07:30:36.432] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/8566 -1 0 - - CC-- 2/1/1/1/0 0/0 "POST https://jellyfin.MYDOMAIN.dedyn.io/Sessions/Capabilities/Full HTTP/2.0"
2023-03-28T07:30:44 Error haproxy xx.xxx.xx.xx:7837 [28/Mar/2023:07:30:36.432] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/8566 -1 0 - - CC-- 2/1/2/2/0 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432 HTTP/2.0"
2023-03-28T07:30:44 Error haproxy xx.xxx.xx.xx:7837 [28/Mar/2023:07:30:36.432] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/8558 -1 0 - - CC-- 2/1/3/3/0 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/DisplayPreferences/usersettings?userId=0d64d22975ba4cd6a57253d0e138c432&client=emby HTTP/2.0"
2023-03-28T07:30:35 Error haproxy xx.xxx.xx.xx:7837 [28/Mar/2023:07:30:05.079] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/30011/3/30014 302 121 - - ---- 2/1/0/0/1 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/ HTTP/2.0"
2023-03-27T18:30:31 Error haproxy xx.xxx.xx.xx:23349 [27/Mar/2023:18:29:31.309] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/60030 -1 0 - - CC-- 1/1/0/0/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=7a2175bccb1f1a94152cbd2b2bae8f6d HTTP/2.0"
2023-03-27T18:30:31 Error haproxy xx.xxx.xx.xx:23349 [27/Mar/2023:18:29:30.979] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/-1/-1/60361 -1 0 - - CC-- 1/1/1/1/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432 HTTP/2.0"
2023-03-27T18:30:31 Error haproxy xx.xxx.xx.xx:23349 [27/Mar/2023:18:29:31.309] 1_HTTPS_frontend~ Jellyfin_Backend/Jellyfin_Server 0/0/60021/-1/60030 -1 1220 - - CD-- 1/1/2/2/2 0/0 "GET https://jellyfin.MYDOMAIN.dedyn.io/Users/0d64d22975ba4cd6a57253d0e138c432/Items/Latest?Limit=16&Fields=PrimaryImageAspectRatio%2CBasicSyncInfo%2CPath&ImageTypeLimit=1&EnableImageTypes=Primary%2CBackdrop%2CThumb&ParentId=2c29078a81f6a6f262bb18c85a177434 HTTP/2.0"

Your HAProxy config looks fine.

I am also running Jellyfin and don't have the issues you are describing.
Either your firewall hardware just can't handle that big of a datastream with encryption or something is wrong with your Jellyfin server.

Either way your best bet is to configure and test other local services through HAProxy and see if the error is also present for them.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brandorf on April 19, 2023, 08:29:39 pm

Very detailed guide, but I've gone sideways somewhere.

I have two different wildcard certs, and that may be the cause of issues, but here's how I'm setup :

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on port 80, 443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTPS_frontend (Listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/64401520bca808.51986799.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/644012fb3e50a8.19725727.txt)]

# Frontend: 1_HTTP_frontend (Listening on localhost:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL
    acl acl_64401278359449.48644659 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_64401278359449.48644659

# Backend: PLEX_backend (For Plex Remote Play)
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PLEX_server 192.168.0.197:32400 ssl verify none

# Backend: VTT_backend (Foundry VTT)
backend VTT_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Foundry_VTT 192.168.0.197:30000 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server 127.0.0.1 send-proxy-v2 check-send-proxy



# statistics are DISABLED



Attempting to browse to vtt.brandorf.com just times out.  The odd thing is I'm not getting anything in the haproxy logs.

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: omaha2002@gmail.com on April 23, 2023, 06:53:19 pm

Quote from: TheHellSite on March 22, 2023, 06:49:12 am

Quote from: sapphire on March 21, 2023, 11:07:57 am

OMG, this fixed it! Now I can reach my addresses even with before problematic peers. Thank you so much @thehellsite and @meyergru!

Root Domains
Now I got another question. Did I understand it right, that the tutorial is only working with subdomains, not with root domains? I think, I would have to setup rules to achieve redirects from example.com to www.example.com right?

You are welcome. Like I (we) said most of the times when there are intermidiate issues with VPN tunnels it is because of MTU/MSS beeing to high resulting in packet fragmentation.

You can also make it work for root domains. Either set the backend you want to be served on the root domain as default backend in the HTTPS_frontend or change your mapfile and write the full FQDN before the backend (so not just the subdomain). The first method works for sure the later I am not entirely sure. But you can also create a redirect condition and rule set and place it before the map file rule.

The second method with adding the full FQDN before the backend to the mapfile works too, I tested it and now have two root domain services working beside all the subdomains.

Like this:
#public access subdomains
plex.xxxxxxx.nl            PLEX_backend
kuma.xxxxxxx.nl          KUMA_backend
#public access rootdomain 1
xxxxxxxxxx.nl              CYBERSAFETY_backend
#public access roodomain 2
xxxxxxxxxx.eu             EURO_backend

Title: Re: Tutorial 2023/03: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 27, 2023, 05:02:23 pm

Quote from: brandorf on April 19, 2023, 08:29:39 pm

Very detailed guide, but I've gone sideways somewhere.

I have two different wildcard certs, and that may be the cause of issues, but here's how I'm setup :

Attempting to browse to vtt.domain.tld just times out.  The odd thing is I'm not getting anything in the haproxy logs.

If I where you, I would immidiately remove any sensitive information (domain name, ...) from your post.

1. Your HAProxy config looks fine.

2. You can have as many correct or incorrect certificates in your frontends as you like. The connection will still be established. You will just get a certificate warning or the client will refuse to connect if it is a very badly misconfigured certificate.

3. Just think about... What could be the reasons that you don't even see any connection attempts in the HAProxy logs? That is simple troubleshooting right here!

At this point there are only two possibilities:
a) the traffic is not even reaching HAProxy (firewall issue)
b) HAProxy is not running (unlikely since your config looks fine and you probably didn't get any errors while saving the configuration)

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on April 27, 2023, 09:55:55 pm

Hello, I am very happy to have found this thread as I couldn't find much about all the HAproxy options in the Opnsense gui.  I plan to study this thread and learn how it all works.  I wanted to first introduce myself and show what I am trying to do.  Previously, I spend a coupe of weeks trying to configure HAproxy with moderate success.  But stumped in a couple of important areas.

Before I enabled HAproxy, I only had one domain, which i used a wildcard cert with cloudflare to host my internal services.  I use the swag reverse proxy, and it's all very easy.  In Opnsense, I just forward port 80,443 to the swag server. 

However, now I need another server to have open access to port 80,443 just like the swag server.  However, this seems easier said than done.  So I am trying HAproxy to accomplish this. 

The attached diagram shows what I am trying to do.

I will share my export soon as I troubleshoot this.  Basically, I have set up the real servers, backend pools, public services, conditions and rules.  I am using TCP mode for everything since the servers handle all the cert stuff themselves.  As I said, this is not working perfectly, only partially now.  I can't seem to figure out how to route the ports properly to the two servers in a clean way that makes sense.  All requests seem to go to only one server, again I will share the export and be more specific soon here.

I hope I am not overcomplicating my setup here.  If I could simply forward the ports to multiple places without HAproxy, i would do that instead.  This is a lot of configuring for what I am trying to do.  But if this is the proper way, i will go through with it.  Thanks.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 28, 2023, 01:53:23 am

Quote from: opnuser1 on April 27, 2023, 09:55:55 pm

However, now I need another server to have open access to port 80,443 just like the swag server.  However, this seems easier said than done.  So I am trying HAproxy to accomplish this.
[...]
I hope I am not overcomplicating my setup here.  If I could simply forward the ports to multiple places without HAproxy, i would do that instead.  This is a lot of configuring for what I am trying to do.  But if this is the proper way, i will go through with it.  Thanks.

With plain port forwards this is not possible as you can only forward a port once. So unless you have two public IPs this will not work.
You will have to use HAProxy in TCP mode to redirect to your SWAG proxy and to your NGINX proxy.

Quote from: opnuser1 on April 27, 2023, 09:55:55 pm

I hope I am not overcomplicating my setup here.
[...]
But if this is the proper way, i will go through with it.

In my opinion you are doing exactly this and I have to say you are doing it very good.

Quote from: opnuser1 on April 27, 2023, 09:55:55 pm

But if this is the proper way, i will go through with it.

Again, I really don't think this is the proper way unless you have a good reason for choosing this route.

I mean HAProxy (probably also NGINX and SWAG) support multiple domains including individual certs. Hell even subdomain specific certs.

So If I may ask: Why you don't you just use HAProxy for all this (reverse proxy, SSL, ...)? Is there any specific reason why you would want to start maintaining three reverse proxies in your network?




But to answer your question... Please try the following:
It should work for the first service but I am unsure about the second one.

You will obviously need to disable/delete your port forwarding rule and replace it with the WAN rule of part 4 of my tutorial. So just follow my guide below.

Credits: https://forum.opnsense.org/index.php?topic=18538.msg84958#msg84958
I have rewritten most of the guide by alh (https://forum.opnsense.org/index.php?topic=18538.msg84958#msg84958) so it is easier to understand and can be fully configured in the HAProxy GUI.

Code: [Select]

The guide below lists only those options that need to be changed, all other options need to be left untouched in their default state.

1. Create the real servers
        1.1 TCP_SERVICE1_server: IP, Port, SSL unticked
        1.2 TCP_SERVICE2_server: IP, Port, SSL unticked

2. Create the backends
        2.1 TCP_SERVICE1_backend
                Mode:    TCP
                Servers: TCP_SERVICE1_server

        2.2 TCP_SERVICE2_backend
                Mode:    TCP
                Servers: TCP_SERVICE2_server

3. Create the conditions
        3.1 TCP_SSL_condition
                Condition type: SSL Hello Type
                SSL Hello Type: 1 - client hello

        3.2 TCP_SERVICE1_condition
                Condition type: SNI TLS extension matches (TCP request content inspection)
                SNI Matches:    service1.domain.tld

        3.3 TCP_SERVICE2_condition
                Condition type: SNI TLS extension matches (TCP request content inspection)
                SNI Matches:    service2.domain.tld

4. Create the rules
        4.1 TCP_RequestInspectDelay_rule
                Select conditions:    Nothing selected
                Execute function:     tcp-request inspect delay
                TCP inspection delay: 5s

        4.2 TCP_RequestContentAccept_rule
                Select conditions:               TCP_SSL_condition
                Logical operator for conditions: none
                Execute function:                tcp-request content accept

        4.3 TCP_SERVICE1_rule
                Select conditions:               TCP_SERVICE1_condition
                Logical operator for conditions: none
                Execute function:                Use specified Backend Pool
                Use backend pool:                TCP_SERVICE1_backend

        4.4 TCP_SERVICE2_rule
                Select conditions:               TCP_SERVICE2_condition
                Logical operator for conditions: none
                Execute function:                Use specified Backend Pool
                Use backend pool:                TCP_SERVICE2_backend

5. Edit the SNI_frontend
   Note: This step assumes that you are not following my whole tutorial.
         However you have to at least finish the following steps of my tutorial:
         - Part 4: everything
         - Part 5: step 1-3 and step 10 (only create the SNI_frontend but without a Default Backend Pool)

         The exact order of the rules below is very important here!
         The "TCP_RequestInspectDelay_rule" always has to be the first rule.
         The "TCP_RequestContentAccept_rule" always has to be the second rule.

        5.1 Add the rules to the SNI_frontend in the following order:
                Select Rules: 1. TCP_RequestInspectDelay_rule
                              2. TCP_RequestContentAccept_rule
                              3. TCP_SERVICE1_rule
                              4. TCP_SERVICE2_rule


Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on April 28, 2023, 04:14:46 am

Oh man, thank you for this.  Thanks for confirming I am on the right track, that is comforting loll.

My reason for doing this is mainly because I tinker around with a lot of selfhosted type services.  And I spend way too much time getting all the network particulars work.  And it's always for the same reason, for one reason or another, I just need a server that has free and open access to port 80 and 443.  Every time.  Now, I usually work around it.  But now I just want to be able to do it once and for all.  it's been bugging me for like 10 years my inability to accomplish this.

So that's why lol.

Ok, I will post more later after I study your tips.  One thing I noticed, I did much of this already but I notice you created more rules than I would have ever thought of like those nothing selected rules.  Anyway, i think the rest made sense but I'll be back.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 28, 2023, 01:44:15 pm

Quote from: opnuser1 on April 28, 2023, 04:14:46 am

Oh man, thank you for this.  Thanks for confirming I am on the right track, that is comforting loll.

My reason for doing this is mainly because I tinker around with a lot of selfhosted type services.  And I spend way too much time getting all the network particulars work.  And it's always for the same reason, for one reason or another, I just need a server that has free and open access to port 80 and 443.  Every time.  Now, I usually work around it.  But now I just want to be able to do it once and for all.  it's been bugging me for like 10 years my inability to accomplish this.

So that's why lol.

If you want to set and forget it, then go with one reverse proxy handling port 80+443.
My choice would be HAProxy (or NGINX) on OPNsense directly.

Quote from: opnuser1 on April 28, 2023, 04:14:46 am

Ok, I will post more later after I study your tips.  One thing I noticed, I did much of this already but I notice you created more rules than I would have ever thought of like those nothing selected rules.  Anyway, i think the rest made sense but I'll be back.

Please report back if this worked so I can add it to the tutorial.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on April 28, 2023, 09:43:30 pm

Quote from: TheHellSite on April 28, 2023, 01:44:15 pm

If you want to set and forget it, then go with one reverse proxy handling port 80+443.
My choice would be HAProxy (or NGINX) on OPNsense directly.

I can do this yes and have considered.  but a couple of questions.  The reason i do this is because some software installers I am using expects fresh server with clear, open access to port 80 443, the way a VPS would from a paid hosting service.  And if it's not exactly like that, then it will get errors, then I have to do a lot of work solving all that.  And some software work better with different reverse proxies for whatever reason.  So if I have haproxy be the main reverse proxy, I still will want swag on a server behind it running yet another reverse proxy for a bunch of services.  This is why i concluded what I really want for ultimate flexibility is simply to forward port 80 and 443 to multiple places in environments with a single IP. 

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on April 29, 2023, 08:30:15 pm

Quick Update, I am following everything and it makes sense.  Except for this part where you say I only have to do steps 1-3 and 10 of part 5.  In the picture of the global preferences, you show using a rule to redirect HTTP to HTTPS.  That was created in earlier steps.  I'm guessing I need to create that rule, is that correct?

Another thing, I am actually trying to do your other steps because I like your idea of the A+ rating.  It seems everything goes well, but when I check the ACME log it is empty.  But if I go next to it for the system log, it says the cert was issued/renewed successfully and all the other things were successfull.

Does it necessarily have to show up in the ACME log?  I double checked my configuration and see no differences from yours.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 30, 2023, 02:01:01 am

Quote from: opnuser1 on April 29, 2023, 08:30:15 pm

Quick Update, I am following everything and it makes sense.  Except for this part where you say I only have to do steps 1-3 and 10 of part 5.  In the picture of the global preferences, you show using a rule to redirect HTTP to HTTPS.  That was created in earlier steps.  I'm guessing I need to create that rule, is that correct?

Another thing, I am actually trying to do your other steps because I like your idea of the A+ rating.  It seems everything goes well, but when I check the ACME log it is empty.  But if I go next to it for the system log, it says the cert was issued/renewed successfully and all the other things were successfull.

Does it necessarily have to show up in the ACME log?  I double checked my configuration and see no differences from yours.

You should start reading properly!

Code: [Select]

         - Part 5: step 1-3 and step 10 (only create the SNI_frontend but without a Default Backend Pool)

Quote

only create the SNI_frontend

Quote from: opnuser1 on April 29, 2023, 08:30:15 pm

Another thing, I am actually trying to do your other steps because I like your idea of the A+ rating.  It seems everything goes well, but when I check the ACME log it is empty.  But if I go next to it for the system log, it says the cert was issued/renewed successfully and all the other things were successfull.

Does it necessarily have to show up in the ACME log?  I double checked my configuration and see no differences from yours.

If you want the A+ rating you will have to use HAProxy for SSL offloading. But you told me twice now that you don't want to do this but rather keep using your other proxies.
Therefore you will have to configure this in them. --> No help from me here!

If you changed your opinion about that... Then please do me a favour and just follow my tutorial one-by-one.
You will save yourself a lot of trouble.

Also you said earlier that you have services that need port 80... I would really like to know which service requires port 80 on WAN if you can also run it on port 443 with a reverse proxy.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on April 30, 2023, 05:50:20 am

Quote

If you want the A+ rating you will have to use HAProxy for SSL offloading. But you told me twice now that you don't want to do this but rather keep using your other proxies.
Therefore you will have to configure this in them. --> No help from me here!

If you changed your opinion about that... Then please do me a favour and just follow my tutorial one-by-one.
You will save yourself a lot of trouble.

No!  You are absolutely right, I meant what I said initially.  I just got sidetracked.  Forget all that, I am still trying to do it the original way with the instructions you posted yesterday.

Here are my use cases for port 80 443 on multiple servers.
i run domain1.com and subdomains on one reverse proxy
i would like to run domain2.com on a similar reverse proxy for that
some applications work better on other reverse proxies, I'd like to have that option avaiable.
Some apps, they come with installation scripts.  It assumes you have a domain already pointed to the server with port 80 443 open as it has an nginx install for the domain in the install script.  So it will give errors and PIA to fix without the ports available.  i tried on my system, and it would not work since I am running that reverse proxy on the ports.  Then I tried on a spare Oracle cloud server and it works.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on April 30, 2023, 05:55:23 am

Here are the requirements for the app with that install script.  Again, I tried on several servers and the only one it worked smoothly was the oracle cloud with ports 80 443 open and free.

If I am not mistaken, this is the reason for needing port 80 open:
To install the service, you first download some files.  Then you need to build it out.  Once it is built, then you go to the website on port 80.  Here there is a button to run an installation script.  Once that is finished, now the website is secured on an nginx reverse proxy.  So port 80 is initially needed to install.  You can do without, but then you can't use the handy script.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 02, 2023, 12:05:46 am

I apologize for being a moron.  I entered 0:0:0:0 instead of 0.0.0.0
You can disregard all below, I just left it for reference.  I am now checking if the servers are working as expected, etc.  But HAProxy shows no errors.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 02, 2023, 05:14:24 pm

Quote from: opnuser1 on April 30, 2023, 05:55:23 am

Here are the requirements for the app with that install script.  Again, I tried on several servers and the only one it worked smoothly was the oracle cloud with ports 80 443 open and free.

If I am not mistaken, this is the reason for needing port 80 open:
To install the service, you first download some files.  Then you need to build it out.  Once it is built, then you go to the website on port 80.  Here there is a button to run an installation script.  Once that is finished, now the website is secured on an nginx reverse proxy.  So port 80 is initially needed to install.  You can do without, but then you can't use the handy script.

I think you don't quite understand the basics of networking very well.

If you place a reverse proxy, configured as described in my tutorial, in front any app this app can then run on ANY port and you will still be able to access it from external networks via haproxy on port 443/https.

The port 80 requirement might be valid. However I see absolutely no need to run one reverse proxy per app because of it. This goes against any sort of common sense. You don't need that at all. Not even during installation!

Just use haproxy and configure it exactly like in my tutorial.
Remove all other reverse proxies in your network.
Then configure all your currently active apps in haproxy.

If you now need to install another app using some install script (that for whatever reason needs port 80 during install)....
You just configure the app/service in haproxy on port 80, so you can access the install script.
After the installation is complete you simply change the port of the service in the haproxy server settings to the port the app has after installation.
Dead simple.

BTW: I don't get why you are making such a mistery out of the install scripts you are using. If you would have posted them in the beginning I could have told you right then what it is necessary.

Also remove your haproxy conf file. You exposed your domain.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 02, 2023, 05:58:45 pm

Let me see if I understand...

You are recommending that I stop using my reverse proxy, which is super easy to use, and move the 40 services I have running on that and to configure them in HAproxy instead?  So, you are asking me to configure 40 services in HAproxy?  That sounds daunting to me.

Quote

If you now need to install another app using some install script (that for whatever reason needs port 80 during install)....
You just configure the app/service in haproxy on port 80, so you can access the install script.
After the installation is complete you simply change the port of the service in the haproxy server settings to the port the app has after installation.
Dead simple.

Isn't this what I've been trying to do?  What do you mean by configure the app in haproxy on port 80?  In order for it to work, the domain needs to already be pointing to this app and accessible already from the outside on port 80.  I can't do it internally.  It's a web install script.

I like your recommendation, but it is not yet completely making sense to me, sorry.  I may have to chew on it for a while.

And there will come another time where I will want port 80 forwarded to yet another server.  I just want to be able to do it without modifying all the other apps that are already running fine.  If I do what you are saying and then later I want port 80 going somewhere else, then I'm in the same boat once again.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 02, 2023, 06:23:53 pm

Quote from: opnuser1 on May 02, 2023, 05:58:45 pm

Isn't this what I've been trying to do?  What do you mean by configure the app in haproxy on port 80?  In order for it to work, the domain needs to already be pointing to this app and accessible already from the outside on port 80.  I can't do it internally.  It's a web install script.

Again, please just link the install script? It is a bit hard to help without them.

Apart from that. Please verify with me...
The solution I posted here (https://forum.opnsense.org/index.php?topic=23339.msg163279#msg163279) for you is working correctly, right?
The traffic is beeing routed as desired using the SNI conditions (service1.domain.com, service2.domain.com, ...) to the respective backends?

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 02, 2023, 07:56:03 pm

(deleted)

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 02, 2023, 10:23:31 pm

No matter what I do, I can't seem to get activity into HAproxy.  I think I might be having this issue:

Quote

Warning

If you configure a port that is already in use, the configuration test will be successful but the start of HAProxy will fail silently. Please ensure that the used port is free - especially if the number conflicts with the web configuration of OPNsense.

But I disconnected the camera that was intercepting port 80, and it still wont work.  I must have 10-20 servers on the LAN that use port 80 443.  What am I supposed to do?  Shut them all down, and then enable haproxy?

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 02, 2023, 11:35:24 pm

Quote from: opnuser1 on May 02, 2023, 10:23:31 pm

No matter what I do, I can't seem to get activity into HAproxy.  I think I might be having this issue:

Quote

Warning

If you configure a port that is already in use, the configuration test will be successful but the start of HAProxy will fail silently. Please ensure that the used port is free - especially if the number conflicts with the web configuration of OPNsense.

But I disconnected the camera that was intercepting port 80, and it still wont work.  I must have 10-20 servers on the LAN that use port 80 443.  What am I supposed to do?  Shut them all down, and then enable haproxy?

Yes. This is how it works. You can't have multiple services listening on the same port. Pure logic.

Again, you DON'T need all services to have direct access to port 80 + 443, that is why you are setting up this reverse proxy.
It will handle this for you based on the subdomain.

Before you continue I HIGHLY suggest you look up what a reverse proxy does and what the difference to a basic port forward is.

Also disconnecting the camera won't help you in any way. You have to delete all port forwards for port 80+443.

Quote

A reverse proxy is a type of server that sits between a client and one or more servers, forwarding client requests to those servers. When a client makes a request to a website, for example, the request is first sent to the reverse proxy server, which then forwards the request on to the appropriate web server.

The main difference between a reverse proxy and a basic port forward is that a basic port forward simply forwards traffic from one IP address and port number to another IP address and port number. A reverse proxy, on the other hand, can act as an intermediary, providing additional services such as load balancing, caching, SSL termination, and more.

A reverse proxy can serve as a single point of contact for several different servers, distributing client requests across those servers and allowing them to work together to handle the load. In contrast, a basic port forward simply redirects traffic from one destination to another, without providing any additional functionality or benefits.

Overall, a reverse proxy is a more advanced and flexible solution for managing connections between clients and servers, particularly in situations where a large number of servers need to be balanced and coordinated to handle incoming traffic efficiently.

... from chatgpt.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 03, 2023, 12:33:11 am

(disregard)

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 03, 2023, 01:25:07 am

(deleted)

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 03, 2023, 03:23:54 pm

Quote from: opnuser1 on May 03, 2023, 01:25:07 am

I made the change I was suggesting above for the service1 condition.  I changed it from SNI matches to "contains" and used "domain1.com" rather than *.domain1.com.

Sorry, I missed that part in the uploaded haproxy config file you uploaded.
Yes, "host matches" strictly only matches the exact host specified, no additional third-level subdomains or the main domain will be matched with such a condition.
For your usecase the host contains condition is the best choice.

Quote from: opnuser1 on May 03, 2023, 01:25:07 am

Now the counter shows activity in both servers as requested, and it is routing correctly from the right place.

However, it still doesn't totally work, and for the server1 subdomains, I am getting a 526 error for invalid SSL certificate.  So I believe the traffic is going to the right place, but something is missing.
Should I have two SNI frontends?  One for port 80 and another for 443?

"it still doesn't totally work" What doesn't totally work (apart from the SSL error)?

I am a little confused now.
So the routing which FQDN of each of your domains goes to which reverse proxy and then from there to the desired service. This part if working, right?


About the SSL error, this one seems to be Cloudflare specific: https://community.cloudflare.com/t/community-tip-fixing-error-526-invalid-ssl-certificates/44273
I am not using them since I don't trust them so I can't help here.

But I can tell you that the method we just implemented can only inspect and route HTTPS/TLS traffic since SNI is an extension of the TLS protocol.
So unencrypted connections (like HTTP) won't work with the rules.

Therefore you will have to either use HTTPS from external networks only OR simply follow my whole tutorial and then create these conditions (host contains "domain1", host contains "domain2", the NoSSL_condition from my tutorial) and two rules (rule1: domain1_condition + NoSSL_condition goes to domain1_backend / rule2: same like rule1). Then place these rules on the HTTP_frontend of my tutorial.
You will not need the HTTPtoHTTPS_rule since you insist on using HTTP traffic.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 03, 2023, 07:27:15 pm

Ah thank you very much.
Ultimately, I think everything you instructed is working.  Even though that is a cloudflare specific error, it tells me that I probably need a different frontend for https and http, like your tutorial does.  This way you customized for me does indeed forward 80 and 443 successfully, but I need different conditions on each.

Yes, I like your suggestion to follow the original tutorial, I have been studying it and I like everything about it.  And yes, i do insist on the port 80 as well, so that would be the difference.  I was looking at it already and trying to figure out how to modify it for port 80, but you answered in the post above.

I am also keeping track of all this in a separate article, and I will post it when it is successful here to add to yours, but if all it is is not have that redirect rule, then not really a big deal I guess.  Let me try, and I'll report back.  Thanks, and it was amazing to see the traffic get routed properly.

edit...by "doesn't work" I mean the sites I am visiting externally don't load or anything.  I don't know the exact reason why, but I'm certain it has to do with what you said about SNI and TLS and there need to be different things happening in each pipe/port. 

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 04, 2023, 03:55:34 am

(deleted)

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 04, 2023, 11:12:59 am

Quote from: opnuser1 on May 04, 2023, 03:55:34 am

I made a diagram first to help.   Following your instructions, wouldn't I also have to create additional backends, rules, and servers since I am processing 80and 443 separately?  I included my attempt in the picture.

Not necessary, just leave the port in the server settings blank.
This way you can use a single server + backend combination (server1 + backend1) simultaneously for port 80/443/... as the port used to connect to the front will be used to connect to the server.

So in your diagram you would simply remove the port from server1+server2 and use rule1+rule2 on both the HTTP and HTTPS frontends.

Also you have a VERY big mistake in your current diagram. Unless you really want to load balance between server1 and server2 you HAVE to use a dedicated backend for each server. I think you still didn't quite understand this. Having both servers (server1+server2) in the same backend (http_backend or https_backend) will load balance between the servers and therefore send requests to different servers for each outside connection. I highly doubt you would want this, UNLESS load balancing is really the intend of your setup. But I thought you just wanted to route/forward http(s) traffic to the correct backend proxy?

You need...
server1_backend containing server1_server (port left blank)
server2_backend containing server2_server (port left blank)
server1_rule containing server1_backend as target
server2_rule containing server2_backend as target
http_frontend:80 containing server1_rule and server2_rule
https_frontend:443 containing server1_rule and server2_rule

This way no load balancing is done but the outside requests will be forwarded to the correct server:port target.


Also I thought you where already redirecting the HTTPS traffic using the SNI_frontend and steps I provided earlier?!
Sorry but your whole situation is very confusing!

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 04, 2023, 09:58:30 pm

Thank you.  yes it is confusing due to my lack of knowledge.  I'm going to redo the diagram and post it again based on these latest comments.

Yes, i do not intend to load balance.  You already understand that better than I do.

I didn't realize I could keep using my previous setup you helped me with.  I will continue with that.  I was about to go through your whole tutorial from scratch, and just remove that redirect rule.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 04, 2023, 10:08:39 pm

Quote

Not necessary, just leave the port in the server settings blank.

Holy moly!  I just blanked the port on the real servers of the previous configuration you helped me with and now things work quite well!!

server1
this is the one that handles my reverse proxy and all the subdomains.  everything is working now!!

server2
I think this is the one that needs help due to the fact that it needs 80 first before setting up 443.  Anyway, this one is not working as in i can't get in from outside on port 80 to run the install script.  So this one gets a 500 internal server error.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 04, 2023, 10:19:35 pm

Quote from: opnuser1 on May 04, 2023, 09:58:30 pm

Thank you.  yes it is confusing due to my lack of knowledge.  I'm going to redo the diagram and post it again based on these latest comments.

Yes, i do not intend to load balance.  You already understand that better than I do.

I didn't realize I could keep using my previous setup you helped me with.  I will continue with that.  I was about to go through your whole tutorial from scratch, and just remove that redirect rule.

Okay since you want the loadbalancing done on your other reverse proxy and your script things on the other server you will need this in total.

For the HTTP access
server1_condition host contains server1.domain.com
server2_condition host contains server2.domain.com
server1_backend containing server1_server (port left blank)
server2_backend containing server2_server (port left blank)
server1_rule containing server1_backend as target
server2_rule containing server2_backend as target
http_frontend:80 containing server1_rule and server2_rule


AND
for the https access/redirection the things I told you to configure earlier.

Code: [Select]

The guide below lists only those options that need to be changed, all other options need to be left untouched in their default state.

1. Create the real servers
        1.1 TCP_SERVICE1_server: IP, Port, SSL unticked
        1.2 TCP_SERVICE2_server: IP, Port, SSL unticked

2. Create the backends
        2.1 TCP_SERVICE1_backend
                Mode:    TCP
                Servers: TCP_SERVICE1_server

        2.2 TCP_SERVICE2_backend
                Mode:    TCP
                Servers: TCP_SERVICE2_server

3. Create the conditions
        3.1 TCP_SSL_condition
                Condition type: SSL Hello Type
                SSL Hello Type: 1 - client hello

        3.2 TCP_SERVICE1_condition
                Condition type: SNI TLS extension matches (TCP request content inspection)
                SNI Matches:    service1.domain.tld

        3.3 TCP_SERVICE2_condition
                Condition type: SNI TLS extension matches (TCP request content inspection)
                SNI Matches:    service2.domain.tld

4. Create the rules
        4.1 TCP_RequestInspectDelay_rule
                Select conditions:    Nothing selected
                Execute function:     tcp-request inspect delay
                TCP inspection delay: 5s

        4.2 TCP_RequestContentAccept_rule
                Select conditions:               TCP_SSL_condition
                Logical operator for conditions: none
                Execute function:                tcp-request content accept

        4.3 TCP_SERVICE1_rule
                Select conditions:               TCP_SERVICE1_condition
                Logical operator for conditions: none
                Execute function:                Use specified Backend Pool
                Use backend pool:                TCP_SERVICE1_backend

        4.4 TCP_SERVICE2_rule
                Select conditions:               TCP_SERVICE2_condition
                Logical operator for conditions: none
                Execute function:                Use specified Backend Pool
                Use backend pool:                TCP_SERVICE2_backend

5. Edit the SNI_frontend
   Note: This step assumes that you are not following my whole tutorial.
         However you have to at least finish the following steps of my tutorial:
         - Part 4: everything
         - Part 5: step 1-3 and step 10 (only create the SNI_frontend but without a Default Backend Pool)

         The exact order of the rules below is very important here!
         The "TCP_RequestInspectDelay_rule" always has to be the first rule.
         The "TCP_RequestContentAccept_rule" always has to be the second rule.

        5.1 Add the rules to the SNI_frontend in the following order:
                Select Rules: 1. TCP_RequestInspectDelay_rule
                              2. TCP_RequestContentAccept_rule
                              3. TCP_SERVICE1_rule
                              4. TCP_SERVICE2_rule

Set the SNI_frontend to listen on 0.0.0.0:443
Set the HTTP_frontend to listen on 0.0.0.0:80



When you are done post the haproxy config export.
But please in a codebox, no need to upload the file.
And don't forget to remove sensitive info.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 06, 2023, 04:32:23 am

(deleted)

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 06, 2023, 10:51:57 pm

OK, I gave it a shot.  Not sure if my http config is correct.  Servers seem to be working except for the one that needs http, I am currently checking it out.

But here is my config.

Code: [Select]


#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         0.0.0.0 local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend ()
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp

    # logging options
    option tcplog
    # ACL: TCP_SSL_condition
    acl acl_644c56b6785678.47181279 req.ssl_hello_type 1
    # ACL: NoSSL_condition
    acl acl_644d62959d73a1.59974462 ssl_fc
    # ACL: server1_condition
    acl acl_644c5700ee7657.09485748 req.ssl_sni -m sub -i domain1.com
    # ACL: server2_condition
    acl acl_644c5719768e71.87060950 req.ssl_sni -i domain2.com

    # ACTION: TCP_RequestInspectDelay_rule
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: TCP_RequestContentAccept_rule
    tcp-request content accept if acl_644c56b6785678.47181279
    # ACTION: server1_rule
    use_backend server1_backend if !acl_644d62959d73a1.59974462 acl_644c5700ee7657.09485748
    # ACTION: server2_rule
    use_backend server2_backend if !acl_644d62959d73a1.59974462 acl_644c5719768e71.87060950

# Frontend: HTTP_frontend ()
frontend HTTP_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp

    # logging options
    option tcplog
    # ACL: NoSSL_condition
    acl acl_644d62959d73a1.59974462 ssl_fc
    # ACL: server2_condition
    acl acl_644c5719768e71.87060950 req.ssl_sni -i domain2.com
    # ACL: server1_condition
    acl acl_644c5700ee7657.09485748 req.ssl_sni -m sub -i domain1.com
    acl acl_644c56b6785678.47181279 req.ssl_hello_type 1

    # ACTION: server2_rule
    use_backend server2_backend if !acl_644d62959d73a1.59974462 acl_644c5719768e71.87060950
    # ACTION: server1_rule
    use_backend server1_backend if !acl_644d62959d73a1.59974462 acl_644c5700ee7657.09485748
    # ACTION: TCP_RequestInspectDelay_rule
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: TCP_RequestContentAccept_rule
    tcp-request content accept if acl_644c56b6785678.47181279

# Backend: server1_backend ()
backend server1_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server server1_server 192.168.1.234

# Backend: server2_backend ()
backend server2_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server server2_server 192.168.1.231



# statistics are DISABLED



Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 06, 2023, 11:06:42 pm

(sorry)

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 06, 2023, 11:55:54 pm

Quote from: opnuser1 on May 06, 2023, 11:06:42 pm

I was reading this link:
https://blog.entrostat.com/routing-multiple-domains-using-haproxy-http-and-https-ssl/

And it says:

Quote

What you'll notice here is that I bind to port 80 using mode http but I bind to port 443 using mode tcp. This is to avoid the need for certificates on the 443 bind. Basically, what I'm doing here is routing 443 to a host and I expect that host to have the certificate set up.

Which makes me wonder do I need separate servers per port?  So 4 total...so that I can use different modes for each port?

Please don't take this the wrong way....
But dude, you are getting to a point where I am questioning your ability to simply do what you have been told to do.
You could have been done with whatever you are trying to achieve at this point by just following what I am saying. But no, you are out there looking for other solutions and then asking ME if this is the way?!

I am even telling you why you don't need separate servers+backends for this and you just don't listen/read or refuse to understand.

Remember I am providing a free tutorial and you are asking questions that are out of scope of this tutorial since the beginning. And I still helped you. However this is slowly getting to a point where I am just refusing to help any further.
So for the sake of my time (and yours) just do what I told you to do.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 07, 2023, 01:27:34 am

yes, this is indeed a wonderful totrial.  Sorry again.

I was just wondering.  My actual attempt was indeed trying to follow your instructions.

I see where I didn't follow your instructions.  My bad, yes I'm a moron.  Sorry about that too.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 07, 2023, 06:42:19 am

Quote

When you are done post the haproxy config export.
But please in a codebox, no need to upload the file.
And don't forget to remove sensitive info.

haproxy config below:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         0.0.0.0 local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend ()
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp

    # logging options
    option tcplog
    # ACL: TCP_SSL_condition
    acl acl_644c56b6785678.47181279 req.ssl_hello_type 1
    # ACL: TCP_server1_condition
    acl acl_644c5700ee7657.09485748 req.ssl_sni -m sub -i domain1.com
    # ACL: TCP_server2_condition
    acl acl_644c5719768e71.87060950 req.ssl_sni -i domain2.com

    # ACTION: TCP_RequestInspectDelay_rule
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: TCP_RequestContentAccept_rule
    tcp-request content accept if acl_644c56b6785678.47181279
    # ACTION: TCP_SERVICE1_rule
    use_backend TCP_SERVICE1_backend if acl_644c5700ee7657.09485748
    # ACTION: TCP_SERVICE2_rule
    use_backend TCP_SERVICE2_backend if acl_644c5719768e71.87060950

# Frontend: HTTP_frontend ()
frontend HTTP_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp

    # logging options
    option tcplog
    # ACL: http_server1_condition
    acl acl_6457247ca14984.71641345 hdr_sub(host) -i domain1.com
    # ACL: NoSSL_condition
    acl acl_644d62959d73a1.59974462 ssl_fc
    # ACL: http_server2_condition
    acl acl_64572496aeac32.73416688 hdr_sub(host) -i domain2.com

    # ACTION: http_server1_rule
    use_backend TCP_SERVICE1_backend if acl_6457247ca14984.71641345 !acl_644d62959d73a1.59974462
    # ACTION: http_server2_rule
    use_backend TCP_SERVICE2_backend if acl_64572496aeac32.73416688 !acl_644d62959d73a1.59974462

# Backend: TCP_SERVICE1_backend ()
backend TCP_SERVICE1_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server TCP_SERVICE1_server 192.168.1.234

# Backend: TCP_SERVICE2_backend ()
backend TCP_SERVICE2_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server TCP_SERVICE2_server 192.168.1.231



# statistics are DISABLED



Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on May 07, 2023, 07:43:59 am

Hello sir,

I followed your guide and everything worked for the past year. However, today for some reason, my server doesn't respond to my domain anymore. I have adguard and truenas setup so when I run those command ad.xxxx.xxxx, it resolved. However, if I just run mydomain.xxx then it doesn't resolve to my public IP address anymore.

May I ask what maybe the problem?

Thank you so much for your time sir.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 07, 2023, 10:48:17 am

Quote from: phamd4 on May 07, 2023, 07:43:59 am

Hello sir,

I followed your guide and everything worked for the past year. However, today for some reason, my server doesn't respond to my domain anymore. I have adguard and truenas setup so when I run those command ad.xxxx.xxxx, it resolved. However, if I just run mydomain.xxx then it doesn't resolve to my public IP address anymore.

May I ask what maybe the problem?

Thank you so much for your time sir.

Since you didn't provide any error codes you will have to give me some more info.
But I can assure you that if you didn't change anything on the OPNsense or in HAProxy it is likely an error in your DNS Zone.

Please post an haproxy Export. (check for sensitive info!)

Also tell me your certificate Common Name and Alt Names (replace your real domain name) as configured in the Let's Encrypt plugin certificate settings.

Lastly the relevant content of your DNS Zone (A record, CNAME record) at your domain registrar that are pointing to your domain.(replace sensitive Info)


Please keep any replaced info in the same style everywhere.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on May 07, 2023, 11:19:59 pm

Hello sir,

Thank you for getting back to me.

Attached is my HAProxy config

Code: [Select]

logo

    root@localdomain

Lobby
Reporting
System
Interfaces
Firewall
VPN
Services
ACME Client
Adguardhome
Captive Portal
DHCPv4
DHCPv6
Dnsmasq DNS
Dynamic DNS
FreeRADIUS
HAProxy
Settings
Statistics
Maintenance
Log File
Config Export
Intrusion Detection
Monit
Network Time
OpenDNS
Unbound DNS
Wake on LAN
Web Proxy
Zenarmor
Power
Help

    Services: HAProxy: Config Export

    Config Export
    Config Diff

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    6
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_630c2xxx9944 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_630c25xxx249944

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/630c280xxx7137226.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_630c2982907624.21524463 src 20.xxx.0/24 20xxx0/24 20xxx.0/24

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/630c2909dfxxx.10265915.txt)] if acl_630c2xxx7624.21524463
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/630c262xxx9c50.82551607.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: opnsense_backend ()
backend opnsense_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server opnsense_server 20.14.0.1:55443 ssl verify none

# Backend: truenas_backend ()
backend truenas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server truenas_server 20.xxx.4 ssl verify none

# Backend: proxmox_backend ()
backend proxmox_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server proxmox_server 20.xxx47:8xxx6 ssl verify none

# Backend: ad_backend ()
backend ad_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server ad_server 20.xxx.1:56xxx43 ssl verify none

# Backend (DISABLED): proxmox_backend_1 ()

# Backend (DISABLED): truenas_backend_1 ()



# statistics are DISABLED



OPNsense (c) 2014-2023 Deciso B.V.
As for my domain for the certificate I followed your guide and have it at
Common name *.xxxan.to

As for the DNS Zone I'm using your guide https://desec.io
CNAME is *.mydomainname.to
I don't have A record (I didn't see it in your guide)

I checked and my dyanmic DNS server still update my IP address to desec. Also when I tried to restart my authentication service, I see my token on desec being rewnewed. So I think that aspect it work.

I can access my domain if i have that server running i.e. adguard, plex. However, if I just type in my domain name it doesn't resolve my public ip anymore. I think it at desec end but I've tried to add and delete my domain but nothing seemed to work.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 07, 2023, 11:28:25 pm

Quote from: phamd4 on May 07, 2023, 11:19:59 pm

Hello sir,

Thank you for getting back to me.

Attached is my HAProxy config

Code: [Select]

logo

    root@localdomain

Lobby
Reporting
System
Interfaces
Firewall
VPN
Services
ACME Client
Adguardhome
Captive Portal
DHCPv4
DHCPv6
Dnsmasq DNS
Dynamic DNS
FreeRADIUS
HAProxy
Settings
Statistics
Maintenance
Log File
Config Export
Intrusion Detection
Monit
Network Time
OpenDNS
Unbound DNS
Wake on LAN
Web Proxy
Zenarmor
Power
Help

    Services: HAProxy: Config Export

    Config Export
    Config Diff

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    6
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_630c2xxx9944 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_630c25xxx249944

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/630c280xxx7137226.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_630c2982907624.21524463 src 20.xxx.0/24 20xxx0/24 20xxx.0/24

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/630c2909dfxxx.10265915.txt)] if acl_630c2xxx7624.21524463
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/630c262xxx9c50.82551607.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: opnsense_backend ()
backend opnsense_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server opnsense_server 20.14.0.1:55443 ssl verify none

# Backend: truenas_backend ()
backend truenas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server truenas_server 20.xxx.4 ssl verify none

# Backend: proxmox_backend ()
backend proxmox_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server proxmox_server 20.xxx47:8xxx6 ssl verify none

# Backend: ad_backend ()
backend ad_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server ad_server 20.xxx.1:56xxx43 ssl verify none

# Backend (DISABLED): proxmox_backend_1 ()

# Backend (DISABLED): truenas_backend_1 ()



# statistics are DISABLED



OPNsense (c) 2014-2023 Deciso B.V.
As for my domain for the certificate I followed your guide and have it at
Common name *.xxxan.to

As for the DNS Zone I'm using your guide https://desec.io
CNAME is *.mydomainname.to
I don't have A record (I didn't see it in your guide)

I checked and my dyanmic DNS server still update my IP address to desec. Also when I tried to restart my authentication service, I see my token on desec being rewnewed. So I think that aspect it work.

I can access my domain if i have that server running i.e. adguard, plex. However, if I just type in my domain name it doesn't resolve my public ip anymore. I think it at desec end but I've tried to add and delete my domain but nothing seemed to work.

Please post a screenshot of your desec dns zone. (remove sensitive info) But please overwrite the removed info with something so I can still understand how you set this up.
Are you using a domain of type "example.dedyn.io" or do you own a real domain like "example.yourdomain.com"?

Please also note that if you want to cover your domain "subdomain.DOMAIN.com", given that you have one, you will have to use Common Name "domain.com" and Alt Name "*.domain.com" otherwise only the subdomains of your domain will be covered by your certificate!

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on May 07, 2023, 11:33:57 pm

Hello,

I read your suggestion and just blindly tried to add the A Record in my desec, with my public IP address and voila, I were able to access my public IP. May I ask what happened or changed? I apologize if my question is not very smart. If I add my A record like this, will I be able to have my IP address automatically update? I know with your guide, through the token management, It receive IP update from my Opnsense dynamicDNS. However, if my record is using A Record, I have to manually enter the IP in myself.

THank you so much for your time.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 07, 2023, 11:36:35 pm

Quote from: opnuser1 on May 06, 2023, 10:51:57 pm

OK, I gave it a shot.  Not sure if my http config is correct.  Servers seem to be working except for the one that needs http, I am currently checking it out.

But here is my config.

Code: [Select]


#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         0.0.0.0 local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend ()
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp

    # logging options
    option tcplog
    # ACL: TCP_SSL_condition
    acl acl_644c56b6785678.47181279 req.ssl_hello_type 1
    # ACL: NoSSL_condition
    acl acl_644d62959d73a1.59974462 ssl_fc
    # ACL: server1_condition
    acl acl_644c5700ee7657.09485748 req.ssl_sni -m sub -i domain1.com
    # ACL: server2_condition
    acl acl_644c5719768e71.87060950 req.ssl_sni -i domain2.com

    # ACTION: TCP_RequestInspectDelay_rule
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: TCP_RequestContentAccept_rule
    tcp-request content accept if acl_644c56b6785678.47181279
    # ACTION: server1_rule
    use_backend server1_backend if !acl_644d62959d73a1.59974462 acl_644c5700ee7657.09485748
    # ACTION: server2_rule
    use_backend server2_backend if !acl_644d62959d73a1.59974462 acl_644c5719768e71.87060950

# Frontend: HTTP_frontend ()
frontend HTTP_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp

    # logging options
    option tcplog
    # ACL: NoSSL_condition
    acl acl_644d62959d73a1.59974462 ssl_fc
    # ACL: server2_condition
    acl acl_644c5719768e71.87060950 req.ssl_sni -i domain2.com
    # ACL: server1_condition
    acl acl_644c5700ee7657.09485748 req.ssl_sni -m sub -i domain1.com
    acl acl_644c56b6785678.47181279 req.ssl_hello_type 1

    # ACTION: server2_rule
    use_backend server2_backend if !acl_644d62959d73a1.59974462 acl_644c5719768e71.87060950
    # ACTION: server1_rule
    use_backend server1_backend if !acl_644d62959d73a1.59974462 acl_644c5700ee7657.09485748
    # ACTION: TCP_RequestInspectDelay_rule
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: TCP_RequestContentAccept_rule
    tcp-request content accept if acl_644c56b6785678.47181279

# Backend: server1_backend ()
backend server1_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server server1_server 192.168.1.234

# Backend: server2_backend ()
backend server2_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server server2_server 192.168.1.231



# statistics are DISABLED



Yes, this shows me your haproxy export, but it doesn't tell me wether this is working for you like intended or not.

Also why do you have a "NoSSL_condition" and why did you link it to the serviceX_rules of the HTTP_frontend?
Remove it, this is totally unecessary and I never said that you need this.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 07, 2023, 11:40:31 pm

Quote from: phamd4 on May 07, 2023, 11:33:57 pm

Hello,

I read your suggestion and just blindly tried to add the A Record in my desec, with my public IP address and voila, I were able to access my public IP. May I ask what happened or changed? I apologize if my question is not very smart. If I add my A record like this, will I be able to have my IP address automatically update? I know with your guide, through the token management, It receive IP update from my Opnsense dynamicDNS. However, if my record is using A Record, I have to manually enter the IP in myself.

THank you so much for your time.

Again, please describe your domain setup (main domain name, subdomains, dns zone content, dyndns update TARGET domain/subdomain, ...)!
Otherwise I will not be able to help you very much.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on May 07, 2023, 11:47:49 pm

Hello,

Attached is my screenshot on my desec

main domain name is example.com
sub domain is truenas.example.com
dns zone content (attached in screenshot)
dydns update followed your guide ( using custom dydns2) username is my domain password is the token

I do hope I provide the information you have asked for.

Thank you so much for your help

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 08, 2023, 12:02:21 am

Correct me if I'm wrong below...

This tells me that you own a real domain and that you are not just using a subdomain of "dedyn.io".
It also tells me that you are probably using a managed DNS account at desec. --> You imported the DNS Zone of your domain to desec so that you manage it with them.

Because of that you need your DynDNS client to interact with desec and have "yourdomainname.com" set as the target which should get updated.
In the tutorial I used "tutorial.dedyn.io" as the target which will then automatically create the necessary A record in the DNS Zone.
Since you have your own domain and also want to use it within haproxy and not just subdomains of it, you will have to set the target of the DynDNS update to "yourdomainname.com".

Then a CNAME record with Subname "*" and Content "yourdomainname.com". So all subdomains point to the IP that gets associated to "yourdomainname.com" by the DynDNS plugin. You already have this configured correctly.

The thing you need to fix is the DynDNS part! There is something misconfigured.
Also check the logs! Maybe you copied the wrong credentials / access tokens from desec into the DynDNS config.

Part 2 - Step 8: username and hostname need to be "yourdomainname.com".

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on May 08, 2023, 12:14:36 am

Hello sir,

Attached is my DDNS configuration.

I think I understand what you are saying a little bit. So what I configured is right but you afraid my DDNS doesn't push the right update to my desec domain correct? I think i did it right since I have username is my domain name i.e. "example.com" password is my token. then hostname is just my domain name which is "example.com" correct? or do you want me to have it at *.example.com?

The weird thing is everything were working well. It just suddenly doesn't resolve if i just type in example.com. I have to use mysubdomain.example.com to get it working.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 08, 2023, 12:21:28 am

Quote

Yes, this shows me your haproxy export, but it doesn't tell me wether this is working for you like intended or not.

Also why do you have a "NoSSL_condition" and why did you link it to the serviceX_rules of the HTTP_frontend?
Remove it, this is totally unecessary and I never said that you need this.

Thank you.  I removed the NoSSL condition.

I believe everything is working as far as I can tell.  I will list how I tested.

I tested from a cell connection all the addresses, and everything is going to the right places and the sites load fine and work fine.  So, mostly this confirms that the SNI frontend is working perfectly I think.

Is Http frontend working?  I believe so, hard for me to confirm.  Nothing is really using the http traffic (don't get mad, yes I requested it, hear me out).  So I used it very briefly to set up that website that uses the install script on port 80 initially.  And during the setup, I saw in the Counters area of HAproxy that traffic went through the http frontend.  So that confirms that port 80 is working I think, and the site got successfully set up. 

So i think everything is working, and it is quite a nice setup.  I can add multiple reverse proxies with this setup, all on a single ip, and all the proxies can get ports 80 and 443 going there cleanly.

here is my updated config:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         0.0.0.0 local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend ()
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp

    # logging options
    option tcplog
    # ACL: TCP_SSL_condition
    acl acl_644c56b6785678.47181279 req.ssl_hello_type 1
    # ACL: TCP_server1_condition
    acl acl_644c5700ee7657.09485748 req.ssl_sni -m sub -i domain1.com
    # ACL: TCP_server2_condition
    acl acl_644c5719768e71.87060950 req.ssl_sni -i domain2.com

    # ACTION: TCP_RequestInspectDelay_rule
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: TCP_RequestContentAccept_rule
    tcp-request content accept if acl_644c56b6785678.47181279
    # ACTION: TCP_SERVICE1_rule
    use_backend TCP_SERVICE1_backend if acl_644c5700ee7657.09485748
    # ACTION: TCP_SERVICE2_rule
    use_backend TCP_SERVICE2_backend if acl_644c5719768e71.87060950

# Frontend: HTTP_frontend ()
frontend HTTP_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp

    # logging options
    option tcplog
    # ACL: http_server1_condition
    acl acl_6457247ca14984.71641345 hdr_sub(host) -i domain1.com
    # ACL: http_server2_condition
    acl acl_64572496aeac32.73416688 hdr_sub(host) -i domain2.com

    # ACTION: http_server1_rule
    use_backend TCP_SERVICE1_backend if acl_6457247ca14984.71641345
    # ACTION: http_server2_rule
    use_backend TCP_SERVICE2_backend if acl_64572496aeac32.73416688

# Backend: TCP_SERVICE1_backend ()
backend TCP_SERVICE1_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server TCP_SERVICE1_server 192.168.1.234

# Backend: TCP_SERVICE2_backend ()
backend TCP_SERVICE2_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server TCP_SERVICE2_server 192.168.1.231



# statistics are DISABLED




Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: phamd4 on May 08, 2023, 01:26:33 am

Hello,

Just letting you know everything worked now. I went back and delete and re-add the configuration in the ddns and it were able to update the ip address now.

I had the exact same configuration and uses the same token and it worked now but not before. I don't know what changed but it worked. THank you so much

Quote from: phamd4 on May 08, 2023, 12:14:36 am

Hello sir,

Attached is my DDNS configuration.

I think I understand what you are saying a little bit. So what I configured is right but you afraid my DDNS doesn't push the right update to my desec domain correct? I think i did it right since I have username is my domain name i.e. "example.com" password is my token. then hostname is just my domain name which is "example.com" correct? or do you want me to have it at *.example.com?

The weird thing is everything were working well. It just suddenly doesn't resolve if i just type in example.com. I have to use mysubdomain.example.com to get it working.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 08, 2023, 02:13:27 am

Quote from: opnuser1 on May 08, 2023, 12:21:28 am

Is Http frontend working?  I believe so, hard for me to confirm.  Nothing is really using the http traffic (don't get mad, yes I requested it, hear me out).  So I used it very briefly to set up that website that uses the install script on port 80 initially.  And during the setup, I saw in the Counters area of HAproxy that traffic went through the http frontend.  So that confirms that port 80 is working I think, and the site got successfully set up. 

I still stand by my opinion that port 80 is not necessary at all. But since you never posted/linked that super duper script that would require port 80 this will forever be a myth to me.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 08, 2023, 04:01:33 am

Quote

I still stand by my opinion that port 80 is not necessary at all. But since you never posted/linked that super duper script that would require port 80 this will forever be a myth to me.

I did link it!
here it is:
https://v4-docs.chevereto.com/guides/docker/#create-https-proxy
That's the page with the installation instructions.  The port 80 is used for the place where i linked or the next command after.  I think it is used where I linked which sets up nginx.

the actual code is on the github:
https://github.com/chevereto/docker

but here's another use case that I like:
lets say i am using nginx for subdomains for domain1.com

but for another domain2.com, I'd like to use traefik as the reverse proxy.  So then, again, I'd need port 80 443 for both servers.  Some reverse proxy software work better with certain apps.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 08, 2023, 02:57:14 pm

Quote from: opnuser1 on May 08, 2023, 04:01:33 am

I did link it!
here it is:
https://v4-docs.chevereto.com/guides/docker/#create-https-proxy
That's the page with the installation instructions.  The port 80 is used for the place where i linked or the next command after.  I think it is used where I linked which sets up nginx.

I stand corrected in terms of not providing the script.
Still I have not yet seen any part that would make it need port 80! If this is only needed in order to obtain https certificates for the nginx web server than this is just a lazy part on your end. Should be very easy to switch from HTTP-01 to DNS-01 mode in the ACME tool for the certificate creation.

Quote from: opnuser1 on May 08, 2023, 04:01:33 am

but for another domain2.com, I'd like to use traefik as the reverse proxy.  So then, again, I'd need port 80 443 for both servers.  Some reverse proxy software work better with certain apps.

This is a pretty strong statement without any proof!
It is not about wether one reverse proxy works better than the other. This is all a matter of are you able to configure a (or any) reverse proxy correctly so that the services behind it can be accessed without any issues.

Again your "need" for port in this scenario 80 is pretty outdated and IMO comes from a lack of knowledge about the requirements of the services you would like to run. Everyone nowadays aims to secure outside going connections using https. Some do this by redirecting any port 80 connection to port 443 (like my tutorial). Others keep port 80 right out closed.
Why? Because using a reverse proxy makes port 80 on WAN side totally overdue.
If the service is internally running on port "god knows" with or without https (who cares) just put it behind a reverse proxy and secure the outside going connection using its ability to SSL offload and present it one port 443 and a valid HTTPS certificate.

The only possible scenario where a reverse proxy would need port 80 for a (good) reason is because they want to obtain their certificates using the HTTP-01 challenge. This on the other hand is mostly because the DNS provider / domain registrar doesn't have a DNS API to support the DNS-01 challenge. This alone should get one thinking about the choice of a domain registrar in 2023.
So we are back to the, IMO, laziness of the person setting things up.

Whenever I deploy a new service I read the documentation and try to lock it down as much as possible. Opening only the minimum required ports of the features I actually need. I also always configure them with self-signed SSL certificates to secure the connection to and from the reverse proxy.



I hope you don't take any offense from my writing, I don't mean to judge how you do your things.
I just think your current setup is... a bit clumsy.

Nonetheless I am glad that everything is working now how you wanted it to.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mauro on May 08, 2023, 04:16:12 pm

503 Service Unavailable

Dear all,
at the moment I have my webserver onine with portforwarding and before move on with HAProxy I'm thinking to test it setting it up locally. To keep also setting simple and possibly easier I'm considering reverse proxy of port 80 only for test.example.com
In other terms (IP as reference):
LAN (192.168.1.0/24, LAN Address 192.168.1.1)--> |HAProxy| --> DMZ (webserver 192.168.2.0/24, server, 192.168.2.2)

The only achievement I reached so far when I try to browse test.example.com is

Quote

503 Service Unavailable
No server is available to handle this request

this is haproxy setup:

Quote

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     100
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# Frontend: test_http (Test http)
frontend test_http
    bind 192.168.1.1:80 name 192.168.1.1:80
    mode http
    option http-keep-alive
    default_backend example_backend

    # logging options
    # ACL: kanboard_c
    acl acl_6452ce5a700492.11355253 hdr(host) -i test.example.com

    # ACTION: kanboard_r
    use_backend test_backend if acl_6452ce5a700492.11355253

# Backend: test_backend (example pool)
backend example_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server s1_server 192.168.2.1:80 proto h2

# statistics are DISABLED

Any hint?

Next step will be using SSL. the webapplication have individual SSL certificae which I think I can import in opnsense to set up HTTPS redirection. This will be next gig :)

I've flattened HAProxy few times and reset but I always end up with error 503 :(

I checked the firewall LAN -> DMZ and I don't see anything blocking the connection..

Thanks and please let me know if I can provide more information

cheers

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 08, 2023, 08:54:15 pm

Quote from: mauro on May 08, 2023, 04:16:12 pm

503 Service Unavailable

Dear all,
at the moment I have my webserver onine with portforwarding and before move on with HAProxy I'm thinking to test it setting it up locally. To keep also setting simple and possibly easier I'm considering reverse proxy of port 80 only for test.example.com
In other terms (IP as reference):
LAN (192.168.1.0/24, LAN Address 192.168.1.1)--> |HAProxy| --> DMZ (webserver 192.168.2.0/24, server, 192.168.2.2)

The only achievement I reached so far when I try to browse test.example.com is

Quote

503 Service Unavailable
No server is available to handle this request

this is haproxy setup:

Quote

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     100
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# Frontend: test_http (Test http)
frontend test_http
    bind 192.168.1.1:80 name 192.168.1.1:80
    mode http
    option http-keep-alive
    default_backend example_backend

    # logging options
    # ACL: kanboard_c
    acl acl_6452ce5a700492.11355253 hdr(host) -i test.example.com

    # ACTION: kanboard_r
    use_backend test_backend if acl_6452ce5a700492.11355253

# Backend: test_backend (example pool)
backend example_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server s1_server 192.168.2.1:80 proto h2

# statistics are DISABLED

Any hint?

Next step will be using SSL. the webapplication have individual SSL certificae which I think I can import in opnsense to set up HTTPS redirection. This will be next gig :)

I've flattened HAProxy few times and reset but I always end up with error 503 :(

I checked the firewall LAN -> DMZ and I don't see anything blocking the connection..

Thanks and please let me know if I can provide more information

cheers

Honestly please just follow my tutorial. I will not provide support for something else here.
If you want to do it your way then just ask in the appropriate forum.
But I will say if you keep on testing your way you will need much more time.

If it is not working with my way you can simply disable the WAN firewall rule and re-enable the NAT portforward.
This way you can also test this.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mauro on May 09, 2023, 12:39:04 am

hi @TheHellSite,

before posting here I posted in the general forum and considering nobody answered I thhought it was because there is a more specific tutorial.

Anyway, thanks for your help and your tutorial which I found very interesting and I'll give it a go asap with all features

Cheers

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on May 09, 2023, 04:22:28 am

Quote

I hope you don't take any offense from my writing, I don't mean to judge how you do your things.
I just think your current setup is... a bit clumsy.

Nonetheless I am glad that everything is working now how you wanted it to.

No problem at all, I take it as you are trying to give helpful advice.

I'm no networking expert.  If I were a network genius like you I wouldn't be here, would I?  So not sure what you want me to do.  This is the way I've able to wrap my mind around and it works, and I quite like it lol.  I'm sure there are better ways!  Thank you very much.  I've been looking forward to this configuration for a while.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brandorf on May 09, 2023, 06:04:37 am

I had this working before, but I accidentialy hosed the install and didn't have my config back up.  (Lesson learned).

Now the issue I'm having is getting a 503 no matter what I've tried so far.  Not sure where the issue is.

For example, trying to hit my domain vtt.*.com I get the correct cert to the browser, but still a 503, and here's all I see in the haproxy log.

Code: [Select]

2023-05-08T23:00:55-05:00 Informational haproxy 192.168.0.198:60994 [08/May/2023:23:00:52.860] 1_HTTPS_frontend~ VTT_backend/FoundryVTT_server 0/3032/-1/-1/3035 503 217 - - SC-- 2/1/0/0/3 0/0 "GET https://vtt.*.com/favicon.ico HTTP/2.0"
2023-05-08T23:00:52-05:00 Informational haproxy 192.168.0.198:60994 [08/May/2023:23:00:49.702] 1_HTTPS_frontend~ VTT_backend/FoundryVTT_server 0/3045/-1/-1/3048 503 217 - - SC-- 2/1/0/0/3 0/0 "GET https://vtt.*.com/ HTTP/2.0"

Here's the haproxy config.  I hope you can help me see what I'm not seeing.

Code: [Select]

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on port 80, 443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option tcplog

# Frontend: 1_HTTP_frontend (Listening on :80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_645996ff1a8d85.67011734 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_645996ff1a8d85.67011734

# Frontend: 1_HTTPS_frontend (Listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/64599b1898e146.09447169.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option httplog

    # ACTION: PUBLIC_DOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/645997471e1f42.25745091.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PLEX_server 192.168.0.197:32400 ssl verify none

# Backend: VTT_backend ()
backend VTT_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server FoundryVTT_server 192.168.0.197:30000 ssl verify none

# Backend: Homeassistant_backend ()
backend Homeassistant_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Homeassistant_server 192.168.0.196:8123 ssl verify none


Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 09, 2023, 04:47:15 pm

Quote from: brandorf on May 09, 2023, 06:04:37 am

I had this working before, but I accidentialy hosed the install and didn't have my config back up.  (Lesson learned).

Now the issue I'm having is getting a 503 no matter what I've tried so far.  Not sure where the issue is.

For example, trying to hit my domain vtt.*.com I get the correct cert to the browser, but still a 503, and here's all I see in the haproxy log.

Code: [Select]

2023-05-08T23:00:55-05:00 Informational haproxy 192.168.0.198:60994 [08/May/2023:23:00:52.860] 1_HTTPS_frontend~ VTT_backend/FoundryVTT_server 0/3032/-1/-1/3035 503 217 - - SC-- 2/1/0/0/3 0/0 "GET https://vtt.*.com/favicon.ico HTTP/2.0"
2023-05-08T23:00:52-05:00 Informational haproxy 192.168.0.198:60994 [08/May/2023:23:00:49.702] 1_HTTPS_frontend~ VTT_backend/FoundryVTT_server 0/3045/-1/-1/3048 503 217 - - SC-- 2/1/0/0/3 0/0 "GET https://vtt.*.com/ HTTP/2.0"

Here's the haproxy config.  I hope you can help me see what I'm not seeing.

Code: [Select]

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on port 80, 443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option tcplog

# Frontend: 1_HTTP_frontend (Listening on :80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_645996ff1a8d85.67011734 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_645996ff1a8d85.67011734

# Frontend: 1_HTTPS_frontend (Listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/64599b1898e146.09447169.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option httplog

    # ACTION: PUBLIC_DOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/645997471e1f42.25745091.txt)]

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PLEX_server 192.168.0.197:32400 ssl verify none

# Backend: VTT_backend ()
backend VTT_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server FoundryVTT_server 192.168.0.197:30000 ssl verify none

# Backend: Homeassistant_backend ()
backend Homeassistant_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Homeassistant_server 192.168.0.196:8123 ssl verify none


Your HAProxy config looks good.
And since you get the 503 error this means connections are getting to HAProxy.
Your issue is mostlikely the SSL (yes or no) checkbox in the Real Server settings. Verify which service needs SSL and which doesn't for the local access between HAProxy and the local service.

Also unless you get your certificates using the HTTP-01 method, which I doubt, since there is no rule for it on the HTTP_frontend remove do the following:
in the HAProxy plugin: delete the acme_challenge_backend and acme_challenge_host and all other haproxy entries auto generated by the ACME plugin.

in the ACME plugin: Go the the settings and disable the "HAProxy Integration", hit Apply.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mauro on May 09, 2023, 06:12:17 pm

Quote from: TheHellSite on May 08, 2023, 08:54:15 pm

Honestly please just follow my tutorial. I will not provide support for something else here.
If you want to do it your way then just ask in the appropriate forum.
But I will say if you keep on testing your way you will need much more time.

If it is not working with my way you can simply disable the WAN firewall rule and re-enable the NAT portforward.
This way you can also test this.

I there, I followed the suggestion and at the end of the process i have this 2 issues which I can't figure out:
1) Certificate is not valid. I also run the ssllab test and I received the same answer (rating T) showing certificate not trusted
2) again error 503 Service unavailable

Checking the HAProxy log, it shows:

Quote

Informational   haproxy   public_IP:9911 [09/May/2023:17:25:07.299] 2_HTTPS_frontend/127.4.4.3:443: SSL handshake failure

which I think I solved removing the SSL tick on the  real server set up. I have the apache virtual server only listening on port 80

#1: is it possible it is because at the moment I'm using a staging cert?

#2: this is the issue I'm investigating now for few days without any luck. I'll go over your tutorial but hints are welcome

cheers

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 09, 2023, 06:20:04 pm

Quote from: mauro on May 09, 2023, 06:12:17 pm

Quote from: TheHellSite on May 08, 2023, 08:54:15 pm

Honestly please just follow my tutorial. I will not provide support for something else here.
If you want to do it your way then just ask in the appropriate forum.
But I will say if you keep on testing your way you will need much more time.

If it is not working with my way you can simply disable the WAN firewall rule and re-enable the NAT portforward.
This way you can also test this.

I there, I followed the suggestion and at the end of the process i have this 2 issues which I can't figure out:
1) Certificate is not valid. I also run the ssllab test and I received the same answer (rating T) showing certificate not trusted
2) again error 503 Service unavailable

Checking the HAProxy log, it shows:

Quote

Informational   haproxy   public_IP:9911 [09/May/2023:17:25:07.299] 2_HTTPS_frontend/127.4.4.3:443: SSL handshake failure

which I think I solved removing the SSL tick on the  real server set up. I have the apache virtual server only listening on port 80

#1: is it possible it is because at the moment I'm using a staging cert?

#2: this is the issue I'm investigating now for few days without any luck. I'll go over your tutorial but hints are welcome

cheers

#1 yes.

#2 just follow the tutorial... You are wasting your time here.

Also if the webserver/service behind haproxy answers locally on port 80 then of course you need to untick the SSL chekbox in the associated Real Server. (as you figured correctly)

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: brandorf on May 09, 2023, 07:06:16 pm

Quote from: TheHellSite on May 09, 2023, 04:47:15 pm

Your HAProxy config looks good.
And since you get the 503 error this means connections are getting to HAProxy.
Your issue is mostlikely the SSL (yes or no) checkbox in the Real Server settings. Verify which service needs SSL and which doesn't for the local access between HAProxy and the local service.

Also unless you get your certificates using the HTTP-01 method, which I doubt, since there is no rule for it on the HTTP_frontend remove do the following:
in the HAProxy plugin: delete the acme_challenge_backend and acme_challenge_host and all other haproxy entries auto generated by the ACME plugin.

in the ACME plugin: Go the the settings and disable the "HAProxy Integration", hit Apply.

That got it, thanks!  So the SSL check on the real server setting would only be if that server was serving its own cert?

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 10, 2023, 01:41:40 pm

Quote from: brandorf on May 09, 2023, 07:06:16 pm

That got it, thanks!  So the SSL check on the real server setting would only be if that server was serving its own cert?

Yes. But I already clearly explained this in the tutorial. It is also clearly explained by the hint next to the checkbox...

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mauro on May 16, 2023, 07:46:21 pm

Hi there.

After last post Iand sugestion to follow the tutorial prior to ask advice, I did and I still end up with the same error when I try to browse the webapp

Quote

503 Service Unavailable
No server is available to handle this request.

So I assume:
redirect 80->443 is working fine because I start the connection on port 80
certificate is set well despite it is the staging one giving error

I'd say the issue lays between the firewall and the backend.

The backend (webserver) accept both on port 80 and 443 with a running cert for SSL connection. can this be an issue?

Considering I tried few tutorial included this one I think the issue stays withing the firewall.

The webserver is up and running and has no restrictions to the Apache ports

Any sugestion?

thanks a lot in advance

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 17, 2023, 12:05:48 am

@mauro
HAProxy config export and a basic network diagram. That is what you will have to provide now, not just error codes.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mauro on May 17, 2023, 06:06:57 pm

Quote from: TheHellSite on May 17, 2023, 12:05:48 am

@mauro
HAProxy config export and a basic network diagram. That is what you will have to provide now, not just error codes.

Roger, @TheHellSite

NETWORK Topography (simplified)

LAN (IP address) <--> | FW, lo_IP (127.x.x.x) | <--> DMZ (server_ip)

IP_Address is my FW LAN Address as per OPNsense meaning
lo_IP is the equivalent of your 127.4.4.3 but customized
server_IP is the static ip of the webserver in the DMZ network

To keep the webapp available from the outside world I have the SNI Frontend based on the LAN address, port 80,443. I'm doing this way because I already tried following your tutorial using the WAN address on the SNI frontend with same result plus server unreachable. I created a Firewall rule for LAN to acceppt incoming requests on port 80,443

HAProxy set up file:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 warning
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend (listening on LAN address port 80/443)
frontend 0_SNI_Frontend
    bind lan_ip:80 name lan_ip:80
    bind lan_ip:443 name lan_ip:443
    mode tcp
    default_backend SSL_Backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on lo_ip:80)
frontend 1_HTTP_frontend
    bind lo_ip:80 name lo_ip:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option dontlognull
    option log-separate-errors
    option httplog
    # ACL: NoSSL_condition
    acl acl_6462b25dd3fc08.98092716 ssl_fc

    # ACTION: HTTP2HTTPS_r
    http-request redirect scheme https code 301 if !acl_6462b25dd3fc08.98092716

# Frontend: 2_HTTPS_frontend (Listening on lo_ip:443)
frontend 2_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind lo_ip:443 name lo_ip:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6463bbbf543239.59805119.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option dontlognull
    option log-separate-errors
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_r
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/645151c9cb3ae5.07476878.txt)]

# Backend: s1_backend (s1 server backend)
backend s1_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server s1_server server_ip

# Backend: SSL_Backend (SNI backend)
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server lo_ip send-proxy-v2 check-send-proxy



# statistics are DISABLED

Thanks a lot

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 17, 2023, 11:13:57 pm

Quote from: mauro on May 17, 2023, 06:06:57 pm

To keep the webapp available from the outside world I have the SNI Frontend based on the LAN address, port 80,443. I'm doing this way because I already tried following your tutorial using the WAN address on the SNI frontend with same result plus server unreachable.

It can not work - plain logic.
1. Your apache is listening on port 80 (no ssl) and 443 (probably with ssl).
2. My tutorial assumes that the user wants all connections to be upgraded from port 80 to 443, what you also configured by using the HTTP_frontend on port 80 with the HTTPtoHTTPS_rule.
3. The HTTPS_frontend has SSL offloading enabled, so it decrypts any connection and then forwards it to the real server based on the real server connection configuration.
4. However, in your scenario you didn't configure your apache real server correctly since you left the port blank and didn't tell haproxy if the real server expects SSL or not.

Sorry but this is yet another error of someone not reading the tutorial properly.
All your issues are explained very clearly there.

Quote from: mauro on May 17, 2023, 06:06:57 pm

I created a Firewall rule for LAN to acceppt incoming requests on port 80,443

Shouldn't even be necessary on a fresh install since it allows anything from LAN to anything by default.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mauro on May 18, 2023, 01:15:05 am

Thanks again for your time.

As I said I'm new to revers proxy and I'm trying to learn. Your tutorial is for a plex server and I'm working on a webserver so at some point I need to take some decision/assumptions based on my understanding and it can be wrong but in the learning curve.

Going over your remarks:

Quote

1. Your apache is listening on port 80 (no ssl) and 443 (probably with ssl).

correct, 443 with SSL

Quote

2. My tutorial assumes that the user wants all connections to be upgraded from port 80 to 443, what you also configured by using the HTTP_frontend on port 80 with the HTTPtoHTTPS_rule.

this is also apache does for my application, redirect any call on port 80 to port 443 and use encrypted communication therefore so far your set up fit the requirements

Quote

3. The HTTPS_frontend has SSL offloading enabled, so it decrypts any connection and then forwards it to the real server based on the real server connection configuration.

Reading your tutorial #9 about SSL connection with the back end I thought ur set ups should work because u use an SSL connection to the plex server.
The webserver listen on port 443 with SSL
This is the tricky part for me and also from your notes I can't follow if and how I should change my reported setup.
Could you please be more specific?

Quote

4. However, in your scenario you didn't configure your apache real server correctly since you left the port blank and didn't tell haproxy if the real server expects SSL or not.

I set up the webserver as u did for the SSL_backend. both listening to port 80,443 and both will use SSL on port 443. If the webserver backend should be different, can you please guide me how to adjust?

bottom line, I read the tutorial several time. This is the CLOSEST tutorial to my needs (u set up a ples server and I a webserver) I have found but not exactly what I need. In this difference come the potential confusion. If I knew what to do I didn't need a tutorial. I found online also other blogs and always arrived to a dead end with error 503. I mention it on previous post under this tthread already

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 19, 2023, 09:20:51 am

This is the solution.

Quote

4. However, in your scenario you didn't configure your apache real server correctly since you left the port blank and didn't tell haproxy if the real server expects SSL or not.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: moophy on May 19, 2023, 01:15:18 pm

Hi!

I've followed the guide here and it has worked flawlessly for all my services internally! So thank you for that! So much better without the warning every time going into a server.

I'm struggling with nextcloud though and feel like i'm not passing something through that i should be. I've been tearing my hair out on this one! When i type in the browser nextcloud.my.domain it says site can't be reached but if i add /login to the end then it works perfectly (for internal) with padlock sign. I can also access internally via IP:11000 (This is the Apache port) which diverts me to nextcloud.my.domain/login as expected and works. I just can't get it to do this via reverse proxy so neither internal or external works with just nextcloud.my.domain.

This leads me to think there is something missing going from HAProxy to Nextcloud (Which is in Portainer). I have a bunch of other servers which are all working fine (Truenas, Proxmox etc) in HAProxy. I've tried putting Nextcloud in as a VM on Proxmox and also in Truenas to see if it was Portainer causing any issue but same problem. Code below with all my other servers removed and left Portainer and Nextcloud as they are on the same IP. Portainer works, Nextcloud doesn't...

Code: [Select]

# Automatically generated configuration.
# Do not edit this file manually.
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats

# Frontend: 0_SNI_frontend (listening on 80 and 443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option tcplog

# Frontend: 1_HTTP_frontend (listening on localhost:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_6451d6d41f14e3.72189927 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6451d6d41f14e3.72189927

# Frontend: 1_HTTPS_frontend (listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6451dd0fe21db7.63563341.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option httplog
    # ACL: cardav-endpoint
    acl acl_6461de0380c7b3.75062629 path_end -i /.well-known/carddav
    # ACL: caldav-endpoint
    acl acl_6461dde5d15634.54704624 path_end -i /.well-known/caldav
    # ACL: nc_nodeinfo
    acl acl_6466fe1b2e8ff2.47035478 path /.well-known/nodeinfo
    # ACL: nc_webfinger
    acl acl_6466fe303acb97.89104263 path /.well-known/webfinger
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6451d1c3d510f5.88841915 src 10.1.1.0/24 192.168.0.0/24

    # ACTION: cardav-endpoint
    http-request redirect code 301 location /remote.php/dav if acl_6461de0380c7b3.75062629
    # ACTION: caldav-endpoint
    http-request redirect code 301 location /remote.php/dav if acl_6461dde5d15634.54704624
    # ACTION: nc_nodeinfo
    http-request redirect code 301 location /index.php/%[capture.req.uri] if acl_6466fe1b2e8ff2.47035478
    # ACTION: nc_webfinger
    http-request redirect code 301 location /index.php/%[capture.req.uri] if acl_6466fe303acb97.89104263
    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6451d0414cb7f2.59080919.txt)] if acl_6451d1c3d510f5.88841915
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6451d7a6418a58.11785496.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: portainer_backend (portainer backend)
backend portainer_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server portainer 10.1.1.59:9443

# Backend: nextcloud_backend (nextcloud_backend)
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server nextcloud 10.1.1.59:11000
# statistics are DISABLED
 
In the HAProxy log it says:

Code: [Select]

2023-05-19T16:35:18 | Informational | haproxy | 10.1.1.59:58792 [19/May/2023:16:35:18.215] 0_SNI_frontend SSL_backend/SSL_server 1/-1/0 0 CC 2/2/1/1/0 0/0 | 
-- | -- | -- | -- | --
2023-05-19T16:35:07 | Informational | haproxy | 10.1.1.103:46386 [19/May/2023:16:34:37.775] 1_HTTPS_frontend~ 1_HTTPS_frontend/<NOSRV> -1/-1/-1/-1/30003 0 0 - - PR-- 2/2/0/0/0 0/0 "<BADREQ>" | 
2023-05-19T16:35:07 | Informational | haproxy | 10.1.1.103:46386 [19/May/2023:16:34:37.761] 0_SNI_frontend SSL_backend/SSL_server 1/0/30017 5134 cD 2/2/1/1/0 0/0 | 
2023-05-19T16:34:48 | Informational | haproxy | 10.1.1.59:45564 [19/May/2023:16:34:48.055] 0_SNI_frontend SSL_backend/SSL_server 1/0/1 0 -- 3/3/2/2/0 0/0 | 
2023-05-19T16:34:48 | Informational | haproxy | 10.1.1.59:45564 [19/May/2023:16:34:48.055] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake
I do have DNS A record in Unbound pointing at the firewall IP as with all my other services. I added the rules for Cardav/Caldav/webfinger/nodeinfo to see if that helps (was noted in other guides) but that didn't change anything.

Thanks!

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: smoked-proposal on May 20, 2023, 02:47:10 pm

Thanks for this tutorial TheHellSite.
The only difficulty I had was a mistake that I made: I set `Verify SSL Certificate: true` on the real server (nextcloud). Then I "couldn't connect" to my server (I remember I could with IP but not domain name, and on internal network not from outside) and waste several hours to spot what was wrong...

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: moophy on May 21, 2023, 04:23:07 am

smoked-proposal would you mind posting in your HAproxy config? To my knowledge everything i've set is as per the tutorial but i'd like to see your config for Nextcloud if you have it working (I don't have SSL verify either). I've asked over in the Nextcloud forums and they do believe HAproxy is dropping something in the redirect considering hitting nextcloud directly works correctly.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 21, 2023, 07:40:50 pm

Quote from: moophy on May 21, 2023, 04:23:07 am

smoked-proposal would you mind posting in your HAproxy config? To my knowledge everything i've set is as per the tutorial but i'd like to see your config for Nextcloud if you have it working (I don't have SSL verify either). I've asked over in the Nextcloud forums and they do believe HAproxy is dropping something in the redirect considering hitting nextcloud directly works correctly.

I can't be of any help here as this is a service specific issue.
Search the thread, there might be some people that had similiar problems with nextcloud.

Quote

... they do believe HAproxy is dropping something in the redirect considering hitting nextcloud directly works correctly.

Wrong, HAProxy is forwarding everything hitting "nextcloud.domain.com" to your Nextcloud server, nothing gets dropped for no reason.
I recently installed Nextcloud for testing purposes on Proxmox in an LXC. No issues at all while running it through HAProxy with "nextcloud.domain.com".

The most likely cause of error is the Apache server.
Also try clearing your browser cache.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: moophy on May 22, 2023, 07:09:49 am

Thank you, i did manage to resolve part of the issue from your help with clearing my browser cache. I had been testing via incognito mode often but had forgotten last few changes. Locally I can now access nextcloud via domain name as expected.
 
Externally however, no joy. Map file entry under #public access subdomains
nextcloud nextcloud_backend in both local and public maps (in that order). Would this point to an issue somewhere on Opnsense? Whether that's firewall, HAproxy etc not sure. Considering nextcloud itself can accept connection via url locally? Happy for your guidance and if you think that issue is still the target server then i'll go deep dive further there. Halfway there :)

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 22, 2023, 10:36:17 am

Quote from: moophy on May 22, 2023, 07:09:49 am

Thank you, i did manage to resolve part of the issue from your help with clearing my browser cache. I had been testing via incognito mode often but had forgotten last few changes. Locally I can now access nextcloud via domain name as expected.
 
Externally however, no joy.

No error codes, no logs, no map file content, ...
Is it just nextcloud not working from external or all services in haproxy?

So many questions and possible solutions but no context provided, therefore ---> Unable to help.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: moophy on May 23, 2023, 05:09:42 am

Nextcloud was the only service i was making publicly available. I moved one of my internal services out from local subdomain to public and no i cannot reach it externally. app files:

Code: [Select]

# local access only subdomains
iprox iprox_backend
autgtp autgtp_backend
portainer portainer_backend

#public access subdomains
nextcloud nextcloud_backend
truenas truenas_backend

Code: [Select]

#public access subdomains
nextcloud nextcloud_backend
truenas truenas_backend
logs from HAproxy when trying an external connection for nextcloud:

Code: [Select]

2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.718] 0_SNI_frontend SSL_backend/SSL_server 1/0/254 5920 -- 3/2/1/1/0 0/0
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.740] 1_HTTPS_frontend~ nextcloud_backend/nextcloud 0/0/1/229/230 304 1194 - - ---- 4/2/0/0/0 0/0 "GET /apps/richdocuments/settings/fonts.json HTTP/1.1"2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 0_SNI_frontend SSL_backend/SSL_server 1/0/0 0 -- 3/2/1/1/0 0/0
2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake
logs from HAproxy when trying an external connection for truenas:

Code: [Select]

2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 0_SNI_frontend SSL_backend/SSL_server 1/0/0 0 -- 3/2/1/1/0 0/0
2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.718] 0_SNI_frontend SSL_backend/SSL_server 1/0/254 5920 -- 3/2/1/1/0 0/0
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.740] 1_HTTPS_frontend~ nextcloud_backend/nextcloud 0/0/1/229/230 304 1194 - - ---- 4/2/0/0/0 0/0 "GET /apps/richdocuments/settings/fonts.json HTTP/1.1"

Interesting that the Truenas call seems to be going to nextcloud?

Unbound DNS: Overides

   truenas.mydomain     dedyn.io   A (IPv4 address)   10.1.1.1            
   nextcloud.mydomain   dedyn.io   A (IPv4 address)   10.1.1.1       

Firewall rules:

HAProxy_ports (80, 443) allowed to WAN address

I don't see any deny results in firewall log. Let me know what other logs or config would be useful. Thank you

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sc00by on May 23, 2023, 12:27:20 pm

Good morning,

I am having some challenge with getting HAproxy to handle internal requests.  I am not using unbound for DNS since I have active directory deployed.  I did try using unbound with overrides.  Externally, HAproxy is working, can access plex from the office and able to access a web page.

I've gone through the tutorial a few times and am assuming that something is missed.

The goal is to use HAProxy internally and externally to encrypt traffic with a real cert.  I'd appreciate any help or shove back to what was missed in the tutorial.

Code: [Select]


Config:

Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     1000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 1000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listenting on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options

# Frontend: 1_HTTPS_frontend (Listenting on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6467f8cd2c0025.92252317.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6468020a093145.77501579 src 192.168.1.0/24

    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/646801b883fa30.90520723.txt)] if acl_6468020a093145.77501579
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6467f584ef9e54.41997502.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PLEX_server 192.168.1.62:32400 ssl verify none

# Backend: remote_dc02_backend (remote dc02)
backend remote_dc02_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server remote_dc02_server 192.168.1.6:443 ssl verify none

# Backend: BMC_backend (bmc test)
backend BMC_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server BMC_server 192.168.1.35 ssl verify none



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED






Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 24, 2023, 11:00:06 pm

Quote from: sc00by on May 23, 2023, 12:27:20 pm

Good morning,

I am having some challenge with getting HAproxy to handle internal requests.  I am not using unbound for DNS since I have active directory deployed.  I did try using unbound with overrides.  Externally, HAproxy is working, can access plex from the office and able to access a web page.

I've gone through the tutorial a few times and am assuming that something is missed.

The goal is to use HAProxy internally and externally to encrypt traffic with a real cert.  I'd appreciate any help or shove back to what was missed in the tutorial.

Code: [Select]


Config:

Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     1000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 1000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listenting on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options

# Frontend: 1_HTTPS_frontend (Listenting on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6467f8cd2c0025.92252317.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6468020a093145.77501579 src 192.168.1.0/24

    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/646801b883fa30.90520723.txt)] if acl_6468020a093145.77501579
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6467f584ef9e54.41997502.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PLEX_server 192.168.1.62:32400 ssl verify none

# Backend: remote_dc02_backend (remote dc02)
backend remote_dc02_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server remote_dc02_server 192.168.1.6:443 ssl verify none

# Backend: BMC_backend (bmc test)
backend BMC_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server BMC_server 192.168.1.35 ssl verify none



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED






First off, you did not provide any sort of error code or error description. Just some "it is not working, help help"...

Sorry, but this is also out of scope of this tutorial.
The DNS override is most likely your issue! Since you are not using Unbound, you are on your own here.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 24, 2023, 11:47:59 pm

Quote from: moophy on May 23, 2023, 05:09:42 am

Nextcloud was the only service i was making publicly available. I moved one of my internal services out from local subdomain to public and no i cannot reach it externally. app files:

Code: [Select]

# local access only subdomains
iprox iprox_backend
autgtp autgtp_backend
portainer portainer_backend

#public access subdomains
nextcloud nextcloud_backend
truenas truenas_backend

Code: [Select]

#public access subdomains
nextcloud nextcloud_backend
truenas truenas_backend
logs from HAproxy when trying an external connection for nextcloud:

Code: [Select]

2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.718] 0_SNI_frontend SSL_backend/SSL_server 1/0/254 5920 -- 3/2/1/1/0 0/0
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.740] 1_HTTPS_frontend~ nextcloud_backend/nextcloud 0/0/1/229/230 304 1194 - - ---- 4/2/0/0/0 0/0 "GET /apps/richdocuments/settings/fonts.json HTTP/1.1"2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 0_SNI_frontend SSL_backend/SSL_server 1/0/0 0 -- 3/2/1/1/0 0/0
2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake
logs from HAproxy when trying an external connection for truenas:

Code: [Select]

2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 0_SNI_frontend SSL_backend/SSL_server 1/0/0 0 -- 3/2/1/1/0 0/0
2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.718] 0_SNI_frontend SSL_backend/SSL_server 1/0/254 5920 -- 3/2/1/1/0 0/0
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.740] 1_HTTPS_frontend~ nextcloud_backend/nextcloud 0/0/1/229/230 304 1194 - - ---- 4/2/0/0/0 0/0 "GET /apps/richdocuments/settings/fonts.json HTTP/1.1"

Interesting that the Truenas call seems to be going to nextcloud?

Unbound DNS: Overides

   truenas.mydomain     dedyn.io   A (IPv4 address)   10.1.1.1            
   nextcloud.mydomain   dedyn.io   A (IPv4 address)   10.1.1.1       

Firewall rules:

HAProxy_ports (80, 443) allowed to WAN address

I don't see any deny results in firewall log. Let me know what other logs or config would be useful. Thank you

Please also provide the complete current haproxy config.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sc00by on May 25, 2023, 12:40:02 am

Quote from: TheHellSite on May 24, 2023, 11:00:06 pm

Quote from: sc00by on May 23, 2023, 12:27:20 pm

Good morning,

I am having some challenge with getting HAproxy to handle internal requests.  I am not using unbound for DNS since I have active directory deployed.  I did try using unbound with overrides.  Externally, HAproxy is working, can access plex from the office and able to access a web page.

I've gone through the tutorial a few times and am assuming that something is missed.

The goal is to use HAProxy internally and externally to encrypt traffic with a real cert.  I'd appreciate any help or shove back to what was missed in the tutorial.

Code: [Select]


Config:

Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     1000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 1000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listenting on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options

# Frontend: 1_HTTPS_frontend (Listenting on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6467f8cd2c0025.92252317.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6468020a093145.77501579 src 192.168.1.0/24

    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/646801b883fa30.90520723.txt)] if acl_6468020a093145.77501579
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6467f584ef9e54.41997502.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: PLEX_backend ()
backend PLEX_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PLEX_server 192.168.1.62:32400 ssl verify none

# Backend: remote_dc02_backend (remote dc02)
backend remote_dc02_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server remote_dc02_server 192.168.1.6:443 ssl verify none

# Backend: BMC_backend (bmc test)
backend BMC_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server BMC_server 192.168.1.35 ssl verify none



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED






First off, you did not provide any sort of error code or error description. Just some "it is not working, help help"...

Sorry, but this is also out of scope of this tutorial.
The DNS override is most likely your issue! Since you are not using Unbound, you are on your own here.

Shoot.  I thought I put a note in there about logs... :-[

Outside of the informational log, there is nothing logged.  In the informational, I see connections originating from the outside.  Nothing for hitting this particular internal IP/host, no error.  Logs accessed from Services/HAProxy/Logs, enabled all.  Nothing here is pertinent near as I can tell.  When this internal (BMC) resource loads, I see an unsigned cert which tells me it is not hitting haproxy.

When I access plex, it originates from lan IP then hits the public IP and I can see this logging client IP, it loads.  This works from inside/outside.  My other external service also works inside/outside.  An override with plex

I created a backend, server, and map file for internal services.

I've tried with and without unbound.  With and without overrides.  Right now I am using unbound, opnsense is my DNS. It loads...attempt to access the web service...I get a self signed cert.  This is the head scratcher, same result whether I use AD or Unbound DNS so assuming something I've missed.

What I am really after is 'what did I miss in the configuration' because I'll be setting up another HAproxy box targeted at internal only resources as well.

So, AD DNS is not in the picture, in this context, I am in the same spot.

I'll take another stroll through the tutorial.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: moophy on May 25, 2023, 04:58:49 am

Quote from: TheHellSite on May 24, 2023, 11:47:59 pm

Quote from: moophy on May 23, 2023, 05:09:42 am

Nextcloud was the only service i was making publicly available. I moved one of my internal services out from local subdomain to public and no i cannot reach it externally. app files:

Code: [Select]

# local access only subdomains
iprox iprox_backend
autgtp autgtp_backend
portainer portainer_backend

#public access subdomains
nextcloud nextcloud_backend
truenas truenas_backend

Code: [Select]

#public access subdomains
nextcloud nextcloud_backend
truenas truenas_backend
logs from HAproxy when trying an external connection for nextcloud:

Code: [Select]

2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.718] 0_SNI_frontend SSL_backend/SSL_server 1/0/254 5920 -- 3/2/1/1/0 0/0
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.740] 1_HTTPS_frontend~ nextcloud_backend/nextcloud 0/0/1/229/230 304 1194 - - ---- 4/2/0/0/0 0/0 "GET /apps/richdocuments/settings/fonts.json HTTP/1.1"2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 0_SNI_frontend SSL_backend/SSL_server 1/0/0 0 -- 3/2/1/1/0 0/0
2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake
logs from HAproxy when trying an external connection for truenas:

Code: [Select]

2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 0_SNI_frontend SSL_backend/SSL_server 1/0/0 0 -- 3/2/1/1/0 0/0
2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.718] 0_SNI_frontend SSL_backend/SSL_server 1/0/254 5920 -- 3/2/1/1/0 0/0
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.740] 1_HTTPS_frontend~ nextcloud_backend/nextcloud 0/0/1/229/230 304 1194 - - ---- 4/2/0/0/0 0/0 "GET /apps/richdocuments/settings/fonts.json HTTP/1.1"

Interesting that the Truenas call seems to be going to nextcloud?

Unbound DNS: Overides

   truenas.mydomain     dedyn.io   A (IPv4 address)   10.1.1.1            
   nextcloud.mydomain   dedyn.io   A (IPv4 address)   10.1.1.1       

Firewall rules:

HAProxy_ports (80, 443) allowed to WAN address

I don't see any deny results in firewall log. Let me know what other logs or config would be useful. Thank you

Please also provide the complete current haproxy config.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs

# autogenerated entries for config in backends/frontends

# autogenerated entries for stats

# Frontend: 0_SNI_frontend (listening on 80 and 443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend testssl

    # logging options

# Frontend: 1_HTTP_frontend (listening on localhost:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_6451d6d41f14e3.72189927 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6451d6d41f14e3.72189927

# Frontend: 1_HTTPS_frontend (listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6451dd0fe21db7.63563341.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option httplog
    # ACL: cardav-endpoint
    acl acl_6461de0380c7b3.75062629 path /.well-known/carddav
    # ACL: caldav-endpoint
    acl acl_6461dde5d15634.54704624 path /.well-known/caldav
    # ACL: nc_nodeinfo
    acl acl_6466fe1b2e8ff2.47035478 path /.well-known/nodeinfo
    # ACL: nc_webfinger
    acl acl_6466fe303acb97.89104263 path /.well-known/webfinger
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6451d1c3d510f5.88841915 src 10.1.1.0/24 192.168.0.0/24

    # ACTION: cardav-endpoint
    http-request redirect code 301 location /remote.php/dav if acl_6461de0380c7b3.75062629
    # ACTION: caldav-endpoint
    http-request redirect code 301 location /remote.php/dav if acl_6461dde5d15634.54704624
    # ACTION: nc_nodeinfo
    http-request redirect code 301 location /index.php/%[capture.req.uri] if acl_6466fe1b2e8ff2.47035478
    # ACTION: nc_webfinger
    http-request redirect code 301 location /index.php/%[capture.req.uri] if acl_6466fe303acb97.89104263
    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6451d0414cb7f2.59080919.txt)] if acl_6451d1c3d510f5.88841915
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6451d7a6418a58.11785496.txt)]

# Backend: truenas_backend (NAS backend)
backend truenas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server truenas 10.1.1.73 ssl verify none

# Backend: iprox_backend ()
backend iprox_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server iprox 10.1.1.101:8006 ssl verify none

# Backend: mbfirewall_backend ()
backend mbfirewall_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server mbfirewall 127.0.0.1:55443 ssl verify none

# Backend: autgtp_backend ()
backend autgtp_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server autgtp 10.1.1.113:7070

# Backend: portainer_backend (portainer backend)
backend portainer_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server portainer 10.1.1.59:9443 ssl alpn h2,http/1.1 verify none

# Backend: nextcloud_backend (nextcloud_backend)
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server nextcloud 10.1.1.59:11000

# Backend: idrac_backend ()
backend idrac_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server idrac 192.168.0.120 ssl verify none

# Backend: testssl (test1)
backend testssl
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server testSSL 127.4.4.3 send-proxy-v2 check-send-proxy

# statistics are DISABLE

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 25, 2023, 10:36:55 am

Quote from: sc00by on May 25, 2023, 12:40:02 am

Shoot.  I thought I put a note in there about logs... :-[

Outside of the informational log, there is nothing logged.  In the informational, I see connections originating from the outside.  Nothing for hitting this particular internal IP/host, no error.  Logs accessed from Services/HAProxy/Logs, enabled all.  Nothing here is pertinent near as I can tell.  When this internal (BMC) resource loads, I see an unsigned cert which tells me it is not hitting haproxy.

When I access plex, it originates from lan IP then hits the public IP and I can see this logging client IP, it loads.  This works from inside/outside.  My other external service also works inside/outside.  An override with plex

I created a backend, server, and map file for internal services.

I've tried with and without unbound.  With and without overrides.  Right now I am using unbound, opnsense is my DNS. It loads...attempt to access the web service...I get a self signed cert.  This is the head scratcher, same result whether I use AD or Unbound DNS so assuming something I've missed.

What I am really after is 'what did I miss in the configuration' because I'll be setting up another HAproxy box targeted at internal only resources as well.

So, AD DNS is not in the picture, in this context, I am in the same spot.

I'll take another stroll through the tutorial.

Sorry I still don't know what your issue is exactly now?
Your text is very confusing to read.

What I took from it is that the internal and external accesses are working but it is not presenting a trusted cert?

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 25, 2023, 10:42:42 am

Quote from: moophy on May 25, 2023, 04:58:49 am

Quote from: TheHellSite on May 24, 2023, 11:47:59 pm

Quote from: moophy on May 23, 2023, 05:09:42 am

Nextcloud was the only service i was making publicly available. I moved one of my internal services out from local subdomain to public and no i cannot reach it externally. app files:

Code: [Select]

# local access only subdomains
iprox iprox_backend
autgtp autgtp_backend
portainer portainer_backend

#public access subdomains
nextcloud nextcloud_backend
truenas truenas_backend

Code: [Select]

#public access subdomains
nextcloud nextcloud_backend
truenas truenas_backend
logs from HAproxy when trying an external connection for nextcloud:

Code: [Select]

2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.718] 0_SNI_frontend SSL_backend/SSL_server 1/0/254 5920 -- 3/2/1/1/0 0/0
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.740] 1_HTTPS_frontend~ nextcloud_backend/nextcloud 0/0/1/229/230 304 1194 - - ---- 4/2/0/0/0 0/0 "GET /apps/richdocuments/settings/fonts.json HTTP/1.1"2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 0_SNI_frontend SSL_backend/SSL_server 1/0/0 0 -- 3/2/1/1/0 0/0
2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake
logs from HAproxy when trying an external connection for truenas:

Code: [Select]

2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 0_SNI_frontend SSL_backend/SSL_server 1/0/0 0 -- 3/2/1/1/0 0/0
2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.718] 0_SNI_frontend SSL_backend/SSL_server 1/0/254 5920 -- 3/2/1/1/0 0/0
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.740] 1_HTTPS_frontend~ nextcloud_backend/nextcloud 0/0/1/229/230 304 1194 - - ---- 4/2/0/0/0 0/0 "GET /apps/richdocuments/settings/fonts.json HTTP/1.1"

Interesting that the Truenas call seems to be going to nextcloud?

Unbound DNS: Overides

   truenas.mydomain     dedyn.io   A (IPv4 address)   10.1.1.1            
   nextcloud.mydomain   dedyn.io   A (IPv4 address)   10.1.1.1       

Firewall rules:

HAProxy_ports (80, 443) allowed to WAN address

I don't see any deny results in firewall log. Let me know what other logs or config would be useful. Thank you

Please also provide the complete current haproxy config.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs

# autogenerated entries for config in backends/frontends

# autogenerated entries for stats

# Frontend: 0_SNI_frontend (listening on 80 and 443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend testssl

    # logging options

# Frontend: 1_HTTP_frontend (listening on localhost:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_6451d6d41f14e3.72189927 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6451d6d41f14e3.72189927

# Frontend: 1_HTTPS_frontend (listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6451dd0fe21db7.63563341.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option httplog
    # ACL: cardav-endpoint
    acl acl_6461de0380c7b3.75062629 path /.well-known/carddav
    # ACL: caldav-endpoint
    acl acl_6461dde5d15634.54704624 path /.well-known/caldav
    # ACL: nc_nodeinfo
    acl acl_6466fe1b2e8ff2.47035478 path /.well-known/nodeinfo
    # ACL: nc_webfinger
    acl acl_6466fe303acb97.89104263 path /.well-known/webfinger
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6451d1c3d510f5.88841915 src 10.1.1.0/24 192.168.0.0/24

    # ACTION: cardav-endpoint
    http-request redirect code 301 location /remote.php/dav if acl_6461de0380c7b3.75062629
    # ACTION: caldav-endpoint
    http-request redirect code 301 location /remote.php/dav if acl_6461dde5d15634.54704624
    # ACTION: nc_nodeinfo
    http-request redirect code 301 location /index.php/%[capture.req.uri] if acl_6466fe1b2e8ff2.47035478
    # ACTION: nc_webfinger
    http-request redirect code 301 location /index.php/%[capture.req.uri] if acl_6466fe303acb97.89104263
    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6451d0414cb7f2.59080919.txt)] if acl_6451d1c3d510f5.88841915
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6451d7a6418a58.11785496.txt)]

# Backend: truenas_backend (NAS backend)
backend truenas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server truenas 10.1.1.73 ssl verify none

# Backend: iprox_backend ()
backend iprox_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server iprox 10.1.1.101:8006 ssl verify none

# Backend: mbfirewall_backend ()
backend mbfirewall_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server mbfirewall 127.0.0.1:55443 ssl verify none

# Backend: autgtp_backend ()
backend autgtp_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server autgtp 10.1.1.113:7070

# Backend: portainer_backend (portainer backend)
backend portainer_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server portainer 10.1.1.59:9443 ssl alpn h2,http/1.1 verify none

# Backend: nextcloud_backend (nextcloud_backend)
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server nextcloud 10.1.1.59:11000

# Backend: idrac_backend ()
backend idrac_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server idrac 192.168.0.120 ssl verify none

# Backend: testssl (test1)
backend testssl
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server testSSL 127.4.4.3 send-proxy-v2 check-send-proxy

# statistics are DISABLE

What you should do first now is to remove those extra rules for nextcloud from the HTTPS_frontend. I think you also said they didn't fix anything and I already said that they shouldn't actually be necessary afaik.
Leave only the local and public subdomains mapfile rules on it, in that order!

Now reboot your OPNsense, sometimes this can also fix strange issues.

Then please tell me the URL (replace your real domain with just domain.com) on which you are trying to access nextcloud and truenas.

External Access
truenas.domain.com --> error/result: ?
nextcloud.domain.com --> error/result: ?

Internal Access
truenas.domain.com --> error/result: ?
nextcloud.domain.com --> error/result: ?

Also nslookup or ping those URLs from an internal device to verify that the DNS override is working!

Is 10.1.1.1 the OPNsense interface IP of the network the local clients are trying to access those URLs from?

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sc00by on May 25, 2023, 01:30:14 pm

Quote from: TheHellSite on May 25, 2023, 10:36:55 am

Quote from: sc00by on May 25, 2023, 12:40:02 am

Shoot.  I thought I put a note in there about logs... :-[

Outside of the informational log, there is nothing logged.  In the informational, I see connections originating from the outside.  Nothing for hitting this particular internal IP/host, no error.  Logs accessed from Services/HAProxy/Logs, enabled all.  Nothing here is pertinent near as I can tell.  When this internal (BMC) resource loads, I see an unsigned cert which tells me it is not hitting haproxy.

When I access plex, it originates from lan IP then hits the public IP and I can see this logging client IP, it loads.  This works from inside/outside.  My other external service also works inside/outside.  An override with plex

I created a backend, server, and map file for internal services.

I've tried with and without unbound.  With and without overrides.  Right now I am using unbound, opnsense is my DNS. It loads...attempt to access the web service...I get a self signed cert.  This is the head scratcher, same result whether I use AD or Unbound DNS so assuming something I've missed.

What I am really after is 'what did I miss in the configuration' because I'll be setting up another HAproxy box targeted at internal only resources as well.

So, AD DNS is not in the picture, in this context, I am in the same spot.

I'll take another stroll through the tutorial.

Sorry I still don't know what your issue is exactly now?
Your text is very confusing to read.

What I took from it is that the internal and external accesses are working but it is not presenting a trusted cert?

External works perfect inside and outside.  Get a lets encrypt cert.  We're happy, nothing to do.

Internally?  I get a self signed cert, nothing is logged in HAProxy.

I went through the tutorial again last night.  Step 7.2 for the FQDN rule...I see you have a domain value, but I am unable to save this condition because my domain is not a valid ipv4 address?

I can plug in 'my.test.com', it will accept and tests out OK.  If I use 'lan.scobar.pw', it will not accept the value, gives error below.  Outside of informational log, nothing is recorded.

[NOTICE] (67194) : haproxy version is 2.6.13-234aa6d
[NOTICE] (67194) : path to executable is /usr/local/sbin/haproxy
[ALERT] (67194) : config : parsing [/usr/local/etc/haproxy.conf.staging:73] : error detected while parsing ACL 'acl_64685bef85c4e2.08714329' : 'lan.scobar.pw' is not a valid IPv4 or IPv6 address.
[ALERT] (67194) : config : parsing [/usr/local/etc/haproxy.conf.staging:78] : error detected while parsing switching rule : no such ACL : 'acl_64685bef85c4e2.08714329'.
[ALERT] (67194) : config : Error(s) found in configuration file : /usr/local/etc/haproxy.conf.staging
[ALERT] (67194) : config : Fatal errors found in configuration.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on May 25, 2023, 10:52:54 pm

Quote from: sc00by on May 25, 2023, 01:30:14 pm

Quote from: TheHellSite on May 25, 2023, 10:36:55 am

Quote from: sc00by on May 25, 2023, 12:40:02 am

Shoot.  I thought I put a note in there about logs... :-[

Outside of the informational log, there is nothing logged.  In the informational, I see connections originating from the outside.  Nothing for hitting this particular internal IP/host, no error.  Logs accessed from Services/HAProxy/Logs, enabled all.  Nothing here is pertinent near as I can tell.  When this internal (BMC) resource loads, I see an unsigned cert which tells me it is not hitting haproxy.

When I access plex, it originates from lan IP then hits the public IP and I can see this logging client IP, it loads.  This works from inside/outside.  My other external service also works inside/outside.  An override with plex

I created a backend, server, and map file for internal services.

I've tried with and without unbound.  With and without overrides.  Right now I am using unbound, opnsense is my DNS. It loads...attempt to access the web service...I get a self signed cert.  This is the head scratcher, same result whether I use AD or Unbound DNS so assuming something I've missed.

What I am really after is 'what did I miss in the configuration' because I'll be setting up another HAproxy box targeted at internal only resources as well.

So, AD DNS is not in the picture, in this context, I am in the same spot.

I'll take another stroll through the tutorial.

Sorry I still don't know what your issue is exactly now?
Your text is very confusing to read.

What I took from it is that the internal and external accesses are working but it is not presenting a trusted cert?

External works perfect inside and outside.  Get a lets encrypt cert.  We're happy, nothing to do.

Internally?  I get a self signed cert, nothing is logged in HAProxy.

I went through the tutorial again last night.  Step 7.2 for the FQDN rule...I see you have a domain value, but I am unable to save this condition because my domain is not a valid ipv4 address?

I can plug in 'my.test.com', it will accept and tests out OK.  If I use 'lan.scobar.pw', it will not accept the value, gives error below.  Outside of informational log, nothing is recorded.

[NOTICE] (67194) : haproxy version is 2.6.13-234aa6d
[NOTICE] (67194) : path to executable is /usr/local/sbin/haproxy
[ALERT] (67194) : config : parsing [/usr/local/etc/haproxy.conf.staging:73] : error detected while parsing ACL 'acl_64685bef85c4e2.08714329' : 'lan.scobar.pw' is not a valid IPv4 or IPv6 address.
[ALERT] (67194) : config : parsing [/usr/local/etc/haproxy.conf.staging:78] : error detected while parsing switching rule : no such ACL : 'acl_64685bef85c4e2.08714329'.
[ALERT] (67194) : config : Error(s) found in configuration file : /usr/local/etc/haproxy.conf.staging
[ALERT] (67194) : config : Fatal errors found in configuration.

Sorry but I still can quite follow...
I still don't know what you are trying to achieve here.

Also how can EXTERNAL work from inside and outside?
Per definition external is outside.

Please provide a structured answer. like above: https://forum.opnsense.org/index.php?topic=23339.msg165410#msg165410

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sevet on May 29, 2023, 01:01:16 pm

I followed this great tutorial and everything in it worked on the first hit!!
Great work, great explanation and details, thanks.

The thing that don't work for me is what is ommited from the latest version of this (as far as i understand)
I want to use the no ssl offloading option, and i can't make it work, i did a map file and rule to map my already SSLed services, i put the rule in the 0_SNI_frontend but no rule i make that is not just selecting my pool catches the host name, its like it doesn't have the url yet to select, if i put a rule to just force my pool it works but then all the none ssl services do not work.

What am i missing, If i put my SSLed services as SSL in the real server and map them as regular servers everything works but then i have double SSL which im trying to avoid.

Title: Re: Tutorial 2023/04: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mauro on May 31, 2023, 12:07:29 am

Quote from: TheHellSite on May 19, 2023, 09:20:51 am

This is the solution.

Quote

4. However, in your scenario you didn't configure your apache real server correctly since you left the port blank and didn't tell haproxy if the real server expects SSL or not.

Hi @TheHellSite,

it took me the long way to figure out my issue was the map file.
At the moment I haven't understood what is the cause but so far I created several conditions and rules working fine. The map file will be the next step but for the moment, after so many efforts, I need to get away from Haproxy a little.

Thanks for your tutorial

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: studentpcster on May 31, 2023, 10:51:35 pm

This tutorial works brilliantly, up until the end of Part 5.
Fully encrypted and secured access to my self hosted servers/services, from external networks.

Part 6 - Access from internal networks in the tutorial, does not work for me.
I have tried both the Split DNS method, as well as the port forwarding with NAT reflection.

If my internet is offline, then DNS resolution to the self hosted servers/services fails. I can only access them via internal IP addresses and ports.

Any pointers would be appreciated. Kind of a noob to reverse proxy setups.

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on June 08, 2023, 03:02:11 am

Let me know if you choose not to investigate since my configuration was a modification of the tutorial here....

The setup you helped with was working well for a while.  Now, it doesn't seem to be working.  As in, the sites are not accessible externally (i use my cell phone).  but they work fine internally, with both ip addresses and hostnames.  I also tried going back to port forwarding instead of using haproxy (for ports 80,443).  As soon as i turn on port forwarding, the sites start working externally. 


So this tells me something has changed or stopped working in my haproxy config.  but what?  I will post my config below:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         0.0.0.0 local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend ()
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp

    # logging options
    option tcplog
    # ACL: TCP_SSL_condition
    acl acl_644c56b6785678.47181279 req.ssl_hello_type 1
    # ACL: TCP_server1_condition
    acl acl_644c5700ee7657.09485748 req.ssl_sni -m sub -i domain1
    # ACL: TCP_server2_condition
    acl acl_644c5719768e71.87060950 req.ssl_sni -m sub -i domain2

    # ACTION: TCP_RequestInspectDelay_rule
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: TCP_RequestContentAccept_rule
    tcp-request content accept if acl_644c56b6785678.47181279
    # ACTION: TCP_SERVER1_rule
    use_backend TCP_SERVER1_backend if acl_644c5700ee7657.09485748
    # ACTION: TCP_SERVER2_rule
    use_backend TCP_SERVER2_backend if acl_644c5719768e71.87060950

# Frontend: HTTP_frontend ()
frontend HTTP_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp

    # logging options
    option tcplog
    # ACL: http_server1_condition
    acl acl_6457247ca14984.71641345 hdr_sub(host) -i domain1
    # ACL: http_server2_condition
    acl acl_64572496aeac32.73416688 hdr_sub(host) -i domain2

    # ACTION: http_server1_rule
    use_backend TCP_SERVER1_backend if acl_6457247ca14984.71641345
    # ACTION: http_server2_rule
    use_backend TCP_SERVER2_backend if acl_64572496aeac32.73416688

# Backend: TCP_SERVER1_backend ()
backend TCP_SERVER1_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server server1_server 192.168.1.234

# Backend: TCP_SERVER2_backend ()
backend TCP_SERVER2_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server server2_server 192.168.1.217

# Backend (DISABLED): TCP_SERVER3_backend ()



# statistics are DISABLED





Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on June 10, 2023, 03:25:46 am

I found the issue for my post above.  I had Upnp plugin installed and enabled.  For whatever reason that messed with my setup.  i disabled it and it works again.

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: jonf on June 19, 2023, 10:59:08 am

I'm having problems with my certificate not renewing, automatically and manually. I get the error 'domain validation failed (dns01)' in the 'System log' tab under ACME. I haven't changed my DNS hostname and it can still be accessed from the web (albeit with the 'NET::ERR_CERT_DATE_INVALID' error), including from the SSL Labs server certificate test website. I haven't made any recent changes to my HAProxy config either.

I thought maybe my internal DNS was a problem since I have query forwarding enabled in Unbound that redirects to the DNSCrypt-Proxy app in OPNsense, so I tried disabling it so that my ISP's DNS is used instead and the same thing happened. I also tried resetting the ACME client under Settings and again the same thing happened when I tried to manually renew.

What else should I try, or what other info do I need to give for troubleshooting?

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Grenen on June 20, 2023, 11:08:57 am

Just a headsup for everyone posting here. The author updated the post with the following:

Quote

No More Free Support

Due to the increasing number of support requests I've been receiving, both directly in the topic and via DM, I regret to inform that I can no longer provide free assistance. Balancing my real job and personal life has become extremely challenging. While I genuinely want to help everyone resolve their issues to get things up and running smoothly, I find it difficult to allocate the necessary time without sacrificing my personal commitments.

In addition, it has come to my attention that some individuals seeking help are not thoroughly reading the provided tutorial or lack the fundamental knowledge of networking. This has been a recurring issue and has made the support process increasingly frustrating.

I sincerely appreciate your interest in my expertise and if you would like to receive my assistance, I am more than happy to provide you with the details via DM.

Thank you for your understanding in this matter,
TheHellSite

But perhaps someone else have a solution to my problem. I have HAProxy up and running for a few months, was working fine. In may i added local domains map file for a site. Now i deleted the map file and removed all the local domain map file rules etc. But now my public domains aren't available from my internal network anymore (they work from external access).

Ive gone through the setup and everything seems fine, and i havent changed anything in the domain override in Unbound.

https://ibb.co/vkGLPGF (https://ibb.co/vkGLPGF)

Any suggestions where the confligt might be located? What else block internal access to my public domains?

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: jonf on June 21, 2023, 01:46:53 pm

Quote from: jonf on June 19, 2023, 10:59:08 am

I'm having problems with my certificate not renewing, automatically and manually. I get the error 'domain validation failed (dns01)' in the 'System log' tab under ACME. I haven't changed my DNS hostname and it can still be accessed from the web (albeit with the 'NET::ERR_CERT_DATE_INVALID' error), including from the SSL Labs server certificate test website. I haven't made any recent changes to my HAProxy config either.

I thought maybe my internal DNS was a problem since I have query forwarding enabled in Unbound that redirects to the DNSCrypt-Proxy app in OPNsense, so I tried disabling it so that my ISP's DNS is used instead and the same thing happened. I also tried resetting the ACME client under Settings and again the same thing happened when I tried to manually renew.

What else should I try, or what other info do I need to give for troubleshooting?

Got it working by changing the API key for my desec hostname. Not sure why the old one stopped working, but just in case anyone else is in the same boat as me try deleting your API key, create a new one, then paste that into the challenge type settings.

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on June 24, 2023, 10:23:45 pm

TheHellSite has provided a great, extremely handy tutorial here, so thank you for that.  Very much appreciated.

He does get annoyed when people don't know what they are talking about.  But at the same time, if someone knows all these things they wouldn't be here for help.  So I don't get that.  But it does suck up your time, so either way I get it.

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on June 24, 2023, 10:35:46 pm

Not trying to speak for the man but my humble view is different. That is that he doesn't get annoyed when people don't know what they're talking about or ask for assistance except when they think they can just do their own setup, different to his tutorial and asking why is not working. As if it was generic haproxy help thread.
He has made the point several times that he'd help to get it working _as per tutorial_, and people have continued deviating from it and coming to this thread for help.

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on June 26, 2023, 04:22:25 pm

Actually everything is said in my statement. But I am happy to provide an answer.

Quote from: opnuser1 on June 24, 2023, 10:23:45 pm

TheHellSite has provided a great, extremely handy tutorial here, so thank you for that.  Very much appreciated.

He does get annoyed when people don't know what they are talking about.  But at the same time, if someone knows all these things they wouldn't be here for help.  So I don't get that.  But it does suck up your time, so either way I get it.

Basically, what you are saying is this:
Someone makes a tutorial, people follow it, have issues with it (some because they lack BASIC knowledge of networking), request help from the creator, ...
In your logic it is now the obligation of the tutorial creator to happily provide FREE support in his FREE TIME for everyone using the tutorial?  ??? Nonsense...

(I don't mean any offense to you)

While actually the real issue here (which seems to be rising) is that nowadays many people expect that everything will get done for them by someone else! Hell, hopefully even for free.
"Issue XYZ appears, what do I do now? The tutorial doesn't cover it... Mhhh, I will just ask this random guy I took the tutorial/information from, he will know it for sure."
They even expect this in the concept of homelabbing, selfhosting, ...

Where has the effort gone to learn things on your own? To solve your own issues?
This is what drives me nuts! In todays world with the free availability of AI tools it is really not hard at all to learn new topics or solve issues on your own.

I am not a doctor, so I would never just grab some book/tutorial that explains to me how to remove a kidney, go to a related person and try it, when I don't even know how to hold a scalpel.
The same applies here! If you don't know the basics then maybe this is way beyond your CURRENT skills.

(with "you" I don't mean you in particular)

Quote from: cookiemonster on June 24, 2023, 10:35:46 pm

Not trying to speak for the man but my humble view is different. That is that he doesn't get annoyed when people don't know what they're talking about or ask for assistance except when they think they can just do their own setup, different to his tutorial and asking why is not working. As if it was generic haproxy help thread.
He has made the point several times that he'd help to get it working _as per tutorial_, and people have continued deviating from it and coming to this thread for help.

Someone understood the issue pretty well. :)
But to be fair, I did/do get annoyed when people don't know the basics. And IMO I have all the right in the world to do so since it is not my job to teach them these basics. Especially not if it is just a 5s online search away.
THEY are using a product/software/..., THEY want to achieve something with it, so THEY have to look around how to get it working. Plain simple.

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on June 26, 2023, 04:27:45 pm

I'd call it fair. According to the header, the thread has been read 171056 times as of now. 37 pages of assistance.
Thank you.

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opnuser1 on June 26, 2023, 09:09:13 pm

Quote from: cookiemonster on June 26, 2023, 04:27:45 pm

I'd call it fair. According to the header, the thread has been read 171056 times as of now. 37 pages of assistance.
Thank you.

Totally fair, and above and beyond.  Helped me solve a long standing goal of mine.  I was thinking of starting a similar thread, but maybe not a good idea if i do not even know the basics.

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: blackwing on June 28, 2023, 07:11:33 am

is it possible to use 1 Public IP for the Public Service that will be use by different sub domains with port 80 as its port.

    sub1.domain.com Real Server  172.16.100.20 Port 80
    sub2.domain.com Real Server  172.16.100.21 Port 80
    sub3.domain.com Real Server  172.16.100.22 Port 80
with the condition prefix base on the subdomain

Public Service has the public IP 443 and 80

I was actually trying this setup but it end up loading the same content on all subs.

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sorano on June 29, 2023, 11:42:27 am

Quote from: blackwing on June 28, 2023, 07:11:33 am

is it possible to use 1 Public IP for the Public Service that will be use by different sub domains with port 80 as its port.

    sub1.domain.com Real Server  172.16.100.20 Port 80
    sub2.domain.com Real Server  172.16.100.21 Port 80
    sub3.domain.com Real Server  172.16.100.22 Port 80
with the condition prefix base on the subdomain

Public Service has the public IP 443 and 80

I was actually trying this setup but it end up loading the same content on all subs.

Yes it is possible.

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Twitchiz on July 01, 2023, 05:38:38 am

Is nslookup on a local PC returning the proper IPs?

Quote from: Grenen on June 20, 2023, 11:08:57 am

Just a headsup for everyone posting here. The author updated the post with the following:

Quote

No More Free Support

Due to the increasing number of support requests I've been receiving, both directly in the topic and via DM, I regret to inform that I can no longer provide free assistance. Balancing my real job and personal life has become extremely challenging. While I genuinely want to help everyone resolve their issues to get things up and running smoothly, I find it difficult to allocate the necessary time without sacrificing my personal commitments.

In addition, it has come to my attention that some individuals seeking help are not thoroughly reading the provided tutorial or lack the fundamental knowledge of networking. This has been a recurring issue and has made the support process increasingly frustrating.

I sincerely appreciate your interest in my expertise and if you would like to receive my assistance, I am more than happy to provide you with the details via DM.

Thank you for your understanding in this matter,
TheHellSite

But perhaps someone else have a solution to my problem. I have HAProxy up and running for a few months, was working fine. In may i added local domains map file for a site. Now i deleted the map file and removed all the local domain map file rules etc. But now my public domains aren't available from my internal network anymore (they work from external access).

Ive gone through the setup and everything seems fine, and i havent changed anything in the domain override in Unbound.

https://ibb.co/vkGLPGF (https://ibb.co/vkGLPGF)

Any suggestions where the confligt might be located? What else block internal access to my public domains?

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: tokar86a on July 01, 2023, 11:05:26 am

delete

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: blackwing on July 01, 2023, 08:12:29 pm

Quote from: sorano on June 29, 2023, 11:42:27 am

Quote from: blackwing on June 28, 2023, 07:11:33 am

is it possible to use 1 Public IP for the Public Service that will be use by different sub domains with port 80 as its port.

    sub1.domain.com Real Server  172.16.100.20 Port 80
    sub2.domain.com Real Server  172.16.100.21 Port 80
    sub3.domain.com Real Server  172.16.100.22 Port 80
with the condition prefix base on the subdomain

Public Service has the public IP 443 and 80

I was actually trying this setup but it end up loading the same content on all subs.

Yes it is possible.

I would love to learn how to do it. cause I've been stuck with the content of my other vm that should be on another subdomain and showing up on the other subdomain.

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: sorano on July 01, 2023, 08:26:14 pm

Quote from: blackwing on July 01, 2023, 08:12:29 pm

I would love to learn how to do it. cause I've been stuck with the content of my other vm that should be on another subdomain and showing up on the other subdomain.

Great!

Motivation is usually all you need so what is stopping you from learning it?

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: chaoticmaster on July 05, 2023, 06:23:29 pm

In case anyone runs in to this issue in part 7 of the tutorial:

Quote

You can of course also use the predefined "Source IP is local" condition.

This did not wok for me and was giving me 503 errors I assume because it was using the wrong mapfile.

I defined my subnets and it worked as expected.

192.168.3.0/24, 10.168.5.0/24

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nmiller0113 on July 18, 2023, 12:26:34 am

Quote from: TheHellSite on May 31, 2021, 01:06:11 pm

Part 7 - Advanced Configuration: local-access-only subdomains
Imagine you have a service that you would like to access / protect using your brand new reverse proxy without making it available on the internet?
Well, HAProxy has got you covered!

• In your OPNsense go to: Services --> HAProxy --> Settings --> Advanced --> Map Files
  Here you need to clone the "PUBLIC_SUBDOMAINS_mapfile", rename it to f.e. "LOCAL_SUBDOMAINS_mapfile" and add all your local-access-only subdomains along with their corresponding backends.
  Keep in mind that the content of your "PUBLIC_SUBDOMAINS_mapfile" also has to be put in the "LOCAL_SUBDOMAINS_mapfile"! I will explain why later.
  (https://i.postimg.cc/hJvmymwR/P007-001-HAProxy-Map-Files-Local-Subdomains.png) (https://postimg.cc/hJvmymwR)
• Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
  Now you need a condition that detects if the source of the request is a local IP or a FQDN.
  You can of course also use the predefined "Source IP is local" condition.
  I am however using only specific subnets since the predefined condition is using the entire RFC1918 IP range, which I don't need!
  (https://i.postimg.cc/PPMcSxX5/P007-002-HAProxy-Conditions-Local-Subdomains-Subnets.png) (https://postimg.cc/PPMcSxX5)
  
  As I just said you can also check for a FQDN.
  But please keep in mind that HAProxy resolves those hostnames to their IPs and then checks them. But the resolving is only done once during the start / restart of HAProxy.
  So if the IP of your FQDN is changing regularly this won't work very well, except if you restart your HAProxy using a cron job like every 24 hours or so.
  (https://i.postimg.cc/3dhVjpHq/P007-003-HAProxy-Conditions-Local-Subdomains-FQDN.png) (https://postimg.cc/3dhVjpHq)
• Next go to: Services --> HAProxy --> Settings --> Rules & Checks --> Rules
  Here you need to clone the "PUBLIC_SUBDOMAINS_rule", rename it to f.e. "LOCAL_SUBDOMAINS_rule", select your "LOCAL_SUBDOMAINS_SUBNETS_condition" and select your "LOCAL_SUBDOMAINS_mapfile".
  If you are also using a FQDN condition, like I do, you will need to select both your FQDN and your subnet condition together with the logical "or" operator!
  (https://i.postimg.cc/64JGpmQB/P007-004-HAProxy-Rules-Local-Subdomains.png) (https://postimg.cc/64JGpmQB)
• Next go to: Services --> HAProxy --> Settings --> Virtual Services --> Public Services
  The last thing left to do is to place the "LOCAL_SUBDOMAINS_rule" before your in your "HTTPS_frontend".
  
  Attention!
  Remember that I told you to also put the content of your "PUBLIC_SUBDOMAINS_mapfile" in the "LOCAL_SUBDOMAINS_mapfile"?
  This is because HAProxy is processing the rules in the frontends based on the order they appear!
  So if you place your "PUBLIC_SUBDOMAINS_rule" before your "LOCAL_SUBDOMAINS_rule" in the frontend configuration, you won't get access to your local-access-only subdomains.
  Vice versa this will also happen and you will no longer have access to your public subdomains.
  To avoid this you have to also put the content of your "PUBLIC_SUBDOMAINS_mapfile" in the "LOCAL_SUBDOMAINS_mapfile" and place their rules in the correct order.
  
  The correct way of placing both rules is like this.
  (https://i.postimg.cc/4KMmdS3m/P007-005-HAProxy-Frontends-HTTPS-Rule-Order.png) (https://postimg.cc/4KMmdS3m)
• Done!
  You should now still have access to your public subdomains from any network and also have access to your local-access-only subdomains from the locations you defined.

@TheHellSite

First of all...thank you so so much for this extensive guide! It was awesome and extremely helpful. I got everything working first time without a hitch! I sent ya *some* beer just now! Whatever you can buy with what I sent :)

I am writing because I saw a typo in section 4 of Part 7 I quoted above.

You wrote:

The last thing left to do is to place the "LOCAL_SUBDOMAINS_rule" before your in your "HTTPS_frontend"

And I think you meant to write:

The last thing left to do is to place the "LOCAL_SUBDOMAINS_rule" before your "PUBLIC_SUBDOMAINS_rule" in your "HTTPS_frontend"

Thanks again for everything!

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on July 18, 2023, 01:08:29 am

Quote from: nmiller0113 on July 18, 2023, 12:26:34 am

First of all...thank you so so much for this extensive guide! It was awesome and extremely helpful. I got everything working first time without a hitch! I sent ya *some* beer just now! Whatever you can buy with what I sent :)

I am writing because I saw a typo in section 4 of Part 7 I quoted above.

You wrote:

The last thing left to do is to place the "LOCAL_SUBDOMAINS_rule" before your in your "HTTPS_frontend"

And I think you meant to write:

The last thing left to do is to place the "LOCAL_SUBDOMAINS_rule" before your "PUBLIC_SUBDOMAINS_rule" in your "HTTPS_frontend"

Thanks again for everything!

Thank you for the beers.
Fixed, thanks for the hint!

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: taker on July 19, 2023, 07:28:39 pm

Hello, I tried with this tutorial to set up HA Proxy. Unfortunately, the HA proxy does not start anymore. It throws this error in the log:

Code: [Select]

Error configd.py [85b23125-6c10-4561-81f5-f28b4ca64c4e] Script action failed with Command '/usr/local/opnsense/scripts/OPNsense/HAProxy/syncCerts.py actions --output bootgrid --page-rows '10' --page '1' --search '' --sort-col '' --sort-dir ''' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 44, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/OPNsense/HAProxy/syncCerts.py actions --output bootgrid --page-rows '10' --page '1' --search '' --sort-col '' --sort-dir ''' returned non-zero exit status 1.
This is my Config

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend ()
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_64b0212a904331.12997942 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_64b0212a904331.12997942

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Cloud_backend ()
backend Cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Cloud 10.7.0.206:80

# Backend: Webserver_backend ()
backend Webserver_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server webserver 10.7.0.204:80



# statistics are DISABLED
Greetings

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: jellelle on July 29, 2023, 09:14:25 am

Quote from: Tattoofreak on February 10, 2022, 03:40:27 pm

@TheHellSite
Maybe you would also like on how to enable Websockets on your frontend(s) if your incoming clients are looking for such one(s). Websockets are basically used for example for streaming services over web. I was looking for so long on how to resolve my problem for making HAProxy work with Synology's DS Cam Android app which tries to connect from remote to the Synology Surveillance Station NAS behind HAProxy and I finally found out. You have to insert the following on your frontend (where you have to replace <myBackend(Pool)> with your according backend, of course):

Code: [Select]

acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend <myBackend(Pool)> if is_websocket
Here's the link to my original and solved issue:
 https://www.synoforum.com/threads/connecting-synology-ds-cam-android-app-to-synology-surveillance-station-through-opnsense-haproxy-plugin.7969/ (https://www.synoforum.com/threads/connecting-synology-ds-cam-android-app-to-synology-surveillance-station-through-opnsense-haproxy-plugin.7969/)

Could you tell us how to enable websocket on HAProxy?
Emby and Home Assistant don't work completely without them.

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Steeltitanpro on August 02, 2023, 09:20:12 am

Thank you so much for this tutorial.

I do have (hopfully) a quick question and went through 20 some pages to see if its been asked.

I have all my subdomains working perfectly, however how do I set my example.com domain?

I have all my services under service.example.com and want a website at example.com

Im sure its something I am overlooking. Like do I put something in my Map file?

Thank you again

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nikkon on August 18, 2023, 12:48:19 pm

Hi all,
I'm curious if I can use this method for internal running services (jails on freenas) without exposing them outside.
I can force the DNS override so I resolve them with fqdn from LAN but I can't make HAproxy work and serve the Certificate for them. I already got certificates for all instances in acme (jail1.domain.x, jail2.domain.x)

thank you in advance

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nmiller0113 on August 19, 2023, 01:07:25 am

Quote from: nikkon on August 18, 2023, 12:48:19 pm

Hi all,
I'm curious if I can use this method for internal running services (jails on freenas) without exposing them outside.
I can force the DNS override so I resolve them with fqdn from LAN but I can't make HAproxy work and serve the Certificate for them. I already got certificates for all instances in acme (jail1.domain.x, jail2.domain.x)

thank you in advance

It's all right there in Part 7 of the guide "Advanced Configuration: local-access-only subdomains"

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nikkon on August 19, 2023, 01:07:51 am

Thanks

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: delicious_potatoe on August 26, 2023, 06:37:08 pm

Hello,

thank you very much for this great tutorial. At some points I despaired a bit, but that was more due to my lack of attention to read correctly!  ;D

Meanwhile everything is running, inside and outside of the network. However, there is one thing I didn't quite understand:

I didn't create DNS-Split nor NAT-Reflection rule and still I can reach all services locally through my subdomain, which shouldn't work.
The WAN rule I activate only temporarily to make a few services public for a short time, usually I am via a VPN or physically in the local network.
I see that Unbound forwards local DNS requests to the nameservers and if I set up a DNS-Split (Host Override) to keep the DNS resolution local, I have no access to my services.

Without DNS-Split or Port-Forwarding: Works
Local Device -> Pi-Hole (cache or forward) -> OPNsense -> Unbound (cache or forward) -> Root Nameserver.

With DNS-Split: Does not work
Local Device -> Pi-Hole (cache or forward) -> OPNsense -> Unbound (e.g. my-service.subdomain.dedyn.io = 192.168.1.50)

Even if everything works, I would like to understand why I do not need DNS split or it does not work.

Kind regards.

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: fizykpl on August 26, 2023, 11:21:50 pm

Quote from: TheHellSite on June 07, 2021, 11:29:22 am

Quote from: sorano on June 05, 2021, 03:00:21 pm

1. You dont need to use virtual IP's.
2. Use map files {Advanced --> Map files}

1. You dont need to use virtual IP's.
I totally get your point! This makes indeed sense but I think only if you have a static WAN IP.
As it would break the access from internal networks to the external URLs "service.subdomain.mydomain.tld" if one enabled that access using DNS rewrite rules. I am not aware of a way to rewrite DNS entries in Unbound to the WAN interface address.

With NAT reflection your way of setting this up can of course work.


2. Use map files {Advanced --> Map files}
I haven't used those yet but looks very promising!
This really makes sense in a big environment with lots of subdomains.
Thank you for pointing this out! I will add it to the FAQ.  :)

Hey there and thank you so, so much for this great tutorial! It gave me exactly what I needed!

Yet there is a reason why I'm quoting this particular post.

Configuration made basing on your tutorial was working flawlessly on version 23.7.1 (os-haproxy 4.0, haproxy26 2.6.14), but after update to 23.7.2 and haproxy26 2.6.15 HAProxy service was failing to start.

I followed sorano's suggestion to not use virtual ip and bingo! That was it (it took me hours to find out where the issue is, as there were no message in logs - just a startup failure of HAProxy).

Maybe it would be good to add adnotation or a second way to configure HTTPS_frontend?

I can confirm that it works flawlessly with dynamic WAN ip.

Once again thank you very much and @sorano too :)

Cheers
Paweł

Title: Re: Tutorial: OPNsense, HAProxy, Let's Encrypt, Wildcard Certs, 100% A+ SSLLabs
Post by: TheHellSite on August 28, 2023, 09:04:44 pm

Quote from: fizykpl on August 26, 2023, 11:21:50 pm

Hey there and thank you so, so much for this great tutorial! It gave me exactly what I needed!

Yet there is a reason why I'm quoting this particular post.

Configuration made basing on your tutorial was working flawlessly on version 23.7.1 (os-haproxy 4.0, haproxy26 2.6.14), but after update to 23.7.2 and haproxy26 2.6.15 HAProxy service was failing to start.

I followed sorano's suggestion to not use virtual ip and bingo! That was it (it took me hours to find out where the issue is, as there were no message in logs - just a startup failure of HAProxy).

Maybe it would be good to add adnotation or a second way to configure HTTPS_frontend?

I can confirm that it works flawlessly with dynamic WAN ip.

Once again thank you very much and @sorano too :)

Cheers
Paweł

Cannot reproduce! And as usual not even a config export attached...
Everything working fine here on more than 6 different OPNsense instances (with latest update 23.7.2) configured exactly as per tutorial.

Furthermore there is no reason it wouldn't work anymore with a virtual IP since the localhost is somewhat also just a virtual IP.

About your question. If you read and understood each step of the guide you can easily figure out how to configure the HTTPS_frontend directly without the SNI_frontend.

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Digital_Addict on September 14, 2023, 03:31:21 am

Ok I dont' know if I need to create a new thread for this but I'll ask here first. I've followed your guide and I've set up a service (QuasselIRC) that  uses port 4242. I've got all three public services set up with the appropriate backends, rules and conditions. Long story short I followed Part 5 to the letter. My question is for services like IRC that don't use 80 or 443 do you ahve to set up separate Virtual Services ->  Public Services for things like IRC?

Title: Re: Tutorial 2021/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: le_top on September 21, 2023, 07:26:35 pm

Quote from: jellelle on July 29, 2023, 09:14:25 am

Could you tell us how to enable websocket on HAProxy?
Emby and Home Assistant don't work completely without them.

For the record, I use the setup explained in the tutorial and I also have an issue with Home Assistant's use of websockets.

The issue is that after about 30 seconds of inactivity, the web socket is closed by haproxy, which results in a page reload as this is what Home Assistant does when the socket disconnects or reconnects (probably to ensure that the shown states are up-to-date).

I opened a github issue for that (https://github.com/home-assistant/frontend/issues/17454) and the founder blames the haproxy configuration.

There is at least one UI for nginx that has an option for websockets:
(https://community-assets.home-assistant.io/original/4X/1/e/c/1ec1ba60dc965f02fbf35b2c71e960dc11a875de.png) .

I've tried amending timeout values, but none of those that I changed had a positive effect on the timeout.

I have not found the correct option yet - I guess that there is a setting to fix this.

Edit: it seems that there is a heart-beat @55 seconds and nginx times out at 60 seconds, while ha-proxy does at 30s by default.  https://github.com/home-assistant/home-assistant-js-websocket/issues/108 .

I added "timeout tunnel 60000" to Services>HAPRoxy>Settings>Settings>Default Parameters>Custom Options  which is available after enabling the advanced mode.

Then I checked Services>HAPRoxy>Config Export where I could find this setting in the same section as the one suggested by Home Assistant (https://github.com/ochlocracy/home-assistant.io/commit/9dc92b724586c51cd90334f2dac4492792f04fab)

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: SuperMiguel on September 26, 2023, 01:32:15 am

Since last week i been getting this: MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING when trying to log into my opnsense server, any quick fix for this? Thanks

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on September 26, 2023, 06:31:09 pm

Quote from: SuperMiguel on September 26, 2023, 01:32:15 am

Since last week i been getting this: MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING when trying to log into my opnsense server, any quick fix for this? Thanks

User specific issue.
Working fine here on more than 6 different instances of OPNsense + HAProxy configured as per tutorial.
And, as always, not even a HAProxy config export included, must be hard to read the first post...

Title: HAProxy service is disabled after configuring Public Service
Post by: sleepyal on September 29, 2023, 05:20:27 pm

New here so please take it easy on me.  I'm having a strange issue following this tutorial.  Everything works out fine until I get to the Public Services part.  After I configure "0_SNI_frontend" as pictured the service is disabled and I am unable to enable it again until I either delete or disable the public service.  Haven't been able to find any conversation about this particular issue.  Any help would be greatly appreciated.

My HAProxy config is attached.

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Cromagnonaut on October 03, 2023, 03:35:47 pm

Quote from: TheHellSite on September 26, 2023, 06:31:09 pm

Quote from: SuperMiguel on September 26, 2023, 01:32:15 am

Since last week i been getting this: MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING when trying to log into my opnsense server, any quick fix for this? Thanks

User specific issue.
Working fine here on more than 6 different instances of OPNsense + HAProxy configured as per tutorial.
And, as always, not even a HAProxy config export included, must be hard to read the first post...

My guess is this is somehow related to Firefox. Could be a coincidence, but I did a fresh configuration of HAProxy following the guide today and had the same error on Firefox initially when loading the pages. When using Chromium, everything works as expected (except random 503 errors, but this is another topic ...). Nothing in the error / log files.

Maybe this helps for troubleshooting. For completeness, I have attached my config file.

BTW, @TheHellSite: Do you have another way of giving beers other than buymeacoffee? It uses stripe which some of us (including me) might not have. I would like to sponsor the effort! :)

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 03, 2023, 11:31:41 pm

Quote from: Cromagnonaut on October 03, 2023, 03:35:47 pm

Quote from: SuperMiguel on September 26, 2023, 01:32:15 am

Since last week i been getting this: MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING when trying to log into my opnsense server, any quick fix for this? Thanks

My guess is this is somehow related to Firefox. Could be a coincidence, but I did a fresh configuration of HAProxy following the guide today and had the same error on Firefox initially when loading the pages. When using Chromium, everything works as expected (except random 503 errors, but this is another topic ...). Nothing in the error / log files.

Maybe this helps for troubleshooting. For completeness, I have attached my config file.

BTW, @TheHellSite: Do you have another way of giving beers other than buymeacoffee? It uses stripe which some of us (including me) might not have. I would like to sponsor the effort! :)

Added another option to donate. :)

Check your PM.

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: _Alchemist_ on October 10, 2023, 07:45:04 pm

Hello TheHellSite,

I have recently switched back to using OPNsense and HAProxy and again used your tutorial.

As I mainly use IPv6 today, I had to slightly modify two steps to make it work with my setup:

Part 4 - System preparation

Step 4: To allow IPv4 and IPv6 with the same firewall rule, all I had to do was change "TCP/IP Version" from "IPv4" to "IPv4+IPv6":

(https://i.postimg.cc/gn71c4T2/OPNsense-Firewall-rule.png) (https://postimg.cc/gn71c4T2)

Part 5 - HAProxy configuration

Step 10: To make HAProxy listen on ports 80 and 443 on its IPv6 as well as IPv4 addresses, all I had to add here was "[::]:80" and "[::]:443":

(https://i.postimg.cc/hX0NBgTV/HAPRoxy-SNI-Frontend.png) (https://postimg.cc/hX0NBgTV)

After applying these changes, I can now securely access my services behind HAproxy from IPv4 and IPv6 networks.
Do you think you could add these changes to your tutorial? Anyway, thanks for all your work :)

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on October 10, 2023, 10:18:25 pm

Quote from: _Alchemist_ on October 10, 2023, 07:45:04 pm

Hello TheHellSite,

I have recently switched back to using OPNsense and HAProxy and again used your tutorial.

As I mainly use IPv6 today, I had to slightly modify two steps to make it work with my setup:

Part 4 - System preparation

Step 4: To allow IPv4 and IPv6 with the same firewall rule, all I had to do was change "TCP/IP Version" from "IPv4" to "IPv4+IPv6":

(https://i.postimg.cc/gn71c4T2/OPNsense-Firewall-rule.png) (https://postimg.cc/gn71c4T2)

Part 5 - HAProxy configuration

Step 10: To make HAProxy listen on ports 80 and 443 on its IPv6 as well as IPv4 addresses, all I had to add here was "[::]:80" and "[::]:443":

(https://i.postimg.cc/hX0NBgTV/HAPRoxy-SNI-Frontend.png) (https://postimg.cc/hX0NBgTV)

After applying these changes, I can now securely access my services behind HAproxy from IPv4 and IPv6 networks.
Do you think you could add these changes to your tutorial? Anyway, thanks for all your work :)

Will add if I find some time. However this should be self-explanatory, especially for someone implementing IPv6.

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bucky2780 on October 27, 2023, 06:57:24 am

Part 6 - Access from internal networks using split DNS

I would like to get internal access working... the tutorial is a revelation. Thanks again.

I need some help to understand what you are trying to do here: 
I don't recognise the IP address of the plex server you are pointing to. I did not see that IP address anywhere else in the tutorial.
Essentially I'd like the internal plex be directed to the reverse proxy, as if it originated from outside... that way no changes needed in the backend.

What IP address should I use in the override? Should I point to haproxy front end or directly to the backend service ? What ip are you using in this section?

I'm somewhat confused by this...
---------------------------------------------------
I played with this a little more, and solution is simple. The IP address in part 6 of the tutorial sample threw me.

Using unbound... create an A record with multiple cname records pointing to the lan ip. (opnsense interface). This is similar to what I'm doing today with nginx... so no problems.
When inside the network, directs to the https request to my LAN IP which is my router and haproxy interface. it works well....

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: shuvitcrew on October 27, 2023, 10:41:21 pm

I have posted, that I have troubles with the configuration of a OpenVPN-Server with HAProxy in Post #283. I have to apologise for the confusion I have caused. The HAProxy config was absolut perfect - thanks to author of this thread - the problem was a incorrect certificate in the OpenVPN server - today I've got it solved! The right configuration can be found in post #171 - that works perfect! The hint I could give is that you have to put the virtual adress of your VPN server in the "real server" IP adress field.

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: pitt1717 on November 01, 2023, 01:04:46 am

was wondering if anyone can lend a hand.

I am trying to get the Collabora CODE server running behind the HAProxy. I followed the guide and got nextcloud up. But i am unsure how to translate the apache proxy pass rules from the below link into the gui form of HA Proxy


https://sdk.collaboraonline.com/docs/installation/Proxy_settings.html#id1

thanks all

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on November 01, 2023, 10:58:07 am

Quote from: pitt1717 on November 01, 2023, 01:04:46 am

was wondering if anyone can lend a hand.

I am trying to get the Collabora CODE server running behind the HAProxy. I followed the guide and got nextcloud up. But i am unsure how to translate the apache proxy pass rules from the below link into the gui form of HA Proxy


https://sdk.collaboraonline.com/docs/installation/Proxy_settings.html#id1

thanks all
[/quote
Please read the first post. This isn't a thread for HAProxy support.

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: patrick010 on November 05, 2023, 03:57:54 pm

I'm following this great tutorial, but am running in to some issues;
First the VIP, that doesn't work. As soon as I create the HTTP_frontend Public Service (Part 5, step 10), and apply, HAProxy doesn't start anymore. When I leave the VIP part out and use localhost it does work.
Furthermore, when I get to Part 6, Option B step 2, a NAT rule which supposedly is created in Part 4 - Step 3 must be altered. However, in that particular step no NAT rule is created, only a WAN rule. Did I somehow miss this NAT rule?
Lastly, I get 503 when I try to browse to the public url. I can curl the server's IP from the OPNsense shell and my pc. Is this the missing NAT rule?
Should anyone want to take a look at this, my config is attached :)
Oh, and I'm on OPNsense 23.7.7_3

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: linjojo on November 07, 2023, 02:09:35 pm

Thanks for a great tutorial! I have followed it but tried to adapt it to my use case, but I think I am missing something that someone else perhaps has run into?

Eventually, I plan to have a couple of public services but I am not really there yet :) In the meantime, I have a couple of services running on a docker host which I would like to just expose internally via a proxy with the domain name that I have purchased. These are services running without any ssl so only http but I think I might have missed something around that as I get redirected to https, or is that expected? Trying to reach any of these services I get a 503 Service Unavailable.

Perhaps worth mentioning is that I am using AdGuard with Unbound set as my Upstream DNS, and added the services as overrides in Unbound eg. test.thisismydomain.com with the internal IP of OPNsense as the target (10.0.1.1). Is that correct?

Sorry for all the questions. I could ofc run nginx or traefik on my docker host instead, just thought I would try this out first  :)

Code: [Select]

curl -vvvv http://test.thisismydomain.com
*   Trying 10.0.1.1:80...
* Connected to test.thisismydomain.com (10.0.1.1) port 80 (#0)
> GET / HTTP/1.1
> Host: test.thisismydomain.com
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< content-length: 0
< location: https://test.thisismydomain.com/
<
* Connection #0 to host test.thisismydomain.com left intact

Code: [Select]

2023-11-07T14:23:45 Informational haproxy Connect from 10.0.1.11:62741 to 10.0.1.1:443 (1_HTTPS_frontend/HTTP)
2023-11-07T14:23:45 Informational haproxy Connect from 10.0.1.11:62741 to 10.0.1.1:443 (1_HTTPS_frontend/HTTP)
2023-11-07T14:23:45 Informational haproxy Connect from 10.0.1.11:62742 to 10.0.1.1:80 (0_SNI_frontend/TCP)
2023-11-07T14:23:45 Informational haproxy Connect from 10.0.1.11:62741 to 10.0.1.1:443 (0_SNI_frontend/TCP)
2023-11-07T14:23:45 Informational haproxy Connect from 10.0.1.11:62740 to 10.0.1.1:80 (0_SNI_frontend/TCP)

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs

# autogenerated entries for config in backends/frontends

# autogenerated entries for stats

# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80 and 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_64f0ce32710c92.22370601 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_64f0ce32710c92.22370601

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/64f0da0792f405.45981915.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBNETS_condition
    acl acl_64f0df6633f1c3.71515106 src_is_local

    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/64f0ded2f1b488.73578425.txt)] if acl_64f0df6633f1c3.71515106

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_SERVER 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: test_backend ()
backend test_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server test 10.0.1.110:49005

# statistics are DISABLED

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nmiller0113 on November 08, 2023, 12:35:07 am

Not really asking for help so much that I'm curious if anyone else has had to recently turn off OCSP stapling in order to get their services not to error in Firefox? This was working fine for a year and I've not changed a single setting in HAProxy or ACME, but all of a sudden now it doesn't work properly and I've since had to disable it to get my services accessible in Firefox again. I've dug around and cannot find a clear answer as to why.

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nmiller0113 on November 08, 2023, 01:45:52 am

Quote from: nmiller0113 on November 08, 2023, 12:35:07 am

Not really asking for help so much that I'm curious if anyone else has had to recently turn off OCSP stapling in order to get their services not to error in Firefox? This was working fine for a year and I've not changed a single setting in HAProxy or ACME, but all of a sudden now it doesn't work properly and I've since had to disable it to get my services accessible in Firefox again. I've dug around and cannot find a clear answer as to why.

Interesting. I finally found the spot /tmp/haproxy/ssl where the OCSP update file was placed so I added the CRON back and re-enabled the store setting in HAProxy, and monitored the folder and saw it was updating. So I then re-issued my cert with OCSP stapling required and now it's magically working again. Not sure what I fixed, but it's not like enabling of it is terribly difficult so I'm pretty sure I didn't change anything from the previous configuration when I re-enabled it!

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: @lex on November 11, 2023, 06:32:33 pm

Hi,

I'm having trouble (since today) accessing my server from within my LAN (internally) due to the self-signed certificate on the server itself (I'm OK from the outside as I have a good certificate on the opnsense).  The browsers now refuse an exception.

What can I do ?  Am I the only one ?

Thanks !

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 11, 2023, 07:45:17 pm

Quote from: @lex on November 11, 2023, 06:32:33 pm

Am I the only one ?

Yes you are. Issues with direct access to the local IP of the target service from within the same network has nothing to do with this tutorial.

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nikkon on November 14, 2023, 01:37:52 pm

I am trying to use this howto only for the internal services (I have no intention to expose them out) and I can't make it work.
can anyone give me a hand with this. I tried everything I know to make this work.

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: slykens on November 15, 2023, 05:26:08 am

Hello... another pfsense refugee here.

Still working on getting everything working how I want and tonight's project was wrangling haproxy. I am having a problem with https redirect so I followed the tutorial in this thread with no success.

When an https client hits haproxy, it works as expected.

When an http client hits haproxy, I get the following error in the haproxy log:

Code: [Select]

ssl_redirect/[::]:80: Received something which does not look like a PROXY protocol header
This is my present config export:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    1
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: https ()
frontend https
    bind 0.0.0.0:443 name 0.0.0.0:443 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6554226ca7c6c4.56456894.certlist
    bind [::]:443 name [::]:443 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6554226ca7c6c4.56456894.certlist
    mode http
    option http-keep-alive
    option forwardfor

    # logging options

    # ACTION: sni_translation
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/65542596a04585.83628685.txt)]

# Frontend: ssl_redirect ()
frontend ssl_redirect
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    bind [::]:80 name [::]:80 accept-proxy
    mode http
    option http-keep-alive

    # logging options

    # ACTION: ssl_redirect
    # NOTE: actions with no ACLs/conditions will always match
    http-request redirect scheme https code 301

# Backend: x_openvpn_as ()
backend x_openvpn_as
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server x_openvpn_as 10.11.23.2:443 ssl verify none

# Backend: webui ()
backend webui
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server webui 127.0.0.1:1443 ssl verify none



# statistics are DISABLED

Any ideas or guidance are welcome and appreciated. Thank you.

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: nikkon on November 15, 2023, 08:05:38 pm

does anyone knows what this long means:
2023-11-14T13:41:01   Warning   haproxy   Proxy <HTTPCLIENT> stopped (cumulated conns: FE: 0, BE: 0).   
2023-11-14T13:41:01   Warning   haproxy   Proxy HASS_Backend stopped (cumulated conns: FE: 0, BE: 0).   
2023-11-14T13:41:01   Warning   haproxy   Proxy PLEX_Backend stopped (cumulated conns: FE: 0, BE: 0).   
2023-11-14T13:41:01   Warning   haproxy   Proxy acme_challenge_backend stopped (cumulated conns: FE: 0, BE: 0).   
2023-11-14T13:41:01   Warning   haproxy   Proxy NAS_Backend stopped (cumulated conns: FE: 0, BE: 0).

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: tadchilly on November 18, 2023, 02:51:13 am

I've been using this for months and really like it... but has anyone tried adding another domain?  What steps would I need to take?

For now the PUBLIC_SUBDOMAINS_rule is used to "Map domains to backends using a map file" and

Test Type is "IF", conditions are "Nothing selected", execute function "Map domains to backend pools using a map file".

What conditions would I use to specify one or the other?

Title: Re: Tutorial 2023/05: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: tadchilly on November 18, 2023, 02:59:15 am

Quote from: Steeltitanpro on August 02, 2023, 09:20:12 am

Thank you so much for this tutorial.

I do have (hopfully) a quick question and went through 20 some pages to see if its been asked.

I have all my subdomains working perfectly, however how do I set my example.com domain?

I have all my services under service.example.com and want a website at example.com

Im sure its something I am overlooking. Like do I put something in my Map file?

Thank you again

Oops, I missed this before I posted. I'm trying to do the same.  I assume you have to make a second map file and create separate rules for it, but I have no idea what they would be.

Edit:  I think I would create a second map file, second rule pointing to that mapfile, and then a new condition for each new subdomain pointing to the new rule.  If there's a more efficient way of doing this or if anyone has any pointers, I'd really appreciate it.

I think I might try backing up my config and trying though.

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: ChillinLeon on November 22, 2023, 05:53:33 am

Quote from: slykens on November 15, 2023, 05:26:08 am

Hello... another pfsense refugee here.

Still working on getting everything working how I want and tonight's project was wrangling haproxy. I am having a problem with https redirect so I followed the tutorial in this thread with no success.

When an https client hits haproxy, it works as expected.

When an http client hits haproxy, I get the following error in the haproxy log:

Code: [Select]

ssl_redirect/[::]:80: Received something which does not look like a PROXY protocol header
This is my present config export:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    1
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: https ()
frontend https
    bind 0.0.0.0:443 name 0.0.0.0:443 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6554226ca7c6c4.56456894.certlist
    bind [::]:443 name [::]:443 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6554226ca7c6c4.56456894.certlist
    mode http
    option http-keep-alive
    option forwardfor

    # logging options

    # ACTION: sni_translation
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/65542596a04585.83628685.txt)]

# Frontend: ssl_redirect ()
frontend ssl_redirect
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    bind [::]:80 name [::]:80 accept-proxy
    mode http
    option http-keep-alive

    # logging options

    # ACTION: ssl_redirect
    # NOTE: actions with no ACLs/conditions will always match
    http-request redirect scheme https code 301

# Backend: x_openvpn_as ()
backend x_openvpn_as
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server x_openvpn_as 10.11.23.2:443 ssl verify none

# Backend: webui ()
backend webui
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server webui 127.0.0.1:1443 ssl verify none



# statistics are DISABLED

Any ideas or guidance are welcome and appreciated. Thank you.

Unfortunately, I am stuck with this as well after following the tutorials . My goal is simply to have HA proxy act as a reverse proxy for various sub domains that I aim to make backends and frontends for. Im testing just one for my proxmox server that I'd like to reverse proxy to. Yet here is the only thing i found in the logs Everytime i access my proxmox domain

Quote

2_HTTPS_Frontend/192.168.1.43:443: Received something which does not look like a PROXY protocol header

My browser will give me this error (snippet Firefox)

Quote

PR_END_OF_FILE_ERROR

While chrome will say my connection is closed but the HAProxy logs still give the same error. I'm quite new to HAProxy so I can't initially make heads or tails of this. Any help would be appreciated.

Attached is my conf

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_Frontend (Listening on 0.0.0.0:443 and 0.0.0.0:80)
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option tcplog

# Frontend: 1_HTTP_Frontend (Listening on 127.0.0.1:80)
frontend 1_HTTP_Frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_655d4c7f77c559.77912446 ssl_fc

    # ACTION: HTTPtoHTTPS
    http-request redirect scheme https code 301 if !acl_655d4c7f77c559.77912446

# Frontend: 2_HTTPS_Frontend (Listening on WAN IP:443)
frontend 2_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 192.168.1.43:443 name 192.168.1.43:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/655d518eb205a6.14872799.certlist
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option log-separate-errors
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/655d4cef9a0796.78380664.txt)]

# Backend: SSL_backend (SSL Backend)
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: Proxmox_backend (Proxmox Backend)
backend Proxmox_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Proxmox_server 172.16.1.1:8006



# statistics are DISABLED



Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: BSAfH42 on November 22, 2023, 04:44:27 pm

and another victim of this error here  :-\
both when trying to connect via

Code: [Select]

http and

Code: [Select]

https

Code: [Select]

2023-11-22T16:33:22 Informational haproxy 134.xx.xx.xx:41647 [22/Nov/2023:16:33:22.341] 1_HTTP_frontend/127.4.4.3:80: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:21 Informational haproxy 134.xx.xx.xx:41645 [22/Nov/2023:16:33:21.262] 1_HTTP_frontend/127.4.4.3:80: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:18 Informational haproxy 134.xx.xx.xx:41642 [22/Nov/2023:16:33:18.847] 1_HTTPS_frontend/127.4.4.3:443: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:18 Informational haproxy 134.xx.xx.xx:41641 [22/Nov/2023:16:33:18.795] 1_HTTPS_frontend/127.4.4.3:443: Received something which does not look like a PROXY protocol header
Versions:

Code: [Select]

Name HAProxy
Version 2.6.15-446b02c
Release_date 2023/08/09

Code: [Select]

Versions OPNsense 23.7.8_1-amd64
FreeBSD 13.2-RELEASE-p5
OpenSSL 1.1.1w 11 Sep 2023
I ran out of ideas what to try  ???

config is:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   8192
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local1 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats


# Resolver: opnsense
resolvers 64fcd546611ba3.78740961
    nameserver 127.0.0.1:53 127.0.0.1:53
    nameserver 192.168.178.1:53 192.168.178.1:53
    nameserver 9.9.9.9:53 9.9.9.9:53
    nameserver 192.168.80.2:53 192.168.80.2:53
    parse-resolv-conf
    resolve_retries 3
    timeout resolve 1s
    timeout retry 1s


# NOTE: Mailer alert bofh ignored: not configured in any backend

# Mailer: alert CB
mailers 64fcc379c27b34.94392037
    timeout mail 30s
    mailer blah.blubb.25


# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80,  0.0.0.0:443, )
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL-backend

    # logging options

# Frontend: 1_HTTP_frontend (listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    http-request use-service prometheus-exporter if { path /metrics }

    # logging options
    # ACL: NoSSL_condition
    acl acl_6314a0aad6d518.84034638 ssl_fc
    # ACL: find_acme_challenge
    acl acl_6339cb3bd963e1.30823960 path_beg -i /.well-known/acme-challenge/

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6314a0aad6d518.84034638
    # ACTION: redirect_acme_challenges
    use_backend acme_challenge_backend if acl_6339cb3bd963e1.30823960

# Frontend: 1_HTTPS_frontend (listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6314a6a33cce38.68245567.certlist
    mode http
    option http-keep-alive
    option forwardfor
    http-request use-service prometheus-exporter if { path /metrics }
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6314a164535f16.33310179.txt)]

# Backend (DISABLED): SSL-backend-old ()

# Backend: HomeAssistant_Backend (Homeassistant)
backend HomeAssistant_Backend
    # health checking is DISABLED
    email-alert mailers 64fcc379c27b34.94392037
    email-alert from a@b.c
    email-alert to a@b.c
    email-alert level alert
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server HomeAssistant 192.168.80.21:8123 resolve-prefer ipv4

# Backend: PhotoPrism (PhotoPrism App on TrueNAS)
backend PhotoPrism
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PhotoPrism 192.168.80.30:2342

# Backend: Syncthing (Syncthing on TRueNAS)
backend Syncthing
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Syncthing 192.168.80.17:20910

# Backend: Paperless (paperless-ngx DMS)
backend Paperless
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PaperLess 192.168.80.30:8000

# Backend: FileBrowser (filebrowser on TrueNAS)
backend FileBrowser
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server FileBrowser 192.168.80.17:10187

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL-backend (SSL backend pool)
backend SSL-backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Libre_photos_backend (LibrePhotos in VM)
backend Libre_photos_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server LibrePhotos 192.168.80.30:3000

# Backend: Nextcloud_Backend (Nextcloud Backend)
backend Nextcloud_Backend
    # health checking is DISABLED
    email-alert mailers 64fcc379c27b34.94392037
    email-alert from a@b.c
    email-alert to a@b.c
    email-alert level alert
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nextcloud 192.168.80.30:80 resolve-prefer ipv4

# Backend: Jellyfin_backend (Jellyfin in VM)
backend Jellyfin_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Jellyfin 192.168.80.30:8096

# Backend: PaperMerge (papermerge DMS)
backend PaperMerge
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PaperMerge 192.168.80.17:10141



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED

frontend prometheus_exporter
   bind *:8404
   mode http
   http-request use-service prometheus-exporter if { path /metrics }

should I switch to nginx as reverse proxy ???

really?

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on November 22, 2023, 05:27:18 pm

To all people above asking for help...
Or "threatening" to use NGINX instead  ;D

Everything was and is working fine for me on 10+ instances of OPNsense (outdated and latest firmware versions) configured exactly as per tutorial.
If you are not able to follow the tutorial EXACTLY as it is, then this is totally a user error on your side.

As stated in the top of the tutorial, no more free support for a perfectly working tutorial if followed correctly.
If you still need help, feel free to PM me for paid support.

Best Regards
TheHellSite

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: BSAfH42 on November 25, 2023, 11:27:43 am

I was not asking YOU explicitly to help, I was asking the community. That's what forums are made for, aren't they?

And for not following the guide: this a an unmodified reloaded backup config from an OPNsense machine that broke down 2 months ago.

And thanks for the insult. I am able to follow a guide and I triple checked. Nice way to approach potential customers, really.  :(

And I was not "threating" to switch to nginx - but I have a problem to solve. so that would be a last resort.

Paid support is fine - send me a mesage for a quote, I might take it.

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: techsolo12 on November 25, 2023, 09:31:30 pm

First of all: I use Opnsense with Unbound and AdGuard.

Like some other users, I had the problem that when I configured the public service "0_SNI_fronted", the HAProxy service could no longer be started. If I only set port "443" the problem did not occur.
So the problem was that port "80" was already blocked by another service.

A look at "Interfaces -> Diagnostics -> Netstat -> Socket" revealed that port "80" was already being used by AdGuard.

To change this, I adjusted the default http port in the AdGuard config. From "80" to "81". To do this, simply stop the AdGuard service in the WebUI, then go to the shell and edit the config.

Code: [Select]

nano /usr/local/AdGuardHome/AdGuardHome.yaml
Now you can edit the default port to "81".

Code: [Select]

http:
  pprof:
    port: 6060
    enabled: false
  address: 0.0.0.0:81
  session_ttl: 720h
After all you start the AdGuard service on WebUI. Now you should can configure services in haproxy.

best regards,
techsolo12

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: techsolo12 on November 26, 2023, 08:42:58 pm

First of all, a huge thank you to TheHellSite for this detailed tutorial!

Unfortunately, I need your help. I have configured HAProxy as described in the tutorial. However, with my own domain.

All services that are to be reached externally work as desired. Only the internal service does not seem to be "noticed" by HAProxy. Unfortunately, no accesses to the "node2-ipmi" service from the source IP from the "10.10.10.0/24" network appear in the log. I cannot connect to the service "node2-ipmi".

In firefox i got this warning "SEC_ERROR_UNKNOWN_ISSUER".

Since no log entries appear in the log, I cannot attach any.

Config export:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening to 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (listening on 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_65612d875c4e55.24914702 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_65612d875c4e55.24914702

# Frontend: 1_HTTPS_frontend (listening to 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6561dfa723cb35.23136075.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_FQDN_condition
    acl acl_6563927a593ba4.09519486 src domain.tld
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_65627ea0efa5d5.95729048 src 10.10.5.0/28 10.10.10.0/24 10.10.11.0/24
    # ACL: nextcloud_caldav
    acl acl_65626936202592.20944712 path_beg -i /.well-known/caldav
    # ACL: nextcloud_carddav
    acl acl_656269439b5220.54434789 path_beg -i /.well-known/carddav

    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/656277f5815fc5.43737480.txt)] if acl_6563927a593ba4.09519486 || acl_65627ea0efa5d5.95729048
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/65612e0d931f69.06203948.txt)]
    # ACTION: nextcloud_dav
    http-request set-path /remote.php/dav if acl_65626936202592.20944712 || acl_656269439b5220.54434789

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: cloud_backend ()
backend cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server cloud_server 10.10.20.5:80

# Backend: vw_backend ()
backend vw_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server vw_server 10.10.20.7:80

# Backend: office_backend ()
backend office_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server office_server 10.10.20.8:80

# Backend: rezepte_backend ()
backend rezepte_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server rezepte_server 10.10.20.9:3000

# Backend: cash_backend ()
backend cash_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server cash_server 10.10.20.10:5006

# Backend: node2-ipmi_backend ()
backend node2-ipmi_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server node2-ipmi_server 10.10.5.6:443 ssl verify none



# statistics are DISABLED

With best regards,
techsolo12

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bucky2780 on December 03, 2023, 11:33:09 pm

tadchilly:

adding another domain was simple for me... easier than you might first think.... no need to change the mapfile either as subdomain names do not change with both domains.

in  1_https_frontend, simply add another wildcard cert here... and everything should work as normal.
my existing cert was (acme) *.expand.net.au, and to this I added another cert of *.expand.dedyn.io

Obviously need to ensure your wildcard dns will resolve properly. I found for local use, I can set a wildcard in unbound, the same way I do in cloudlfare, and will will resolve internally just fine.

hope this helps

Quote from: tadchilly on November 18, 2023, 02:51:13 am

I've been using this for months and really like it... but has anyone tried adding another domain?  What steps would I need to take?

For now the PUBLIC_SUBDOMAINS_rule is used to "Map domains to backends using a map file" and

Test Type is "IF", conditions are "Nothing selected", execute function "Map domains to backend pools using a map file".

What conditions would I use to specify one or the other?

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bucky2780 on December 03, 2023, 11:46:53 pm

techsolo12

It is probably a dns issue. You need to setup a nat reflection solution, so that internal dns names will map to the gateway, where haproxy can do its work.(much the same as when you are coming from external).

I use unbound for dns, and setup a wildcard DNS entry much the same as I did on cloudflare and desec. This wildcard entry points to the opnsense gateway, and haproxy then does its magic.

hope that helps... (worked for me)

Quote from: techsolo12 on November 26, 2023, 08:42:58 pm

First of all, a huge thank you to TheHellSite for this detailed tutorial!

Unfortunately, I need your help. I have configured HAProxy as described in the tutorial. However, with my own domain.

All services that are to be reached externally work as desired. Only the internal service does not seem to be "noticed" by HAProxy. Unfortunately, no accesses to the "node2-ipmi" service from the source IP from the "10.10.10.0/24" network appear in the log. I cannot connect to the service "node2-ipmi".

In firefox i got this warning "SEC_ERROR_UNKNOWN_ISSUER".

Since no log entries appear in the log, I cannot attach any.

Config export:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening to 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (listening on 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_65612d875c4e55.24914702 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_65612d875c4e55.24914702

# Frontend: 1_HTTPS_frontend (listening to 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6561dfa723cb35.23136075.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_FQDN_condition
    acl acl_6563927a593ba4.09519486 src domain.tld
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_65627ea0efa5d5.95729048 src 10.10.5.0/28 10.10.10.0/24 10.10.11.0/24
    # ACL: nextcloud_caldav
    acl acl_65626936202592.20944712 path_beg -i /.well-known/caldav
    # ACL: nextcloud_carddav
    acl acl_656269439b5220.54434789 path_beg -i /.well-known/carddav

    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/656277f5815fc5.43737480.txt)] if acl_6563927a593ba4.09519486 || acl_65627ea0efa5d5.95729048
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/65612e0d931f69.06203948.txt)]
    # ACTION: nextcloud_dav
    http-request set-path /remote.php/dav if acl_65626936202592.20944712 || acl_656269439b5220.54434789

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: cloud_backend ()
backend cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server cloud_server 10.10.20.5:80

# Backend: vw_backend ()
backend vw_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server vw_server 10.10.20.7:80

# Backend: office_backend ()
backend office_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server office_server 10.10.20.8:80

# Backend: rezepte_backend ()
backend rezepte_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server rezepte_server 10.10.20.9:3000

# Backend: cash_backend ()
backend cash_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server cash_server 10.10.20.10:5006

# Backend: node2-ipmi_backend ()
backend node2-ipmi_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server node2-ipmi_server 10.10.5.6:443 ssl verify none



# statistics are DISABLED

With best regards,
techsolo12

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DOM_EUWest on December 17, 2023, 03:18:22 pm

Thank you for these detailed instructions.

It has made my life much easier.


However, I have the problem that Plex is not accessible, neither internally nor externally.


The Plex log shows me the error....


CERT: incomplete TLS handshake from xxx.xxx.xxx:28422: stream truncated

HA Log..
Informational   haproxy   xx.x.xx.xxx:21734 [18/Dec/2023:02:33:09.222] 1_HTTPS_frontend/127.4.4.3:443: SSL handshake failure


The certificate is new and was requested via ACME with the exact same settings.


I am at a bit of a loss.

MTU WAN: 1500 auto
MTU LAN 9000 with 10G LAGG interface

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    8
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:443 0.0.0.0:80)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_657ed45319efa3.43352536 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_657ed45319efa3.43352536

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/657ed88b10e6c1.81075400.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/657ed57bcfd057.79414853.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    timeout connect 30s
    server SSL_Server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Plex_DMZ_backend ()
backend Plex_DMZ_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    timeout connect 30s
    http-reuse safe
    server Plex_Server_DMZ 10.10.20.11:32400 ssl verify none resolve-prefer ipv4



# statistics are DISABLED




10.10.2.1 is the LAN IP

When i set the DNS to the Real Server ( 10.10.20.11) and go to https://plex.mydomain.eu:32400 , all work fine.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DOM_EUWest on December 18, 2023, 02:16:06 pm

Ok :-)

Now it works.

Changed the LAN MTU from 9000 to 4076 was the hack. Now all works fine

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Aldo on January 06, 2024, 01:49:01 am

Hi,
This a great tutorial!
I don't why my certificate is B but it helped me handle all my backends with the haproxy.
Ok, my OPNsense and network knowledge noobness made me do some mistake but everything's ok in the end.
And I'm using Cloudflare instead of deSEC with no problem at all.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mater on January 07, 2024, 07:27:04 pm

I'm using port 443 on the WAN Interface for my wireguard VPN.
So I can use it in public wifi (most have port 443 in tcp/udp not blocked).

How do I have to configure HAProxy, so it only uses this Port local?
I only use SSL + Reverse Proxy internal and over the VPN.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Easy4Breezy on January 17, 2024, 08:58:01 am

I followed the tutorial and everything works fine, thank you so much!

But for some reason Seafile is not accessible for me and I only get error 503.
When I open the web interface with the internal IP over port 9080 it works fine.

I configured everything as it says on the website:
https://manual.seafile.com/docker/pro-edition/deploy_seafile_pro_with_docker/ (https://manual.seafile.com/docker/pro-edition/deploy_seafile_pro_with_docker/)

Quote

Since version 10.0.x, if you want to use a reverse proxy and apply for a certificate outside docker, you can use FORCE_HTTPS_IN_CONF to force write https://<your_host> in the configuration file.

e.g.

seafile:
    ...
    environment:
        ...
        - SEAFILE_SERVER_LETSENCRYPT=false
        - SEAFILE_SERVER_HOSTNAME=seafile.example.com
        - FORCE_HTTPS_IN_CONF=true
        ...

Title: Re: Tutorial 2023/09: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: securid on January 21, 2024, 12:22:08 pm

Quote from: BSAfH42 on November 22, 2023, 04:44:27 pm

and another victim of this error here  :-\
both when trying to connect via

Code: [Select]

http and

Code: [Select]

https

Code: [Select]

2023-11-22T16:33:22 Informational haproxy 134.xx.xx.xx:41647 [22/Nov/2023:16:33:22.341] 1_HTTP_frontend/127.4.4.3:80: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:21 Informational haproxy 134.xx.xx.xx:41645 [22/Nov/2023:16:33:21.262] 1_HTTP_frontend/127.4.4.3:80: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:18 Informational haproxy 134.xx.xx.xx:41642 [22/Nov/2023:16:33:18.847] 1_HTTPS_frontend/127.4.4.3:443: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:18 Informational haproxy 134.xx.xx.xx:41641 [22/Nov/2023:16:33:18.795] 1_HTTPS_frontend/127.4.4.3:443: Received something which does not look like a PROXY protocol header
Versions:

Code: [Select]

Name HAProxy
Version 2.6.15-446b02c
Release_date 2023/08/09

Code: [Select]

Versions OPNsense 23.7.8_1-amd64
FreeBSD 13.2-RELEASE-p5
OpenSSL 1.1.1w 11 Sep 2023
I ran out of ideas what to try  ???

config is:

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   8192
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local1 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats


# Resolver: opnsense
resolvers 64fcd546611ba3.78740961
    nameserver 127.0.0.1:53 127.0.0.1:53
    nameserver 192.168.178.1:53 192.168.178.1:53
    nameserver 9.9.9.9:53 9.9.9.9:53
    nameserver 192.168.80.2:53 192.168.80.2:53
    parse-resolv-conf
    resolve_retries 3
    timeout resolve 1s
    timeout retry 1s


# NOTE: Mailer alert bofh ignored: not configured in any backend

# Mailer: alert CB
mailers 64fcc379c27b34.94392037
    timeout mail 30s
    mailer blah.blubb.25


# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80,  0.0.0.0:443, )
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL-backend

    # logging options

# Frontend: 1_HTTP_frontend (listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    http-request use-service prometheus-exporter if { path /metrics }

    # logging options
    # ACL: NoSSL_condition
    acl acl_6314a0aad6d518.84034638 ssl_fc
    # ACL: find_acme_challenge
    acl acl_6339cb3bd963e1.30823960 path_beg -i /.well-known/acme-challenge/

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6314a0aad6d518.84034638
    # ACTION: redirect_acme_challenges
    use_backend acme_challenge_backend if acl_6339cb3bd963e1.30823960

# Frontend: 1_HTTPS_frontend (listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6314a6a33cce38.68245567.certlist
    mode http
    option http-keep-alive
    option forwardfor
    http-request use-service prometheus-exporter if { path /metrics }
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6314a164535f16.33310179.txt)]

# Backend (DISABLED): SSL-backend-old ()

# Backend: HomeAssistant_Backend (Homeassistant)
backend HomeAssistant_Backend
    # health checking is DISABLED
    email-alert mailers 64fcc379c27b34.94392037
    email-alert from a@b.c
    email-alert to a@b.c
    email-alert level alert
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server HomeAssistant 192.168.80.21:8123 resolve-prefer ipv4

# Backend: PhotoPrism (PhotoPrism App on TrueNAS)
backend PhotoPrism
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PhotoPrism 192.168.80.30:2342

# Backend: Syncthing (Syncthing on TRueNAS)
backend Syncthing
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Syncthing 192.168.80.17:20910

# Backend: Paperless (paperless-ngx DMS)
backend Paperless
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PaperLess 192.168.80.30:8000

# Backend: FileBrowser (filebrowser on TrueNAS)
backend FileBrowser
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server FileBrowser 192.168.80.17:10187

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL-backend (SSL backend pool)
backend SSL-backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Libre_photos_backend (LibrePhotos in VM)
backend Libre_photos_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server LibrePhotos 192.168.80.30:3000

# Backend: Nextcloud_Backend (Nextcloud Backend)
backend Nextcloud_Backend
    # health checking is DISABLED
    email-alert mailers 64fcc379c27b34.94392037
    email-alert from a@b.c
    email-alert to a@b.c
    email-alert level alert
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nextcloud 192.168.80.30:80 resolve-prefer ipv4

# Backend: Jellyfin_backend (Jellyfin in VM)
backend Jellyfin_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Jellyfin 192.168.80.30:8096

# Backend: PaperMerge (papermerge DMS)
backend PaperMerge
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PaperMerge 192.168.80.17:10141



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED

frontend prometheus_exporter
   bind *:8404
   mode http
   http-request use-service prometheus-exporter if { path /metrics }

should I switch to nginx as reverse proxy ???

really?

Did you (or anyone else perhaps) ever figure this out?

I've set this up following the guide, everything works fine, but HA doesn't. I find old reddit threads and things about websockets, but it just doesn't want to work.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 21, 2024, 01:52:52 pm

Sure, fixed via PM.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: securid on January 21, 2024, 05:32:14 pm

Apologies @TheHellSite.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 21, 2024, 07:07:31 pm

Quote from: securid on January 21, 2024, 05:32:14 pm

Quote from: TheHellSite on January 21, 2024, 01:52:52 pm

My post was helpful to you?

No, and I wasn't asking you either.

Yes you were, since I was that "anyone" who fixed it for him and many others that contacted me via PM.
And since you are following my guide I think it is somewhat helpful to you. But you do yours I guess.

Quote from: securid on January 21, 2024, 05:32:14 pm

Quote from: TheHellSite on January 21, 2024, 01:52:52 pm

Feel free to click [applaud] to the left underneath my profile.

Not for replies like these.

Fine with me.

Quote from: securid on January 21, 2024, 05:32:14 pm

Quote from: TheHellSite on January 21, 2024, 01:52:52 pm

Additionally you can consider donating

Only in your dreams.

Again, fine with me.

Quote from: securid on January 21, 2024, 05:32:14 pm

Unless you're desperate to earn something, then I can tell you these posts don't work in your advantage.

I think you are pretty smart, so if you take a look at the amount of views this guide has versus the number of people that actually donated you would be able to answer this on your own.

I even helped some people that contacted me via PM free of charge. Because they provided all the necessary information without me asking, have been very nice, friendly and just good to chat with.

Just to be clear though if it was my intention to make a living out of this I would put this whole guide behind a paywall or put it on my personal blog (which doesn't exist and hopefully never will) and fill it up with ads fishy as hell (like others do).
I even know that there are many companies out there that use this guide to bill their clients a big amount of money while I get nothing from it. Guess what, fine with me.

Instead I chose to publish this in the official forum, free of charge, available to anyone, without asking for anything.
Why? Because I believe in and support the "free knowledge for everyone" movement.
Not because I am from a third world country and can barely make a living,. I do this because it will make the world just a tiny little bit better.

Quote from: securid on January 21, 2024, 05:32:14 pm

But then I think perhaps you're not aware how some other people read your messages, so perhaps this makes you aware. If you want to take my advise, stop posting if you don't want to help (its fine!) but don't be like this.

Just so you know your issue and the ones of many others posting here is based on the fact that (you) misread / skipped / oversaw a tiny but very important bit of the configuration.
Additionally you didn't even include the config export, like many others. Which I clearly tell everyone to do so.
Now tell me you wouldn't be annoyed by that at this point.

But yes, you are right, I guess I am the offensive dick in this scenario.

Last but not least, the offensive dick is still happy to help.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: securid on January 21, 2024, 07:53:53 pm

Quote from: TheHellSite on January 21, 2024, 07:07:31 pm

...

apologies.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: chemlud on January 21, 2024, 08:30:02 pm

Really? Please stop, it's not worth it. If you can`t post something positive, don`t post anything at all... ;-)

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: muxflash on January 24, 2024, 03:56:10 pm

Hello,

I use OpnSense and Haproxy for my Portainer.

It works like a charm but I have a problem with WebSockets while opening Console of a container , It doesn't work

I try to add
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Real-IP %[src]

in Option pass-through  of my docker_backend

but this doesn't work

I try with rules ans Acl but no more success

Can someone help me ?

Thank you

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mgrunwald on January 30, 2024, 04:41:13 pm

Thank you very much for the tutorial! This worked great for me until I upgraded to 24.1. This broke my setup and every connection uses the first cert in the list even if this is for a different domain. I get the NET::ERR_CERT_COMMON_NAME_INVALID error

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 31, 2024, 03:36:29 pm

Attention

The latest HAProxy update has a few bugs related to SNI and OCSP updates.

This means the following:
1. If you have multiple certificates on the same frontend for different subdomains/domains then connections will fail.
2. If you have "INVALID_SNI" configured then HAProxy will now always use the INVALID_SNI certificate because of issue 1.
3. The new built-in OCSP update feature is not working as intended just yet.

The good thing is that all issues are already patched and will be fixed in the next update.

Alternatively you can patch manually.
https://github.com/opnsense/plugins/issues/3779#issuecomment-1917956814

In order to apply the patch manually, access the OPNsense shell and run the following commands.

Code: [Select]

1. Reinstall the HAProxy plugin.
pkg install -f os-haproxy
2. Apply the patch.
opnsense-patch -c plugins 404c19f6e
3. Restart HAProxy from the OPNsense dashboard or reboot OPNsense.
EDIT:
HAProxy refuses to start if a self-signed certificate is configured as (default) certificate under the SSL offloading section on a (HTTPS) frontend.
So for now it is best to remove the "INVALID_SNI" certificate as default from the HTTPS frontend.

Kind Regards
TheHellSite

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: rmouthaan on January 31, 2024, 06:00:01 pm

@TheHellSite, Thanks for the quick response and solution 8) The above solution works for me!

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: netnut on January 31, 2024, 08:48:35 pm

Quote from: TheHellSite on January 31, 2024, 03:36:29 pm

EDIT:
HAProxy refuses to start if a self-signed certificate is configured as (default) certificate under the SSL offloading section on a (HTTPS) frontend.
So for now it is best to remove the "INVALID_SNI" certificate as default from the HTTPS frontend.

@TheHellSite

I'm _not_ using your plugin, but I do use HAProxy on other systems with a crt-list, default self-signed cert and ocsp updates. So a shot in the dark, not sure if this "solves" your problem: You might want to declare your "default" certificate with "!*" in a crt-list to prevent errors:

https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#5.1-crt-list

/etc/haproxy/frontend-crt-list.conf

Code: [Select]

/path/to/default.crt.pem !*
/path/to/fqdn.crt.pem [ocsp-update on alpn h2,http/1.1] foo.bar
/path/to/wildcard.crt.pem [ocsp-update on alpn h2,http/1.1] *.foo.bar


Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on January 31, 2024, 09:53:38 pm

Quote from: netnut on January 31, 2024, 08:48:35 pm

Quote from: TheHellSite on January 31, 2024, 03:36:29 pm

EDIT:
HAProxy refuses to start if a self-signed certificate is configured as (default) certificate under the SSL offloading section on a (HTTPS) frontend.
So for now it is best to remove the "INVALID_SNI" certificate as default from the HTTPS frontend.

@TheHellSite

I'm _not_ using your plugin, but I do use HAProxy on other systems with a crt-list, default self-signed cert and ocsp updates. So a shot in the dark, not sure if this "solves" your problem: You might want to declare your "default" certificate with "!*" in a crt-list to prevent errors:

https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#5.1-crt-list

/etc/haproxy/frontend-crt-list.conf

Code: [Select]

/path/to/default.crt.pem !*
/path/to/fqdn.crt.pem [ocsp-update on alpn h2,http/1.1] foo.bar
/path/to/wildcard.crt.pem [ocsp-update on alpn h2,http/1.1] *.foo.bar


Thank you, this should definitely fix the issue.
However most of the people using this guide refrain from editing the OPNsense config files.
Therefore I hope we will see a fix in the OPNsense HAProxy plugin for self-signed certs.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: user78425653 on February 01, 2024, 04:00:45 pm

Hello @TheHellSite

I was using opnsense with HAProxy > 1 year.
Thanks for opffering this free tutorial, it definitely made the world better (at least for me).

The problem with opnsene 24.1 and HAProxy 4.2 hit me as well.

I will be happy when the "strict-sni" update of your guide is released.

Thank you in advance.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 01, 2024, 04:13:31 pm

Quote from: user78425653 on February 01, 2024, 04:00:45 pm

Hello @TheHellSite

I was using opnsense with HAProxy > 1 year.
Thanks for opffering this free tutorial, it definitely made the world better (at least for me).

The problem with opnsene 24.1 and HAProxy 4.2 hit me as well.

I will be happy when the "strict-sni" update of your guide is released.

Thank you in advance.

It just dropped.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: user78425653 on February 01, 2024, 04:24:06 pm

Quote from: TheHellSite on February 01, 2024, 04:13:31 pm

Quote from: user78425653 on February 01, 2024, 04:00:45 pm

Hello @TheHellSite

I was using opnsense with HAProxy > 1 year.
Thanks for opffering this free tutorial, it definitely made the world better (at least for me).

The problem with opnsene 24.1 and HAProxy 4.2 hit me as well.

I will be happy when the "strict-sni" update of your guide is released.

Thank you in advance.

It just dropped.

Thanks so much. My day would be horrible today if that didn't work.

@others I can confirm it works but I patched as TheHellSite described. Can't confirm it working without patching opnsense first.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 01, 2024, 04:26:19 pm

Important changes for OPNsense v24.1 and above

• 20240201
  
  • Simplified and improved Part 8. The "INVALID_SNI" certificate is no longer needed, instead we use the "strict-sni" parameter to achieve an even better result.
    Thanks @mnaiman (https://github.com/opnsense/plugins/issues/3779#issuecomment-1920084820) for the suggestion.

To comprehend the changes introduced in OPNsense v24.1 and above (HAProxy version >4.2) please do the following.
This is only necessary if you followed "Part 8 - Advanced Configuration: Hide your certificate on access by IP" of the tutorial!

Modify your configs as shown in the updated part 8 of the tutorial and remove the "INVALID_SNI" certificate as default certificate from your "HTTPS_frontend".

The "INVALID_SNI" certificate can then be deleted at "System: Trust: Certificates".
The "INVALID_SNI" certificate-authority can then be deleted at "System: Trust: Authorities".

Kind Regards
TheHellSite

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: furax on February 02, 2024, 09:23:39 pm

@TheHellSite,

Thank you so much for the things you're doing here!

Regards,

Julien.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: loop0 on February 03, 2024, 12:15:21 pm

@TheHellSite, many thanks and kudos for your tremendous effort, contribution and help!

I try to avoid seeking for help and solve my problems on my own. But after upgrading to 24.1 I stuck in the CRON configuration when it comes to the update HAProxy OCSP Data you mentioned in Part5.4, this feature has disappeared and can no longer be selected. I assume this is needed to get the OCSP must staple extension running.

Is there an alternative way of configuring or what I'm doing wrong or missing.
 
Thanks loop0

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: securid on February 03, 2024, 05:22:25 pm

I have 2 domains, public and internal seperated and its been working fine. Needed the patches of course after the updates which bugged out SNI on haproxy, but fortunately that was an easy fix (thanks!  8)).

I actually have more internal domains in different subnets. Is it possible to add new certificates for each one and then add these to the list the same way as the first internal domain? I would need to setup a map for each one too and I am thinking that as long as they are before the public map this should work?

I don't wanna bork things up so I figured I'd ask first  :D .

Has anyone done this and is it as straightforward as I think it is?

Thanks!

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: vladnik on February 03, 2024, 10:47:06 pm

Attention!

Hey everyone,

after the upgrade to 24.1, please check your cron job for updating OCSP data.
Since that function is no longer available from the list, mine was set to "Automatic firmware update", which could potentially be *really* bad :o

(https://private-user-images.githubusercontent.com/48983623/302065614-86c2263e-3e88-49ac-826a-615588a7968f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDcwMDAxMjYsIm5iZiI6MTcwNjk5OTgyNiwicGF0aCI6Ii80ODk4MzYyMy8zMDIwNjU2MTQtODZjMjI2M2UtM2U4OC00OWFjLTgyNmEtNjE1NTg4YTc5NjhmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAyMDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMjAzVDIyMzcwNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTQyMDJjODBhOTcyMTU3ODk5YzJkMWQ5MjdhMzg0MmMyMjlmZjU1ZmQ3ODZhODk4ODU4YmVmMmQ1YjNhYzJkZWYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.sruqUwFLvSw6RXkYMxTkr8wijqXrYRQKGyuC7Lh2pVY)

@TheHellSite: Thank you for the guide and the ongoing maintenance, much appreciated! :) (beer is on the way)

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: techsolo12 on February 03, 2024, 11:26:50 pm

Quote from: loop0 on February 03, 2024, 12:15:21 pm

@TheHellSite, many thanks and kudos for your tremendous effort, contribution and help!

I try to avoid seeking for help and solve my problems on my own. But after upgrading to 24.1 I stuck in the CRON configuration when it comes to the update HAProxy OCSP Data you mentioned in Part5.4, this feature has disappeared and can no longer be selected. I assume this is needed to get the OCSP must staple extension running.

Is there an alternative way of configuring or what I'm doing wrong or missing.
 
Thanks loop0

As i know the OCSP update cronjob isn't needed anymore since the OCSP feature was completely revamped with the actual version of haproxy 4.2 which is bundled in opnsense 24.1

I had some errors with the OCSP updates so i opened a issue in the opnsense/plugins github repo.
https://github.com/opnsense/plugins/issues/3755

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 03, 2024, 11:32:49 pm

Quote from: vladnik on February 03, 2024, 10:47:06 pm

Attention!

Hey everyone,

after the upgrade to 24.1, please check your cron job for updating OCSP data.
Since that function is no longer available from the list, mine was set to "Automatic firmware update", which could potentially be *really* bad :o

@TheHellSite: Thank you for the guide and the ongoing maintenance, much appreciated! :) (beer is on the way)

Thank you for pointing this out, I opened a bug report.
https://github.com/opnsense/core/issues/7197

Quote from: techsolo12 on February 03, 2024, 11:26:50 pm

Quote from: loop0 on February 03, 2024, 12:15:21 pm

@TheHellSite, many thanks and kudos for your tremendous effort, contribution and help!

I try to avoid seeking for help and solve my problems on my own. But after upgrading to 24.1 I stuck in the CRON configuration when it comes to the update HAProxy OCSP Data you mentioned in Part5.4, this feature has disappeared and can no longer be selected. I assume this is needed to get the OCSP must staple extension running.

Is there an alternative way of configuring or what I'm doing wrong or missing.
 
Thanks loop0

As i know the OCSP update cronjob isn't needed anymore since the OCSP feature was completely revamped with the actual version of haproxy 4.2 which is bundled in opnsense 24.1

I had some errors with the OCSP updates so i opened a issue in the opnsense/plugins github repo.
https://github.com/opnsense/plugins/issues/3755

I will update the tutorial once again to comprehend the changes, please bear with me.

EDIT: Fixed, see Part 5.

As the automatic OCSP updates are now fully built into HAProxy, there is no need for the previously used Cron job, which has been removed from the OPNsense system as of update 24.1.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DarkCorner on February 04, 2024, 11:07:28 pm

In Public Services 1_HTTPS_Frontend
the "SSL option pass-through" field in the tutorial ther is "curves secp384r1".
Having generated the certificate with the simple 4096, what value should be indicated in this field?

How and where should logging be enabled to pinpoint where errors occur?

Thanks in advance.

=== Update ===
I generated the certificate again with ec384, as suggested in the tutorial. I reset the "SSL option pass-through" field with the value "curves secp384r1".

Now:
1) From Internet, using the certificate for web GUI access works and the certificate was recognized.
2) From Internet, a NAS and Debian Server in DMZ are not recognized. In Firefox the error is "Connection timed out".
3) In Statistic Status the Backends are No Check.
4) The SSL Labs test on the two subdomains returns the value "A" and not "A+" in both tests. DNS CAA = No is reported. The documentation reports that for both servers https://URL (HTTP/1.1 503 Service Unavailable)
5) In Firewall /Log / Live view the WAN rule is executed.

Evidently there is something blocking the call of the two servers in DMZ. Maybe there's a rule missing?
How can I enable HAProxy logging?

=== New Update ===
I checked twice, all the parameters.
Servers continue to be unreachable with error 503.

I don't understand why I only have to listen to ports 80 and 443.
It is true that the service port is indicated in the Real Server (for example, 32400 is used in the tutorial), but in the browser I have to type MY-DOMAIN.TLD:NumberPort.
Already in the WAN rule NumberPort is not filtered because it is not in the alias.

Thanks in advance, again.

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Squiggley on February 05, 2024, 08:59:39 pm

@TheHellSite, Just wanted to say thank you soooooo much for this tutorial. It has made a very complicated task much easier for me. I converted over from pfsense because this tutorial was exactly what I wanted to setup and I have not been disappointed. It turns out I like opnsense much better too!

Title: Re: Tutorial 2023/12: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: DarkCorner on February 06, 2024, 01:42:56 pm

I'm sorry, but I stillo get errors instead.

After moving the two servers to the LAN and creating a new VirtualIP, now all the configurations correspond to the tutorial (obviously if there hadn't been some oversight on my part).

The only difference I find is in the Public Services / Type option "http-keep-alive [default]" in the snapshot of point 9 and that in this version of HAProxy is not present.

• The SSL Labs test on my-domain.tld returns "Certificate name mismatch" because it searches for *.my-domain.tld
  Instead, if I search for server.my-domain.tld the test is A and not A+
• When calling server.my-domain.tld:NumPort from the internet with Firefox I get the "connection timed out" error and in Firewall Log Live View, the public IP address of the laptop being tested reports the error "Default
  deny / state violation rule"
• When calling server.my-domain.tld from the internet with Firefox I get the "503 Service Unavailable" error without errors in Firewall Log Live View.

I'm sorry, because the architecture proposed in this Tutorial is interesting.

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: mgrunwald on February 07, 2024, 02:42:05 pm

I am not sure this was mentioned before but https://desec.io no longer new registrations for DynDNS.
For the German speaking audience I can highly recommend https://ipv64.net/
Many texts on the website are English, but someone not speaking German might have problems understanding everything

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: keyboardDabbler on February 07, 2024, 09:58:28 pm

EDIT

Nevermind, my first issue was related to https://forum.opnsense.org/index.php?topic=38435.0 (https://forum.opnsense.org/index.php?topic=38435.0) which has since been patched.

As always, i appreciate the upkeep of this guide.

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 08, 2024, 11:02:47 am

Quote from: mgrunwald on February 07, 2024, 02:42:05 pm

I am not sure this was mentioned before but https://desec.io no longer new registrations for DynDNS.
For the German speaking audience I can highly recommend https://ipv64.net/
Many texts on the website are English, but someone not speaking German might have problems understanding everything

This is not entirely true, but I am already aware of that.
deSec had a "bot registration attack" and therefore paused all new domain registrations temporarly.

https://talk.desec.io/t/dyndns-service-down/823/3?u=thehellsite

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bunchofreeds on February 08, 2024, 09:23:41 pm

Hi all,

I'm trying to get my internally hosted services to report the originating client IP when going through a proxy chain starting with Cloudflare then to HAproxy. Currently HAproxy logs shows the local CloudFlare CDN address.

Cloudflare is setup to proxy and is Full (Strict) meaning I'm using the Cloudflare origin cert offloaded at HAproxy

I've found that cloudflare do collect the Client IP within cf-connecting-ip
https://developers.cloudflare.com/support/troubleshooting/restoring-visitor-ips/restoring-original-visitor-ips/

And I have found this post that helps someone with pfSense to do what I want
https://forum.netgate.com/topic/176777/haproxy-cloudflare-restoring-original-ip/3

What I'm not sure about is how (if possible) to get HAproxy to reference the cloudflare IP address list to know what sessions to insert the cf-connecting-ip into x-forwarded-for
Ideally this is in the form of some alias or map that dynamically checks https://www.cloudflare.com/ips-v4

Thanks for any help with this, also it's not urgent at all and just for my home setup and for fun really.

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: stefan21 on February 22, 2024, 12:37:49 am

OPNsense is up-to-date -->

OPNsense 24.1.2_1-amd64
FreeBSD 13.2-RELEASE-p10
OpenSSL 3.0.13

Firefox: 121.0 (64-bit), archlinux (doesn't matter, latest update on windows brings up the same issue)

DEFAULT in FF: security.ssl.enable_ocsp_stapling = true

--> leads to no access on any pages with certs following the tutorial. At least if pages are secured to local access only. I assume, same error for public access.

Changing the default in FF to false gives access back.

Question: anyone else with this experiance? Something wrong in my config? Did I overlook something? I tried to follow exactly the tutorial (not using wild card certs). Did work before the last update of the certs... OCSP-server down? If so, does OCSP making sense for servers meant for internal access only? Not at all for me. Even for public access - if an OCSP server is blocked or down, no access to a server would be possible. No control to third party (man in the middle) confirm. Does this really make sense?

Opinions welcome.

regards,
stefan

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: johnmcallister on February 22, 2024, 01:35:06 am

Just chiming in here --

Thanks very much doing all the work on this How-To, OP, and for keeping it updated, etc.
I successfully implemented it in my modest OPNsense instances/networks, before realizing that for small networks where there may never be more than perhaps 1 to 3 people logging in to a given OPNsense instance, in fact it's far more secure to simply shut off all HTTP listening on external network ports, and instead use SSH tunnels / port redirects, i.e. ssh -L 9450:localhost:80 my.opnsense.host to connect directly to the opnSense instance and access the webgui that way.  Then it doesn't matter at all whether HTTPS is active as the entire connection takes place inside the highly-secured SSH network connection.  With SSL tunnels there is no need for a webgui process to be listening anywhere except localhost:80.

It avoids the major overhead and ongoing maintenance complexity of getting a public/real SSL certificate issued, emplaced, and dealing with ongoing renewals, etc.

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: cookiemonster on February 22, 2024, 10:38:54 am

Quote from: johnmcallister on February 22, 2024, 01:35:06 am

Just chiming in here --

Thanks very much doing all the work on this How-To, OP, and for keeping it updated, etc.
I successfully implemented it in my modest OPNsense instances/networks, before realizing that for small networks where there may never be more than perhaps 1 to 3 people logging in to a given OPNsense instance, in fact it's far more secure to simply shut off all HTTP listening on external network ports, and instead use SSH tunnels / port redirects, i.e. ssh -L 9450:localhost:80 my.opnsense.host to connect directly to the opnSense instance and access the webgui that way.  Then it doesn't matter at all whether HTTPS is active as the entire connection takes place inside the highly-secured SSH network connection.  With SSL tunnels there is no need for a webgui process to be listening anywhere except localhost:80.

It avoids the major overhead and ongoing maintenance complexity of getting a public/real SSL certificate issued, emplaced, and dealing with ongoing renewals, etc.

The main purpose of the tutorial is not to to access the OPN UI, for which your method makes perfect sense, but instead to reverse proxy services that are hosted internally in a LAN.

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on February 22, 2024, 10:44:42 am

Quote from: cookiemonster on February 22, 2024, 10:38:54 am

Quote from: johnmcallister on February 22, 2024, 01:35:06 am

Just chiming in here --

Thanks very much doing all the work on this How-To, OP, and for keeping it updated, etc.
I successfully implemented it in my modest OPNsense instances/networks, before realizing that for small networks where there may never be more than perhaps 1 to 3 people logging in to a given OPNsense instance, in fact it's far more secure to simply shut off all HTTP listening on external network ports, and instead use SSH tunnels / port redirects, i.e. ssh -L 9450:localhost:80 my.opnsense.host to connect directly to the opnSense instance and access the webgui that way.  Then it doesn't matter at all whether HTTPS is active as the entire connection takes place inside the highly-secured SSH network connection.  With SSL tunnels there is no need for a webgui process to be listening anywhere except localhost:80.

It avoids the major overhead and ongoing maintenance complexity of getting a public/real SSL certificate issued, emplaced, and dealing with ongoing renewals, etc.

The main purpose of the tutorial is not to to access the OPN UI, for which your method makes perfect sense, but instead to reverse proxy services that are hosted internally in a LAN.

Exactly.
To add to this, exposing any administrative web interfaces using a reverse proxy is the worst thing you could possibly do.
Only expose those things using either your SSH tunnel method or VPN tunnels.

The whole point of a reverse proxy is to make web services available using a single port, a nice URL and a valid certificate.

Quote

It avoids the major overhead and ongoing maintenance complexity of getting a public/real SSL certificate issued, emplaced, and dealing with ongoing renewals, etc.

We are using ACME to avoid exactly all of that. It is a one time only set and forget process.

I think you didn't really understand the tutorial at all.
Also public certificates, as you can see in my tutorial, are so easy and completely free to obtain nowadays.

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: bunchofreeds on February 23, 2024, 12:36:59 am

Quote from: bunchofreeds on February 08, 2024, 09:23:41 pm

Hi all,

I'm trying to get my internally hosted services to report the originating client IP when going through a proxy chain starting with Cloudflare then to HAproxy. Currently HAproxy logs shows the local CloudFlare CDN address.

Cloudflare is setup to proxy and is Full (Strict) meaning I'm using the Cloudflare origin cert offloaded at HAproxy

I've found that cloudflare do collect the Client IP within cf-connecting-ip
https://developers.cloudflare.com/support/troubleshooting/restoring-visitor-ips/restoring-original-visitor-ips/

And I have found this post that helps someone with pfSense to do what I want
https://forum.netgate.com/topic/176777/haproxy-cloudflare-restoring-original-ip/3

What I'm not sure about is how (if possible) to get HAproxy to reference the cloudflare IP address list to know what sessions to insert the cf-connecting-ip into x-forwarded-for
Ideally this is in the form of some alias or map that dynamically checks https://www.cloudflare.com/ips-v4

Thanks for any help with this, also it's not urgent at all and just for my home setup and for fun really.

Found some answers
https://github.com/home-assistant/core/issues/40421#issuecomment-1667019787

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: stefan21 on February 23, 2024, 08:17:20 am

Quote from: stefan21 on February 22, 2024, 12:37:49 am

OPNsense is up-to-date -->

OPNsense 24.1.2_1-amd64
FreeBSD 13.2-RELEASE-p10
OpenSSL 3.0.13

Firefox: 121.0 (64-bit), archlinux (doesn't matter, latest update on windows brings up the same issue)

DEFAULT in FF: security.ssl.enable_ocsp_stapling = true

--> leads to no access on any pages with certs following the tutorial. At least if pages are secured to local access only. I assume, same error for public access.

Changing the default in FF to false gives access back.

Made a few changes:

Did a roll back to

OPNsense 23.7.12_5-amd64
FreeBSD 13.2-RELEASE-p7
OpenSSL 1.1.1w

For all certs I changed to key lenght 2048. OCSP must staple "off". Renewed all certs and deployed to the servers. Now on all clients FF and TB are working again with no errors.

Adding local and public subdomain rules in public services HTTPS_frontend are not working as expected in my setup. Adding in option pass through

acl lan_vpn src 192.168.x.x/24
acl lan_vpn src 10.0.x.x/24
http-request deny if ! lan_vpn

is working. Access from public is not possible - as intended. Maybe it helps someone.

@thehellsite: thx very much for this great tutorial.

regards,
stefan

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: ckocank on February 25, 2024, 09:45:24 am

Edit: After a restart, every thing works again.
My HAproxy stopped working after the update and I cannot make it work again.
I tried all the latest fixes in the comment and updated part 5 and part 8.
This is my config

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.ocsp-update.mindelay 300
    tune.ssl.ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr libc,last

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening on 0000:443 0000:80)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend ssl_backend

    # logging options

# Frontend: 1_HTTP_frontend (listen on 10.1.2.3:80)
frontend 1_HTTP_frontend
    bind 10.1.2.3:80 name 10.1.2.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: no_ssl_condition
    acl acl_642ff4b1bd6b30.27652312 ssl_fc

    # ACTION: http_to_https_rule
    http-request redirect scheme https code 301 if !acl_642ff4b1bd6b30.27652312

# Frontend: 1_HTTPS_frontend (listen on 10.1.2.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 10.1.2.3:443 name 10.1.2.3:443 accept-proxy ssl curves secp384r1 strict-sni  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/642ffac3a289a1.74357812.certlist
    mode http
    option http-keep-alive
    option forwardfor

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/642ff59e3f1923.99537840.txt)]

# Backend (DISABLED): acme_challenge_backend (Added by ACME Client plugin)


# Backend: ssl_backend (ssl virtual ip backend)
backend ssl_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server ssl_server 10.1.2.3 send-proxy-v2 check-send-proxy

# Backend: test_backend (test backend pool)
backend test_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server test_server 10.1.1.100:81



# statistics are DISABLED

frontend prometheus_exporter
   bind *:8404
   mode http
   http-request use-service prometheus-exporter if { path /metrics }

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: JohnnyBeGood on February 26, 2024, 12:18:34 am

TheHellSite has been incredibly helpful with both HAProxy and setting up a Postfix SMTP relay for me.
My scenario was a little bit complicated, and although I was able to follow up to a certain point, the complexity of my setup required me to reach out for assistance. Not only was the assistance top-notch, but the pricing was also quite reasonable. I highly recommend reaching out if you need any similar services. Thanks again for the great support!

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Squiggley on February 28, 2024, 07:48:32 pm

Hi @HellSite

I have followed your tutorial I believe exactly, and I have almost all of it working.
For all of my services I have real servers and matching backends.
The local and public map files are working great.
It redirects http to https just fine.
I get an A+ SSL rating

All of that works great (would have taken me forever without your tutorial)

However I do not seem to be able to access any services on my LAN from a VLAN via the fqdn.

I am more than happy to work things out myself I am really just looking to try and understand how it works a bit better and perhaps get a pointer to which part is not functioning properly.

Can you tell me if these assumptions are correct.
My laptop is on vlan B and all my services are on vlan A.

1) If I can access the services via the IP address successfully then my firewall rules are allowing the correct traffic through.
2) If I do an nslookup or ping from my laptop and I get the opnsense ip address on vlan A then unbound is working correctly as traffic should be sent to haproxy for it to work out where its really going.

So it seems to me that my issue must be haproxy is not giving the correct address back when I enter the fqdn on the laptop

I have intentionally not put my configs in here as after reading all the pages in this thread I see the enormous amount of work you put into this and I would like to understand/work things out myself.

Thanks

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: Squiggley on March 01, 2024, 05:30:13 am

Disregard, I don't know why but its all working fine now.

Thanks again for the tutorial

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: opojomo on March 21, 2024, 02:30:12 pm

Problem solved.

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: WickedBone on March 29, 2024, 04:36:36 pm

Hello,
  First of all i want to thank everyone in this community for the very helpful comments and descriptions and TheHellSite for the beautiful  tutorial and all the updates to it.

   That being said I'm here because I'm out of ideas and my head is about to explode. So I've followed the tutorial almost 2 years ago now and manged to get everything to work perfectly for my  environment. But 2 days ago i decided to update opnsense from 23.7.something to the newest version 24.4.4 like I've done many times in those past 2 years whenever I remembered to do it. But out of the not so many servers I have behind the haproxy, one didn't work properly and gave me the famous error 503 Service Unavailable No server is available to handle this request. For context this is a very very old server which, due to reasons, works on debian  6 and cannot be upgraded, with self signed openssl cert for its apache website which makes it inaccessible unless you downgrade your browser tls to 1 min. That's why I put this abomination behind my haproxy so the site can be reached more easily. Without any changes (at least I'm not aware of any) to the configuration it just stopped working after the update.

    I guess my question is: did the update change anything to how opnsense is connecting to real servers with ssl enabled (because the connection works if its disabled, but then the site configuration becomes a problem) for old sites and servers and in general where do i look to fix the issue (apache on the server, opnsense, haproxy)?  :'(

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: oLeDfrEeZe on April 01, 2024, 03:01:37 pm

Hello,

first - i will thank you so much for the tutorial!

Now my question:
I followed the instructions up to point 5 and configured everything exactly as described.
But now on Part 6: "If you try to access your URL 'your_service.your_subdomain.dedyn.io' from a device on your internal network, it should fail." but It doesn't fail... it works  ???

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on April 16, 2024, 09:25:20 am

I have a question about HAproxy SSL performance with large downloads:

Using a NAT port forward to an internal HTTPS nginx server, I get full wire speed i.e. (45 MByte/s) from the outside, but using HAproxy following this tutorial, I am limited to download speeds of ~4-5 MByte/s.

I have tried to use HTTP instead of HTTPs for the backend server connection to no avail.

I suspected that the SSL termination would lead to an interrupt of the streaming flow in that each packet must be acknowledged over the whole chain before the next one can get requested, so I have somewhat optimized my throughput by a factor of 2 via this tuning:

Code: [Select]

    tune.h2.max-frame-size 4048576
    tune.pipesize 4048576
    tune.h2.initial-window-size       1048576
    tune.h2.be.initial-window-size    1048576
    tune.h2.fe.initial-window-size    1048576

But it seems that the HTTPS termination within HAproxy itself is the culprit. I found some other source (https://forum.netgate.com/topic/152492/haproxy-slow-on-wan-jagged-throughput) that suggest there might be a problem with only some ISPs showing that behaviour when SSL traffic is terminated by HAproxy.

Does everybody else see this or may my ISP really be the culprit?

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: oLeDfrEeZe on April 21, 2024, 10:11:09 pm

Yes, I can confirm this, I also have performance issues with HAProxy and Plex. When I use NAT port forward then I get full speed, but when I use HAProxy then my transfer rate is also limited to only ~1Mb/s

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 24, 2024, 10:14:51 pm

Quote from: meyergru on April 16, 2024, 09:25:20 am

I have a question about HAproxy SSL performance with large downloads:

Using a NAT port forward to an internal HTTPS nginx server, I get full wire speed i.e. (45 MByte/s) from the outside, but using HAproxy following this tutorial, I am limited to download speeds of ~4-5 MByte/s.

I have tried to use HTTP instead of HTTPs for the backend server connection to no avail.

I suspected that the SSL termination would lead to an interrupt of the streaming flow in that each packet must be acknowledged over the whole chain before the next one can get requested, so I have somewhat optimized my throughput by a factor of 2 via this tuning:

Code: [Select]

    tune.h2.max-frame-size 4048576
    tune.pipesize 4048576
    tune.h2.initial-window-size       1048576
    tune.h2.be.initial-window-size    1048576
    tune.h2.fe.initial-window-size    1048576

But it seems that the HTTPS termination within HAproxy itself is the culprit. I found some other source (https://forum.netgate.com/topic/152492/haproxy-slow-on-wan-jagged-throughput) that suggest there might be a problem with only some ISPs showing that behaviour when SSL traffic is terminated by HAproxy.

Does everybody else see this or may my ISP really be the culprit?

No issues on my side here.

VDSL 100/40 MBit/s on HAProxy side.
VDSL 250/40 MBit/s on Client side.

Client side downloads files from HAProxy side using File Browser with max speed of 40 MBit/s.
This is using a direct connection between two parties and no peering service (f.e. Cloudflare) involved.

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: oLeDfrEeZe on April 24, 2024, 10:30:25 pm

That's strange... Do you have OPNsense version 24.1.6 installed?

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: TheHellSite on April 24, 2024, 10:59:45 pm

Quote from: oLeDfrEeZe on April 24, 2024, 10:30:25 pm

That's strange... Do you have OPNsense version 24.1.6 installed?

Yes.

Title: Re: Tutorial 2024/02: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating
Post by: meyergru on April 25, 2024, 10:11:04 am

I also use direct peering, no tunnels or anything. FWIW, the issue does not happen when I access the port from inside, i.e. without any delays. So it is not a general bottleneck. I see it happen only when I access the service from outside, with different counterparts.

I also tried a packet trace, and my TCP window size gets reduced to < 400 bytes when the problem occurs. From what I have read about HAproxy, it controls TCP directly in order to stream the data between frontend and backend, be able to fail-over (hence "HA") and things like that. Over the years, there have been discussions on subtle problems like this over and over. I get the impression that HAproxy is a very complex piece of software.

I have tried many parameters to no avail, but it seems fruitless considering that there are so many tuning parameters for both frontend and backend (i.e. buffers sizes and such). My connections seem to use HTTP/1.1, so it seems it is not a HTTP/2 or QUIC issue.

Other mentions of the problem also suggested that this is not a FreeBSD or OpnSense problem, since it occurs on Linux as well.