[SOLVED] Problems with HAProxy plugin

[alh]

I run the HAProxy plugin to do SSL termination for a Bitwarden_rs container and SSL passthrough for a MailStore server. So far the experience has been terrible. The first connection nearly ALWAYS fails with the following entries in the log:

haproxy[27090]: x.x.x.x:50621 [11/Aug/2020:10:12:05.146] https_tcp https_tcp/<NOSRV> -1/-1/0 0 SC 1/1/0/0/0 0/0

Firefox doesn't work at all and other Browsers need a lot of reloads to start working. Contacting these servers internally, bypassing the proxy, works flawlessly.

I wonder if someone has had a similar experience or even a fix. Config is pretty basic, at least for MailStore, just a TCP frontend/backend that checks SNI and forwards accordingly.

[alh]

Looks like adding the following two lines/options to frontend config solves the issue (advanced mode):

Code: [Select]

tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }

Of course you better create conditions/rules for this since it avoids an warning during config check and integrates more nicely with the GUI of the plugin

[huuich]

Could you share step by step how do you configure SSL passthrough? Thanks!

[alh]

This should work for any TCP-based SSL/TLS encrypted service in passthrough (HAProxy: TCP) mode... It does NOT work for STARTTLS!

In this example I use TCP port 443.

• HAProxy plugin: Create "Real Server" (enter name, IP/FQDN and port number if different from 443, the rest can be left at default)
• HAProxy plugin: Create "Backend Pool" (enter name, set mode to TCP and select the real server from step 1)
• HAProxy plugin: Create "Condition" (enter name ["traffic_ssl"], condition type is "custom condition (option pass-through)" with value "req_ssl_hello_type 1")
• HAProxy plugin: Create "condition" (enter name ["myservice_sni"], condition type is "SNI TLS extension matches (TCP request content inspection)" with value "myservice.example.com" or whatever your FQDN is)
• HAProxy plugin: Create "Rule" (enter name ["request_inspect_delay"], select no condition, function is "tcp-request inspect delay" with value "5s" or whatever suits you)
• HAProxy plugin: Create "Rule" (enter name ["request_content_accept_ssl"], select condition of 3 ["traffic_ssl"], function is "tcp-request content accept")
• HAProxy plugin: Create "Rule" (enter name ["myservice_sni"], select condition of 4 ["myservice_sni"], function is "Use specific backend pool" with your pool from 2)
• HAProxy plugin: Create "Public service" (enter name ["https_passthrough"], choose a listen address [":443" for all], type is "TCP" and select the 3 rules created earlier)
• HAProxy plugin: Enable plugin or test/apply
• Firewall: allow incoming traffic to WAN (address) or whatever for TCP port 443.

That works at least for me. If you have double NAT you would need to disable port randomization for the proxied port...

Does that help you?

[huuich]

This should work for any TCP-based SSL/TLS encrypted service in passthrough (HAProxy: TCP) mode... It does NOT work for STARTTLS!

In this example I use TCP port 443.

• HAProxy plugin: Create "Real Server" (enter name, IP/FQDN and port number if different from 443, the rest can be left at default)
• HAProxy plugin: Create "Backend Pool" (enter name, set mode to TCP and select the real server from step 1)
• HAProxy plugin: Create "Condition" (enter name ["traffic_ssl"], condition type is "custom condition (option pass-through)" with value "req_ssl_hello_type 1")
• HAProxy plugin: Create "condition" (enter name ["myservice_sni"], condition type is "SNI TLS extension matches (TCP request content inspection)" with value "myservice.example.com" or whatever your FQDN is)
• HAProxy plugin: Create "Rule" (enter name ["request_inspect_delay"], select no condition, function is "tcp-request inspect delay" with value "5s" or whatever suits you)
• HAProxy plugin: Create "Rule" (enter name ["request_content_accept_ssl"], select condition of 3 ["traffic_ssl"], function is "tcp-request content accept")
• HAProxy plugin: Create "Rule" (enter name ["myservice_sni"], select condition of 4 ["myservice_sni"], function is "Use specific backend pool" with your pool from 2)
• HAProxy plugin: Create "Public service" (enter name ["https_passthrough"], choose a listen address [":443" for all], type is "TCP" and select the 3 rules created earlier)
• HAProxy plugin: Enable plugin or test/apply
• Firewall: allow incoming traffic to WAN (address) or whatever for TCP port 443.

That works at least for me. If you have double NAT you would need to disable port randomization for the proxied port...

Does that help you?

Thanks for quick reply and detail steps, I've follow your steps and make first server runs with ssl passthrough ok. Because I have more than 1 server, I've added server 2 by steps:

• HAProxy plugin: Create "Real Server 2" (enter name: real_server_2, still use port 443, the rest can be left at default)
• HAProxy plugin: Edit "Backend Pool" (enter name, set mode to TCP and select the real server from step 1)
• HAProxy plugin: Create "condition" (enter name ["myservice_sni_server 2"], condition type is "SNI TLS extension matches (TCP request content inspection)" with value "server2.example.com")
• HAProxy plugin: Edit "Rule" name ["myservice_sni"], select 2 condition ["myservice_sni_server_1"] and ["myservice_sni_server_2"], function is "Use specific backend pool" with your pool from 2
• HAProxy plugin: Edit "Public service" (enter name ["https_passthrough"], choose a listen address ["0.0.0.0:443" for all], type is "TCP" and select the 3 rules created earlier)
• HAProxy plugin: Enable plugin or test/apply
• Firewall: allow incoming traffic to WAN (address) or whatever for TCP port 443.

But both of my website on 2 servers can not run? How can I make 2 server working with ssl passthrough. Thanks!

[alh]

I understand that both servers run both website, correct?

Did you make sure that in rule ["myservice_sni"] you changed the "Logical operator for conditions" to "OR"? It is obviously not possible to match both conditions at the same time...

[huuich]

I understand that both servers run both website, correct?

Did you make sure that in rule ["myservice_sni"] you changed the "Logical operator for conditions" to "OR"? It is obviously not possible to match both conditions at the same time...

I've run 2 separate website on 2 servers. I've found a way to run by add 2 separate conditions and rule for 2 website. Thanks very much!

[huuich]

I have another problem when create a new site with ssl. I've used webinoly (https://webinoly.com/en/) in a server and it has Free SSL certificates with Let’s Encrypt integrated. When I've tried to create ssl for a site on this server it shows error below, I still don't know how to fix. When search on google, a topic said that "A working port 80 is required." (https://community.letsencrypt.org/t/timeout-during-connect-likely-firewall-problem/99719). Now I only apply your steps above, not know how to allow port 80 with 443 with your steps. Please help me to fix. Thanks!

Code: [Select]

Please, be sure that the mydomain.com subdomain is currently pointing (DNS) to this server.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for mydomain.com
Using the webroot path /var/www/mydomain.com/htdocs for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. mydomain.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://mydomain.com/.well-known/acme-challenge/eI3IjJ4OmYJ2RVQkDwm3QsBmp5Jl0QnlbC9RuEYkvtY: Timeout during connect (likely firewall problem)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mydomain.com
   Type:   connection
   Detail: Fetching
   http://mydomain.com/.well-known/acme-challenge/eI3IjJ4OmYJ2RVQkDwm3QsBmp5Jl0QnlbC9RuEYkvtY:
   Timeout during connect (likely firewall problem)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

[ERROR] Unable to create the new certificate!

[engelant]

@alh I would like to express my gratitude for your detailed list of settings, as this solved my issue after hours of trying to get this to work.

[huuich]

I've install HAProxy using SSL passthrough follow your guide and every works ok except my Nginx log can not receive real ip from client, it's only show ip  local of my OPNSense server in file /var/log/nginx/access.log

I follow this guide from BeNe but nothing change, Do you have any solution in this? Thanks!

Code: [Select]

Services -> HAProxy -> Virtual Services -> Backend Pool -> Edit Server
Enable "advanced mode"

Field:
Option pass-through

Code: [Select]
# add X-FORWARDED-FOR
option forwardfor

# add X-CLIENT-IP
# http-request add-header X-CLIENT-IP %[src]

This should work for any TCP-based SSL/TLS encrypted service in passthrough (HAProxy: TCP) mode... It does NOT work for STARTTLS!

In this example I use TCP port 443.

• HAProxy plugin: Create "Real Server" (enter name, IP/FQDN and port number if different from 443, the rest can be left at default)
• HAProxy plugin: Create "Backend Pool" (enter name, set mode to TCP and select the real server from step 1)
• HAProxy plugin: Create "Condition" (enter name ["traffic_ssl"], condition type is "custom condition (option pass-through)" with value "req_ssl_hello_type 1")
• HAProxy plugin: Create "condition" (enter name ["myservice_sni"], condition type is "SNI TLS extension matches (TCP request content inspection)" with value "myservice.example.com" or whatever your FQDN is)
• HAProxy plugin: Create "Rule" (enter name ["request_inspect_delay"], select no condition, function is "tcp-request inspect delay" with value "5s" or whatever suits you)
• HAProxy plugin: Create "Rule" (enter name ["request_content_accept_ssl"], select condition of 3 ["traffic_ssl"], function is "tcp-request content accept")
• HAProxy plugin: Create "Rule" (enter name ["myservice_sni"], select condition of 4 ["myservice_sni"], function is "Use specific backend pool" with your pool from 2)
• HAProxy plugin: Create "Public service" (enter name ["https_passthrough"], choose a listen address [":443" for all], type is "TCP" and select the 3 rules created earlier)
• HAProxy plugin: Enable plugin or test/apply
• Firewall: allow incoming traffic to WAN (address) or whatever for TCP port 443.

That works at least for me. If you have double NAT you would need to disable port randomization for the proxied port...

Does that help you?