haproxy tcp mode passthrough to existing server not working

[Beleggrodion]

Hi,

I searched the forum and read all the threads (with the tutorials) that i found about haproxy configuration, tried different approaches but nothing worked as expected. ssl connection always fails (ex. firefox SSL_ERROR_RX_RECORD_TOO_LONG) or when i try it with openssl s_client to check the certificate it looks like more , no certficate is given or it runs in non-ssl mode or something.

Currently i had a running setup with classic nat directly to a windows exchange 2016 server. Currently the server do the complete let's encrypt stuff by himself and this shoul'd not change. The server is in productive mode and is used internal and externaly. (split dns).

Because their is only one external ip i need to share this ip on port 443, so as i read haproxy is a good solution to solve this. So a secondary server with web services shoul'd be installed now behind the opnsense too.

So for this solution the let's encrypt stuff shoul'd not be happening on the opnsense, the server themselfs shoul'd be maintain the renewal, and so on. So for this the "tcp (layer 4) modes are the way to work with, or are i wrong?

Because the complete stuff on 80 and 443 is productive, i use 9080 and 9443 for testing purposes as ports on the firewall.

Below is my config (some stuff is defined but not used like the resolver currently) . i tried it last evening around 4-5 hours without sucess, so perhaps i oversea something that someone other, which had a working setup, see directly and i don't see it like the trees in the forest. ;-)

Code Select Expand


#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    1
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats


# Resolver: dc-cust-01
resolvers 63d8cfcde3f718.78402437
    nameserver 192.168.99.11:53 192.168.99.11:53
    resolve_retries 3
    timeout resolve 1s
    timeout retry 1s



# Frontend: http ()
frontend http
    bind 192.168.99.1:9080 name 192.168.99.1:9080
    bind 1.x.x.x:9080 name 1.x.x.x:9080
    mode http
    option http-keep-alive
    default_backend ex-cust-01_http_backend
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: http_mail_customer_ch
    acl acl_63d834ead32456.82956086 hdr_sub(host) -i mail.customer.ch

    # ACTION: http_mail_customer_ch
    use_backend ex-cust-01_http_backend if acl_63d834ead32456.82956086

# Frontend: https_sni ()
frontend https_sni
    bind 192.168.99.1:9443 name 192.168.99.1:9443
    bind 1.x.x.x:9443 name 1.x.x.x:9443
    mode tcp
    default_backend ex-cust-01_https_backend
    # tuning options
    timeout client 30s

    # logging options
    option tcplog
    # ACL: ssl_hello
    acl acl_63d84150d59f08.49426761 req_ssl_hello_type 1
    # ACL: https_mail_customer_ch
    acl acl_63d83503a02259.11459738 req.ssl_sni -m sub -i mail.customer.ch

    # ACTION: tcp_request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: tcp_request_content_accept_ssl
    tcp-request content accept if acl_63d84150d59f08.49426761
    # ACTION: https_mail_customer_ch
    use_backend ex-cust-01_https_backend if acl_63d83503a02259.11459738
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Real-IP %[src]

# Backend: ex-cust-01_https_backend (Exchange 2016 - HTTPS)
backend ex-cust-01_https_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server ex-cust-01_https 192.168.99.11:443 ssl verify none send-proxy-v2 check-send-proxy send-proxy-v2 check-send-proxy

# Backend (DISABLED): matrix-cust-01_https_backend (Matrix Chat HTTPS)

# Backend: ex-cust-01_http_backend (Exchange 2016 - HTTP)
backend ex-cust-01_http_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    option forwardfor
    http-reuse safe
    server ex-cust-01_http 192.168.99.11:80

# Backend (DISABLED): matrix-cust-01_http_backend (Matrix Chat HTTP)


on the "ex-cust-01_https_backend" i also tried with and without ssl in the "server" line. the owa, and other services are only via ssl reachable on the server, port 80 always gets empty white pages expect the .well-known directory for let's encrypt.

[Fright]

Hi!
sorry, wrong topic.
But once it happened)
It is not entirely clear you want passthrough TLS or terminate?
In addition, I doubt that Ex2k16 supports the proxy protocol (""send-proxy-v2 check-send-proxy").
It's also interesting how it turned out to double "send-proxy-v2 check-send-proxy send-proxy-v2 check-send-proxy"?
And of course, it's always good to enable logging (Raise Log Level & Detailed Logging) on HAProxy frontend and take a look at the logs)

[Beleggrodion]

Oh, i don't see your answer yet. In the meantime (around one day before your post)  i found a solution which works currently perfect for my setup. (its not perfect people who want a clean setup but it works at the moment)

Code Select Expand


#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    1
    hard-stop-after             60s
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats


# Resolver: dc-company-01
resolvers 63d8cfcde3f718.78402437
    nameserver 172.xxx.xxx.yyy:53 172.xxx.xxx.yyy:53
    resolve_retries 3
    timeout resolve 1s
    timeout retry 1s



# Frontend: https_443_frontend (Access to HTTPS Services on Exchange and Matrix Server)
frontend https_443_frontend
    bind 4x.xxx.xxx.xxx:9443 name 4x.xxx.xxx.xxx:9443
    bind 172.xxx.xxx.xxx:9443 name 172.xxx.xxx.xxx:9443
    mode tcp
    default_backend ex-company-01_backend
    # tuning options
    timeout client 30s

    # logging options
    # ACL: sni_check_admin_matrix
    acl acl_63dbd637445d88.60329440 req.ssl_sni -i admin.matrix.customer-domain.net
    # ACL: sni_check_matrix
    acl acl_63dc2304400b73.51558638 req.ssl_sni -i matrix.customer-domain.net
    # ACL: sni_check_element
    acl acl_63dc2313a4d6a3.83535618 req.ssl_sni -i element.customer-domain.net

    # ACTION: check_sni_admin_matrix
    use_backend matrix-company-01_backend if acl_63dbd637445d88.60329440
    # ACTION: check_sni_matrix
    use_backend matrix-company-01_backend if acl_63dc2304400b73.51558638
    # ACTION: check_sni_element
    use_backend matrix-company-01_backend if acl_63dc2313a4d6a3.83535618
    # WARNING: pass through options below this line
    option tcplog
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Real-IP %[src]
   

# Frontend: https_8448_frontend (Matrix Federation Service)
frontend https_8448_frontend
    bind 4x.xxx.xxx.xxx:8448 name 4x.xxx.xxx.xxx:8448 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63dbc3b0d08131.92200544.certlist
    mode http
    option http-keep-alive
    default_backend matrix-company-01_federation_backend
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

# Backend: ex-company-01_backend (Microsoft Exchange 2016 Services)
backend ex-company-01_backend
    # health checking is DISABLED
    mode tcp
    balance roundrobin
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    option ssl-hello-chk
    server ex-company-01_https 172.xxx.xxx.yyy:443

# Backend: matrix-company-01_federation_backend (Matrix Federation Service)
backend matrix-company-01_federation_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server matrix-company-01_federation 172.xxx.xxx.zzz:8080

# Backend: matrix-company-01_backend (Matrix Synapse & Elements Services)
backend matrix-company-01_backend
    # health checking is DISABLED
    mode tcp
    balance roundrobin
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    # WARNING: pass through options below this line
    option ssl-hello-chk
    option forwardfor
    server matrix-company-01_https 172.xxx.xxx.zzz:443 send-proxy-v2


for the matrix server is setup now two ip's on one all "server" in nginx running with:

Code Select Expand

server {
    listen 172.xxx.xxx.zzz:443 ssl http2 proxy_protocol;
    listen 172.xxx.xxx.zzy:443 ssl http2;
    ...
}

one ip is used for external communication from haproxy (the proxy_protocol) and the other for internal communcation from the lan side. so split dns is used here.

Coul'd be that Ex2k16 has proxy support protocol but i think the used iis behind it not, because the former company, who managed this customer used a 2012r2 server for the exchange server (so an other construction side comes up in the next months). I also tried on one apreach to use the iis of the exchange as reverse proxy to the matrix server, which worked expect that some url's which are generated from the matrix server when federation are used, not realy rfc conform are and so wrongly interpreted from iis side and then not correct sended to the matrix server.