[SOLVED] HAproxy port 80+443 to multiple backends not working

[opnuser1]

Hello, I used a modified version of TheHellSite's guide to create a HAproxy setup where ports 80,443 are fowarded to multiple internal servers.  This was working last month, and not working any longer.  I am having a hard time finding where the issue is. 

The goal was:
Get 80+443 forwarded to multiple backends.  Before this, I used NAT port forward to send to just one server.  But now I need more.  The environment I am in has only a single public IP.

This worked initially, but not anymore.  Please see my haproxy export below.

Things I have tested:
If I disable haproxy, and enable the NAT port forward (individually per backend), it works instantly and flawlessly.
If I enable haproxy, external access stops working for any server.  Internally, the sites/servers are accessible since I use an Unbound override.

Even though haproxy is not working all the way, when I check the counters, there are bytes coming in and out.  However, there is something weird as it takes a long time for any stats/counters to appear.  It used to be faster.  Also I looked in the haproxy logs, and there are no logs since the beginning of May, and I don't understand that either.

If I test if ports 80 and 443 are open from external.  I think this is hitting haproxy properly, because I see a byte in the counters when I test the port.  In the opnsense documentation there is a warning about haproxy silent error if the ports are in use.  I don't think that is happening for me, because then haproxy would not receive any data, right?

I also have a WAN rule allowing the ports to the destinations.

Code: [Select]

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         0.0.0.0 local0
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend ()
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp

    # logging options
    option log-separate-errors
    option tcplog
    # ACL: TCP_SSL_condition
    acl acl_644c56b6785678.47181279 req.ssl_hello_type 1
    # ACL: TCP_server1_condition
    acl acl_644c5700ee7657.09485748 req.ssl_sni -m sub -i domain1.com
    # ACL: TCP_server2_condition
    acl acl_644c5719768e71.87060950 req.ssl_sni -m sub -i domain2.com

    # ACTION: TCP_RequestInspectDelay_rule
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 5s
    # ACTION: TCP_RequestContentAccept_rule
    tcp-request content accept if acl_644c56b6785678.47181279
    # ACTION: TCP_SERVER1_rule
    use_backend TCP_SERVER1_backend if acl_644c5700ee7657.09485748
    # ACTION: TCP_SERVER2_rule
    use_backend TCP_SERVER2_backend if acl_644c5719768e71.87060950

# Frontend: HTTP_frontend ()
frontend HTTP_frontend
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp

    # logging options
    option tcplog
    # ACL: http_server1_condition
    acl acl_6457247ca14984.71641345 hdr_sub(host) -i domain1.com
    # ACL: http_server2_condition
    acl acl_64572496aeac32.73416688 hdr_sub(host) -i domain2.com

    # ACTION: http_server1_rule
    use_backend TCP_SERVER1_backend if acl_6457247ca14984.71641345
    # ACTION: http_server2_rule
    use_backend TCP_SERVER2_backend if acl_64572496aeac32.73416688

# Backend: TCP_SERVER1_backend ()
backend TCP_SERVER1_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server server1_server 192.168.1.234

# Backend: TCP_SERVER2_backend ()
backend TCP_SERVER2_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server server2_server 192.168.1.217

# Backend (DISABLED): TCP_SERVER3_backend ()



# statistics are DISABLED




[Taomyn]

I'm no HAProxy expert but I feel your pain as I too have hit weird HAProxy issues over the years. I do recall some changes some time back where I needed to rethink my HAProxy settings and one of these is to not bind it to all firewall IPs (0.0.0.0) nor to port 443 (I don't use 80 for anything). For example, my main frontend binding looks like this:

Code: [Select]

frontend Web_Server_SSL
    http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    bind 192.168.1.1:5581 name 192.168.1.1:5581 ssl alpn h2 crt-list /tmp/haproxy/ssl/57cb08105a3629.39396088.certlist
    mode http
    option http-keep-alive
    default_backend Apache_Server_Maggie
    option forwardfor


I then set the NAT rule for external access to send 443 to 192.168.1.1:5581 on the firewall.

In your config if the OPNsense GUI is listening on port 443 you're also trying to do that, which can't be good.


Hope that helps.

[opnuser1]

Ah thanks for the response.  So you think it's the listen address as 0.0.0.0 ?  That would make sense as I am having a hard time finding any other reason.  Do you think if I use a Virtual IP interface and listen to that, would that work? 

[Taomyn]

Ah thanks for the response.  So you think it's the listen address as 0.0.0.0 ?  That would make sense as I am having a hard time finding any other reason.  Do you think if I use a Virtual IP interface and listen to that, would that work?

Yes as long as that IP is not being used by another service on the same IP and ports - this is why I have HAProxy listening on 192.168.1.1:5581 - just in case, and let the NAT sort that out. The Sockets tab in INTERFACES: DIAGNOSTICS: NETSTAT will help you see what's in use. In fact I actually have a second listen bind on a virtual IP and port 443 to allow internal devices to go through the proxy - I have some services that work better that way.

[opnuser1]

so you have it set up where anything coming in on 443 gets NAT to 5581, and you have haproxy listen to 5581 ?

but then the internal services are expecting 443....do you change it back to 443 somewhere?  I don't quite understand. 

What I am currently trying is a virtual ip in loopback mode.  I will use this ip for haproxy to listen to on 443 and 80.  I hope this solves my issue.  I don't even understand what the problem is really.  I checked netstat and the only binding there is haproxy for 443 and 80.

[opnuser1]

Whoa, i believe I found the culprit.  I had installed and enabled Upnp plugin.  I disabled it and now everything works.  I do not understand.