Messages

That's great news! I've now solved it with HAproxy, I then had the challenge that only the IP adres of the proxy was visible to the hosting panel. Logical behaviour but I wasn't aware that customers were using fail2ban-like functionality on their websites. Luckily I've been able to activate the PROXY protocol on the hosting services to solve that challenge.

I really like Caddy and the ease to use it, but given the IP address challenge I would still not be able to switch I'm afraid. Anyway my problem is solved for now. If someone needs help with a similar use-case feel free to dm me  :)

I've came to this setup:

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.ocsp-update.mindelay 300
    tune.ssl.ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): SNI-listener (Public service)

# Frontend (DISABLED): HA-Listener (public)

# Frontend: Public-service-sni-listener ()
frontend Public-service-sni-listener
    bind [::]:443 name [::]:443
    bind [::]:80 name [::]:80
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:8123 name 0.0.0.0:8123
    bind [::]:8123 name [::]:8123
    mode tcp
    default_backend pool-all

    # logging options
    # WARNING: pass through options below this line
      tcp-request inspect-delay 5s
      tcp-request content accept if { req_ssl_hello_type 1 }

# Backend (DISABLED): homeassistant-pool ()

# Backend (DISABLED): nextcloudpool ()

# Backend (DISABLED): directadminpool ()

# Backend: pool-all ()
backend pool-all
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # ACL: homeassistant_sni
    acl acl_668517d7e34a26.66992240 req.ssl_sni -i app1.example1.org
    # ACL: nextcloud_sni
    acl acl_668517cca10095.43472848 req.ssl_sni -i app2.example2.org

    # ACTION: ha_sni_rule
    use-server homeassistant if acl_668517d7e34a26.66992240
    # ACTION: nextcloud_sni_rule
    use-server office if acl_668517cca10095.43472848
    # ACTION: other_sni_rule
    use-server directadmin unless acl_668517d7e34a26.66992240 acl_668517cca10095.43472848
    server directadmin 192.168.10.102:443
    server homeassistant 192.168.1.88:8123
    server office 192.168.1.35:443



# statistics are DISABLED

Unfortunately it doesn't work, anyone a suggestion?

[Update 4/7:]

First of all great tutorial and topic, it really helped me to understand how HAproxy works. That said I do need a bit of a differend setup since I don't want HAproxy to manage any of the ssl stuff.

I have three services I want to route based on SNI using HAproxy.

- Two domains/services are static (homeassistant: app1.example1.org & Nextcloud: app2.example2.org)

The third service is directadmin server with remote users, those domains are dynamic meaning that the users using the server are adding, editing and removing domains all the time.

setup should be,
IF SNI1 -> server X
IF SNI2 -> server Y
IF ANY-OTHER-SNI -> server Z

This is my current config, the two static services (homeassistant and Nextloud) are working flawless. Unfotunately the directadmin domains are not working :(

Code Select Expand

removed old config to cleanup

I've read every post and almost every related other topic but unfortunately I can't get it to work, pulling my hairs for days now. Happy to pay someone to help me out if needed.

update: to add, i've basically recreated the second example on this page https://www.haproxy.com/blog/enhanced-ssl-load-balancing-with-server-name-indication-sni-tls-extension#choose-a-server-using-sni-aka-ssl-routing So I believe I'm close, yet not close enough.

Update 4/7:

Got it working!

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.ocsp-update.mindelay 300
    tune.ssl.ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: Public-service-sni-listener ()
frontend Public-service-sni-listener
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind 0.0.0.0:8123 name 0.0.0.0:8123
    mode tcp

    # logging options
    # ACL: homeassistant_sni
    acl acl_668517d7e34a26.66992240 req.ssl_sni -m sub -i app1.example1.org
    # ACL: nextcloud_sni
    acl acl_668517cca10095.43472848 req.ssl_sni -m sub -i app2.example2.org

    # ACTION: other_sni_rule
    use_backend directadminpool unless acl_668517d7e34a26.66992240 || acl_668517cca10095.43472848
    # ACTION: ha_sni_rule
    use_backend homeassistant-pool if acl_668517d7e34a26.66992240
    # ACTION: nextcloud_sni_rule
    use_backend nextcloudpool if acl_668517cca10095.43472848
    # ACTION: PUBLIC_DOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/667995c7e25e94.80171493.txt,directadminpool)]
    # WARNING: pass through options below this line
      tcp-request inspect-delay 5s
      tcp-request content accept if { req_ssl_hello_type 1 }

# Backend: homeassistant-pool ()
backend homeassistant-pool
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server homeassistant 192.168.1.88:8123

# Backend: nextcloudpool ()
backend nextcloudpool
    # health check: Nextcloud-Healthcheck
    mode tcp
    balance roundrobin
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server office 192.168.1.35:443 check inter 5s port 443

# Backend: directadminpool ()
backend directadminpool
    # health checking is DISABLED
    mode tcp
    balance roundrobin
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server directadmin 192.168.10.102:443

Now the only thing I would like to achieve is that HAproxy forwards the correct client ip to the real servers due to some security solutions running on their. How can I achieve that? Anyone a clue?

Aah that's what I thought, thank you for confirming.
Now the question is, how to configure HAproxy  ;D

I have it working for HA in HAproxy. but unfortunately the nextcloud config keeps forwarding to the DA host. I've tried to setup just one listener as Patrick suggested but then I'm not able to define a default pool which breaks everything:

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    tune.ssl.ocsp-update.mindelay 300
    tune.ssl.ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: HA-listener (Public service)
frontend HA-listener
    bind ha.example.org:8123 name ha.example.org:8123
    bind nextcloud.example.com:443 name nextcloud.example.com:443
    mode tcp

    # logging options
    option tcplog
    # WARNING: pass through options below this line
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }

# Frontend (DISABLED): Nextcloud (Nextcloud)

# Backend: homeassistant-pool ()
backend homeassistant-pool
    # health check: HA-Healthcheck
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server homeassistant 192.168.1.88:8123 check inter 2s port 8123

# Backend: nextcloudpool ()
backend nextcloudpool
    # health check: Nextcloud-Healthcheck
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server nextcloud 192.168.1.35:443 check inter 5s port 443



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

listen  remote_statistics
    bind            192.168.1.1:8999
    mode            http
    stats uri       /haproxy?stats
    stats hide-version


With multiple listeners configured only the HA implementation works.

That could be the solution. But can I just use "*" as a domain? Or how should the config look like?

That's not really a desired situation. The directadmin server should handle the certificates etc. Isn't there a type of proxy which simply proxies

1.domain.org -> server 1
2.domain.org -> server 2
if none of the above -> server 3 (transparant)

When I disable the NAT rule to the directadmin server caddy works flawless. The question is, how can I make sure that it first runs over caddy and if no domain in the caddy config was found it needs to be forwarded to the DA server.

Just installed caddy and created a domain and handler as described in the documentation. Obviously disabled HAproxy. But when I try to connect to the configured domain it's trying to connect to the directadmin server again... :(

Also a lot of error regarding obtaining a ssl certificate in the log.

Code Select Expand

error","ts":"2024-06-22T12:25:59Z","logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"blabla.example.org","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/"}

Thank you for your quick reply! Aah that might be the issue. Does caddy have the same functionality? And could I achieve my goal with Caddy?

Seems easier to maintain so might consider a shift to caddy.

Hi all!

For several days I'm pulling my hairs to solve the following. I have three servers: Directadmin, Home Assistant and Nextcloud. The direct admin server is used by external users and has variable domains which those users add and remove all the time. The HA and Nextcloud servers are more static.

As for now I've configured the directadmin server with plain and simple port forwarding. I want to achieve that if certain domains are used (ha.example.org or nxt.example.org) the firewalls forwards the request to the right internal server, but for all the other requests it should go to directadmin. I've tried configuring it using HAproxy and got it working for Home Assistant since that's running on a different port. But nextcloud is not working (I believe because of overlapping ports 80 & 443 with directadmin) When I do a CURL command to nxt.example.org is shows a certificate from Directadmin.

Does anyone have the golden idea on how to achieve this? And/or what do you need from me (like configs to check what's wrong)

this is my haproxy config:

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    tune.ssl.ocsp-update.mindelay 300
    tune.ssl.ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: HA (Home Assistant)
frontend HA
    bind ha.example.org:8123 name ha.example.org:8123
    mode tcp
    default_backend homeassistant-pool

    # logging options
    option tcplog
    # WARNING: pass through options below this line
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }

# Frontend: Nextcloud (Nextcloud)
frontend Nextcloud
    bind nxt.example.org:443 name nxt.example.org:443
    bind nxt.example.org:80 name nxt.example.org:80
    mode tcp
    default_backend nextcloudpool

    # logging options
    option tcplog
    # WARNING: pass through options below this line
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }

# Backend: homeassistant-pool ()
backend homeassistant-pool
    # health check: HA-Healthcheck
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server homeassistant 192.168.1.88:8123 check inter 2s port 8123

# Backend: nextcloudpool ()
backend nextcloudpool
    # health check: Nextcloud-Healthcheck
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server qfeeds-office 192.168.1.35:443 check inter 5s port 443



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

listen  remote_statistics
    bind            192.168.1.1:8999
    mode            http
    stats uri       /haproxy?stats
    stats hide-version

HI all,

I'm trying to setup HAproxy but in OPNsense but I can't figure out a way to setup a default backend.
F.E. I have 3 servers running. server 1 & 2 have specific domains but server 3 is a shared hosting platform with hundreds of domains which change on daily basis.

I want HAproxy to serve server 1 or 2, but if none of these conditions are met it has to redirect to server 3. How can I set this up?

Kind regards,

Robert

Oh stupid me :-) It works now! Thank you very much!

When I changed the firewall rule it worked for a second. But that could have been a false positive.
This is how the rule is setup now.

Thanks for your quick reply. Unfortunately it doesn't work though. That's exactly what I've setup on the WAN interface.

default gateway on the server seems to work as well:

Code Select Expand

ip -6 route show
::1 dev lo proto kernel metric 256 pref medium
2a02:a45a:9668:1::c3c3 dev ens18 proto kernel metric 100 pref medium
2a02:a45a:9668:1::/64 dev ens18 proto ra metric 100 pref medium
fe80::/64 dev ens18 proto kernel metric 1024 pref medium
default via fe80::21a:8cff:fe6c:1a56 dev ens18 proto ra metric 100 pref medium

The "fe80::21a:8cff:fe6c:1a56" address also shows on the DMZ interface in the admin GUI.

I should not use the Link-local address to accept the traffic right? (I didn't but just double checking)

Hi All,

I'm running into an issue. (Quite new to IPV6)

I've enabled IPV6 successfully on both DMZ and LAN. IPV6 tests are all successful.
This is the situation:
WAN prefix: 2a02:a45a:9668::/48
LAN: 2a02:a45a:9668:0::
DMZ (webserver adress): 2a02:a45a:9668:1::c3c3

I can locally connect to the webserver IPV6 adres without any problems.

In my DMZ I'm trying to make my webserver publicly available: 2a02:a45a:9668:1::c3c3 . But that's where the challenge starts. I thought the only thing I need to do is to Pass 443 to the IPV6 address on the WAN port, but that doesn't cut it.

(Tested with http://ipv6.my-addr.com/ipv6-ready-ipv6-connectivity-ipv6-website-test.php.php?site=2a02%3Aa45a%3A9668%3A1%3A%3Ac3c3)

Any ideas on what I'm doing wrong?

Kind regards,

Robert