Messages

Nextcloud was the only service i was making publicly available. I moved one of my internal services out from local subdomain to public and no i cannot reach it externally. app files:

Code Select Expand

# local access only subdomains
iprox iprox_backend
autgtp autgtp_backend
portainer portainer_backend

#public access subdomains
nextcloud nextcloud_backend
truenas truenas_backend

Code Select Expand

#public access subdomains
nextcloud nextcloud_backend
truenas truenas_backend

logs from HAproxy when trying an external connection for nextcloud:

Code Select Expand

2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.718] 0_SNI_frontend SSL_backend/SSL_server 1/0/254 5920 -- 3/2/1/1/0 0/0
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.740] 1_HTTPS_frontend~ nextcloud_backend/nextcloud 0/0/1/229/230 304 1194 - - ---- 4/2/0/0/0 0/0 "GET /apps/richdocuments/settings/fonts.json HTTP/1.1"2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 0_SNI_frontend SSL_backend/SSL_server 1/0/0 0 -- 3/2/1/1/0 0/0
2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake

logs from HAproxy when trying an external connection for truenas:

Code Select Expand

2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 0_SNI_frontend SSL_backend/SSL_server 1/0/0 0 -- 3/2/1/1/0 0/0
2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.718] 0_SNI_frontend SSL_backend/SSL_server 1/0/254 5920 -- 3/2/1/1/0 0/0
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.740] 1_HTTPS_frontend~ nextcloud_backend/nextcloud 0/0/1/229/230 304 1194 - - ---- 4/2/0/0/0 0/0 "GET /apps/richdocuments/settings/fonts.json HTTP/1.1"


Interesting that the Truenas call seems to be going to nextcloud?

Unbound DNS: Overides

   truenas.mydomain     dedyn.io   A (IPv4 address)   10.1.1.1            
   nextcloud.mydomain   dedyn.io   A (IPv4 address)   10.1.1.1       

Firewall rules:

HAProxy_ports (80, 443) allowed to WAN address

I don't see any deny results in firewall log. Let me know what other logs or config would be useful. Thank you

Please also provide the complete current haproxy config.

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs

# autogenerated entries for config in backends/frontends

# autogenerated entries for stats

# Frontend: 0_SNI_frontend (listening on 80 and 443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend testssl

    # logging options

# Frontend: 1_HTTP_frontend (listening on localhost:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_6451d6d41f14e3.72189927 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6451d6d41f14e3.72189927

# Frontend: 1_HTTPS_frontend (listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6451dd0fe21db7.63563341.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option httplog
    # ACL: cardav-endpoint
    acl acl_6461de0380c7b3.75062629 path /.well-known/carddav
    # ACL: caldav-endpoint
    acl acl_6461dde5d15634.54704624 path /.well-known/caldav
    # ACL: nc_nodeinfo
    acl acl_6466fe1b2e8ff2.47035478 path /.well-known/nodeinfo
    # ACL: nc_webfinger
    acl acl_6466fe303acb97.89104263 path /.well-known/webfinger
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6451d1c3d510f5.88841915 src 10.1.1.0/24 192.168.0.0/24

    # ACTION: cardav-endpoint
    http-request redirect code 301 location /remote.php/dav if acl_6461de0380c7b3.75062629
    # ACTION: caldav-endpoint
    http-request redirect code 301 location /remote.php/dav if acl_6461dde5d15634.54704624
    # ACTION: nc_nodeinfo
    http-request redirect code 301 location /index.php/%[capture.req.uri] if acl_6466fe1b2e8ff2.47035478
    # ACTION: nc_webfinger
    http-request redirect code 301 location /index.php/%[capture.req.uri] if acl_6466fe303acb97.89104263
    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6451d0414cb7f2.59080919.txt)] if acl_6451d1c3d510f5.88841915
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6451d7a6418a58.11785496.txt)]

# Backend: truenas_backend (NAS backend)
backend truenas_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server truenas 10.1.1.73 ssl verify none

# Backend: iprox_backend ()
backend iprox_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server iprox 10.1.1.101:8006 ssl verify none

# Backend: mbfirewall_backend ()
backend mbfirewall_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server mbfirewall 127.0.0.1:55443 ssl verify none

# Backend: autgtp_backend ()
backend autgtp_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server autgtp 10.1.1.113:7070

# Backend: portainer_backend (portainer backend)
backend portainer_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server portainer 10.1.1.59:9443 ssl alpn h2,http/1.1 verify none

# Backend: nextcloud_backend (nextcloud_backend)
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server nextcloud 10.1.1.59:11000

# Backend: idrac_backend ()
backend idrac_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server idrac 192.168.0.120 ssl verify none

# Backend: testssl (test1)
backend testssl
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server testSSL 127.4.4.3 send-proxy-v2 check-send-proxy

# statistics are DISABLE

Nextcloud was the only service i was making publicly available. I moved one of my internal services out from local subdomain to public and no i cannot reach it externally. app files:

Code Select Expand

# local access only subdomains
iprox iprox_backend
autgtp autgtp_backend
portainer portainer_backend

#public access subdomains
nextcloud nextcloud_backend
truenas truenas_backend

Code Select Expand

#public access subdomains
nextcloud nextcloud_backend
truenas truenas_backend

logs from HAproxy when trying an external connection for nextcloud:

Code Select Expand

2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.718] 0_SNI_frontend SSL_backend/SSL_server 1/0/254 5920 -- 3/2/1/1/0 0/0
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.740] 1_HTTPS_frontend~ nextcloud_backend/nextcloud 0/0/1/229/230 304 1194 - - ---- 4/2/0/0/0 0/0 "GET /apps/richdocuments/settings/fonts.json HTTP/1.1"2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 0_SNI_frontend SSL_backend/SSL_server 1/0/0 0 -- 3/2/1/1/0 0/0
2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake

logs from HAproxy when trying an external connection for truenas:

Code Select Expand

2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 0_SNI_frontend SSL_backend/SSL_server 1/0/0 0 -- 3/2/1/1/0 0/0
2023-05-23T12:47:31 Informational haproxy 10.1.1.59:46236 [23/May/2023:12:47:31.646] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.718] 0_SNI_frontend SSL_backend/SSL_server 1/0/254 5920 -- 3/2/1/1/0 0/0
2023-05-23T12:47:05 Informational haproxy 10.1.1.59:46936 [23/May/2023:12:47:05.740] 1_HTTPS_frontend~ nextcloud_backend/nextcloud 0/0/1/229/230 304 1194 - - ---- 4/2/0/0/0 0/0 "GET /apps/richdocuments/settings/fonts.json HTTP/1.1"


Interesting that the Truenas call seems to be going to nextcloud?

Unbound DNS: Overides

   truenas.mydomain     dedyn.io   A (IPv4 address)   10.1.1.1            
   nextcloud.mydomain   dedyn.io   A (IPv4 address)   10.1.1.1       

Firewall rules:

HAProxy_ports (80, 443) allowed to WAN address

I don't see any deny results in firewall log. Let me know what other logs or config would be useful. Thank you

Thank you, i did manage to resolve part of the issue from your help with clearing my browser cache. I had been testing via incognito mode often but had forgotten last few changes. Locally I can now access nextcloud via domain name as expected.

Externally however, no joy. Map file entry under #public access subdomains
nextcloud nextcloud_backend in both local and public maps (in that order). Would this point to an issue somewhere on Opnsense? Whether that's firewall, HAproxy etc not sure. Considering nextcloud itself can accept connection via url locally? Happy for your guidance and if you think that issue is still the target server then i'll go deep dive further there. Halfway there :)

smoked-proposal would you mind posting in your HAproxy config? To my knowledge everything i've set is as per the tutorial but i'd like to see your config for Nextcloud if you have it working (I don't have SSL verify either). I've asked over in the Nextcloud forums and they do believe HAproxy is dropping something in the redirect considering hitting nextcloud directly works correctly.

Hi!

I've followed the guide here and it has worked flawlessly for all my services internally! So thank you for that! So much better without the warning every time going into a server.

I'm struggling with nextcloud though and feel like i'm not passing something through that i should be. I've been tearing my hair out on this one! When i type in the browser nextcloud.my.domain it says site can't be reached but if i add /login to the end then it works perfectly (for internal) with padlock sign. I can also access internally via IP:11000 (This is the Apache port) which diverts me to nextcloud.my.domain/login as expected and works. I just can't get it to do this via reverse proxy so neither internal or external works with just nextcloud.my.domain.

This leads me to think there is something missing going from HAProxy to Nextcloud (Which is in Portainer). I have a bunch of other servers which are all working fine (Truenas, Proxmox etc) in HAProxy. I've tried putting Nextcloud in as a VM on Proxmox and also in Truenas to see if it was Portainer causing any issue but same problem. Code below with all my other servers removed and left Portainer and Nextcloud as they are on the same IP. Portainer works, Nextcloud doesn't...

Code Select Expand

# Automatically generated configuration.
# Do not edit this file manually.
#
global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats

# Frontend: 0_SNI_frontend (listening on 80 and 443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option tcplog

# Frontend: 1_HTTP_frontend (listening on localhost:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option httplog
    # ACL: NoSSL_condition
    acl acl_6451d6d41f14e3.72189927 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6451d6d41f14e3.72189927

# Frontend: 1_HTTPS_frontend (listening on localhost:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6451dd0fe21db7.63563341.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option httplog
    # ACL: cardav-endpoint
    acl acl_6461de0380c7b3.75062629 path_end -i /.well-known/carddav
    # ACL: caldav-endpoint
    acl acl_6461dde5d15634.54704624 path_end -i /.well-known/caldav
    # ACL: nc_nodeinfo
    acl acl_6466fe1b2e8ff2.47035478 path /.well-known/nodeinfo
    # ACL: nc_webfinger
    acl acl_6466fe303acb97.89104263 path /.well-known/webfinger
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6451d1c3d510f5.88841915 src 10.1.1.0/24 192.168.0.0/24

    # ACTION: cardav-endpoint
    http-request redirect code 301 location /remote.php/dav if acl_6461de0380c7b3.75062629
    # ACTION: caldav-endpoint
    http-request redirect code 301 location /remote.php/dav if acl_6461dde5d15634.54704624
    # ACTION: nc_nodeinfo
    http-request redirect code 301 location /index.php/%[capture.req.uri] if acl_6466fe1b2e8ff2.47035478
    # ACTION: nc_webfinger
    http-request redirect code 301 location /index.php/%[capture.req.uri] if acl_6466fe303acb97.89104263
    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6451d0414cb7f2.59080919.txt)] if acl_6451d1c3d510f5.88841915
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6451d7a6418a58.11785496.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: portainer_backend (portainer backend)
backend portainer_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # WARNING: pass through options below this line
    timeout tunnel 3600s
    http-reuse safe
    server portainer 10.1.1.59:9443

# Backend: nextcloud_backend (nextcloud_backend)
backend nextcloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server nextcloud 10.1.1.59:11000
# statistics are DISABLED


In the HAProxy log it says:

Code Select Expand

2023-05-19T16:35:18 | Informational | haproxy | 10.1.1.59:58792 [19/May/2023:16:35:18.215] 0_SNI_frontend SSL_backend/SSL_server 1/-1/0 0 CC 2/2/1/1/0 0/0 | 
-- | -- | -- | -- | --
2023-05-19T16:35:07 | Informational | haproxy | 10.1.1.103:46386 [19/May/2023:16:34:37.775] 1_HTTPS_frontend~ 1_HTTPS_frontend/<NOSRV> -1/-1/-1/-1/30003 0 0 - - PR-- 2/2/0/0/0 0/0 "<BADREQ>" | 
2023-05-19T16:35:07 | Informational | haproxy | 10.1.1.103:46386 [19/May/2023:16:34:37.761] 0_SNI_frontend SSL_backend/SSL_server 1/0/30017 5134 cD 2/2/1/1/0 0/0 | 
2023-05-19T16:34:48 | Informational | haproxy | 10.1.1.59:45564 [19/May/2023:16:34:48.055] 0_SNI_frontend SSL_backend/SSL_server 1/0/1 0 -- 3/3/2/2/0 0/0 | 
2023-05-19T16:34:48 | Informational | haproxy | 10.1.1.59:45564 [19/May/2023:16:34:48.055] 1_HTTPS_frontend/127.0.0.1:443: Connection closed during SSL handshake

I do have DNS A record in Unbound pointing at the firewall IP as with all my other services. I added the rules for Cardav/Caldav/webfinger/nodeinfo to see if that helps (was noted in other guides) but that didn't change anything.

Thanks!