HAproxy fails to establish connection from LAN - WAN works

[fastflo]

Hi everybody,

i have read a lot, followed instructions und tutorials, but somehow it doesnt seem to work.

I have a nextcloud instance running behind HAproxy with SSL offloading (thanks to the TheHellSite's tutorial, great work)
But somehow I cannot get access from LAN working.
WAN connections from the outside world are running prefectly fine, and when connecting to the WAN subdomain, it also works from a local subnet. But it doesn't connect directly, it goes over the ISPs router. Problem with that is speed.

Long story short:

This is the curl output when connecting from WAN (currently working domain replaced with "wandomain" for privay reasons. IP address changed as well)

Code: [Select]

curl -vv https://cloud.wandomain.dedyn.io
*   Trying 123.123.123.123:443...
* Connected to cloud.wandomain.dedyn.io (123.123.123.123) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=wandomain.dedyn.io
*  start date: Jan 31 17:54:22 2024 GMT
*  expire date: Apr 30 17:54:21 2024 GMT
*  subjectAltName: host "cloud.wandomain.dedyn.io" matched cert's "*.wandomain.dedyn.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55c7bcea7eb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: cloud.wandomain.dedyn.io
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 302
< date: Thu, 15 Feb 2024 12:25:16 GMT
< server: Apache
< content-security-policy: default-src 'self'; script-src 'self' 'nonce-OTZCYndWSlk0bytkeURsMVZiTklQTzhKSkpzcStKU1dMS0pjU2c2RG93MD06bXM0OWhpTVVvY3pKKzFSYVpkWXdFOXhPSE9nRnRQckFaSkZ6ZkVYTTFrTT0='; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';
< expires: Thu, 19 Nov 1981 08:52:00 GMT
< cache-control: no-store, no-cache, must-revalidate
< pragma: no-cache
< set-cookie: oc_sessionPassphrase=VcCM3N8fqHDln%2Bprw0dn1iScTTY8%2F9PUdEMivMayfJ8ryangSy8dhPKWCBVCYUkvp2F9z0AnWeXW3f4z5NOpV%2FoBph9uI6kT2mdCRO3FQKLaROuDgl0v7Bbe8AyVBsHJ; path=/; secure; HttpOnly; SameSite=Lax
< set-cookie: __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< set-cookie: __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< set-cookie: ocyp2lhyr740=6plc1iqo19ua2pe6nqhqu6chkd; path=/; secure; HttpOnly; SameSite=Lax
< referrer-policy: no-referrer
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-permitted-cross-domain-policies: none
< x-robots-tag: noindex, nofollow
< x-xss-protection: 1; mode=block
< location: https://cloud.wandomain.dedyn.io/index.php/login
< content-length: 0
< content-type: text/html; charset=UTF-8
< strict-transport-security: max-age=63072000; includeSubDomains; preload
<
* Connection #0 to host cloud.wandomain.dedyn.io left intact


This is the curl output from a temporary (identical) setup subdomain. In Unbound an overwrite to the SNI is in place. Works also from WAN when the override is not in place. wildcard cert, etc... is all working (from WAN. domain replaced with wandomain2, IP 10.10.30.1 is the SNI frontend, listening on 0.0.0.0:443 <- That desperate i am already)
Gateway is running into a timeout! When the Unboud override is disabled, DNS cache flushed, a fresh connection works again, but over the WAN address only.

Code: [Select]

curl -vv https://cloud.wandomain2.dedyn.io
*   Trying 10.10.30.1:443...
* Connected to cloud.wandomain2.dedyn.io (10.10.30.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=wandomain2.dedyn.io
*  start date: Jan 31 19:01:10 2024 GMT
*  expire date: Apr 30 19:01:09 2024 GMT
*  subjectAltName: host "cloud.wandomain2.dedyn.io" matched cert's "*.wandomain2.dedyn.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55bca4010eb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: cloud.wandomain2.dedyn.io
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 504
< content-length: 92
< cache-control: no-cache
< content-type: text/html
<
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
* Connection #0 to host cloud.wandomain2.dedyn.io left intact


I am stuck. Any help is highly appreciated.

PS: Is there a way to forward the real source IP to the backend server? Had a nextcloud user which typed in the password wrong multiple time, which disabled everybody, as all request are coming from 10.10.30.1, and not the real source IP. Thanks

[fastflo]

Additional info of the HAproxy logs:

I don't know where to loom at, and i am totally stuck!

Code: [Select]

2024-02-15T14:47:28 Error haproxy 10.20.10.59:60180 [15/Feb/2024:14:46:28.698] 0_SNI_Frontend SSL_backend/SSL_Server 1/0/60242 4962 cD 12/6/5/5/0 0/
2024-02-15T14:46:58 Informational haproxy 10.20.10.59:60180 [15/Feb/2024:14:46:58.938] 2_HTTPS_Frontend~ nextcloud_backend/nextcloud 0/-1/-1/-1/0 -1 0 - - CR-- 6/3/0/0/0 0/0 "GET https://cloud.wandomain2.dedyn.io/favicon.ico HTTP/2.0"
2024-02-15T14:46:58 Informational haproxy 10.20.10.59:60180 [15/Feb/2024:14:46:28.808] 2_HTTPS_Frontend~ nextcloud_backend/nextcloud 0/0/0/-1/30006 504 198 - - sH-- 6/3/0/0/0 0/0 "GET https://cloud.wandomain2.dedyn.io/ HTTP/2.0"