2021-11-13T15:50:07 haproxy[18223] Connect from 1.2.3.4:29968 to 127.0.0.1:8443 (PublicService_HTTPS/HTTP)
1.2.3.4:17799 [14/Nov/2021:12:01:26.338] PublicService_HTTPS~ BackendPool_Default/RealServer_Default 0/3030/-1/-1/3036 503 222 - - SC-- 1/1/0/0/3 0/0 "GET https://server.com/start HTTP/2.0"
BackendPool_Default/RealServer_Default 0/3030/-1/-1/3036 503 222 - - SC-- 1/1/0/0/3 0/0
"GET https://server.com/start HTTP/2.0"
Performing a health audit at System: Firmware > Run an audit
***GOT REQUEST TO AUDIT HEALTH***Currently running OPNsense 21.7.5 (amd64/OpenSSL) at Sun Nov 14 18:12:47 CET 2021>>> Check installed kernel versionVersion 21.7.5 is correct.>>> Check for missing or altered kernel filesNo problems detected.>>> Check installed base versionVersion 21.7.5 is correct.>>> Check for missing or altered base filesNo problems detected.>>> Check for missing package dependenciesChecking all packages: .......... done>>> Check for missing or altered package filesChecking all packages: .......... done>>> Check for core packages consistencyCore package "opnsense" has 66 dependencies to check.Checking packages: .................................................................... done***DONE***
Carefully check HAProxy configuration items in GUI / Carefully check HAProxy configuration items in the actual config file
Running HAProxy in check mode in the foreground at the console (with -c)
haproxy -f /usr/local/etc/haproxy.conf -c[WARNING] 317/181810 (10164) : parsing [/usr/local/etc/haproxy.conf:64] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.[WARNING] 317/181810 (10164) : parsing [/usr/local/etc/haproxy.conf:89] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.Warnings were found.
Running HAProxy in the foreground at the console (without daemon option, i.e. do not use -D) perhaps with verbose (-V), maybe debug (-d)
00000003:PublicService_HTTPS.accept(0007)=000a from [1.2.3.4:47425] ALPN=h200000003:PublicService_HTTPS.clireq[000a:ffffffff]: GET https://<server.com>/favicon.ico HTTP/2.000000003:PublicService_HTTPS.clihdr[000a:ffffffff]: host: <server.com>00000003:PublicService_HTTPS.clihdr[000a:ffffffff]: sec-ch-ua: "Microsoft Edge";v="95", "Chromium";v="95", ";Not A Brand";v="99"00000003:PublicService_HTTPS.clihdr[000a:ffffffff]: dnt: 100000003:PublicService_HTTPS.clihdr[000a:ffffffff]: sec-ch-ua-mobile: ?100000003:PublicService_HTTPS.clihdr[000a:ffffffff]: user-agent: Mozilla/5.0 (Linux; Android 11; LE2123) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Mobile Safari/537.36 EdgA/95.0.1020.4800000003:PublicService_HTTPS.clihdr[000a:ffffffff]: sec-ch-ua-platform: "Android"00000003:PublicService_HTTPS.clihdr[000a:ffffffff]: accept: image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.800000003:PublicService_HTTPS.clihdr[000a:ffffffff]: sec-fetch-site: same-origin00000003:PublicService_HTTPS.clihdr[000a:ffffffff]: sec-fetch-mode: no-cors00000003:PublicService_HTTPS.clihdr[000a:ffffffff]: sec-fetch-dest: image00000003:PublicService_HTTPS.clihdr[000a:ffffffff]: referer: https://<server.com>/00000003:PublicService_HTTPS.clihdr[000a:ffffffff]: accept-encoding: gzip, deflate, br00000003:PublicService_HTTPS.clihdr[000a:ffffffff]: accept-language: de-AT,de;q=0.9,en-US;q=0.8,en;q=0.7,de-DE;q=0.6,en-GB;q=0.500000003:PublicService_HTTPS.clihdr[000a:ffffffff]: cookie: __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; oc_sessionPassphrase=<REDACTED>; <REDACTED>=<REDACTED>fd[0xb] OpenSSL error[0x1416f086] tls_process_server_certificate: certificate verify failed00000004:GLOBAL.accept(0004)=000b from [unix:1] ALPN=<none>00000004:GLOBAL.srvcls[adfd:ffffffff]00000004:GLOBAL.clicls[adfd:ffffffff]00000004:GLOBAL.closed[adfd:ffffffff]fd[0xb] OpenSSL error[0x1416f086] tls_process_server_certificate: certificate verify failedfd[0xb] OpenSSL error[0x1416f086] tls_process_server_certificate: certificate verify failedfd[0xb] OpenSSL error[0x1416f086] tls_process_server_certificate: certificate verify failed00000003:BackendPool_Nextcloud.clicls[000a:000b]00000003:BackendPool_Nextcloud.closed[000a:000b]00000005:GLOBAL.accept(0004)=000b from [unix:1] ALPN=<none>00000005:GLOBAL.srvcls[adfd:ffffffff]00000005:GLOBAL.clicls[adfd:ffffffff]00000005:GLOBAL.closed[adfd:ffffffff]
can you try to connect to backend from opnsense shell with "openssl s_client" (dont forget -servername if you use SNI) and share results?
openssl s_client <REDACTED>5513582366720:error:0200203D:system library:connect:Connection refused:/usr/src/crypto/openssl/crypto/bio/b_sock2.c:110:5513582366720:error:2008A067:BIO routines:BIO_connect:connect error:/usr/src/crypto/openssl/crypto/bio/b_sock2.c:111:connect:errno=61
Doing anything funky, maybe with plugins?
Do you have any Resolver Options configured in your Backend Pool?
2021-11-15T06:56:55 haproxy[30508] <REDACTED>:42259 [15/Nov/2021:06:56:55.895] PublicService_HTTPS~ BackendPool_Nextcloud/RealServer_Nextcloud 0/0/0/51/51 200 4953 - - ---- 1/1/0/0/0 0/0 "GET https://<REDACTED>/core/js/oc.js?v=8007c44e HTTP/2.0" 2021-11-15T06:56:55 haproxy[30508] <REDACTED>:42259 [15/Nov/2021:06:56:55.624] PublicService_HTTPS~ BackendPool_Nextcloud/RealServer_Nextcloud 0/0/10/126/136 200 5348 - - ---- 1/1/0/0/0 0/0 "GET https://<REDACTED>/login HTTP/2.0"
curl -v https://<REDACTED> --cacert <PATH-TO-R3-CERT>.crt* Trying <REDACTED>:443...* Connected to <REDACTED> (<REDACTED>) port 443 (#0)* ALPN, offering h2* ALPN, offering http/1.1* successfully set certificate verify locations:* CAfile: <PATH-TO-R3-CERT>.crt* CApath: /etc/ssl/certs* TLSv1.3 (OUT), TLS handshake, Client hello (1):* TLSv1.3 (IN), TLS handshake, Server hello (2):* TLSv1.2 (IN), TLS handshake, Certificate (11):* TLSv1.2 (IN), TLS handshake, Server key exchange (12):* TLSv1.2 (IN), TLS handshake, Server finished (14):* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):* TLSv1.2 (OUT), TLS handshake, Finished (20):* TLSv1.2 (IN), TLS handshake, Finished (20):* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384* ALPN, server accepted to use h2* Server certificate:* subject: CN=<REDACTED>* start date: Nov 12 17:06:56 2021 GMT* expire date: Feb 10 17:06:55 2022 GMT* subjectAltName: host "<REDACTED>" matched cert's "<REDACTED>"* issuer: C=US; O=Let's Encrypt; CN=R3* SSL certificate verify ok.* Using HTTP2, server supports multi-use* Connection state changed (HTTP/2 confirmed)* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0* Using Stream ID: 1 (easy handle 0x55e35be79560)> GET / HTTP/2> Host: <REDACTED>> user-agent: curl/7.74.0> accept: */*> * Connection state changed (MAX_CONCURRENT_STREAMS == 128)!< HTTP/2 302 < server: nginx/1.21.4< date: Mon, 15 Nov 2021 06:17:46 GMT< content-type: text/html; charset=UTF-8< location: https://<REDACTED>/login< expires: Thu, 19 Nov 1981 08:52:00 GMT< cache-control: no-store, no-cache, must-revalidate< pragma: no-cache< set-cookie: <REDACTED>; path=/; secure; HttpOnly; SameSite=Lax< set-cookie: <REDACTED>; path=/; secure; HttpOnly; SameSite=Lax< content-security-policy: default-src 'self'; script-src 'self' '<REDACTED>'; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';< set-cookie: __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax< set-cookie: __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict< strict-transport-security: max-age=15768000; includeSubDomains; preload< referrer-policy: no-referrer< x-content-type-options: nosniff< x-download-options: noopen< x-frame-options: SAMEORIGIN< x-permitted-cross-domain-policies: none< x-robots-tag: none< x-xss-protection: 1; mode=block< * Connection #0 to host <REDACTED> left intact
R3 (Let's Encrypt) (as you expected) is valid
SSL Verify CA = LE - ISRG Root X1 (+ R3 (ACME Client))
Applied the changes. It works
Contains anchor