HAproxy 503 errors since upgrade to 21.7.5

[opn_minded]

Yes, obviously - just wanted to point out that it has expired (.."as you expected"..) :).

I have removed "R3 (ACME Client)" from the list of CAs that haproxy would verify against. Although haproxy doesn't show the "apply" button once I click "save" in the real server config, it seems to work.

Update to this statement: once I "force-apply" the new settings, without R3 (ACME Client), haproxy throws 503s again - so I'm back to the state where I select both CAs (as mentioned in https://forum.opnsense.org/index.php?topic=25558.msg123000#msg123000).

As for the preferred chain selection - maybe it would make sense to add a slider "advanced mode" like in other plugins to enable further options?

[Fright]

without R3 (ACME Client), haproxy throws 503s again - so I'm back to the state where I select both CAs

hah. yet another use of ssl libraries )
really HAProxy does not use the chain provided by server. the file specified in the ca-file directive must contain the entire chain of intermediate and root certificates. didn't know about it, I'm sorry  :o

[Fright]

without R3 (ACME Client), haproxy throws 503s again

I must correct myself: although the docs (https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#5.1-ca-verify-file) and some forums claim that the file must include intermediate certificates, I checked on one of my sites using LE and on the letsencrypt.org itself. works well if only the root certificate is added to the file on ca-file option.

[benyamin]

Presuming the "R3 (ACME Client)" has this SHA1 fingerprint:

Code Select Expand

A0:53:37:5B:FE:84:E8:B7:48:78:2C:7C:EE:15:82:7A:6A:F5:A4:05

...shouldn't it just be the "R3 (ACME Client)" cert at SSL Verify CA?

Just make sure the ISRG Root X1 is imported at System: Trust: Authorities.

btw, you can get the fingerprint by using:

Code Select Expand

openssl x509 -in  <PATH-TO-R3-CERT>.crt -noout -fingerprint

[opn_minded]

@benyamin:

"My" R3 (Acme Client) has:

Code Select Expand

openssl x509 -in R3_AcmeClient.crt -noout -fingerprint
SHA1 Fingerprint=A0:53:37:5B:FE:84:E8:B7:48:78:2C:7C:EE:15:82:7A:6A:F5:A4:05

.. and yes, LE - ISRG Root X1 is imported as system > authorities > trust:

I have, as you suggested, imported https://letsencrypt.org/certs/isrgrootx1.pem /w alias "LE - ISRG Root X1" (it shows up as "self-signed") - immediately afterwards the existing R3 (ACME Client) would show issuer = LE - ISRG Root X1.

As far as I understand from https://letsencrypt.org/certificates/ => ISRG Root X1 (as root CA) signed R3 (as intermediate CA) => R3 (as intermediate CA) is signing LE ACME challenges.

Sidenote - I'm using LEs automation to provide the LE-CRTs to a SFTP. If I run for the export..

Code Select Expand

openssl x509 -in ca.pem -noout -fingerprint
SHA1 Fingerprint=A0:53:37:5B:FE:84:E8:B7:48:78:2C:7C:EE:15:82:7A:6A:F5:A4:05

.. you can see that also the exported fullchain.pem would contain the R3 (Acme Client).

Anyhow - I've now run several tests, the only working option is that haproxy validates against "R3 (Acme Client)", if I remove it and only put "LE - ISRG Root X1" there, it fails with 503.

[Fright]

@opn_minded
still wondering why the behavior is different. is there a chance nginx does not supply the entire certificate chain, but only the end-entity certificate? Openssl in the https://forum.opnsense.org/index.php?topic=25558.msg122974#msg122974 did not show the result (forgot to specify the port?). can you try again please. it is possible with the SNI if you use it: like

Code Select Expand

openssl s_client -connect <backend ip>:443 -servername <servername>
thanks!

[opn_minded]

hi @fright!

i didn't add the port in the first request as i thought nginx would re-route traffic (automatically, as it's configured) to SSL/443.

Code Select Expand

openssl s_client -connect <REDACTED>:443
CONNECTED(00000003)
depth=0 CN = <REDACTED>
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = <REDACTED>
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:CN = <REDACTED>
   i:C = US, O = Let's Encrypt, CN = R3
---
Server certificate
-----BEGIN CERTIFICATE-----
<REDACTED>
-----END CERTIFICATE-----
subject=CN = <REDACTED>

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2482 bytes and written 407 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: <REDACTED>
    Session-ID-ctx:
    Master-Key: <REDACTED>
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    <REDACTED>

    Start Time: 1637571166
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: yes
---

[Fright]

Hi!

depth=0 CN = <REDACTED>
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = <REDACTED>
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:CN = <REDACTED>
   i:C = US, O = Let's Encrypt, CN = R3
---

thanks!
looks like it is. nginx not sending certs chain. leaf cert only. so haproxy needs this CAs certs in ca-file

[benyamin]

looks like it is. nginx not sending certs chain. leaf cert only. so haproxy needs this CAs certs in ca-file

That's for servers in the backend pool, right? Fix the chain there?

[Fright]

yes. i think that if backend will provide full intermediates chain then HAProxy will not need intermediate CAs certs in ca-file