OPNsense cannot connect via TLS to any server with an Let's Encrypt certificate.

[KHE]

After having issues with both updating OPNsense and DNS over TLS it seems to me that there is an issue with LE certificates.

Code Select Expand

[admin@OPNsense ~]$ fetch -o mimugmail.conf https://www.routerperformance.net/mimugmail.conf
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
5843273977856:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: https://www.routerperformance.net/mimugmail.conf: Authentication error

after I remove the DST Root CA X3 certificate from /etc/ssl/certs.pem and /usr/local/etc/ssl/certs.pem I get the following:

Code Select Expand

[admin@OPNsense ~]# fetch -o mimugmail.conf https://www.routerperformance.net/mimugmail.conf
Certificate verification failed for /C=US/O=Internet Security Research Group/CN=ISRG Root X1
898400673792:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: https://www.routerperformance.net/mimugmail.conf: Authentication error

And in both places the ISRG Root X1 is valid till 2035.

Running openssl s_client -connect unicast.uncensoreddns.org:853 (using a LE cert) gives the following (shortend):

Code Select Expand

CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
notAfter=Sep 30 18:14:03 2024 GMT
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
notAfter=Sep 15 16:00:00 2025 GMT
verify return:1
depth=0 CN = unicast.censurfridns.dk
notAfter=Nov 18 18:38:31 2021 GMT
verify return:1
---
Certificate chain
0 s:CN = unicast.censurfridns.dk
   i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
...
subject=CN = unicast.censurfridns.dk

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA384
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4404 bytes and written 409 bytes
Verification error: certificate has expired
...


KH

[Fright]

let me guess..
added cross-signed ISRG Root X1 cert to Trusted?)

[sgenchev]

This includes package updates from opnsense itself:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 21.7.2_1 (amd64/OpenSSL) at Thu Sep 30 12:40:03 CDT 2021
Fetching changelog information, please wait... Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4703086047232:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/sets/changelog.txz.sig: Authentication error
Updating OPNsense repository catalogue...
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
1578206494720:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3

[Tupsi]

I was about to answer that as well. Getting a update which would be able to fix that will be hard if a pkg update fails (which it does).

[Tupsi]

The fix seems to lie in this thread, but I am still trying to figure out what to do exactly from there. But people got it working again over there.
https://forum.opnsense.org/index.php?topic=24950.0

[mokai]

It also affects ldaps:// connections (!!) and let in our case to broken vpn authentication for our users.

seems the openssl version is the problem. the only thing that helped us for now is to import the ldap servers LE-Certificate into Trust->Authorities. It is then directly trusted by the ssl client and the (assumably broken) chain is not checked.

[mokai]

i think this is a bigger issue hitting a lot of people and should be raised in priority

https://twitter.com/Scott_Helme/status/1443293844292919304

[KHE]

let me guess..
added cross-signed ISRG Root X1 cert to Trusted?)

Kind of. I did delete the old DST Root CA X3 (called R3 (Let's Encrypt) from Trusted and recreated all the certificates.
This then seems to have added the ISRG Root X1 (called R3 (ACME Client). I removed this and recreated all of them again. This solved the issues.
I can update from mirrors with LE certs, DoT is working again with uncensoreddns.org and the openssl s_client also verifies the LE certs again.

Thanks
KH

[opnsense1972]

this steps worked for me. maybe some benefit too;

1. remove old ca (go System - Trust - Authorities)
2. change update mirror to http (MARWAN)
3. logon to commandline (as mine also seem to fail on ntop repository),
pkg remove ntop
pkg clean
pkg update
pkg upgrade
reboot

[joeyboon]

I've followed these steps and my webgui cert works like a charm, I'm also able to check for updates.

let me guess..
added cross-signed ISRG Root X1 cert to Trusted?)

Kind of. I did delete the old DST Root CA X3 (called R3 (Let's Encrypt) from Trusted and recreated all the certificates.
This then seems to have added the ISRG Root X1 (called R3 (ACME Client). I removed this and recreated all of them again. This solved the issues.
I can update from mirrors with LE certs, DoT is working again with uncensoreddns.org and the openssl s_client also verifies the LE certs again.

Thanks
KH

But I'm having other problems that still persist.

I'm on:
OPNsense 21.7.3_3-amd64
FreeBSD 12.1-RELEASE-p20-HBSD
LibreSSL 3.3.4

Monit can't send any more emails:

2021-10-01T18:11:49   monit[38088]   Aborting event   
2021-10-01T18:11:49   monit[38088]   SMTP: Error sending data to the mailserver -- No error: 0   
2021-10-01T18:11:49   monit[38088]   SSL: write error -- error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed   
2021-10-01T18:11:49   monit[38088]   Mail: SSL server certificate verification error: certificate has expired

And Nextcloud backups also stopped working.

2021-10-02T10:34:39   php-cgi[17483]   {"url":"https:\/\/DOMAINREPLACEDFORPRIVACY\/ocs\/v1.php\/cloud\/user","content_type":null,"http_code":0,"header_size":0,"request_size":0,"filetime":-1,"ssl_verify_result":10,"redirect_count":0,"total_time":0.081055,"namelookup_time":0.061367,"connect_time":0.066082,"pretransfer_time":0,"size_upload":0,"size_download":0,"speed_download":0,"speed_upload":0,"download_content_length":-1,"upload_content_length":-1,"starttransfer_time":0,"redirect_time":0,"redirect_url":"","primary_ip":"IPRELACEDFORPRIVACY","certinfo":[],"primary_port":443,"local_ip":"45.83.233.8","local_port":60733,"http_version":0,"protocol":2,"ssl_verifyresult":0,"scheme":"HTTPS","appconnect_time_us":0,"connect_time_us":66082,"namelookup_time_us":61367,"pretransfer_time_us":0,"redirect_time_us":0,"starttransfer_time_us":0,"total_time_us":81055}   
2021-10-02T10:34:39   php-cgi[17483]   Cannot get real username


The Nextcloud server has a let's encrypt certificate but it's not signed with the old CA.


Any suggestions?

[opnsense1972]

maybe, forse reset of existing account in acme, and then renew certificate will help

[joeyboon]

Since the router in my case (initiating email or connection to Nextcloud) is the client this should have nothing to do with the certificate, but with the CA's marked as trusted by the OS right? So shouldn't this be an update of the OS (BSD)? Maybe it's handling the cross signed versions of the CA wrong or something?

[joeyboon]

Does anyone have similar problems? Or maybe even someone who has fixed them by now?

[KHE]

Hi,

my solution in the end:

• Remove all LE CA/intermediate certificates with 0 certificates attached from System: Trust: Authorities
• Follow the solution from Felix: https://forum.opnsense.org/index.php?topic=24950.msg119837#msg119837, to download the ISRG root X1 and the LE R3 certificates and add a new Authority to System: Trust: Authorities with the R3 cert first and the ISRG after it in the Certificate data field and give it a nice name.
• All my ACME Client certificates where now assigned to this certificate. YMMV.
• Delete the old R3 certificate created by the ACME client, which should now have no certificates attached.

If you do not use the ACME client plug-in, the first two steps might be sufficient.

KH

[franco]

If you do not use the ACME client plug-in, the first two steps might be sufficient.

From what we have seen acme-client caused CA stuffing (multiple CA certificates in one entry) which then got flushed into the trust store causing the wrong chain to be verified by OpenSSL. When not using acme-client the condition is nearly impossible to trigger in the first place unless you maybe imported Let's Encrypt CA certificates manually into System: Trust: Authorities.

There is a code change to make handling the trust store more careful with regard to arbitrary System: Trust: Authorities input.

We since replaced the certificates away from Let's Encrypt for the community updates server to allow updates even in the botched system state.

But in the end consider Let's Encrypt doing this stunt to keep old Android phones from breaking. It nearly broke the rest of the Internet for a bit in the process. Not a good plan.


Cheers,
Franco