Let's Encrypt changing its root CA certificate -- Possible breaking changes

[SecAficionado]

UPDATED Dec 21, 2020
Please see post in the thread below about fix for older Android devices from IdenTrust and Let's Encrypt

UPDATED Dec 5, 2020
Let's Encrypt switched to a new CA on Dec 3, 2020, and any certificates renewed or issued with default settings are affected. There is a hotfix for 20.7.5 to prevent Opnsense from reporting issues with the validity of renewed/new certificates. Please see the thread below for the link.

The original post mentions that the change will happen in January 2021, but Let's Encrypt already made the change. Presumably to coincide with their 5th year anniversary.

--- Original Post ---

If you use Let's Encrypt certificates for your firewall and perhaps other internal servers, this might affect you. Your certificates may start giving certain users/clients warnings that they are not valid, starting in January 2021. Please read on.

Currently, the root CA for Let's Encrypt is cross signed by another CA, which was widely available 5 years ago. This made Let's Encrypt's certificates valid from day 1 on many systems, including legacy systems. That root CA is up for renewal on September, 2021, and Let's Encrypt will replace it with a new CA, which is not cross-signed. This should not be a problem for any system that is regularly patched, but it is likely to be an issue with legacy systems that are not regularly updated, or for IoT setups that don't get new certificate store updates.

When you renew any Let's Encrypt certificates after January 2021, you will get certificates signed by the new CA. This may break SSL/TLS for those older/IoT systems. To help with the transition, Let's Encrypt will allow clients to request certificates signed with the old root. That will give you time to make whatever changes you need (including migrating to a different CA) before the September 2021 deadline when all new certificates will be signed by the new CA.

More info at https://letsencrypt.org/2020/11/06/own-two-feet.html

There are two other free alternatives to Let's Encrypt, which use the same setup: Buypass, and ZeroSSL. Migrating to either one could be as simple as changing the URL for the certificate request.

[Steve79]

Thanks SecAficionado for bringing this to general discussion.

I cannot confirm, that it will start in January 2021. Might it be when the certs is valid till January 2021? My first LE-cert was affected two days ago, when renewed on Dec 2nd.

Code: [Select]

        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity
            Not Before: Dec  2 07:41:11 2020 GMT
            Not After : Mar  2 07:41:11 2021 GMT

        Authority Information Access:
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/

I presume, it will affect more and more users over the next days and weeks. It causes warnings on several android clients e.g. nextcloud. And no, they aren't old nor outdated versions. Updated Android 10 is also affected. From my understanding, the client warnings cannot be solved by opnsense, but through root CA updates on the clients.

But there is "problems" within opnsense with the new LE CA as well.

 I use monit plugin to watch the validity and expiration on my LE certs on my haproxy:

Code: [Select]

failed port 443 protocol https with ssl options {verify: enable} and certificate valid > 28 days retry 3
It basically warns me when LE plugin automatic renew might have failed, but before the cert expires. Now it tells me, that the certs issuer could not be verified:

Code: [Select]

SSL server certificate verification error: unable to get local issuer certificate
This becomes also a problem, if you backup your opnsense config on a webdav with this cert, like i do. It does fail with out a cert, which opnsense considers valid (this is -of cause- correct behavior, but makes config backup fail).

Code: [Select]

System Log shows entry from "php" with [...]"ssl_verify_result":20[...]
From my understanding (which i consider limited), opnsense requires a root CA update for the new LE CA as well. Can anyone confirm this and / or is there a way to fix this without an update of opnsense e.g. from console?
You mention a switch to keep the old CA. Is there a way to use this with the LE plugin?

Any help or hints are much appreciated.

Thanks Steve

[Gauss23]

When you renew any Let's Encrypt certificates after January 2021

So all certs issued now are still with the old CA. It starts for certs which are renewed/issued in January 2021.
So if you renew a cert at the the end of this month it will be running with the old ca for 3 months.

[Steve79]

So all certs issued now are still with the old CA. It starts for certs which are renewed/issued in January 2021.
So if you renew a cert at the the end of this month it will be running with the old ca for 3 months.

As i was trying to say in my post: I can not confirm this. It seems to me, that they are using the new CA already...

[mimugmail]

Look at GitHub issues, there is already a hotfix command available

[Steve79]

Look at GitHub issues, there is already a hotfix command available

https://github.com/opnsense/plugins/issues/2126

Thanks!

[SecAficionado]

@Steve79, yes, it looks like they switched on December 3. I got the impression, from reading their blog post, that they would not make the change until January 2021, but here we are.
https://twitter.com/letsencrypt/status/1334568843927228418

I'll try to edit my original post to correct that part.

You should still be able to renew the certificates by issuing the renew command with the alternate option, as referred in the Let's Encrypt blog post, but I am not sure the Opnsense GUI is aware of that option yet. I'm also happy to see that @Fraenki was on top of the issue and issued a hotfix with lightning speed!

I did not realize that so many systems would be impacted (even Android 10!) . I am lucky that my cert was renewed before the switch. I just happened to run into the announcement on Twitter and I thought it was important to post here.

If you can, please post progress on this thread. I know I will need to follow the same steps in the next few weeks.

[Steve79]

@SecAficionado My problems are gone with the LE plugin hotfix and another renewal after that. Thanks for updating the original post.

On github fraenki explains the problem like this:

...When doing this the certificate is referenced to the CA by using the caref attribute. However, this attribute is never updated. As a result applications like HAProxy will send an invalid certificate chain, effectively breaking SSL communication.

I'm wildly guessing here, but it think LE changed only the intermediate CA and the LE plugin run into a bug with building the chain for that. So it might be true, that the root CA has not been changed by LE yet.
 
As marked in orange here
https://letsencrypt.org/2020/09/17/new-root-and-intermediates.html#the-new-certificates

[franco]

We will push an update next week with the workaround included. Manual renewal is still mandatory to fix this though.


Cheers,
Franco

[SecAficionado]

@Steve79, thanks for the update and the clarification. Yeah, without being able to see an actual certificate, my guess was, well, just a guess  .

@Franco, thank you for your help. I am very glad this did not affect me directly, but still very thankful that you guys are so quick to fix stuff.

[SecAficionado]

New update (Dec 21, 2020) Let's Encrypt announced on their blog (https://letsencrypt.org/2020/12/21/extending-android-compatibility.html) that they have a workaround for older Android devices.

This fix relies on an Android-specific implementation of certificate validation, which may or may not match other systems, like IoT devices or other OSs.

They also say that, since this workaround is supposed to prevent Android devices from having any issues, they will not implement changes they had in mind for January or February of 2021. This could be critical for anyone having certificates nearing expiration. On the other hand, if the Android issue is causing you pain with newly renewed certificates, you might want to force an early renewal to get things to work again.

It is important to test now and look for workarounds or fixes.