opnsense using wrong letsencrypt R3 intermediate certificate

[mfedv]

Hi,

opnsense/acme still uses an old Let's Encrypt R3 intermediate
certificate, pointing to a root CA (DST Root CA X3) that is about to
expire tomorrow (Sep. 30):

    https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

Ubuntu decided to jump ahead and removed the DST Root CA X3 already in
yesterday's update. While Firefox uses its own truststore and thus still
accepts these certificates, many cli commands on Ubuntu now don't accept
them anymore. Lost some of tonight's backups (restic) because of that.
Other, non-Ubuntu systems might show the same problems on/after
September 30.

Old trust path:

  local cert
    -> C = US, O = Let's Encrypt, CN = R3
       ->  O = Digital Signature Trust Co., CN = DST Root CA X3

New trust path:

local cert
    -> C = US, O = Let's Encrypt, CN = R3 (same entity as above, but different signature)
       ->  C = US, O = Internet Security Research Group, CN = ISRG Root X1

In System / Trust / Authorities I had both versions of the R3
intermediate certificate, but all of the local certs referred to the
old, now untrusted one.
It seems not to be possible in the GUI to just remove the old
certificate without also removing all those local certs referring to it.

I had to resort to manually editing /conf/config.xml, replacing all
occurances of
    <caref>600b59276e541</caref>
with
    <caref>60ac21f018263</caref>
and then rebooting (there is probably some less disrupting way).

Note: the IDs _will_ be different on every installation. You can find
the IDs for your installation on the command line using

    # grep -B 1 '<descr>R3 ' /conf/config.xml
        <refid>5fd0f040a02cd</refid>
        <descr>R3 (Let's Encrypt)</descr>
    --
        <refid>6093156cc2158</refid>
        <descr>R3 (ACME Client)</descr>

The one labeld "ACME Client" will be the current version of the R3
intermediate certificate.


You might want to check with your opnsense installations, too, if you
use the ACME plugin.

Regards
Matthias

[franco]

Hi Matthias,

I believe a fix will be part of this update to acme-client plugin: https://github.com/opnsense/plugins/pull/2551

ETA unclear, but looks like a hotfix candidate given the timing.


Cheers,
Franco

[Felix.]

Can confirm this issue.
On Ubuntu 20.04 the trust chain is already broken because ca-certificates removed the old DST Root Ca crt.  :-[
In my case it basically stops APT from working, because I am using an package repository with Let's Encrypt cert on it.
On Debian Buster / Proxmox the old cert is still trusted.

Also, I've found this documentation about the preferred chain: https://github.com/acmesh-official/acme.sh/wiki/Preferred-Chain
Sadly the default is (whyever) still the old DST Root, but for future proofing, it'd be great to have the preferred chain configurable via GUI, so we can avoid such last-minute moves.  ;D

Any chance we can get that acme plugin update to OPNsense today, because of the certificate expiry?
I assume all web sites / mailserver and whatnot protected by these certificates will break tonight.

And thank you Matthias for your research on how to workaround this issue, I'll check that out!  :D
And also thanks to franco for your quick response on that issue!  :)

Regards
Felix

[KHE]

It is even more urgent. The DST Root CA X3 certificate is valid till Thursday 30. September 2021 at 16:01:15, but the R3 intermediate certificate is only valid till Wednesday, 29. September 2021 at 21:21:40.

This means today at 9:21:41 pm the certificate chain will break. At least for my certificates.

KH

PS: Timezone is CEST

[Felix.]

I just found the GitHub issue that discusses this exact topic and there seems to be a patch already available!
https://github.com/opnsense/plugins/issues/2550
https://github.com/opnsense/plugins/issues/2550#issuecomment-929380587

So, if it doesn't make it into a publicly available hotfix today, we can at least patch by ourselves and call it a day.
I'll test it and report how / if it worked shortly!

[mimugmail]

You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all, then go to your sevices and look if they are correctly linked and restart.

No patch necessary.

[Felix.]

Tried the patches, they work.
Press the new re-import button (and if required, ymmv) renew your certificates.

[IsaacFL]

It looks like the certificate for https://forum.opnsense.org/ also will have the same issue today.

[KHE]

You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all, then go to your sevices and look if they are correctly linked and restart.

No patch necessary.

You have to assign the certificates to the webservice/HA-Proxy Public Services again manually afterwards. I have automatic restart of the services enabled. So in my case the links where gone.

It looks like the certificate for https://forum.opnsense.org/ also will have the same issue today.

For me https://forum.opnsense.org/ has a good one.

KH

[mimugmail]

You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all, then go to your sevices and look if they are correctly linked and restart.

No patch necessary.

You have to assign the certificates to the webservice/HA-Proxy Public Services again manually afterwards. I have automatic restart of the services enabled. So in my case the links where gone.

It looks like the certificate for https://forum.opnsense.org/ also will have the same issue today.

For me https://forum.opnsense.org/ has a good one.

KH

" then go to your sevices and look if they are correctly linked and restart" :)

[Fright]

@mimugmail thanks for reminding!! (i couldn't remember why my 20.7.7's was already giving out the correct chain. it seemed to me that I was not doing anything  :o  )

[mfedv]

You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all, then go to your sevices and look if they are correctly linked and restart.

Thats a good one.
Removing the expiring R3 cert was the first thing I tried, but with all my LE certs gone from System:Trust:Certificates I panicked and grabbed a backup config. Did not think of renewing them at that point.

Will be a busy day at letsencrypt when everybody renews all of their certs on the same day :-)

Matthias

[IsaacFL]

You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all, then go to your sevices and look if they are correctly linked and restart.

No patch necessary.

This did not work for me.  I created a new Cert but the Certification Path still showed the old Root. Even though the System/Trust/Authority Certificate shows expiration of 2025.

It did throw an error at:
/usr/local/opnsense/mvc/app/library/OPNsense/AcmeClient/LeValidation/Base.php Line: 240

[Felix.]

Also tried mimugmail's solution on another instance - works too!
So, less hassle than expected, really.

Let's see what burns on friday...  ;D

[IsaacFL]

It looks like the certificate for https://forum.opnsense.org/ also will have the same issue today.

You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all, then go to your sevices and look if they are correctly linked and restart.

No patch necessary.

This did not work for me.  I created a new Cert but the Certification Path still showed the old Root. Even though the System/Trust/Authority Certificate shows expiration of 2025.

It did throw an error at:
/usr/local/opnsense/mvc/app/library/OPNsense/AcmeClient/LeValidation/Base.php Line: 240

I found my problem.  Mimugmail process did work, but my MS Edge browser still had the intermediate R3 Certificate so it showed in the Cert Path.  Once I deleted on MS Edge it shows the correct path.

I assume once the expiration date was reached, then Edge would have downloaded the new R3 Cert on its own.