OPNsense Forum

Archive => 21.7 Legacy Series => Topic started by: mfedv on September 29, 2021, 11:27:34 am

Title: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: mfedv on September 29, 2021, 11:27:34 am
Hi,

opnsense/acme still uses an old Let's Encrypt R3 intermediate
certificate, pointing to a root CA (DST Root CA X3) that is about to
expire tomorrow (Sep. 30):

    https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

Ubuntu decided to jump ahead and removed the DST Root CA X3 already in
yesterday's update. While Firefox uses its own truststore and thus still
accepts these certificates, many cli commands on Ubuntu now don't accept
them anymore. Lost some of tonight's backups (restic) because of that.
Other, non-Ubuntu systems might show the same problems on/after
September 30.

Old trust path:

  local cert
    -> C = US, O = Let's Encrypt, CN = R3
       ->  O = Digital Signature Trust Co., CN = DST Root CA X3

New trust path:

 local cert
    -> C = US, O = Let's Encrypt, CN = R3 (same entity as above, but different signature)
       ->  C = US, O = Internet Security Research Group, CN = ISRG Root X1

In System / Trust / Authorities I had both versions of the R3
intermediate certificate, but all of the local certs referred to the
old, now untrusted one.
It seems not to be possible in the GUI to just remove the old
certificate without also removing all those local certs referring to it.

I had to resort to manually editing /conf/config.xml, replacing all
occurances of
    <caref>600b59276e541</caref>
with
    <caref>60ac21f018263</caref>
and then rebooting (there is probably some less disrupting way).

Note: the IDs _will_ be different on every installation. You can find
the IDs for your installation on the command line using

    # grep -B 1 '<descr>R3 ' /conf/config.xml
        <refid>5fd0f040a02cd</refid>
        <descr>R3 (Let's Encrypt)</descr>
    --
        <refid>6093156cc2158</refid>
        <descr>R3 (ACME Client)</descr>

The one labeld "ACME Client" will be the current version of the R3
intermediate certificate.


You might want to check with your opnsense installations, too, if you
use the ACME plugin.

Regards
Matthias
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: franco on September 29, 2021, 12:24:49 pm
Hi Matthias,

I believe a fix will be part of this update to acme-client plugin: https://github.com/opnsense/plugins/pull/2551

ETA unclear, but looks like a hotfix candidate given the timing.


Cheers,
Franco
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: Felix. on September 29, 2021, 01:21:13 pm
Can confirm this issue.
On Ubuntu 20.04 the trust chain is already broken because ca-certificates removed the old DST Root Ca crt.  :-[
In my case it basically stops APT from working, because I am using an package repository with Let's Encrypt cert on it.
On Debian Buster / Proxmox the old cert is still trusted.

Also, I've found this documentation about the preferred chain: https://github.com/acmesh-official/acme.sh/wiki/Preferred-Chain
Sadly the default is (whyever) still the old DST Root, but for future proofing, it'd be great to have the preferred chain configurable via GUI, so we can avoid such last-minute moves.  ;D

Any chance we can get that acme plugin update to OPNsense today, because of the certificate expiry?
I assume all web sites / mailserver and whatnot protected by these certificates will break tonight.

And thank you Matthias for your research on how to workaround this issue, I'll check that out!  :D
And also thanks to franco for your quick response on that issue!  :)

Regards
Felix
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: KHE on September 29, 2021, 02:15:08 pm
It is even more urgent. The DST Root CA X3 certificate is valid till Thursday 30. September 2021 at 16:01:15, but the R3 intermediate certificate is only valid till Wednesday, 29. September 2021 at 21:21:40.

This means today at 9:21:41 pm the certificate chain will break. At least for my certificates.

KH

PS: Timezone is CEST
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: Felix. on September 29, 2021, 03:04:50 pm
I just found the GitHub issue that discusses this exact topic and there seems to be a patch already available!
https://github.com/opnsense/plugins/issues/2550
https://github.com/opnsense/plugins/issues/2550#issuecomment-929380587

So, if it doesn't make it into a publicly available hotfix today, we can at least patch by ourselves and call it a day.
I'll test it and report how / if it worked shortly!
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: mimugmail on September 29, 2021, 04:10:23 pm
You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all, then go to your sevices and look if they are correctly linked and restart.

No patch necessary.
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: Felix. on September 29, 2021, 04:19:06 pm
Tried the patches, they work.
Press the new re-import button (and if required, ymmv) renew your certificates.

Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: IsaacFL on September 29, 2021, 04:36:58 pm
It looks like the certificate for https://forum.opnsense.org/ also will have the same issue today.
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: KHE on September 29, 2021, 04:46:10 pm
You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all, then go to your sevices and look if they are correctly linked and restart.

No patch necessary.

You have to assign the certificates to the webservice/HA-Proxy Public Services again manually afterwards. I have automatic restart of the services enabled. So in my case the links where gone.

It looks like the certificate for https://forum.opnsense.org/ also will have the same issue today.

For me https://forum.opnsense.org/ has a good one.

KH
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: mimugmail on September 29, 2021, 04:54:49 pm
You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all, then go to your sevices and look if they are correctly linked and restart.

No patch necessary.

You have to assign the certificates to the webservice/HA-Proxy Public Services again manually afterwards. I have automatic restart of the services enabled. So in my case the links where gone.

It looks like the certificate for https://forum.opnsense.org/ also will have the same issue today.

For me https://forum.opnsense.org/ has a good one.

KH

" then go to your sevices and look if they are correctly linked and restart" :)
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: Fright on September 29, 2021, 05:09:42 pm
@mimugmail thanks for reminding!! (i couldn’t remember why my 20.7.7's was already giving out the correct chain. it seemed to me that I was not doing anything  :o  )
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: mfedv on September 29, 2021, 05:35:21 pm
You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all, then go to your sevices and look if they are correctly linked and restart.

Thats a good one.
Removing the expiring R3 cert was the first thing I tried, but with all my LE certs gone from System:Trust:Certificates I panicked and grabbed a backup config. Did not think of renewing them at that point.

Will be a busy day at letsencrypt when everybody renews all of their certs on the same day :-)

Matthias
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: IsaacFL on September 29, 2021, 06:20:55 pm
You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all, then go to your sevices and look if they are correctly linked and restart.

No patch necessary.

This did not work for me.  I created a new Cert but the Certification Path still showed the old Root. Even though the System/Trust/Authority Certificate shows expiration of 2025.

It did throw an error at:
/usr/local/opnsense/mvc/app/library/OPNsense/AcmeClient/LeValidation/Base.php Line: 240


Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: Felix. on September 29, 2021, 06:32:41 pm
Also tried mimugmail's solution on another instance - works too!
So, less hassle than expected, really.

Let's see what burns on friday...  ;D
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: IsaacFL on September 29, 2021, 07:48:26 pm
It looks like the certificate for https://forum.opnsense.org/ also will have the same issue today.


You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all, then go to your sevices and look if they are correctly linked and restart.

No patch necessary.

This did not work for me.  I created a new Cert but the Certification Path still showed the old Root. Even though the System/Trust/Authority Certificate shows expiration of 2025.

It did throw an error at:
/usr/local/opnsense/mvc/app/library/OPNsense/AcmeClient/LeValidation/Base.php Line: 240

I found my problem.  Mimugmail process did work, but my MS Edge browser still had the intermediate R3 Certificate so it showed in the Cert Path.  Once I deleted on MS Edge it shows the correct path.

I assume once the expiration date was reached, then Edge would have downloaded the new R3 Cert on its own.
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: Felix. on September 29, 2021, 09:44:29 pm
Immediately after the old R3 expired, the browsers showed the new chains correctly.
I attached some examples. Also catched a site that still had the old trust chain in use, it breaks as expected.

Good luck y'all!  ;)
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: andrew on September 30, 2021, 04:11:42 pm
Thank you mfedv and mimugmail! You saved my day!
EDIT: And Frank (see below) and franco and everyone else who helped fixing this!
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: franco on September 30, 2021, 04:13:17 pm
FWIW, acme-client 3.2 is now available for update... special thanks to Frank on this one.


Cheers,
Franco
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: Felix. on September 30, 2021, 05:53:18 pm
Nice work!  ;D
Looking forward to run the update.


Now that the old DST Root expired, too, I got a little problem trying to update.
I removed the expired R3 from System -> Trust -> Authorities but this could not fix this issue.
The pkg.opnsense.org page reports the new trustchain, though, IN THE BROWSER! (see openssl below)

OpenSSL (openssl s_client -showcerts -connect pkg.opnsense.org:443) tells me:

CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
notAfter=Sep 30 18:14:03 2024 GMT
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
notAfter=Sep 15 16:00:00 2025 GMT
verify return:1
depth=0 CN = pkg.opnsense.org
notAfter=Dec  2 04:35:59 2021 GMT
verify return:1
---
Certificate chain
 0 s:CN = pkg.opnsense.org
   i:C = US, O = Let's Encrypt, CN = R3
-----BEGIN CERTIFICATE-----
MIIFJjCCBA6gAwIBAgISAwnrN5poWj6huxamPB7KKObHMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MDMwNDM2MDBaFw0yMTEyMDIwNDM1NTlaMBsxGTAXBgNVBAMT
EHBrZy5vcG5zZW5zZS5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQDLuegAAsvkP2JKDIinPk32wmxki8rWvWIG/9qWrzhX7uSneOnXsaMGg08FiDRL
V90INKc6ah4tJP62MB9LAaieTy/RvgetWvXJSAHVxI/qdGQbzO519zU45poxqngd
wkyQUOKg9PzQpcMPPFmicxx8ZGVJpeDlXA0RvFpDOcq3ilp1yYRzziVL3P9AmpZj
HpP8hYwtTi6ZgQIFYXfjsSryXOXqAKcwHkMGo8ybXk+CPrvOqztGcyiqyMhqa3Lf
OgLmIFJ2Fwji9GXiYnR0B8ovp18psvI4jio3ueTj34SrcvHm0rpNDK4tmkBZHMy3
MiDzQ0YI94xGZk4tetXhGuMXAgMBAAGjggJLMIICRzAOBgNVHQ8BAf8EBAMCBaAw
HQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYD
VR0OBBYEFCsV4Qsui4+pEVjC+GwfxyC5EVOWMB8GA1UdIwQYMBaAFBQusxe3WFbL
rlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0cDov
L3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5sZW5jci5v
cmcvMBsGA1UdEQQUMBKCEHBrZy5vcG5zZW5zZS5vcmcwTAYDVR0gBEUwQzAIBgZn
gQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5s
ZXRzZW5jcnlwdC5vcmcwggEEBgorBgEEAdZ5AgQCBIH1BIHyAPAAdQB9PvL4j/+I
VWgkwsDKnlKJeSvFDngJfy5ql2iZfiLw1wAAAXuqKBw5AAAEAwBGMEQCIBCHE6jR
njQEF/6It4qSuSPUSpSzNBYjJ90Z6N4RpDu9AiBaAkb5ZA95J/ixPXuaY5+ZjhT9
fMCGJLC6nAflr0P8tQB3AJQgvB6O1Y1siHMfgosiLA3R2k1ebE+UPWHbTi9YTaLC
AAABe6ooHe0AAAQDAEgwRgIhALpNbmfz4/ovnuuBbvBzegPU60r2Kk2PcG6acDLh
tyklAiEAzrIl1bMEAGeV0H7/aUJVfWqobZmOUdFb4Z3xyzvetzIwDQYJKoZIhvcN
AQELBQADggEBAE9YsG8jP9mCotJYPgAKIry2A4fTFReihcATDwhG/iEsLZ3Y2fwE
oCW5C51nAf7GNDzMlI+RCZzLHDRvg+5XJ1rjOWDAmm7DFx1AlsAvvK3jwAuvPA6h
jTIQyQwj1ULsqhbtGqHudrICh+XLs+i/ROJUqC40tM25rgPYDuRyYZ/rrSAy4DQh
tm7bdU/FfHTaPsO0Bb4hD75pOX8vDcD4xFO39XM5zML4zOvD0hKtVOWRzI1Jg0dX
NLpuO3rqwbZiSJfxM0LeGbbNNouM9jxkL3lhr7VsYTJix0mq0clAnUnIjnP2euPm
w3QJTpUmC7BgoxjliF+0UsQe2kqxEOm/5WM=
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----
---
Server certificate
subject=CN = pkg.opnsense.org

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4683 bytes and written 405 bytes
Verification error: certificate has expired
---
New, TLSv1.2, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
    Session-ID: CA62F24EAC5A00CEF8CB1CA04B986793D50D784591F2B4B5BDA66DC937388A06
    Session-ID-ctx:
    Master-Key: 1BF31F6E4565CA8B17AAE51E22B2189A724160464B685BB8232E0C2389CECE0252D357A46D3975179048D0A7A3E81BB8
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 48 3d 03 f0 5e e0 b5 3e-9a aa 90 8b a8 52 52 61   H=..^..>.....RRa
    0010 - 08 cc 87 15 3e 2e 30 76-00 a5 a6 38 8a ea 9a ff   ....>.0v...8....
    0020 - 8c e1 92 f9 0c e0 32 93-c1 f9 1c 4c 2e 83 1e 3d   ......2....L...=
    0030 - 9a 47 ab fe ae 5d 91 f9-47 d7 07 5a 4e 2a 6e 28   .G...]..G..ZN*n(
    0040 - 20 79 6c 2e 94 b9 5d f3-76 b8 a1 e2 f7 be e6 8f    yl...].v.......
    0050 - b2 70 28 d1 d8 be dc 94-4e 05 ab 3a 50 e5 2a 71   .p(.....N..:P.*q
    0060 - 83 bc fa f7 80 f4 68 98-cd 39 9c 84 c8 cf 12 74   ......h..9.....t
    0070 - 10 73 7d ed 06 fb 14 fe-24 10 bb 7a ff 05 2a 20   .s}.....$..z..*
    0080 - f7 d9 9b 18 4c 66 21 3d-0f 6c 62 3e 40 fb b3 dc   ....Lf!=.lb>@...
    0090 - 82 41 cb 55 9e 94 12 c0-ec e7 60 9b 83 c7 26 d9   .A.U......`...&.
    00a0 - 7a 34 d0 07 d2 74 f2 e9-bf a0 10 ed 2d 3a 07 cc   z4...t......-:..
    00b0 - 3a e8 eb 17 4f 94 f0 89-39 36 e1 e3 1f 01 81 05   :...O...96......
    00c0 - 21 af 52 3a 16 8f dc 79-67 4d e8 ef fa 82 dc bf   !.R:...ygM......

    Start Time: 1633017373
    Timeout   : 7200 (sec)
    Verify return code: 10 (certificate has expired)
    Extended master secret: yes
---

Full Log from Package Updater:

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 21.7.3_1 (amd64/OpenSSL) at Thu Sep 30 17:51:37 CEST 2021
Fetching changelog information, please wait... Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
5472367599616:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/sets/changelog.txz.sig: Authentication error
Updating OPNsense repository catalogue...
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
3858323083264:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
3858323083264:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
3858323083264:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
3858323083264:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
3858323083264:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
3858323083264:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz: Authentication error
repository OPNsense has no meta file, using default settings
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
3858323083264:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
3858323083264:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
3858323083264:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/packagesite.txz: Authentication error
Unable to update repository OPNsense
Error updating repositories!
pkg: Repository OPNsense cannot be opened. 'pkg update' required
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: Felix. on September 30, 2021, 06:10:16 pm
For my services that are hit by this issue, I built the following chain that gets delivered by nginx (should work with apache/haproxy etc. too):

wget https://letsencrypt.org/certs/isrgrootx1.pem
wget https://letsencrypt.org/certs/lets-encrypt-r3.pem

cat myCert.pem lets-encrypt-r3.pem isrgrootx1.pem > fullchain.pem

This excludes the old DST root from the chain and clients like openssl / pkg / apt will stop complaining.
The browsers seem to be more tolerant at this point, though.
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: Tupsi on September 30, 2021, 08:08:55 pm
how would I fix my opnsense installation if pkg update already stopped working because if the issue? I do not seem to have the LE plugin installed (if its not in the base system I do not have it installed myself), so from what I get from this thread I can't just delete the old cert and let the plugin catch the new one, right?

Felix posted someting in his last post which seems to get me the correct cert chain, put where do I have to put this in order to get a pkg update running again?
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: Tupsi on September 30, 2021, 08:30:37 pm
nm, figured it out myself. Used Felix wget on another computer and cat together the two (without a 3rd own) and imported that under Authorities as new trusted authrorities.

After that a pkg update worked again, so thanks for the pointers @Felix!
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: Felix. on September 30, 2021, 08:49:16 pm
Great catch, Tupsi!
Works for me, too!

How to get package updates to work:

- Remove the Let's Encrypt's R3 cert from System -> Trust -> Authorities.
- Add a new Authority Certificate and paste both R3 and ISRG Root X1 into the "Certificate data" field.

"certificate data" should look like this:
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----

Or download them directly from the Let's Encrypt links that I pasted above.

After you updated, re-issue any of your LE certificates (or all of them, to fix your services like HAproxy).
This will load the correct Authority from LE again and replace your just added custom Authority and the system should be good to go again.

Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: razamatan on September 30, 2021, 09:14:02 pm
even after getting the new R3 cert in as a cert authority, i still get tls issues when trying to check for updates in the webui...  how do you fix that?
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: KHE on September 30, 2021, 10:05:36 pm
Did you delete the old R3 cert?
I had to repeat the process of deleting the LE CA cert and reissue them again in order to get my update and DoT working again.
After the reissue of the first LA cert with the ACME Client the LE CA cert was added again and from then on everything started to work again.

KH
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: razamatan on September 30, 2021, 10:50:54 pm
yea, i deleted the old R3 cert.
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: ThyOnlySandman on October 01, 2021, 06:32:13 am
I'm confused...
I updated via http without 3rd party repos.  Got Acme import.
Deleted old + new R3.
Renew all certs.
Run Acme import on all certs - verify R3 CA cert has total # of associated certs.
Set cert in settings administration.  Reboot webui.
Opnsense now authenticates https mirrors.
Set NGINX / NTOPNG certs.
All certs / chains in use are happy.
Reboot Opnsense and won't authenticate against upate mirrors again.  All certs / NGINX chains are fine, including Opnsense GUI cert.
I've done this twice now.  What am I missing?
Thank you.
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: RamSense on October 01, 2021, 08:20:15 am
@Felix.
Thanks! That fixed it for me also!
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: GreenMatter on October 01, 2021, 08:52:57 am
I had two R3 certs in the system. After deleting old one, updates didn't work. When I deleted second R3 (Acme client) cert. I was able to update the system...
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: ThyOnlySandman on October 01, 2021, 08:55:18 am
Does it continue to update for you following a reboot?
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: evox on October 01, 2021, 09:49:46 am
Still having issues authenticating to https mirrors...

Some actions:
1. updated all using http mirror (acme version 3.2)
2. removed LE CA and certs
3. rerun acme, get new certs
4. add new cert to webgui

It seems like the correct cert/CA (LE R3, ISRG Root X1) is in place, not sure what is breaking package updater?

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 21.7.3_3 (amd64/OpenSSL) at Fri Oct  1 09:44:10 CEST 2021
Fetching changelog information, please wait... Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4599340929024:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/sets/changelog.txz.sig: Authentication error
Updating OPNsense repository catalogue...
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
1458337177600:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
1458337177600:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
1458337177600:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
1458337177600:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
1458337177600:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
1458337177600:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz: Authentication error
repository OPNsense has no meta file, using default settings
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
1458337177600:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
1458337177600:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
1458337177600:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/packagesite.txz: Authentication error
Unable to update repository OPNsense
Error updating repositories!
pkg: Repository OPNsense cannot be opened. 'pkg update' required
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: meepmeep on October 01, 2021, 11:42:41 am
I removed the old LE authority, and changed my mirror to LeaseWeb/http. I could get the update to 21.7.3_3 ..

... But i still get this error message :

Code: [Select]
***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 21.7.3_3 (amd64/OpenSSL) at Fri Oct  1 11:39:17 CEST 2021
Fetching changelog information, please wait... Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
6526875459584:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/sets/changelog.txz.sig: Authentication error
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.txz: .......... done
Processing entries: .......... done
OPNsense repository update completed. 767 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (1 candidates): . done
Processing candidates (1 candidates): . done
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: comvid on October 01, 2021, 11:48:32 am
Same here.
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: dinguz on October 01, 2021, 12:01:28 pm
Great catch, Tupsi!
Works for me, too!

How to get package updates to work:

- Remove the Let's Encrypt's R3 cert from System -> Trust -> Authorities.
- Add a new Authority Certificate and paste both R3 and ISRG Root X1 into the "Certificate data" field.

Or download them directly from the Let's Encrypt links that I pasted above.

After you updated, re-issue any of your LE certificates (or all of them, to fix your services like HAproxy).
This will load the correct Authority from LE again and replace your just added custom Authority and the system should be good to go again.

When I do these steps, secure (https) updates work again, but only until the next reboot. After this reboot the errors are back again. Can I do something to make these 'stick'?
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: bill.gertz on October 01, 2021, 01:33:35 pm
Sadly this still doesn't fix the problem - after reboot it comes back from the grave:

Code: [Select]
***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 21.7.3_3 (amd64/OpenSSL) at Fri Oct  1 13:27:05 CEST 2021
Fetching changelog information, please wait... Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4522656063488:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/sets/changelog.txz.sig: Authentication error
Updating OPNsense repository catalogue...
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
2029010690048:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
2029010690048:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
2029010690048:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
2029010690048:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
2029010690048:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz: No address record
repository OPNsense has no meta file, using default settings
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
2029010690048:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
2029010690048:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/packagesite.txz: Authentication error
Unable to update repository OPNsense
Error updating repositories!
pkg: Repository OPNsense cannot be opened. 'pkg update' required
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***

No use in trying to hack the /conf/config.xml as outlined by @mfedv at the top of this thread. After fully patching the CA reference all point to the correct LE R3 Cert. Something is deeply broken in the repository config.
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: chemlud on October 01, 2021, 01:44:36 pm
I have no problems with updates on LibreSSL flavor. Maybe switch for the moment?
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: bill.gertz on October 01, 2021, 01:55:51 pm
@chemlud

Switched to LibreSSL, rebooted and:

Code: [Select]
***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 21.7.3_3 (amd64/OpenSSL) at Fri Oct  1 13:53:06 CEST 2021
Fetching changelog information, please wait... Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
2368641363968:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/sets/changelog.txz.sig: Authentication error
Updating OPNsense repository catalogue...
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4369107513344:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4369107513344:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4369107513344:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4369107513344:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4369107513344:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4369107513344:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/libressl/meta.txz: Authentication error
repository OPNsense has no meta file, using default settings
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4369107513344:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4369107513344:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4369107513344:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/libressl/packagesite.txz: Authentication error
Unable to update repository OPNsense
Error updating repositories!
pkg: Repository OPNsense cannot be opened. 'pkg update' required
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***

Made absolutely no difference. Suspect that something is deeply broken in the repository config. Although it seems I need to update the firmware first but cannot as I cannot connect to the repository.

I've ordered a Chicken and an Egg from Amazon, I'll let you know...
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: chemlud on October 01, 2021, 01:57:42 pm
That didn't work, as it came back after reboot with OpenSSL.... You have to do an update after changing to LibreSSL, to get it installed.

Tried another server with http?
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: bill.gertz on October 01, 2021, 02:12:30 pm
@chemlud

Thanks for your suggestion but it didn't help.

Recreated the LE Fullchain R3 Intermediate X1 Authority Cert as outlined earlier, and deleted the LE R3 Cert. Recreated the Web GUI Cert to reprime ACME.SH. Was then able to update the firmware to the LibreSSL Flavour.

Rebooted and tried another firmware update and:

Code: [Select]
***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 21.7.3_3 (amd64/LibreSSL) at Fri Oct  1 14:07:17 CEST 2021
Fetching changelog information, please wait... Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4034015752192:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: transfer timed out
Updating OPNsense repository catalogue...
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
455974096896:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
455974096896:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
455974096896:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
455974096896:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
455974096896:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
455974096896:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/libressl/meta.txz: Authentication error
repository OPNsense has no meta file, using default settings
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
455974096896:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
455974096896:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
455974096896:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/libressl/packagesite.txz: Authentication error
Unable to update repository OPNsense
Error updating repositories!
pkg: Repository OPNsense cannot be opened. 'pkg update' required
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***

it made absolutely no difference. Again suspect something is deeply wrong with the repository configuration. Starting to dig into that. Setting back to OpenSSL as changing to LibreSSL made no difference.

By the way, found that repriming ACME.SH was not needed as I was able to check config status without repriming the authority certificates.

After reverting back to the OpenSSL Flavour and then resetting one of the ACME.SH certs to reprime the Authority Certificates and rebooting, the issue still comes back. Why on earth the repository seems to fall back on a long-dead cert that relies on the dead X3 Intermediate Authority is beyond my understanding at the moment. Very, very, very perplexed.
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: Felix. on October 01, 2021, 03:12:00 pm
The certificate chain of pkg.opnsense.org changed recently.
They are now using a Sectigo Wildcard Certificate, so I'm wondering if you still have these issues.

OpenSSL Log:
openssl s_client -showcerts -connect pkg.opnsense.org:443


CONNECTED(00000003)
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
verify return:1
depth=0 CN = *.opnsense.org
verify return:1
---
Certificate chain
 0 s:CN = *.opnsense.org
   i:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
-----BEGIN CERTIFICATE-----
MIIGMzCCBRugAwIBAgIRAKql/zLP5nCsBkX7C2AJZVQwDQYJKoZIhvcNAQELBQAw
gY8xCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE3MDUGA1UE
AxMuU2VjdGlnbyBSU0EgRG9tYWluIFZhbGlkYXRpb24gU2VjdXJlIFNlcnZlciBD
QTAeFw0yMDAxMjIwMDAwMDBaFw0yMjAxMjkyMzU5NTlaMBkxFzAVBgNVBAMMDiou
b3Buc2Vuc2Uub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0uOP
ZAoPfLMLew26Jbck9pppvQhHkWlXiz1pOV+YP446zMLJfJhoOOSSb74kf3vdXLdb
vM1Nxmdu2+G409pjfj+KUWEIN6eqZzRQ/ti4H9Xx2utuuENJ4kOrEFLWu+GH9Kfk
A36Pig20/M9PUvam6FB2y/oS3eOioCl9BkitfJsv5TC7PdaPYx5jqUmWP7Exe6pE
vvxPFU6lHlN6eGElW5BZClFtsQdt71SMOEK+fVpuKtSCpmkMxZR7VU9N50gp4u26
CWDEOSdAZEazDZCLMqLLeGr4fI0bxAp2+tjqtPccuywQzyoxTgjjhfvOY5fDHx03
FIQJ8fvSZDtyfcw/KQIDAQABo4IC/TCCAvkwHwYDVR0jBBgwFoAUjYxexFStiuF3
6Zv5mwXhuAGNYeEwHQYDVR0OBBYEFI5agxXurWZ4TCnFaLt9BJVrE3fwMA4GA1Ud
DwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggr
BgEFBQcDAjBJBgNVHSAEQjBAMDQGCysGAQQBsjEBAgIHMCUwIwYIKwYBBQUHAgEW
F2h0dHBzOi8vc2VjdGlnby5jb20vQ1BTMAgGBmeBDAECATCBhAYIKwYBBQUHAQEE
eDB2ME8GCCsGAQUFBzAChkNodHRwOi8vY3J0LnNlY3RpZ28uY29tL1NlY3RpZ29S
U0FEb21haW5WYWxpZGF0aW9uU2VjdXJlU2VydmVyQ0EuY3J0MCMGCCsGAQUFBzAB
hhdodHRwOi8vb2NzcC5zZWN0aWdvLmNvbTAnBgNVHREEIDAegg4qLm9wbnNlbnNl
Lm9yZ4IMb3Buc2Vuc2Uub3JnMIIBfQYKKwYBBAHWeQIEAgSCAW0EggFpAWcAdgBG
pVXrdfqRIDC1oolp9PN9ESxBdL79SbiFq/L8cP5tRwAAAW/OOTUcAAAEAwBHMEUC
IQCqpQuVnJWhzFFonCzybDP+Dfm3VUQJ/OiLP7a9I3eO7AIgP0dyeUsY0SBavzy1
l+uXu/xbcumKSKz4MbYBI0S9TUkAdgBvU3asMfAxGdiZAKRRFf93FRwR2QLBACkG
jbIImjfZEwAAAW/OOTVpAAAEAwBHMEUCIQCv5mW+Y/ptu+eU4mrA/BYg8AzH2ex3
kSy+FBXQE9K/GAIgJJOvwRuu1YP24A++Peis50x4H80dG92gYbX7oC76KUkAdQAi
RUUHWVUkVpY/oS/x922G4CMmY63AS39dxoNcbuIPAgAAAW/OOTULAAAEAwBGMEQC
IBSdWWClPXY7OPNOLkGIdPPrIXlX5cWq0j+cF2qgd/6BAiBX9JHarLhnS6ApPfYm
22nR2s3PILUQFZYCluVdBkJV4jANBgkqhkiG9w0BAQsFAAOCAQEAKftaKWepNIbZ
+1HFSIpucjukG54xlEM+jsOPweVP89P85Oskb5o3PtsiT+Cw2tAgXPiRXdxA1i1E
s5U1VPx7AFDag7SlMU+Te+eG3d206LNTbjZkTRO5KpeagZMaVa+Cq4jzvZx0riue
XTWKdtF/HNXzGXWla8Nx1w8Z9a28ZJSIGhXS0M08Od0RcESDyxiB5YTOQd6Y/TGJ
ZydNocV5piavM0ZQpoKL20ux5K4lcq0c0w+IBu0Jg2SBXMM0aM0LA+lka9QGeuI6
phO9dHuN1yPbSpeQPyyZ1SVzKtIRywb8nnhMBzpb2Z46ADt+ZS+ONQDyXCTbQUE8
YdFGtsVZ4A==
-----END CERTIFICATE-----
 1 s:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
   i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
-----BEGIN CERTIFICATE-----
MIIGEzCCA/ugAwIBAgIQfVtRJrR2uhHbdBYLvFMNpzANBgkqhkiG9w0BAQwFADCB
iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl
cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV
BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTgx
MTAyMDAwMDAwWhcNMzAxMjMxMjM1OTU5WjCBjzELMAkGA1UEBhMCR0IxGzAZBgNV
BAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEYMBYGA1UE
ChMPU2VjdGlnbyBMaW1pdGVkMTcwNQYDVQQDEy5TZWN0aWdvIFJTQSBEb21haW4g
VmFsaWRhdGlvbiBTZWN1cmUgU2VydmVyIENBMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEA1nMz1tc8INAA0hdFuNY+B6I/x0HuMjDJsGz99J/LEpgPLT+N
TQEMgg8Xf2Iu6bhIefsWg06t1zIlk7cHv7lQP6lMw0Aq6Tn/2YHKHxYyQdqAJrkj
eocgHuP/IJo8lURvh3UGkEC0MpMWCRAIIz7S3YcPb11RFGoKacVPAXJpz9OTTG0E
oKMbgn6xmrntxZ7FN3ifmgg0+1YuWMQJDgZkW7w33PGfKGioVrCSo1yfu4iYCBsk
Haswha6vsC6eep3BwEIc4gLw6uBK0u+QDrTBQBbwb4VCSmT3pDCg/r8uoydajotY
uK3DGReEY+1vVv2Dy2A0xHS+5p3b4eTlygxfFQIDAQABo4IBbjCCAWowHwYDVR0j
BBgwFoAUU3m/WqorSs9UgOHYm8Cd8rIDZsswHQYDVR0OBBYEFI2MXsRUrYrhd+mb
+ZsF4bgBjWHhMA4GA1UdDwEB/wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMB0G
A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAbBgNVHSAEFDASMAYGBFUdIAAw
CAYGZ4EMAQIBMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9jcmwudXNlcnRydXN0
LmNvbS9VU0VSVHJ1c3RSU0FDZXJ0aWZpY2F0aW9uQXV0aG9yaXR5LmNybDB2Bggr
BgEFBQcBAQRqMGgwPwYIKwYBBQUHMAKGM2h0dHA6Ly9jcnQudXNlcnRydXN0LmNv
bS9VU0VSVHJ1c3RSU0FBZGRUcnVzdENBLmNydDAlBggrBgEFBQcwAYYZaHR0cDov
L29jc3AudXNlcnRydXN0LmNvbTANBgkqhkiG9w0BAQwFAAOCAgEAMr9hvQ5Iw0/H
ukdN+Jx4GQHcEx2Ab/zDcLRSmjEzmldS+zGea6TvVKqJjUAXaPgREHzSyrHxVYbH
7rM2kYb2OVG/Rr8PoLq0935JxCo2F57kaDl6r5ROVm+yezu/Coa9zcV3HAO4OLGi
H19+24rcRki2aArPsrW04jTkZ6k4Zgle0rj8nSg6F0AnwnJOKf0hPHzPE/uWLMUx
RP0T7dWbqWlod3zu4f+k+TY4CFM5ooQ0nBnzvg6s1SQ36yOoeNDT5++SR2RiOSLv
xvcRviKFxmZEJCaOEDKNyJOuB56DPi/Z+fVGjmO+wea03KbNIaiGCpXZLoUmGv38
sbZXQm2V0TP2ORQGgkE49Y9Y3IBbpNV9lXj9p5v//cWoaasm56ekBYdbqbe4oyAL
l6lFhd2zi+WJN44pDfwGF/Y4QA5C5BIG+3vzxhFoYt/jmPQT2BVPi7Fp2RBgvGQq
6jG35LWjOhSbJuMLe/0CjraZwTiXWTb2qHSihrZe68Zk6s+go/lunrotEbaGmAhY
LcmsJWTyXnW0OMGuf1pGg+pRyrbxmRE1a6Vqe8YAsOf4vmSyrcjC8azjUeqkk+B5
yOGBQMkKW+ESPMFgKuOXwIlCypTPRpgSabuY0MLTDXJLR27lk8QyKGOHQ+SwMj4K
00u/I5sUKUErmgQfky3xxzlIPK1aEn8=
-----END CERTIFICATE-----
 2 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
   i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
-----BEGIN CERTIFICATE-----
MIIF3jCCA8agAwIBAgIQAf1tMPyjylGoG7xkDjUDLTANBgkqhkiG9w0BAQwFADCB
iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl
cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV
BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAw
MjAxMDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBiDELMAkGA1UEBhMCVVMxEzARBgNV
BAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVU
aGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBSU0EgQ2Vy
dGlmaWNhdGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK
AoICAQCAEmUXNg7D2wiz0KxXDXbtzSfTTK1Qg2HiqiBNCS1kCdzOiZ/MPans9s/B
3PHTsdZ7NygRK0faOca8Ohm0X6a9fZ2jY0K2dvKpOyuR+OJv0OwWIJAJPuLodMkY
tJHUYmTbf6MG8YgYapAiPLz+E/CHFHv25B+O1ORRxhFnRghRy4YUVD+8M/5+bJz/
Fp0YvVGONaanZshyZ9shZrHUm3gDwFA66Mzw3LyeTP6vBZY1H1dat//O+T23LLb2
VN3I5xI6Ta5MirdcmrS3ID3KfyI0rn47aGYBROcBTkZTmzNg95S+UzeQc0PzMsNT
79uq/nROacdrjGCT3sTHDN/hMq7MkztReJVni+49Vv4M0GkPGw/zJSZrM233bkf6
c0Plfg6lZrEpfDKEY1WJxA3Bk1QwGROs0303p+tdOmw1XNtB1xLaqUkL39iAigmT
Yo61Zs8liM2EuLE/pDkP2QKe6xJMlXzzawWpXhaDzLhn4ugTncxbgtNMs+1b/97l
c6wjOy0AvzVVdAlJ2ElYGn+SNuZRkg7zJn0cTRe8yexDJtC/QV9AqURE9JnnV4ee
UB9XVKg+/XRjL7FQZQnmWEIuQxpMtPAlR1n6BB6T1CZGSlCBst6+eLf8ZxXhyVeE
Hg9j1uliutZfVS7qXMYoCAQlObgOK6nyTJccBz8NUvXt7y+CDwIDAQABo0IwQDAd
BgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/BAQDAgEGMA8G
A1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEMBQADggIBAFzUfA3P9wF9QZllDHPF
Up/L+M+ZBn8b2kMVn54CVVeWFPFSPCeHlCjtHzoBN6J2/FNQwISbxmtOuowhT6KO
VWKR82kV2LyI48SqC/3vqOlLVSoGIG1VeCkZ7l8wXEskEVX/JJpuXior7gtNn3/3
ATiUFJVDBwn7YKnuHKsSjKCaXqeYalltiz8I+8jRRa8YFWSQEg9zKC7F4iRO/Fjs
8PRF/iKz6y+O0tlFYQXBl2+odnKPi4w2r78NBc5xjeambx9spnFixdjQg3IM8WcR
iQycE0xyNN+81XHfqnHd4blsjDwSXWXavVcStkNr/+XeTWYRUc+ZruwXtuhxkYze
Sf7dNXGiFSeUHM9h4ya7b6NnJSFd5t0dCy5oGzuCr+yDZ4XUmFF0sbmZgIn/f3gZ
XHlKYC6SQK5MNyosycdiyA5d9zZbyuAlJQG03RoHnHcAP9Dc1ew91Pq7P8yF1m9/
qS3fuQL39ZeatTXaw2ewh0qpKJ4jjv9cJ2vhsE/zB+4ALtRZh8tSQZXq9EfX7mRB
VXyNWQKV3WKdwrnuWih0hKWbt5DHDAff9Yk2dDLWKMGwsAvgnEzDHNb842m1R0aB
L6KCq9NjRHDEjf8tM7qtj3u1cIiuPhnPQCjY/MiQu12ZIvVS5ljFH4gxQ+6IHdfG
jjxDah2nGN59PRbxYvnKkKj9
-----END CERTIFICATE-----
 3 s:CN = *.opnsense.org
   i:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
-----BEGIN CERTIFICATE-----
MIIGMzCCBRugAwIBAgIRAKql/zLP5nCsBkX7C2AJZVQwDQYJKoZIhvcNAQELBQAw
gY8xCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE3MDUGA1UE
AxMuU2VjdGlnbyBSU0EgRG9tYWluIFZhbGlkYXRpb24gU2VjdXJlIFNlcnZlciBD
QTAeFw0yMDAxMjIwMDAwMDBaFw0yMjAxMjkyMzU5NTlaMBkxFzAVBgNVBAMMDiou
b3Buc2Vuc2Uub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0uOP
ZAoPfLMLew26Jbck9pppvQhHkWlXiz1pOV+YP446zMLJfJhoOOSSb74kf3vdXLdb
vM1Nxmdu2+G409pjfj+KUWEIN6eqZzRQ/ti4H9Xx2utuuENJ4kOrEFLWu+GH9Kfk
A36Pig20/M9PUvam6FB2y/oS3eOioCl9BkitfJsv5TC7PdaPYx5jqUmWP7Exe6pE
vvxPFU6lHlN6eGElW5BZClFtsQdt71SMOEK+fVpuKtSCpmkMxZR7VU9N50gp4u26
CWDEOSdAZEazDZCLMqLLeGr4fI0bxAp2+tjqtPccuywQzyoxTgjjhfvOY5fDHx03
FIQJ8fvSZDtyfcw/KQIDAQABo4IC/TCCAvkwHwYDVR0jBBgwFoAUjYxexFStiuF3
6Zv5mwXhuAGNYeEwHQYDVR0OBBYEFI5agxXurWZ4TCnFaLt9BJVrE3fwMA4GA1Ud
DwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggr
BgEFBQcDAjBJBgNVHSAEQjBAMDQGCysGAQQBsjEBAgIHMCUwIwYIKwYBBQUHAgEW
F2h0dHBzOi8vc2VjdGlnby5jb20vQ1BTMAgGBmeBDAECATCBhAYIKwYBBQUHAQEE
eDB2ME8GCCsGAQUFBzAChkNodHRwOi8vY3J0LnNlY3RpZ28uY29tL1NlY3RpZ29S
U0FEb21haW5WYWxpZGF0aW9uU2VjdXJlU2VydmVyQ0EuY3J0MCMGCCsGAQUFBzAB
hhdodHRwOi8vb2NzcC5zZWN0aWdvLmNvbTAnBgNVHREEIDAegg4qLm9wbnNlbnNl
Lm9yZ4IMb3Buc2Vuc2Uub3JnMIIBfQYKKwYBBAHWeQIEAgSCAW0EggFpAWcAdgBG
pVXrdfqRIDC1oolp9PN9ESxBdL79SbiFq/L8cP5tRwAAAW/OOTUcAAAEAwBHMEUC
IQCqpQuVnJWhzFFonCzybDP+Dfm3VUQJ/OiLP7a9I3eO7AIgP0dyeUsY0SBavzy1
l+uXu/xbcumKSKz4MbYBI0S9TUkAdgBvU3asMfAxGdiZAKRRFf93FRwR2QLBACkG
jbIImjfZEwAAAW/OOTVpAAAEAwBHMEUCIQCv5mW+Y/ptu+eU4mrA/BYg8AzH2ex3
kSy+FBXQE9K/GAIgJJOvwRuu1YP24A++Peis50x4H80dG92gYbX7oC76KUkAdQAi
RUUHWVUkVpY/oS/x922G4CMmY63AS39dxoNcbuIPAgAAAW/OOTULAAAEAwBGMEQC
IBSdWWClPXY7OPNOLkGIdPPrIXlX5cWq0j+cF2qgd/6BAiBX9JHarLhnS6ApPfYm
22nR2s3PILUQFZYCluVdBkJV4jANBgkqhkiG9w0BAQsFAAOCAQEAKftaKWepNIbZ
+1HFSIpucjukG54xlEM+jsOPweVP89P85Oskb5o3PtsiT+Cw2tAgXPiRXdxA1i1E
s5U1VPx7AFDag7SlMU+Te+eG3d206LNTbjZkTRO5KpeagZMaVa+Cq4jzvZx0riue
XTWKdtF/HNXzGXWla8Nx1w8Z9a28ZJSIGhXS0M08Od0RcESDyxiB5YTOQd6Y/TGJ
ZydNocV5piavM0ZQpoKL20ux5K4lcq0c0w+IBu0Jg2SBXMM0aM0LA+lka9QGeuI6
phO9dHuN1yPbSpeQPyyZ1SVzKtIRywb8nnhMBzpb2Z46ADt+ZS+ONQDyXCTbQUE8
YdFGtsVZ4A==
-----END CERTIFICATE-----
---
Server certificate
subject=CN = *.opnsense.org

issuer=C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 6925 bytes and written 393 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
    Session-ID: 4FD5FC0C24454A7799ACBEA546CF350C8F2BBF4084410D9E232E0908038D0EF7
    Session-ID-ctx:
    Master-Key: DDCFC5E40810A4CA39C66382D6DB3767BEEE1C41D805AA8DFD8478B5504679CD5BBA0B3C5AEC0D4F3694B23B7A4F5141
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 63 b7 9d 71 18 42 88 62-e5 51 46 c7 db de 4f f3   c..q.B.b.QF...O.
    0010 - b3 0e 27 66 65 03 be a2-35 fd 3f d8 3c fb b8 75   ..'fe...5.?.<..u
    0020 - ae 07 88 96 ab 4f 26 a3-67 cb 4d d5 62 3c e0 74   .....O&.g.M.b<.t
    0030 - 73 a3 47 9f 6f 16 9d 30-44 17 26 d0 24 8c 69 91   s.G.o..0D.&.$.i.
    0040 - 95 c9 94 41 10 33 0f 53-02 e2 37 f2 fb 20 37 1e   ...A.3.S..7.. 7.
    0050 - a1 3f f1 fa 2a 26 3c 72-7e bb 0a 99 1a e5 50 ba   .?..*&<r~.....P.
    0060 - 3c 4b 5c 4c ab f2 ff ac-5c 16 b1 8b 4a c8 9c e1   <K\L....\...J...
    0070 - 50 0b 13 ce ea f3 82 14-8d ac 9c e7 b5 45 7d ee   P............E}.
    0080 - 35 28 df a3 7f f8 31 38-a1 90 3e 54 c0 05 96 2b   5(....18..>T...+
    0090 - 47 b1 48 d4 31 fc 61 19-b9 0d 7c d2 52 b6 5b fe   G.H.1.a...|.R.[.
    00a0 - 71 78 c4 81 ee 8b 18 eb-19 43 b3 ce 4f ad 84 ac   qx.......C..O...
    00b0 - bd fc 01 bd 2d 61 93 e4-e3 62 07 b2 0e b2 22 18   ....-a...b....".
    00c0 - b2 eb f9 fc fb 63 8c 2f-b1 92 35 cc d9 52 1a 6c   .....c./..5..R.l

    Start Time: 1633093579
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

Renewing their Let's Encrypt Certificate using the new Chain (not the android-compatible cross-signed one) would've worked too, but using a completely different Authority should fix this, too, of course.
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: evox on October 01, 2021, 03:19:45 pm
OPNsense repo is working now, that is nice.

What an issue this has been  ::)
Title: Re: opnsense using wrong letsencrypt R3 intermediate certificate
Post by: bill.gertz on October 01, 2021, 03:27:55 pm
@chemlud and Everyone,

Found my, self-inflicted, issue!  :D

Had my own copy of acme.sh installed as /root/.acme.sh. As I was the developer for the acme.sh DNS01 MailinaBox DNSAPI, I used this copy during development of both the DNSAPI and the OPNsense glue code and content. Looks like that old configuration was being used instead of the OPNsense configured acme.sh.

Oh well, no good deed goes unpunished. Simply deleted this old directory at /root/.acme.sh and everything was right with the world.

For everyone else having issues after updating the Authority Certs to include a Fullchain Cert, then only to find the problem is resurrected after a reboot:

Check for and kill any stray copies of acme.sh you find and verify the Let's Encrypt config through OPNsense Web GUI.