DoT dns resolution stopped working - Related to Let's Encrypt Root CA Expiry

[thingy]

Recently (Sept 30th 2021), a Let's Encrypt root CA expired.

The DST Root CA X3 expired on September 30, 2021.

See: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

Ever since then, the Unbound DNS service on my OPNsense device (OPNsense 21.7.3_3-amd64, FreeBSD 12.1-RELEASE-p20-HBSD, OpenSSL 1.1.1l 24 Aug 2021) fails to resolve IP addresses because I have the following DNS over TLS servers configured:

• 91.239.100.100:853 - anycast.censurfridns.dk
• 145.100.185.15:853 - dnsovertls.sinodun.com
• 145.100.185.16:853 - dnsovertls1.sinodun.com
• 185.49.141.37:853 - getdnsapi.net

The error msg I see in Unbound's logs is:

Code: [Select]

2021-10-02T23:11:30 unbound[38176] [38176:0] error: ssl handshake failed crypto error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
2021-10-02T23:11:30 unbound[38176] [38176:0] notice: ssl handshake failed 91.239.100.100 port 853

I can reproduce this error on my Linux desktop (openSUSE Tumbleweed) by trying to use a dns utility called dog (https://github.com/ogham/dog):

Code: [Select]

:~> dog google.com --tls @91.239.100.100:853
Error [tls]: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (IP address mismatch)

The same utility shows that DoT works fine for other DoT providers e.g. quad 9

Code: [Select]

:~> dog google.com --tls @9.9.9.9:853
A google.com. 1m58s   142.250.187.238

So it is definitely related to a Let's Encrypt cert issue since all of the DoT servers I've configured use a Let's Encrypt certificate.

What I'm struggling to understand is why and is it something I need to workaround on my end or should I be contacting the server admins of the above services and informing them.

On the OPNsense device, when running:

Code: [Select]

echo | openssl s_client -connect 91.239.100.100:853 --showcerts

...I get the following:

Code: [Select]

---
Certificate chain
 0 s:CN = rgnet-iad.anycast.censurfridns.dk
   i:C = US, O = Let's Encrypt, CN = R3
-----BEGIN CERTIFICATE-----
MIIFcTCCBFmgAwIBAgISBGl25rZca1TDey9ke6sjhSuDMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA4MjYyMDUwNTZaFw0yMTExMjQyMDUwNTVaMCwxKjAoBgNVBAMT
IXJnbmV0LWlhZC5hbnljYXN0LmNlbnN1cmZyaWRucy5kazB2MBAGByqGSM49AgEG
BSuBBAAiA2IABCm2DNXRZevNbLyJJSuJ9JJqlfegxFF1Lv5lEiG6+EjW7yBpkEtF
O7f3Uj8QyYLKdOJkBzc4E84SW4nFiZm1apNdqTvyWRWVdN5U6at/Kb6nHmD2tuXK
gKuQmIBolnlEg6OCAzMwggMvMA4GA1UdDwEB/wQEAwIHgDAdBgNVHSUEFjAUBggr
BgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUqlt0IKeu
XrYLQaQO8XkgELZB8J0wHwYDVR0jBBgwFoAUFC6zF7dYVsuuUAlA5h+vnYsUwsYw
VQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVodHRwOi8vcjMuby5sZW5jci5v
cmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxlbmNyLm9yZy8wggEBBgNVHREE
gfkwgfaCF2FueWNhc3QuY2Vuc3VyZnJpZG5zLmRrghdhbnljYXN0LmNlbnN1cmZy
aWRucy5udYIYYW55Y2FzdC51bmNlbnNvcmVkZG5zLmRrghlhbnljYXN0LnVuY2Vu
c29yZWRkbnMub3JngiFyZ25ldC1pYWQuYW55Y2FzdC5jZW5zdXJmcmlkbnMuZGuC
IXJnbmV0LWlhZC5hbnljYXN0LmNlbnN1cmZyaWRucy5udYIicmduZXQtaWFkLmFu
eWNhc3QudW5jZW5zb3JlZGRucy5ka4IjcmduZXQtaWFkLmFueWNhc3QudW5jZW5z
b3JlZGRucy5vcmcwTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEw
KDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEEBgor
BgEEAdZ5AgQCBIH1BIHyAPAAdgD2XJQv0XcwIhRUGAgwlFaO400TGTO/3wwvIAvM
TvFk4wAAAXuEcdDQAAAEAwBHMEUCIQDayjoq3M4yWH4VKgpm4XHyo94BMbURJsO4
xWjOGvrybQIgFDw/p2p7xcyw88TKHWLPXemNaGpL9Hx8opTzoZw6hb8AdgCUILwe
jtWNbIhzH4KLIiwN0dpNXmxPlD1h204vWE2iwgAAAXuEcdDVAAAEAwBHMEUCIQCu
/BwR/jMGDIJ6g18ebi70xMrf016MjqUl/ztKQv0pmwIgQ3d2xYM/PxFu69e6/kTk
l7rSJAVfCU3s+CkrIymIGw8wDQYJKoZIhvcNAQELBQADggEBAFbWt5P7+1XFRXQl
k5JGaKK03Zu8bQZMCnidU5sylEQj4tFA+S7KMd6xPkGwG7Ee+ySmzObUW4sdbY2T
P5n5T/mIQ/yZmVS+fcwdFPNtSXTzViRPuhHWmJU4i/KoR8kDp5lBdUVeRMdw8tp8
ZmLlM2BpNhojRvfSRQ0KHVt93AenLa2phBoSsm+WkgkP4O2McUyY/P8cIKqNCS+x
oPjIbuRuC8bYfNVZC1R9xvhq+rk57J/DXZT6fKwYZkoV3UkXK9YjkAcAaDAdo6Bl
rNYi1sg3scxt1hJlWUz3VMizEj7aSBxMhtObp2jGSRsvhvbg2IUAxVt5+dAHRMUS
B4G8Y4M=
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----
---
Server certificate
subject=CN = rgnet-iad.anycast.censurfridns.dk

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA384
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4496 bytes and written 375 bytes
Verification error: certificate has expired
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 384 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 10 (certificate has expired)
---


Notice that openssl does report: "Verification error: certificate has expired" even though all 3 certs visible above have not expired.

Running the same command on my Linux desktop does not return that error and I get the same certs being output as above.

On Linux (OpenSSL 1.1.1l  24 Aug 2021) I get this instead:

Code: [Select]

.
.
<same certs as above snipped>
.
.
Server certificate
subject=CN = rgnet-iad.anycast.censurfridns.dk

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA384
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4495 bytes and written 381 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 384 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Installing bash + os-git-backup plugin on the OPNsense device and then running:

Code: [Select]

git clone --depth 1 https://github.com/drwetter/testssl.sh.git
cd testssl.sh
mount -t fdescfs fdesc /dev/fd
./testssl.sh --fast 91.239.100.100:853

results in the following output from testssl.sh:

Code: [Select]

.
.
.
  Server Certificate #1 (in response to request w/o SNI)
   Signature Algorithm          SHA256 with RSA
   Server key size              RSA 4096 bits (exponent is 65537)
   Server key usage             Digital Signature, Key Encipherment
   Server extended key usage    TLS Web Server Authentication, TLS Web Client Authentication
   Serial / Fingerprints        031A211648897F4AB8464EB2CA7356E3AC3F / SHA1 436C8F12BD9784FABD951D036311396C647D1524
                                SHA256 F9EAADFD781155620C1295C34E2E504E574597E77451D3EDF2C04D43E54B575F
   Common Name (CN)             rgnet-iad.anycast.censurfridns.dk
   subjectAltName (SAN)         anycast.censurfridns.dk anycast.censurfridns.nu anycast.uncensoreddns.dk anycast.uncensoreddns.org rgnet-iad.anycast.censurfridns.dk
                                rgnet-iad.anycast.censurfridns.nu rgnet-iad.anycast.uncensoreddns.dk rgnet-iad.anycast.uncensoreddns.org
   Trust (hostname)             certificate does not match supplied URI
   Chain of trust               Ok   
   EV cert (experimental)       no
   Certificate Validity (UTC)   52 >= 30 days (2021-08-26 20:50 --> 2021-11-24 20:50)
   ETS/"eTLS", visibility info  not present
   Certificate Revocation List  --
   OCSP URI                     http://r3.o.lencr.org
   OCSP stapling                offered, not revoked
   OCSP must staple extension   --
   DNS CAA RR (experimental)    not offered
   Certificate Transparency     yes (certificate extension)
   Certificates provided        3
   Issuer                       R3 (Let's Encrypt from US)
   Intermediate cert validity   #1: ok > 40 days (2025-09-15 16:00). R3 <-- ISRG Root X1
                                  #2: ok > 40 days (2024-09-30 18:14). ISRG Root X1 <-- DST Root CA X3
   Intermediate Bad OCSP (exp.) Ok

  Server Certificate #2 (in response to request w/o SNI)
   Signature Algorithm          SHA256 with RSA
   Server key size              EC 384 bits (curve P-384)
   Server key usage             Digital Signature
   Server extended key usage    TLS Web Server Authentication, TLS Web Client Authentication
   Serial / Fingerprints        046976E6B65C6B54C37B2F647BAB23852B83 / SHA1 A0BC6997ED2AD84F1B9494FC5398517184267637
                                SHA256 060B69729CE6E55915EE342023756AA7FD26CE2EC11462525FD35F1E8CD30A5C
   Common Name (CN)             rgnet-iad.anycast.censurfridns.dk
   subjectAltName (SAN)         anycast.censurfridns.dk anycast.censurfridns.nu anycast.uncensoreddns.dk anycast.uncensoreddns.org rgnet-iad.anycast.censurfridns.dk
                                rgnet-iad.anycast.censurfridns.nu rgnet-iad.anycast.uncensoreddns.dk rgnet-iad.anycast.uncensoreddns.org
   Trust (hostname)             certificate does not match supplied URI
   Chain of trust               Ok   
   EV cert (experimental)       no
   Certificate Validity (UTC)   52 >= 30 days (2021-08-26 20:50 --> 2021-11-24 20:50)
   ETS/"eTLS", visibility info  not present
   Certificate Revocation List  --
   OCSP URI                     http://r3.o.lencr.org
   OCSP stapling                offered, not revoked
   OCSP must staple extension   --
   DNS CAA RR (experimental)    not offered
   Certificate Transparency     yes (certificate extension)
   Certificates provided        3
   Issuer                       R3 (Let's Encrypt from US)
   Intermediate cert validity   #1: ok > 40 days (2025-09-15 16:00). R3 <-- ISRG Root X1
                                  #2: ok > 40 days (2024-09-30 18:14). ISRG Root X1 <-- DST Root CA X3
   Intermediate Bad OCSP (exp.) Ok
.
.
.


which also confirms that the chain of trust seems to be ok.

I am at a loss to understand why DoT is failing and what to do about it.

Is anyone else experiencing this?

Notes:
- I have checked for and installed all available OPNsense updates.
- I don't wish to use a privacy averse DoT provider like Cloudflare or Quad9 et al so that is not an option.

[chemlud]

...add some of those servers

https://www.privacy-handbuch.de/handbuch_93d.htm

always good to have choices...

[thingy]

I did try the list of servers in the handbook you provided. Most of them used Let's Encrypt certs and so didn't work.

I raise a Github issue for this matter and there is a workaround: https://github.com/opnsense/core/issues/5257#issuecomment-933668219

[chemlud]

...I'm using LibreSSL flavor, no idea if that's reason, but here no problems any more...