DNS Over TLS Broken

[phantomsfbw]

After the 11/2024 update, cannot use DNS Over TLS.  Using Quad9 and IPV4 only.  Worked fine before update.  No access to Internet if turned on.  If turned off, access is fine.  Here are the errors from the DNS/TLS log:

2024-11-20T17:26:26-05:00   Error   unbound   [95068:5] error: ssl handshake cert error: unable to get local issuer certificate   
2024-11-20T17:26:26-05:00   Error   unbound   [95068:5] error: and additionally crypto error:0A000086:SSL routines::certificate verify failed   
2024-11-20T17:26:26-05:00   Error   unbound   [95068:5] error: and additionally crypto error:80000002:system library::No such file or directory   
2024-11-20T17:26:26-05:00   Error   unbound   [95068:5] error: and additionally crypto error:16000069:STORE routines::unregistered scheme

[newsense]

Can you post the output of this command:

Code Select Expand

unbound-anchor -vF

[phantomsfbw]

/usr/local/etc/unbound/root.key does not exist
debug cert update forced
last successful probe: Wed Nov 20 21:33:29 2024
the last successful probe is recent
/usr/local/etc/unbound/icannbundle.pem: No such file or directory
using builtin certificate
have 1 trusted certificates
resolved server address 152.199.24.38
resolved server address 2606:2800:21f:b505:516b:4186:98cd:116
connect to 152.199.24.38
fetched root-anchors/root-anchors.xml (1861 bytes)
connect to 152.199.24.38
fetched root-anchors/root-anchors.p7s (2523 bytes)
signer 0: Subject: /O=ICANN/CN=DNSSEC Trust Anchor Verification/emailAddress=dnssec@iana.org
the PKCS7 signature verified
XML was parsed successfully, 2 keys
success: the anchor has been updated using the cert

[newsense]

So...service operational now ?

[phantomsfbw]

Thank you for the reply and recommendation.  Ran and rebooted.  However, still no DNS over TLS. Log from latest attempt.

2024-11-21T13:24:15-05:00   Informational   unbound   [40958:d] info: 10.0.0.216 lechmere-v1.sslauth.sonos.com.phantom.net. A IN   
2024-11-21T13:24:15-05:00   Informational   unbound   [40958:11] info: resolving lechmere-v1.sslauth.sonos.com.phantom.net. A IN   
2024-11-21T13:24:15-05:00   Informational   unbound   [40958:11] info: 10.0.0.216 lechmere-v1.sslauth.sonos.com.phantom.net. A IN   
2024-11-21T13:24:15-05:00   Informational   unbound   [40958:5] info: 10.0.0.216 lechmere-v1.sslauth.sonos.com. A IN   
2024-11-21T13:24:15-05:00   Informational   unbound   [40958:12] info: resolving lechmere-v1.sslauth.sonos.com. A IN   
2024-11-21T13:24:15-05:00   Informational   unbound   [40958:12] info: 10.0.0.216 lechmere-v1.sslauth.sonos.com. A IN   
2024-11-21T13:24:14-05:00   Notice   unbound   [40958:14] notice: ssl handshake failed 9.9.9.9 port 853   
2024-11-21T13:24:14-05:00   Error   unbound   [40958:14] error: ssl handshake cert error: unable to get local issuer certificate   
2024-11-21T13:24:14-05:00   Error   unbound   [40958:14] error: and additionally crypto error:0A000086:SSL routines::certificate verify failed   
2024-11-21T13:24:14-05:00   Error   unbound   [40958:14] error: and additionally crypto error:80000002:system library::No such file or directory   
2024-11-21T13:24:14-05:00   Error   unbound   [40958:14] error: and additionally crypto error:16000069:STORE routines::unregistered scheme   
2024-11-21T13:24:14-05:00   Error   unbound   [40958:14] error: and additionally crypto error:80000002:system library::No such file or directory   
2024-11-21T13:24:14-05:00   Error   unbound   [40958:14] error: and additionally crypto error:16000069:STORE routines::unregistered scheme   
2024-11-21T13:24:14-05:00   Error   unbound   [40958:14] error: and additionally crypto error:80000002:system library::No such file or directory   
2024-11-21T13:24:14-05:00   Error   unbound   [40958:14] error: ssl handshake failed crypto error:16000069:STORE routines::unregistered scheme   
2024-11-21T13:24:14-05:00   Notice   unbound   [40958:14] notice: ssl handshake failed 9.9.9.9 port 853   
2024-11-21T13:24:14-05:00   Error   unbound   [40958:14] error: ssl handshake cert error: unable to get local issuer certificate   
2024-11-21T13:24:14-05:00   Error   unbound   [40958:14] error: and additionally crypto error:0A000086:SSL routines::certificate verify failed

[newsense]

Ho does your DoT config look like ?

[phantomsfbw]

This is a new install on bare metal.  I had the previous version running without issue until this latest upgrade.  Here are the contents of the DoT:

Custom forwarding
          9.9.9.9                   853   dns.quad9.net   Quad9 Primary IPV4   
          149.112.112.112   853   dns.quad9.net   Quad9 Alternate IPV4   

Domain in blank
Not running IPV6
   

[newsense]

Try this

Code Select Expand

pkg install -f unbound

[phantomsfbw]

Ran the pkg install and it showed reinstalling unbound-1.22.0_1.  Reinstall completed without conflict.  Rebooted and then enabled DoT, and still does not work.  Thank you for the troubleshooting assistance.

Here is the log data for this attempt:

2024-11-21T20:21:09-05:00   Informational   unbound   [37225:16] info: 10.0.0.42 linuxconfig.org. HTTPS IN   
2024-11-21T20:21:09-05:00   Informational   unbound   [37225:16] info: 10.0.0.42 linuxconfig.org. HTTPS IN   
2024-11-21T20:21:09-05:00   Notice   unbound   [37225:16] notice: ssl handshake failed 9.9.9.9 port 853   
2024-11-21T20:21:09-05:00   Error   unbound   [37225:16] error: ssl handshake cert error: unable to get local issuer certificate   
2024-11-21T20:21:09-05:00   Error   unbound   [37225:16] error: and additionally crypto error:0A000086:SSL routines::certificate verify failed   
2024-11-21T20:21:09-05:00   Error   unbound   [37225:16] error: and additionally crypto error:80000002:system library::No such file or directory   
2024-11-21T20:21:09-05:00   Error   unbound   [37225:16] error: and additionally crypto error:16000069:STORE routines::unregistered scheme   
2024-11-21T20:21:09-05:00   Error   unbound   [37225:16] error: and additionally crypto error:80000002:system library::No such file or directory   
2024-11-21T20:21:09-05:00   Error   unbound   [37225:16] error: and additionally crypto error:16000069:STORE routines::unregistered scheme   
2024-11-21T20:21:09-05:00   Error   unbound   [37225:16] error: and additionally crypto error:80000002:system library::No such file or directory   
2024-11-21T20:21:09-05:00   Error   unbound   [37225:16] error: ssl handshake failed crypto error:16000069:STORE routines::unregistered scheme

[dave14305]

Can you successfully connect via router command line?

Code Select Expand

openssl s_client --connect 9.9.9.9 --port 853

[phantomsfbw]

I don't think so...

CONNECTED(00000003)
Can't use SSL_get_servername
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G3
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert Global G3 TLS ECC SHA384 2020 CA1
verify return:1
depth=0 C = CH, ST = Zurich, L = Zurich, O = Quad9, CN = dns.quad9.net
verify return:1
---
Certificate chain
0 s:C = CH, ST = Zurich, L = Zurich, O = Quad9, CN = dns.quad9.net
   i:C = US, O = DigiCert Inc, CN = DigiCert Global G3 TLS ECC SHA384 2020 CA1
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Jul 17 00:00:00 2024 GMT; NotAfter: Jul 16 23:59:59 2025 GMT
1 s:C = US, O = DigiCert Inc, CN = DigiCert Global G3 TLS ECC SHA384 2020 CA1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G3
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Apr 14 00:00:00 2021 GMT; NotAfter: Apr 13 23:59:59 2031 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIHzzCCB1SgAwIBAgIQBBnWyhHgw5zKXJpekiQtkTAKBggqhkjOPQQDAzBZMQsw
CQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMTMwMQYDVQQDEypEaWdp
Q2VydCBHbG9iYWwgRzMgVExTIEVDQyBTSEEzODQgMjAyMCBDQTEwHhcNMjQwNzE3
MDAwMDAwWhcNMjUwNzE2MjM1OTU5WjBXMQswCQYDVQQGEwJDSDEPMA0GA1UECBMG
WnVyaWNoMQ8wDQYDVQQHEwZadXJpY2gxDjAMBgNVBAoTBVF1YWQ5MRYwFAYDVQQD
Ew1kbnMucXVhZDkubmV0MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEHS9/A9uo
8RaaOnVagQbRQPT8PMKdgD2JpTwW4olcwggo7KnvZb2NEJszVxMM9SbfYmWzTCRU
/qwdFBbpOBkXeKOCBf4wggX6MB8GA1UdIwQYMBaAFIoj655r1/k3XfltITl2mqFn
3hCoMB0GA1UdDgQWBBTlfUHsSeB6i0VXayGzIb3LOxlPbjCCAooGA1UdEQSCAoEw
ggJ9gg1kbnMucXVhZDkubmV0ghNkbnMtbm9zZWMucXVhZDkubmV0ghNkb2gtYnJh
dmUucXVhZDkubmV0ghZkbnMucmVzb2x2ZXIucXVhZDkubmV0ghNhbHBoYS1kbnMu
cXVhZDkubmV0ghJiZXRhLWRucy5xdWFkOS5uZXSCDmRuczkucXVhZDkubmV0gg9k
bnMxMC5xdWFkOS5uZXSCD2RuczExLnF1YWQ5Lm5ldIIPZG5zMTIucXVhZDkubmV0
gg9kbnMxMy5xdWFkOS5uZXSCD2RuczE0LnF1YWQ5Lm5ldIIPZG5zMTUucXVhZDku
bmV0ghBkbnMyNTQucXVhZDkubmV0ghFtb3ppbGxhLnF1YWQ5Lm5ldIcECQkJCYcE
CQkJCocECQkJC4cECQkJDIcECQkJDYcECQkJDocECQkJD4cElXBwCYcElXBwCocE
lXBwC4cElXBwDIcElXBwDYcElXBwDocElXBwD4cElXBwcIcQJiAA/gAAAAAAAAAA
AAAA/ocQJiAA/gAAAAAAAAAAAAAACYcQJiAA/gAAAAAAAAAAAAAAEIcQJiAA/gAA
AAAAAAAAAAAAEYcQJiAA/gAAAAAAAAAAAAAAEocQJiAA/gAAAAAAAAAAAAAAE4cQ
JiAA/gAAAAAAAAAAAAAAFIcQJiAA/gAAAAAAAAAAAAAAFYcQJiAA/gAAAAAAAAAA
AP4ACYcQJiAA/gAAAAAAAAAAAP4AEIcQJiAA/gAAAAAAAAAAAP4AEYcQJiAA/gAA
AAAAAAAAAP4AEocQJiAA/gAAAAAAAAAAAP4AE4cQJiAA/gAAAAAAAAAAAP4AFIcQ
JiAA/gAAAAAAAAAAAP4AFTA+BgNVHSAENzA1MDMGBmeBDAECAjApMCcGCCsGAQUF
BwIBFhtodHRwOi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMwDgYDVR0PAQH/BAQDAgOI
MB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjCBnwYDVR0fBIGXMIGUMEig
RqBEhkJodHRwOi8vY3JsMy5kaWdpY2VydC5jb20vRGlnaUNlcnRHbG9iYWxHM1RM
U0VDQ1NIQTM4NDIwMjBDQTEtMi5jcmwwSKBGoESGQmh0dHA6Ly9jcmw0LmRpZ2lj
ZXJ0LmNvbS9EaWdpQ2VydEdsb2JhbEczVExTRUNDU0hBMzg0MjAyMENBMS0yLmNy
bDCBhwYIKwYBBQUHAQEEezB5MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdp
Y2VydC5jb20wUQYIKwYBBQUHMAKGRWh0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNv
bS9EaWdpQ2VydEdsb2JhbEczVExTRUNDU0hBMzg0MjAyMENBMS0yLmNydDAMBgNV
HRMBAf8EAjAAMIIBfwYKKwYBBAHWeQIEAgSCAW8EggFrAWkAdgAS8U40vVNyTIQG
GcOPP3oT+Oe1YoeInG0wBYTr5YYmOgAAAZDBgPPmAAAEAwBHMEUCIQCFAt7DtP7G
lzx1LiBAS/rqbu8d8cVLkrPN+6Q5AIy4QgIgSZj0NWqKU7x5h9Qs5NyBNEIYSREI
P6OWICpWWGAq0RcAdgB9WR4S4XgqexxhZ3xe/fjQh1wUoE6VnrkDL9kOjC55uAAA
AZDBgPPjAAAEAwBHMEUCIQCRO0OuLSJ+duThEJZmCFpuE51c4UUyRpHucg1KtGj3
tgIgfyGM79OjDqH87OzCN/cAd96he3OA0A5/twT7xMtLdn0AdwDm0jFjQHeMwRBB
Btdxuc7B0kD2loSG+7qHMh39HjeOUAAAAZDBgPPtAAAEAwBIMEYCIQC+/HxmsLPE
0tmEOWbrzDH2EjDm//MtP/wPjbe+OkvpiwIhAK1htztI7x5tnTf0QJ3J5zwZJBkZ
R/YQUR9qCfnWA4CqMAoGCCqGSM49BAMDA2kAMGYCMQDSQm4wWeUwG6TQQzs1n0P1
gVqnhZy0lHRRwZ+wvIdoNYhCB52ZEqFLHSiBIbQ0QUACMQCMk8pKlRxHgCfkOdz4
n2OoZQJBzZZL08UO2FbLf+IoCHpOdngPrmHt/hKo8YSUMDg=
-----END CERTIFICATE-----
subject=C = CH, ST = Zurich, L = Zurich, O = Quad9, CN = dns.quad9.net
issuer=C = US, O = DigiCert Inc, CN = DigiCert Global G3 TLS ECC SHA384 2020 CA1
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3271 bytes and written 377 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 812C87A07C8B24011BE622AED9DA212E6553DFDF99E5845A51F93FA89A2C85C0
    Session-ID-ctx:
    Resumption PSK: 5A5B534B7545D9EB4740EC808A296410DB5E44E79459982BD6BC486C604C825477DF9A9100D1F5C91F37FD4BC1DC0C99
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 45 67 f2 f8 42 c5 8b e0-f1 e2 79 51 9c d6 2c 54   Eg..B.....yQ..,T
    0010 - 72 d8 2d 96 b4 17 56 94-0a 23 8a 73 63 3d d2 2d   r.-...V..#.sc=.-
    0020 - 29 a1 b5 1d 40 a0 04 53-3d 24 70 f0 41 29 ab ce   )...@..S=$p.A)..
    0030 - 4c 20 ca 0f 05 3e f1 3e-94 34 74 3e 61 0c 86 8b   L ...>.>.4t>a...
    0040 - 45 59 5c 9f d8 c4 2c 94-d7 0e e8 e3 dc 67 a5 70   EY\...,......g.p
    0050 - c4 c9 06 f1 64 c3 bc 22-68 3d a1 74 b4 ef 32 d0   ....d.."h=.t..2.
    0060 - 20 8a f9 08 f9 ce 2f fe-3c 04 07 70 46 8d 2e 91    ...../.<..pF...
    0070 - 72 5b d7 90 cb 1e 96 b7-bd 00 64 7a e2 e8 83 f0   r[........dz....
    0080 - c5 a7 59 51 76 b6 fe 53-9b c0 10 0a c1 11 0e 8b   ..YQv..S........
    0090 - cc f5 60 d2 8b ae 0e 90-8d 14 bd d8 45 e8 37 42   ..`.........E.7B
    00a0 - ae 5e c5 78 18 a9 17 83-01 64 77 5c 02 f4 16 e0   .^.x.....dw\....
    00b0 - 2e 21 09 a5 8a 3a c4 3e-95 67 59 e0 48 1b 61 09   .!...:.>.gY.H.a.

    Start Time: 1732297173
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 0F0CB8FB01CA3BC29AB7E43BE6A28B46560E2981C09698C3DFDEF049AEC6392B
    Session-ID-ctx:
    Resumption PSK: A6FD458C139924F01D83E521136022B908B7AC1B4C1CDDB7F4DDA8BF0CB19970B45436A8FB4FF27FD1FB8AD4ED197F89
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 45 67 f2 f8 42 c5 8b e0-f1 e2 79 51 9c d6 2c 54   Eg..B.....yQ..,T
    0010 - 4f da cb 1e 73 8d ab f3-7b d8 75 ba aa be d3 c3   O...s...{.u.....
    0020 - ba f8 ae 47 91 85 2a 49-d5 ba 81 46 b5 0e 0e 37   ...G..*I...F...7
    0030 - 44 76 f1 89 69 0d 73 d2-d0 44 2b 86 3a 24 f6 6c   Dv..i.s..D+.:$.l
    0040 - b0 84 f6 b9 42 45 d6 7f-dd 38 9e 58 fc c8 25 15   ....BE...8.X..%.
    0050 - 43 4f 3c e9 90 08 97 82-00 c4 c6 98 1b 02 d5 6b   CO<............k
    0060 - 60 df 54 92 51 eb ea 85-d6 55 99 79 4a 8d 34 64   `.T.Q....U.yJ.4d
    0070 - c9 3c 26 12 7b bc bc a4-20 d9 d9 f4 9d 4a f1 7c   .<&.{... ....J.|
    0080 - d4 60 68 d9 5b 51 62 c8-61 fa 40 0c 05 c6 e5 d5   .`h.[Qb.a.@.....
    0090 - b3 58 d6 2b 75 ec a3 44-ca 1e 8c 12 2e ca 51 0d   .X.+u..D......Q.
    00a0 - 54 62 8e 60 38 e2 f7 e2-b9 6f 17 cc 71 58 cd 98   Tb.`8....o..qX..
    00b0 - 2d 44 68 f4 8f 95 61 5c-8a 08 47 08 89 c9 cd 30   -Dh...a\..G....0

    Start Time: 1732297173
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed

[meyergru]

DoT works for me - also with Quad9.

Your dump of the local connection looks fine - exactly like mine. So, if your unbound cannot handle the SSL connection with a "error: ssl handshake cert error: unable to get local issuer certificate" message, it seems that its certificate chain is off.

I would think that something in your trust settings must be off, although I do not see why the console would work and unbound does not.

I would check system health if there are altered files or a defective file system.

[phantomsfbw]

Given it maybe a certificate issue, I will scrub the drive and reinstall.  Thanks to all for the assistance!

[chemlud]

I would never use DoT with less than 4-5 servers configured...

[Patrick M. Hausen]

I would never use DoT.

Fixed it for you  ;D