Unbound SERVFAIL -> DNSCrypt 127.0.0.1:5353 (and other forwarders)

[DanAnimal]

I have been having no luck setting any Forwarders for Unbound. Both Query Forwarding and DNS over TLS.
(Setting nothing and using the unsecured system nameserver from WAN's upstream DHCP works)
So am now trying to go via DNScrypt. Which still fails.

Here is the settings and results for dig on both DNSCrypt on port 5353 and Unbound on port 53 tested on the OPNSense host:

Code: [Select]

root@uribou:~ # cat /usr/local/etc/unbound.opnsense.d/dot.conf
server:
  do-not-query-localhost: no

# Forward zones
forward-zone:
  name: "."
  forward-addr: 127.0.0.1@5353

root@uribou:~ # dig opnsense.org @127.0.0.1 -p 53

; <<>> DiG 9.18.16 <<>> opnsense.org @127.0.0.1 -p 53
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 4928
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;opnsense.org. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 16:25:22 JST 2023
;; MSG SIZE  rcvd: 41

root@uribou:~ # dig opnsense.org @127.0.0.1 -p 5353

; <<>> DiG 9.18.16 <<>> opnsense.org @127.0.0.1 -p 5353
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46478
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;opnsense.org. IN A

;; ANSWER SECTION:
opnsense.org. 899 IN A 178.162.131.118

;; Query time: 38 msec
;; SERVER: 127.0.0.1#5353(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 16:25:31 JST 2023
;; MSG SIZE  rcvd: 57

logs show:

Code: [Select]

error: SERVFAIL <opnsense.org. A IN>: all the configured stub or forward servers failed, at zone . no server to query nameserver addresses not usable have no nameserver names
This is using OPNsense 23.1.11-amd64

Feel I am going in circles.
Is there something obvious I am missing?
Is there an simple way to reset Unbound to installed defaults so I can try beginning again?

[cookiemonster]

Those test results are what I would expect to see. DoT is saying Unbound listens on loopback port 5353 and then your dig test on it gave a normal result.
I suggest leaving any forwarding alone first and get the "normal" Unbound working.
Leave it as plain resolver and restart. Check with netstat/sockstat that is listening and the desired port. Check from the firewall and from a client.
Then and only then, set DoT and repeat checks.
Only after this point, move to forwarding if necessary.
That's how I would approach it.

[DanAnimal]

Unbound is listening on port 53
DNSCrypt is listening on port 5353

Isn't the dot.conf is reflecting that I have one forwarder set to 127.0.0.1:5353 (screenshot attached).

DNSCrypt is responding on port 5353

[DanAnimal]

sockstat shows Unbound listening on port 53 and DNSCrypt listening on port 5353

Code: [Select]

unbound  unbound    40899 3  stream /tmp/php-fastcgi.socket-1
unbound  unbound    40899 6  udp4   10.11.0.1:53          *:*
unbound  unbound    40899 7  tcp4   10.11.0.1:53          *:*
unbound  unbound    40899 8  udp4   127.0.0.1:53          *:*
unbound  unbound    40899 9  tcp4   127.0.0.1:53          *:*
unbound  unbound    40899 10 udp4   10.11.0.1:53          *:*
unbound  unbound    40899 11 tcp4   10.11.0.1:53          *:*
unbound  unbound    40899 12 udp4   127.0.0.1:53          *:*
unbound  unbound    40899 13 tcp4   127.0.0.1:53          *:*
unbound  unbound    40899 14 udp4   10.11.0.1:53          *:*
unbound  unbound    40899 15 tcp4   10.11.0.1:53          *:*
unbound  unbound    40899 16 udp4   127.0.0.1:53          *:*
unbound  unbound    40899 17 tcp4   127.0.0.1:53          *:*
unbound  unbound    40899 18 udp4   10.11.0.1:53          *:*
unbound  unbound    40899 19 tcp4   10.11.0.1:53          *:*
unbound  unbound    40899 20 udp4   127.0.0.1:53          *:*
unbound  unbound    40899 21 tcp4   127.0.0.1:53          *:*
unbound  unbound    40899 22 tcp4   127.0.0.1:953         *:*
unbound  unbound    40899 23 dgram  -> /var/run/logpriv
unbound  unbound    40899 24 stream -> ??
unbound  unbound    40899 25 stream -> ??
unbound  unbound    40899 26 stream -> ??
unbound  unbound    40899 27 stream -> ??
unbound  unbound    40899 28 stream -> ??
unbound  unbound    40899 29 stream -> ??
unbound  unbound    40899 30 stream -> ??
unbound  unbound    40899 31 stream -> ??
root     dnscrypt-p 26384 7  udp46  *:5353                *:*
root     dnscrypt-p 26384 8  tcp46  *:5353                *:*
Though Unbound seems to have an awful lot going on.

[cookiemonster]

Can you let us know what you want your end setup to be?

[DanAnimal]

I am attempting to get local caching of DNS results and encrypted DNS queries going upstream.

Unbound has shown SERVFAIL for any Query Forwarding or DNS over TLS setting I introduce. (localhost, opendns, cloudflare, google) only working defaulting to the Use System Nameservers without any forwarding set.
Which means plaintext DNS queries going upstream.

[cookiemonster]

Ok. Could you try this:
Unbound:
In Query forwarding, clear it out. This is to use normal forwarding, not a custom one.
In DNS over TLS, disable any you have there i.e. cloudflare 1.1.1.1:853, etc.
Test this "default" Unbound setup. Your queries are plain text on udp/tcp port 53 at this point as we know but will give a baseline.
Next is to enable DoT. Don't do it in "Query Forwarding" but in the DNS over TLS section. Here you enable one upstream like 1.1.1.1:853 and test.
We shall take it from there.

[DanAnimal]

Sure I can retrace my steps again.

DNSCrypt > Configuration > General > Enable DNSCrypt-Proxy: UNTICKED
DNSCrypt now disabled.

Unbound DNS > Query Forwarding > Custom forwarding: All deleted
Unbound DNS > DNS over TLS > Custom forwarding: All disabled;
Restarted Unbound.

Works with plaintext DNS:

Code: [Select]

root@uribou:~ # dig opnsense.org @127.0.0.1 -p 53

; <<>> DiG 9.18.16 <<>> opnsense.org @127.0.0.1 -p 53
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20714
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;opnsense.org. IN A

;; ANSWER SECTION:
opnsense.org. 571 IN A 178.162.131.118

;; Query time: 400 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 20:02:07 JST 2023
;; MSG SIZE  rcvd: 57
Unbound DNS > DNS over TLS > Custom forwarding: Added;
Domain: [BLANK]
Server IP: 1.1.1.1
Server PortPort: 853
Verify CN: one.one.one.one

Restarted Unbound.

Forwarding fails with SERVFAIL

Code: [Select]

root@uribou:~ # dig opnsense.org @127.0.0.1 -p 53

; <<>> DiG 9.18.16 <<>> opnsense.org @127.0.0.1 -p 53
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 33337
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;opnsense.org. IN A

;; Query time: 13 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 20:05:13 JST 2023
;; MSG SIZE  rcvd: 41

[cookiemonster]

Sockstat in this setup, what does it tell us, is Unbound listening on loopback?
Also can you do a dig again but don't specify the loopback and port please. Post results.

[DanAnimal]

Unbound listening on loopback and LAN interfaces

Code: [Select]

root@uribou:~ # sockstat
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS     
root     python3.9  33321 5  dgram  -> /var/run/logpriv
root     daemon     33124 5  dgram  -> /var/run/logpriv
unbound  unbound    31977 5  udp4   10.11.0.1:53          *:*
unbound  unbound    31977 6  tcp4   10.11.0.1:53          *:*
unbound  unbound    31977 7  udp4   127.0.0.1:53          *:*
unbound  unbound    31977 8  tcp4   127.0.0.1:53          *:*
unbound  unbound    31977 9  udp4   10.11.0.1:53          *:*
unbound  unbound    31977 10 tcp4   10.11.0.1:53          *:*
unbound  unbound    31977 11 udp4   127.0.0.1:53          *:*
unbound  unbound    31977 12 tcp4   127.0.0.1:53          *:*
unbound  unbound    31977 13 udp4   10.11.0.1:53          *:*
unbound  unbound    31977 14 tcp4   10.11.0.1:53          *:*
unbound  unbound    31977 15 udp4   127.0.0.1:53          *:*
unbound  unbound    31977 16 tcp4   127.0.0.1:53          *:*
unbound  unbound    31977 17 udp4   10.11.0.1:53          *:*
unbound  unbound    31977 18 tcp4   10.11.0.1:53          *:*
unbound  unbound    31977 19 udp4   127.0.0.1:53          *:*
unbound  unbound    31977 20 tcp4   127.0.0.1:53          *:*
unbound  unbound    31977 21 tcp4   127.0.0.1:953         *:*
unbound  unbound    31977 22 dgram  -> /var/run/logpriv
unbound  unbound    31977 23 stream -> ??
unbound  unbound    31977 24 stream -> ??
unbound  unbound    31977 25 stream -> ??
unbound  unbound    31977 26 stream -> ??
unbound  unbound    31977 27 stream -> ??
unbound  unbound    31977 28 stream -> ??
unbound  unbound    31977 29 stream -> ??
unbound  unbound    31977 30 stream -> ??

dig specifying no server and no port still queries to 127.0.0.1 port 53

Code: [Select]

root@uribou:~ # dig opnsense.org

; <<>> DiG 9.18.16 <<>> opnsense.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 47216
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;opnsense.org. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 20:29:40 JST 2023
;; MSG SIZE  rcvd: 41

[DanAnimal]

Unbound logs for the above failed dig query (same command rerun to grab the pertinent logs)

Code: [Select]

2023-07-20T20:33:15 Debug unbound [31977:2] debug: cache memory msg=135044 rrset=132120 infra=10617 val=0
2023-07-20T20:33:15 Error unbound [31977:2] error: SERVFAIL <opnsense.org. A IN>: all the configured stub or forward servers failed, at zone . no server to query nameserver addresses not usable have no nameserver names
2023-07-20T20:33:15 Debug unbound [31977:2] debug: return error response SERVFAIL
2023-07-20T20:33:15 Debug unbound [31977:2] debug: configured stub or forward servers failed -- returning SERVFAIL
2023-07-20T20:33:15 Informational unbound [31977:2] info: processQueryTargets: opnsense.org. A IN
2023-07-20T20:33:15 Informational unbound [31977:2] info: resolving opnsense.org. A IN
2023-07-20T20:33:15 Debug unbound [31977:2] debug: iterator[module 1] operate: extstate:module_state_initial event:module_event_pass

[cookiemonster]

Thanks, I appreciate name resolution is down whilst doing this.
This is good in a way. We know you get SERVFAIL on DoT when you shouldn't. I would try to get this solved before trying anything else. There's something fundamentally wrong right now.
One thought here. SERVFAIL is a genuine answer. Your setup might be right, just the response for whatever reason is SERVFAIL.
Could you try resolving various other names AND also with different upstreams, one at the time. I.E. disable cloudflare, enable quad9, test various names. Next.

[cookiemonster]

We were typing at the same time it seems.
Nice catch. Do you have anything set in "Custom Options" ? this is made available with a plugin.

[DanAnimal]

I have several work arounds to keep the DNS flowing here in the meantime (lest the streaming services be unavailable for the others in the house).

I did have 'do-not-query-localhost: no' in Custom options (trying to get the forward to DNSCrypt working).
Disabling Custom Options and restarting Unbound I'm getting the same error.

I have configured additional DoT forwarders for:
9.9.9.9 :853 dns.quad9.net
8.8.8.8 :853 dns.google
116.202.176.26 :853 dot.libredns.gr

Testing with 'dig freebsd.org && dig openssl.org && dig icann.org && dig aarnet.edu.au && dig home.cern'

All DoT forwarders disabled (plaintext DNS) All working

Code: [Select]

root@uribou:~ # dig freebsd.org && dig openssl.org && dig icann.org && dig aarnet.edu.au && dig home.cern

; <<>> DiG 9.18.16 <<>> freebsd.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62566
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;freebsd.org. IN A

;; ANSWER SECTION:
freebsd.org. 1300 IN A 96.47.72.84

;; Query time: 16 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:41:33 JST 2023
;; MSG SIZE  rcvd: 56


; <<>> DiG 9.18.16 <<>> openssl.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61269
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openssl.org. IN A

;; ANSWER SECTION:
openssl.org. 3600 IN A 194.97.150.230

;; Query time: 272 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:41:33 JST 2023
;; MSG SIZE  rcvd: 56


; <<>> DiG 9.18.16 <<>> icann.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55531
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;icann.org. IN A

;; ANSWER SECTION:
icann.org. 600 IN A 192.0.43.7

;; Query time: 199 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:41:33 JST 2023
;; MSG SIZE  rcvd: 54


; <<>> DiG 9.18.16 <<>> aarnet.edu.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19319
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aarnet.edu.au. IN A

;; ANSWER SECTION:
aarnet.edu.au. 300 IN A 202.158.207.3

;; Query time: 974 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:41:34 JST 2023
;; MSG SIZE  rcvd: 58


; <<>> DiG 9.18.16 <<>> home.cern
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55844
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;home.cern. IN A

;; ANSWER SECTION:
home.cern. 300 IN A 188.184.37.219

;; Query time: 1016 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:41:35 JST 2023
;; MSG SIZE  rcvd: 54

With only quad9 enabled (All SERVFAIL)

Code: [Select]

root@uribou:~ # dig freebsd.org && dig openssl.org && dig icann.org && dig aarnet.edu.au && dig home.cern

; <<>> DiG 9.18.16 <<>> freebsd.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 53366
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;freebsd.org. IN A

;; Query time: 2 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:44:59 JST 2023
;; MSG SIZE  rcvd: 40


; <<>> DiG 9.18.16 <<>> openssl.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 7142
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openssl.org. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:44:59 JST 2023
;; MSG SIZE  rcvd: 40


; <<>> DiG 9.18.16 <<>> icann.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 47173
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;icann.org. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:44:59 JST 2023
;; MSG SIZE  rcvd: 38


; <<>> DiG 9.18.16 <<>> aarnet.edu.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 29921
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aarnet.edu.au. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:44:59 JST 2023
;; MSG SIZE  rcvd: 42


; <<>> DiG 9.18.16 <<>> home.cern
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 30371
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;home.cern. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:44:59 JST 2023
;; MSG SIZE  rcvd: 38
With only dns.google enabled (All SERVFAIL)

Code: [Select]

root@uribou:~ # dig freebsd.org && dig openssl.org && dig icann.org && dig aarnet.edu.au && dig home.cern

; <<>> DiG 9.18.16 <<>> freebsd.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 22237
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;freebsd.org. IN A

;; Query time: 10 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:46:48 JST 2023
;; MSG SIZE  rcvd: 40


; <<>> DiG 9.18.16 <<>> openssl.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 11238
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openssl.org. IN A

;; Query time: 3 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:46:48 JST 2023
;; MSG SIZE  rcvd: 40


; <<>> DiG 9.18.16 <<>> icann.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 54659
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;icann.org. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:46:48 JST 2023
;; MSG SIZE  rcvd: 38


; <<>> DiG 9.18.16 <<>> aarnet.edu.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 41278
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aarnet.edu.au. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:46:48 JST 2023
;; MSG SIZE  rcvd: 42


; <<>> DiG 9.18.16 <<>> home.cern
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 57233
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;home.cern. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:46:48 JST 2023
;; MSG SIZE  rcvd: 38
With only libredns enabled (All SERVFAIL)

Code: [Select]

root@uribou:~ # dig freebsd.org && dig openssl.org && dig icann.org && dig aarnet.edu.au && dig home.cern

; <<>> DiG 9.18.16 <<>> freebsd.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 59008
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;freebsd.org. IN A

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:48:04 JST 2023
;; MSG SIZE  rcvd: 40


; <<>> DiG 9.18.16 <<>> openssl.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 23335
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openssl.org. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:48:04 JST 2023
;; MSG SIZE  rcvd: 40


; <<>> DiG 9.18.16 <<>> icann.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 41790
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;icann.org. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:48:04 JST 2023
;; MSG SIZE  rcvd: 38


; <<>> DiG 9.18.16 <<>> aarnet.edu.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 16850
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aarnet.edu.au. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:48:04 JST 2023
;; MSG SIZE  rcvd: 42


; <<>> DiG 9.18.16 <<>> home.cern
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 48176
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;home.cern. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:48:04 JST 2023
;; MSG SIZE  rcvd: 38

[cookiemonster]

Once you disable these Custom Options I can't see anything wrong.
The only thoughts right now are to:
- ensure the CN names are correct for the DoT i.e. for the 1.1.1.1 server, the CN is cloudflare-dns.com.
- check that unbound has any necessary security anchors. I think unbound will use OS ones in /etc/ssl/. You should have there a certs directory and a cert.pem bundle.
- increase Unbound log level
- Run a packet cpature