I have been having no luck setting any Forwarders for Unbound. Both Query Forwarding and DNS over TLS.
(Setting nothing and using the unsecured system nameserver from WAN's upstream DHCP works)
So am now trying to go via DNScrypt. Which still fails.
Here is the settings and results for dig on both DNSCrypt on port 5353 and Unbound on port 53 tested on the OPNSense host:
root@uribou:~ # cat /usr/local/etc/unbound.opnsense.d/dot.conf
server:
do-not-query-localhost: no
# Forward zones
forward-zone:
name: "."
forward-addr: 127.0.0.1@5353
root@uribou:~ # dig opnsense.org @127.0.0.1 -p 53
; <<>> DiG 9.18.16 <<>> opnsense.org @127.0.0.1 -p 53
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 4928
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;opnsense.org. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 16:25:22 JST 2023
;; MSG SIZE rcvd: 41
root@uribou:~ # dig opnsense.org @127.0.0.1 -p 5353
; <<>> DiG 9.18.16 <<>> opnsense.org @127.0.0.1 -p 5353
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46478
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;opnsense.org. IN A
;; ANSWER SECTION:
opnsense.org. 899 IN A 178.162.131.118
;; Query time: 38 msec
;; SERVER: 127.0.0.1#5353(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 16:25:31 JST 2023
;; MSG SIZE rcvd: 57
logs show:
error: SERVFAIL <opnsense.org. A IN>: all the configured stub or forward servers failed, at zone . no server to query nameserver addresses not usable have no nameserver names
This is using OPNsense 23.1.11-amd64
Feel I am going in circles.
Is there something obvious I am missing?
Is there an simple way to reset Unbound to installed defaults so I can try beginning again?
Those test results are what I would expect to see. DoT is saying Unbound listens on loopback port 5353 and then your dig test on it gave a normal result.
I suggest leaving any forwarding alone first and get the "normal" Unbound working.
Leave it as plain resolver and restart. Check with netstat/sockstat that is listening and the desired port. Check from the firewall and from a client.
Then and only then, set DoT and repeat checks.
Only after this point, move to forwarding if necessary.
That's how I would approach it.
Unbound is listening on port 53
DNSCrypt is listening on port 5353
Isn't the dot.conf is reflecting that I have one forwarder set to 127.0.0.1:5353 (screenshot attached).
DNSCrypt is responding on port 5353
sockstat shows Unbound listening on port 53 and DNSCrypt listening on port 5353
unbound unbound 40899 3 stream /tmp/php-fastcgi.socket-1
unbound unbound 40899 6 udp4 10.11.0.1:53 *:*
unbound unbound 40899 7 tcp4 10.11.0.1:53 *:*
unbound unbound 40899 8 udp4 127.0.0.1:53 *:*
unbound unbound 40899 9 tcp4 127.0.0.1:53 *:*
unbound unbound 40899 10 udp4 10.11.0.1:53 *:*
unbound unbound 40899 11 tcp4 10.11.0.1:53 *:*
unbound unbound 40899 12 udp4 127.0.0.1:53 *:*
unbound unbound 40899 13 tcp4 127.0.0.1:53 *:*
unbound unbound 40899 14 udp4 10.11.0.1:53 *:*
unbound unbound 40899 15 tcp4 10.11.0.1:53 *:*
unbound unbound 40899 16 udp4 127.0.0.1:53 *:*
unbound unbound 40899 17 tcp4 127.0.0.1:53 *:*
unbound unbound 40899 18 udp4 10.11.0.1:53 *:*
unbound unbound 40899 19 tcp4 10.11.0.1:53 *:*
unbound unbound 40899 20 udp4 127.0.0.1:53 *:*
unbound unbound 40899 21 tcp4 127.0.0.1:53 *:*
unbound unbound 40899 22 tcp4 127.0.0.1:953 *:*
unbound unbound 40899 23 dgram -> /var/run/logpriv
unbound unbound 40899 24 stream -> ??
unbound unbound 40899 25 stream -> ??
unbound unbound 40899 26 stream -> ??
unbound unbound 40899 27 stream -> ??
unbound unbound 40899 28 stream -> ??
unbound unbound 40899 29 stream -> ??
unbound unbound 40899 30 stream -> ??
unbound unbound 40899 31 stream -> ??
root dnscrypt-p 26384 7 udp46 *:5353 *:*
root dnscrypt-p 26384 8 tcp46 *:5353 *:*
Though Unbound seems to have an awful lot going on.
Can you let us know what you want your end setup to be?
I am attempting to get local caching of DNS results and encrypted DNS queries going upstream.
Unbound has shown SERVFAIL for any Query Forwarding or DNS over TLS setting I introduce. (localhost, opendns, cloudflare, google) only working defaulting to the Use System Nameservers without any forwarding set.
Which means plaintext DNS queries going upstream.
Ok. Could you try this:
Unbound:
In Query forwarding, clear it out. This is to use normal forwarding, not a custom one.
In DNS over TLS, disable any you have there i.e. cloudflare 1.1.1.1:853, etc.
Test this "default" Unbound setup. Your queries are plain text on udp/tcp port 53 at this point as we know but will give a baseline.
Next is to enable DoT. Don't do it in "Query Forwarding" but in the DNS over TLS section. Here you enable one upstream like 1.1.1.1:853 and test.
We shall take it from there.
Sure I can retrace my steps again.
DNSCrypt > Configuration > General > Enable DNSCrypt-Proxy: UNTICKED
DNSCrypt now disabled.
Unbound DNS > Query Forwarding > Custom forwarding: All deleted
Unbound DNS > DNS over TLS > Custom forwarding: All disabled;
Restarted Unbound.
Works with plaintext DNS:
root@uribou:~ # dig opnsense.org @127.0.0.1 -p 53
; <<>> DiG 9.18.16 <<>> opnsense.org @127.0.0.1 -p 53
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20714
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;opnsense.org. IN A
;; ANSWER SECTION:
opnsense.org. 571 IN A 178.162.131.118
;; Query time: 400 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 20:02:07 JST 2023
;; MSG SIZE rcvd: 57
Unbound DNS > DNS over TLS > Custom forwarding: Added;
Domain: [BLANK]
Server IP: 1.1.1.1
Server PortPort: 853
Verify CN: one.one.one.one
Restarted Unbound.
Forwarding fails with SERVFAIL
root@uribou:~ # dig opnsense.org @127.0.0.1 -p 53
; <<>> DiG 9.18.16 <<>> opnsense.org @127.0.0.1 -p 53
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 33337
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;opnsense.org. IN A
;; Query time: 13 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 20:05:13 JST 2023
;; MSG SIZE rcvd: 41
Sockstat in this setup, what does it tell us, is Unbound listening on loopback?
Also can you do a dig again but don't specify the loopback and port please. Post results.
Unbound listening on loopback and LAN interfaces
root@uribou:~ # sockstat
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root python3.9 33321 5 dgram -> /var/run/logpriv
root daemon 33124 5 dgram -> /var/run/logpriv
unbound unbound 31977 5 udp4 10.11.0.1:53 *:*
unbound unbound 31977 6 tcp4 10.11.0.1:53 *:*
unbound unbound 31977 7 udp4 127.0.0.1:53 *:*
unbound unbound 31977 8 tcp4 127.0.0.1:53 *:*
unbound unbound 31977 9 udp4 10.11.0.1:53 *:*
unbound unbound 31977 10 tcp4 10.11.0.1:53 *:*
unbound unbound 31977 11 udp4 127.0.0.1:53 *:*
unbound unbound 31977 12 tcp4 127.0.0.1:53 *:*
unbound unbound 31977 13 udp4 10.11.0.1:53 *:*
unbound unbound 31977 14 tcp4 10.11.0.1:53 *:*
unbound unbound 31977 15 udp4 127.0.0.1:53 *:*
unbound unbound 31977 16 tcp4 127.0.0.1:53 *:*
unbound unbound 31977 17 udp4 10.11.0.1:53 *:*
unbound unbound 31977 18 tcp4 10.11.0.1:53 *:*
unbound unbound 31977 19 udp4 127.0.0.1:53 *:*
unbound unbound 31977 20 tcp4 127.0.0.1:53 *:*
unbound unbound 31977 21 tcp4 127.0.0.1:953 *:*
unbound unbound 31977 22 dgram -> /var/run/logpriv
unbound unbound 31977 23 stream -> ??
unbound unbound 31977 24 stream -> ??
unbound unbound 31977 25 stream -> ??
unbound unbound 31977 26 stream -> ??
unbound unbound 31977 27 stream -> ??
unbound unbound 31977 28 stream -> ??
unbound unbound 31977 29 stream -> ??
unbound unbound 31977 30 stream -> ??
dig specifying no server and no port still queries to 127.0.0.1 port 53
root@uribou:~ # dig opnsense.org
; <<>> DiG 9.18.16 <<>> opnsense.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 47216
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;opnsense.org. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 20:29:40 JST 2023
;; MSG SIZE rcvd: 41
Unbound logs for the above failed dig query (same command rerun to grab the pertinent logs)
2023-07-20T20:33:15 Debug unbound [31977:2] debug: cache memory msg=135044 rrset=132120 infra=10617 val=0
2023-07-20T20:33:15 Error unbound [31977:2] error: SERVFAIL <opnsense.org. A IN>: all the configured stub or forward servers failed, at zone . no server to query nameserver addresses not usable have no nameserver names
2023-07-20T20:33:15 Debug unbound [31977:2] debug: return error response SERVFAIL
2023-07-20T20:33:15 Debug unbound [31977:2] debug: configured stub or forward servers failed -- returning SERVFAIL
2023-07-20T20:33:15 Informational unbound [31977:2] info: processQueryTargets: opnsense.org. A IN
2023-07-20T20:33:15 Informational unbound [31977:2] info: resolving opnsense.org. A IN
2023-07-20T20:33:15 Debug unbound [31977:2] debug: iterator[module 1] operate: extstate:module_state_initial event:module_event_pass
Thanks, I appreciate name resolution is down whilst doing this.
This is good in a way. We know you get SERVFAIL on DoT when you shouldn't. I would try to get this solved before trying anything else. There's something fundamentally wrong right now.
One thought here. SERVFAIL is a genuine answer. Your setup might be right, just the response for whatever reason is SERVFAIL.
Could you try resolving various other names AND also with different upstreams, one at the time. I.E. disable cloudflare, enable quad9, test various names. Next.
We were typing at the same time it seems.
Nice catch. Do you have anything set in "Custom Options" ? this is made available with a plugin.
I have several work arounds to keep the DNS flowing here in the meantime (lest the streaming services be unavailable for the others in the house).
I did have 'do-not-query-localhost: no' in Custom options (trying to get the forward to DNSCrypt working).
Disabling Custom Options and restarting Unbound I'm getting the same error.
I have configured additional DoT forwarders for:
9.9.9.9 :853 dns.quad9.net
8.8.8.8 :853 dns.google
116.202.176.26 :853 dot.libredns.gr
Testing with 'dig freebsd.org && dig openssl.org && dig icann.org && dig aarnet.edu.au && dig home.cern'
All DoT forwarders disabled (plaintext DNS) All working
root@uribou:~ # dig freebsd.org && dig openssl.org && dig icann.org && dig aarnet.edu.au && dig home.cern
; <<>> DiG 9.18.16 <<>> freebsd.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62566
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;freebsd.org. IN A
;; ANSWER SECTION:
freebsd.org. 1300 IN A 96.47.72.84
;; Query time: 16 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:41:33 JST 2023
;; MSG SIZE rcvd: 56
; <<>> DiG 9.18.16 <<>> openssl.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61269
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openssl.org. IN A
;; ANSWER SECTION:
openssl.org. 3600 IN A 194.97.150.230
;; Query time: 272 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:41:33 JST 2023
;; MSG SIZE rcvd: 56
; <<>> DiG 9.18.16 <<>> icann.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55531
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;icann.org. IN A
;; ANSWER SECTION:
icann.org. 600 IN A 192.0.43.7
;; Query time: 199 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:41:33 JST 2023
;; MSG SIZE rcvd: 54
; <<>> DiG 9.18.16 <<>> aarnet.edu.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19319
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aarnet.edu.au. IN A
;; ANSWER SECTION:
aarnet.edu.au. 300 IN A 202.158.207.3
;; Query time: 974 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:41:34 JST 2023
;; MSG SIZE rcvd: 58
; <<>> DiG 9.18.16 <<>> home.cern
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55844
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;home.cern. IN A
;; ANSWER SECTION:
home.cern. 300 IN A 188.184.37.219
;; Query time: 1016 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:41:35 JST 2023
;; MSG SIZE rcvd: 54
With only quad9 enabled (All SERVFAIL)
root@uribou:~ # dig freebsd.org && dig openssl.org && dig icann.org && dig aarnet.edu.au && dig home.cern
; <<>> DiG 9.18.16 <<>> freebsd.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 53366
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;freebsd.org. IN A
;; Query time: 2 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:44:59 JST 2023
;; MSG SIZE rcvd: 40
; <<>> DiG 9.18.16 <<>> openssl.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 7142
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openssl.org. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:44:59 JST 2023
;; MSG SIZE rcvd: 40
; <<>> DiG 9.18.16 <<>> icann.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 47173
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;icann.org. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:44:59 JST 2023
;; MSG SIZE rcvd: 38
; <<>> DiG 9.18.16 <<>> aarnet.edu.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 29921
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aarnet.edu.au. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:44:59 JST 2023
;; MSG SIZE rcvd: 42
; <<>> DiG 9.18.16 <<>> home.cern
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 30371
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;home.cern. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:44:59 JST 2023
;; MSG SIZE rcvd: 38
With only dns.google enabled (All SERVFAIL)
root@uribou:~ # dig freebsd.org && dig openssl.org && dig icann.org && dig aarnet.edu.au && dig home.cern
; <<>> DiG 9.18.16 <<>> freebsd.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 22237
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;freebsd.org. IN A
;; Query time: 10 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:46:48 JST 2023
;; MSG SIZE rcvd: 40
; <<>> DiG 9.18.16 <<>> openssl.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 11238
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openssl.org. IN A
;; Query time: 3 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:46:48 JST 2023
;; MSG SIZE rcvd: 40
; <<>> DiG 9.18.16 <<>> icann.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 54659
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;icann.org. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:46:48 JST 2023
;; MSG SIZE rcvd: 38
; <<>> DiG 9.18.16 <<>> aarnet.edu.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 41278
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aarnet.edu.au. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:46:48 JST 2023
;; MSG SIZE rcvd: 42
; <<>> DiG 9.18.16 <<>> home.cern
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 57233
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;home.cern. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:46:48 JST 2023
;; MSG SIZE rcvd: 38
With only libredns enabled (All SERVFAIL)
root@uribou:~ # dig freebsd.org && dig openssl.org && dig icann.org && dig aarnet.edu.au && dig home.cern
; <<>> DiG 9.18.16 <<>> freebsd.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 59008
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;freebsd.org. IN A
;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:48:04 JST 2023
;; MSG SIZE rcvd: 40
; <<>> DiG 9.18.16 <<>> openssl.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 23335
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openssl.org. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:48:04 JST 2023
;; MSG SIZE rcvd: 40
; <<>> DiG 9.18.16 <<>> icann.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 41790
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;icann.org. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:48:04 JST 2023
;; MSG SIZE rcvd: 38
; <<>> DiG 9.18.16 <<>> aarnet.edu.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 16850
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aarnet.edu.au. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:48:04 JST 2023
;; MSG SIZE rcvd: 42
; <<>> DiG 9.18.16 <<>> home.cern
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 48176
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;home.cern. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Thu Jul 20 21:48:04 JST 2023
;; MSG SIZE rcvd: 38
Once you disable these Custom Options I can't see anything wrong.
The only thoughts right now are to:
- ensure the CN names are correct for the DoT i.e. for the 1.1.1.1 server, the CN is cloudflare-dns.com.
- check that unbound has any necessary security anchors. I think unbound will use OS ones in /etc/ssl/. You should have there a certs directory and a cert.pem bundle.
- increase Unbound log level
- Run a packet cpature
Yeah, my Unbound doesn't work with any explicitly set forwarder.
Which we need to have in order to use DoT.
The cert.pem is there ie.
root@uribou:~ # cat /usr/local/etc/unbound.opnsense.d/dot.conf
# Forward zones over TLS
server:
tls-cert-bundle: /etc/ssl/cert.pem
forward-zone:
name: "."
forward-tls-upstream: yes
forward-addr: 1.1.1.1@853#cloudflare-dns.com
root@uribou:~ # ls /etc/ssl/
blacklisted cert.pem certs openssl.cnf
root@uribou:~ # ls /etc/ssl/certs
002c0b4f.0 3513523f.0 626dceaf.0 a3418fda.0 e113c810.0
02265526.0 3bde41ac.0 653b494a.0 a94d09e5.0 e18bfb83.0
03179a64.0 3e44d2f7.0 68dd7389.0 aee5f10d.0 e35234b1.0
062cdee6.0 3e45d192.0 6b99d060.0 b0e59380.0 e36a6752.0
064e0aa9.0 3fb36b73.0 6d41d539.0 b1159c4c.0 e73d606e.0
06dc52d5.0 40193066.0 6fa5da56.0 b433981b.0 e868b802.0
09789157.0 4042bcee.0 706f604c.0 b66938e9.0 e8de2f56.0
0a775a30.0 40547a79.0 749e9e03.0 b727005e.0 ee64a828.0
0b1b94ef.0 406c9bb1.0 75d1b2ed.0 b7a5b843.0 eed8c118.0
0bf05006.0 4304c5e5.0 76cb8f92.0 b81b93f0.0 ef954a4e.0
0f5dc4f3.0 48bec511.0 76faf6c0.0 bf53fb88.0 f081611a.0
0f6fa695.0 4a6481c9.0 7719f463.0 c01eb047.0 f0c70a8d.0
1001acf7.0 4b718d9b.0 773e07ad.0 c28a8a30.0 f249de83.0
106f3e4d.0 4bfab552.0 7aaf71c0.0 ca6e4ad9.0 f30dd6ad.0
14bc7599.0 4f316efb.0 7f3d5d1d.0 cbf06781.0 f3377b1b.0
1636090b.0 5273a94c.0 8160b96c.0 cc450945.0 f387163d.0
18856ac4.0 5443e9e3.0 8cb5ee0f.0 cd58d51e.0 f39fc864.0
1d3472b9.0 54657681.0 8d86cdd1.0 cd8c0d63.0 f51bb24c.0
1e08bfd1.0 57bcb2da.0 8d89cda1.0 ce5e74ef.0 fa5da96b.0
1e09d511.0 5ad8a5d6.0 930ac5d2.0 d4dae3dd.0 fc5a8f99.0
244b5494.0 5cd81ad7.0 93bc0acc.0 d6325660.0 fe8a2cd8.0
2923b3f9.0 5d3033c5.0 9482e63a.0 d7e8dc79.0 feffd413.0
2ae6433e.0 5e98733a.0 988a38cb.0 d887a5bb.0 ff34af3f.0
2b349938.0 5f15c80c.0 9b5697b0.0 dc4d6a89.0
2e5ac55d.0 5f618aec.0 9c8dfbd4.0 dd8e9d41.0
32888f65.0 607986c7.0 9d04f354.0 de6d66f3.0
The DoT forwarders are set as I mentioned. Screenshot attached here as well.
What do you suggest the log level be?
How should I packet capture to illuminate why this won't work even with an unencrypted connection to a separate port on localhost?!
(The erroneous behaviour was the same attempting to use DNSCrypt at localhost:5353...)
ie.
error: SERVFAIL <opnsense.org. A IN>: all the configured stub or forward servers failed, at zone . no server to query nameserver addresses not usable have no nameserver names
Thanks for chasing along with this by the way!
At debug level the logs are hectic.
The forum says they're too large to post here.
Quote from: DanAnimal on July 20, 2023, 04:17:37 PM
How should I packet capture to illuminate why this won't work even with an unencrypted connection to a separate port on localhost?!
Packet capture, select both WAN, loopback and LAN. Change the packet count to a large number like 10000, the rest leave at defaults. Fire your test. Stop the capture and download it. Open the file in Wireshark.
My gut feeling is that the problem isn't related to the flow, i.e. that whether this ordinary way or sending to "with an unencrypted connection to a separate port on localhost?". My gut feels like the packets go the way you configure but for some reason, an upstream server is returning the SERVFAIL and I hope the trace gives a clue as to why. It might not, but I'd like to see the flow.
To your original question, how to set Unbound to defaults without resetting anythign else, I don't know. I don't know what files might be left by just disabling it, that could be picked back up if re-enabled.
Hopefully someone who knows will tell us.
There are other ways we could do. I send my queries upstream to DoT resolvers in a different way but for now I won't confuse things. It is a lot more convoluted and lacks proper logging.
OK, I've got capturing and wireshark but i really don't have an idea what I'm know what I'm doing with it.
I did a little capture from my side to give you some pointers. It can take some scrolling in wireshark to get to what you want but you are looking for destination ip, destination port, that sort of thing in the columns.
I started the capture in OPN's UI (correction - enable promiscuous mode).
Next I ssh'ed into opn and from there I did a dig for example.com.
(https://i.ibb.co/6mVVrn2/one.png) (https://ibb.co/9Nzzc8J)
I went back to UI and stopped and downloaded the capture zip bundle, opened them in wireshark.
I can see the query and response in the loopback. This is the one you want to find.
(https://i.ibb.co/sg8Q2P3/localhost.png) (https://ibb.co/zmqbr5f)
And I can also see the packets in the WAN capture. In my case the response came from 9.9.9.10 and I can confirm my DoT is working fine. The dns response came in a TLS envelope. I can also see other queries to 1.1.1.1 as I spread my DoT queries to multiple upstreams. I did apply a display filter "tcp.port in {53, 853, 8053}" because in my setup those three ports are in play. You probably need only the first two.
(https://i.ibb.co/TmrFm2R/wan.png) (https://ibb.co/Lnv2nR6)
I hope this helps.
I got a capture and found references to my test 'dig home.cern', run several times, on the loopback.
But I could not find anything at all that seemed pertinent on either WAN or LAN!?
(https://i.ibb.co/b6JvSPx/Screenshot-2023-07-21-at-11-16-48.png) (https://ibb.co/YNZdz36)
That suggests the query goes only as far as the loopback and goes no further, although SERVFAIL suggests a response ie it went out. However no trace on WAN suggests it didn't. For now I'm thinking a misleading error.
And when you're doing these tests, you have "Do not use the local DNS service as a nameserver for this system" as no (unticked), right? That is to ensure the loopback is used by the system, and no additional paths ie dnscrypt or any custom options, just DoT from UI, right?
Also "Allow DNS server list to be overridden by DHCP/PPP on WAN" is not enabled, right?
And whilst you're there in System > Settiings > General; Networking, DNS - what do you have there?
You mean the check box 'Use System Nameservers' on SERVICES: UNBOUND DNS: DNS OVER TLS and SERVICES: UNBOUND DNS: QUERY FORWARDING?
They are unticked.
'Enable DNSCrypt-Proxy' is unticked in SERVICES: DNSCRYPT-PROXY: CONFIGURATION
'Enable' is unticked in SERVICES: UNBOUND DNS: CUSTOM OPTIONS (and I will probably uninstall these plugins).
Oops, just found 'Do not use the local DNS service as a nameserver for this system' in SYSTEM: SETTINGS: GENERAL
It was unticked.
'Allow DNS server list to be overridden by DHCP/PPP on WAN' was ticked, but...
Toggling these settings and retesting produced the same results.
'DNS servers' list in SYSTEM: SETTINGS: GENERAL has been empty the whole time.
TBH, I'm thinking to give up on this.
It's taking up a bunch of our time.
I think there must be something filtering port 53 and 853 upstream (which is entirely possible as I am inside CGNAT, and have had to use an OpenWRT host to interpret my ISP's DS-Lite IPv4 over IPv6 because OPNSense is tragically non-trivial and unstable to connect to DS-Lite).
And I have reproduced the issue using Unbound on a fresh Debian VM here too.
I have other non-OPNSense work-arounds to provide this site with DNS over HTTPS.
Including DNSCrypt if I replaced Unbound on port 53.
Although it is very strange that setting DNSCrypt as the forwarder on the same host produced the same error. No weird network environment should affect that. Should it?!
That thought makes wonder if giving up is wrong as DNSCrypt could resolve successfully but Unbound would not connect to DNSCrypt as forwarder!? That SHOULD work regardless of any filtering of 853 or 53, right?
DoH using port 443 would go right through any filtering of 853 or 53.
OPNSense's current Unbound is version 1.17.1 with libnghttp2 so it can do DoH downstream but it is not able to for forwarding alas.
It's frustrating as I have Unbound working with DoT on OPNSense at the site I manage for work.
And the reason I implemented OPNSense at home was to have a working analogue to that.
So each of these failures to be able to do basic things in OPNSense here make me disappointed in myself.
Thank you for taking the effort to chase this with me! You taught me to packet capture and check in wireshark too!
Yes I can't see why it would fail but at least reproducing with a Debian VM running Unbound makes it easier to suspect some other element, like the ISP.
If you wanted too try something different just for kicks, you could try my overcomplicated method to implement DoT.
In summary I install a utility getdns/stubby on OPN. It runs listening on loopback and your choice of port. Then use the Custom Options in Unbound to forward to it. Similar to what you were trying to do with DNSCrypt, of which I have no experience by the way. I could share my stubby config for you to test.
I won't blame you if you instead chose to give up on this for now though.
I'm sorry I haven't been able to help you better.
Don't be sorry, you've been great.
That method does sound just like I was doing with DNSCrypt (Except you are using DoT rather than DoH - which I increasingly suspect cannot function in my environment - I tested with kdig independent of Unbound with similar results).
root@unbound-test:~# kdig -d @1.1.1.1 +dnssec +tls-ca=/etc/ssl/certs/ca-certificates.crt +tls-host=cloudflare-dns.com home.cern
;; WARNING: can't setlocale, disabling IDN
;; DEBUG: Querying for owner(home.cern.), class(1), type(1), server(1.1.1.1), port(853), protocol(TCP)
;; DEBUG: TLS, imported 140 certificates from '/etc/ssl/certs/ca-certificates.crt'
;; WARNING: can't connect to 1.1.1.1@853(TCP)
;; ERROR: failed to query server 1.1.1.1@853(TCP)
So I had DNSCrypt listening on 5353 and resolving perfectly fine. Just like your getdns/stubby.
I could not get Unbound to forward to 127.0.0.1:5353?! (And thus my initial post).
Why are you needing to use Custom Options to forward to a port on localhost?
Shouldn't SERVICES: UNBOUND DNS: QUERY FORWARDING do what it says on the tin and forward requests rather than failing even to address another port on the same host?
Quote from: DanAnimal on July 21, 2023, 01:59:47 PM
Why are you needing to use Custom Options to forward to a port on localhost?
Shouldn't SERVICES: UNBOUND DNS: QUERY FORWARDING do what it says on the tin and forward requests rather than failing even to address another port on the same host?
It might have been that the option wasn't available in the UI at the time I set this and now that I can see it there, I don't know if it does the same thing. It surely looks that way. When I get a chance to mess around I might try it and see to verify it creates the equivalent in the running config.
Anyhow, the contents of my Custom Options are these:
server:
do-not-query-localhost: no
forward-zone:
name: "."
forward-addr: 127.0.0.1@8053And stubby is listening on that port of course.
Take care.
Arigatou Monster san
So after writing my previous post I went back and re-set-up DNSCrypt on port 5353 and the forwarder to 127.0.0.1:5353 in SERVICES: UNBOUND DNS: QUERY FORWARDING (I was wondering about testing if it worked forwarding to the LAN IP)
And the blasted thing is working now!!!
root@uribou:~ # cat /usr/local/etc/unbound.opnsense.d/dot.conf
server:
do-not-query-localhost: no
# Forward zones
forward-zone:
name: "."
forward-addr: 127.0.0.1@5353
Almost verbatim the same as yours bar the port number set through the GUI.
And the same as it was in my first post in this thread.
I have no idea what we did differently or changed (which is not great - I detest repairadoxes).
But it is working now, so thanks.
I'm going to go turn down all the log levels now.
I'm glad to hear is now working. All a bit strange.
You might want to consider marking the original post subject to "SOLVED" :)
I'm still a bit confused about the original issue with Unbound and DoT necessitating DNSCrypt but glad it's working for you.