Unbound SERVFAIL -> DNSCrypt 127.0.0.1:5353 (and other forwarders)

[DanAnimal]

Yeah, my Unbound doesn't work with any explicitly set forwarder.

Which we need to have in order to use DoT.

The cert.pem is there ie.

Code Select Expand

root@uribou:~ # cat /usr/local/etc/unbound.opnsense.d/dot.conf                 

# Forward zones over TLS
server:
  tls-cert-bundle: /etc/ssl/cert.pem

forward-zone:
  name: "."
  forward-tls-upstream: yes
  forward-addr: 1.1.1.1@853#cloudflare-dns.com

root@uribou:~ # ls /etc/ssl/
blacklisted cert.pem certs openssl.cnf
root@uribou:~ # ls /etc/ssl/certs
002c0b4f.0 3513523f.0 626dceaf.0 a3418fda.0 e113c810.0
02265526.0 3bde41ac.0 653b494a.0 a94d09e5.0 e18bfb83.0
03179a64.0 3e44d2f7.0 68dd7389.0 aee5f10d.0 e35234b1.0
062cdee6.0 3e45d192.0 6b99d060.0 b0e59380.0 e36a6752.0
064e0aa9.0 3fb36b73.0 6d41d539.0 b1159c4c.0 e73d606e.0
06dc52d5.0 40193066.0 6fa5da56.0 b433981b.0 e868b802.0
09789157.0 4042bcee.0 706f604c.0 b66938e9.0 e8de2f56.0
0a775a30.0 40547a79.0 749e9e03.0 b727005e.0 ee64a828.0
0b1b94ef.0 406c9bb1.0 75d1b2ed.0 b7a5b843.0 eed8c118.0
0bf05006.0 4304c5e5.0 76cb8f92.0 b81b93f0.0 ef954a4e.0
0f5dc4f3.0 48bec511.0 76faf6c0.0 bf53fb88.0 f081611a.0
0f6fa695.0 4a6481c9.0 7719f463.0 c01eb047.0 f0c70a8d.0
1001acf7.0 4b718d9b.0 773e07ad.0 c28a8a30.0 f249de83.0
106f3e4d.0 4bfab552.0 7aaf71c0.0 ca6e4ad9.0 f30dd6ad.0
14bc7599.0 4f316efb.0 7f3d5d1d.0 cbf06781.0 f3377b1b.0
1636090b.0 5273a94c.0 8160b96c.0 cc450945.0 f387163d.0
18856ac4.0 5443e9e3.0 8cb5ee0f.0 cd58d51e.0 f39fc864.0
1d3472b9.0 54657681.0 8d86cdd1.0 cd8c0d63.0 f51bb24c.0
1e08bfd1.0 57bcb2da.0 8d89cda1.0 ce5e74ef.0 fa5da96b.0
1e09d511.0 5ad8a5d6.0 930ac5d2.0 d4dae3dd.0 fc5a8f99.0
244b5494.0 5cd81ad7.0 93bc0acc.0 d6325660.0 fe8a2cd8.0
2923b3f9.0 5d3033c5.0 9482e63a.0 d7e8dc79.0 feffd413.0
2ae6433e.0 5e98733a.0 988a38cb.0 d887a5bb.0 ff34af3f.0
2b349938.0 5f15c80c.0 9b5697b0.0 dc4d6a89.0
2e5ac55d.0 5f618aec.0 9c8dfbd4.0 dd8e9d41.0
32888f65.0 607986c7.0 9d04f354.0 de6d66f3.0

The DoT forwarders are set as I mentioned. Screenshot attached here as well.

What do you suggest the log level be?

How should I packet capture to illuminate why this won't work even with an unencrypted connection to a separate port on localhost?!
(The erroneous behaviour was the same attempting to use DNSCrypt at localhost:5353...)

ie.

Code Select Expand

error: SERVFAIL <opnsense.org. A IN>: all the configured stub or forward servers failed, at zone . no server to query nameserver addresses not usable have no nameserver names

Thanks for chasing along with this by the way!

[DanAnimal]

At debug level the logs are hectic.
The forum says they're too large to post here.

[cookiemonster]

How should I packet capture to illuminate why this won't work even with an unencrypted connection to a separate port on localhost?!

Packet capture, select both WAN, loopback and LAN. Change the packet count to a large number like 10000, the rest leave at defaults. Fire your test. Stop the capture and download it. Open the file in Wireshark.
My gut feeling is that the problem isn't related to the flow, i.e. that whether this ordinary way or sending to "with an unencrypted connection to a separate port on localhost?". My gut feels like the packets go the way you configure but for some reason, an upstream server is returning the SERVFAIL and I hope the trace gives a clue as to why. It might not, but I'd like to see the flow.

To your original question, how to set Unbound to defaults without resetting anythign else, I don't know. I don't know what files might be left by just disabling it, that could be picked back up if re-enabled.
Hopefully someone who knows will tell us.

There are other ways we could do. I send my queries upstream to DoT resolvers in a different way but for now I won't confuse things. It is a lot more convoluted and lacks proper logging.

[DanAnimal]

OK, I've got capturing and wireshark but i really don't have an idea what I'm know what I'm doing with it.

[cookiemonster]

I did a little capture from my side to give you some pointers. It can take some scrolling in wireshark to get to what you want but you are looking for destination ip, destination port, that sort of thing in the columns.
I started the capture in OPN's UI (correction - enable promiscuous mode).
Next I ssh'ed into opn and from there I did a dig for example.com.

I went back to UI and stopped and downloaded the capture zip bundle, opened them in wireshark.
I can see the query and response in the loopback. This is the one you want to find.

And I can also see the packets in the WAN capture. In my case the response came from 9.9.9.10 and I can confirm my DoT is working fine. The dns response came in a TLS envelope. I can also see other queries to 1.1.1.1 as I spread my DoT queries to multiple upstreams. I did apply a display filter "tcp.port in {53, 853, 8053}" because in my setup those three ports are in play. You probably need only the first two.

I hope this helps.

[DanAnimal]

I got a capture and found references to my test 'dig home.cern', run several times, on the loopback.

But I could not find anything at all that seemed pertinent on either WAN or LAN!?

[cookiemonster]

That suggests the query goes only as far as the loopback and goes no further, although SERVFAIL suggests a response ie it went out. However no trace on WAN suggests it didn't. For now I'm thinking a misleading error.
And when you're doing these tests, you have "Do not use the local DNS service as a nameserver for this system" as no (unticked), right? That is to ensure the loopback is used by the system, and no additional paths ie dnscrypt or any custom options, just DoT from UI, right?
Also "Allow DNS server list to be overridden by DHCP/PPP on WAN" is not enabled, right?
And whilst you're there in System > Settiings > General; Networking, DNS - what do you have there?

[DanAnimal]

You mean the check box 'Use System Nameservers' on SERVICES: UNBOUND DNS: DNS OVER TLS and SERVICES: UNBOUND DNS: QUERY FORWARDING?
They are unticked.

'Enable DNSCrypt-Proxy' is unticked in SERVICES: DNSCRYPT-PROXY: CONFIGURATION

'Enable' is unticked in SERVICES: UNBOUND DNS: CUSTOM OPTIONS (and I will probably uninstall these plugins).

Oops, just found 'Do not use the local DNS service as a nameserver for this system' in SYSTEM: SETTINGS: GENERAL
It was unticked.
'Allow DNS server list to be overridden by DHCP/PPP on WAN' was ticked, but...

Toggling these settings and retesting produced the same results.

'DNS servers' list in SYSTEM: SETTINGS: GENERAL has been empty the whole time.



TBH, I'm thinking to give up on this.
It's taking up a bunch of our time.
I think there must be something filtering port 53 and 853 upstream (which is entirely possible as I am inside CGNAT, and have had to use an OpenWRT host to interpret my ISP's DS-Lite IPv4 over IPv6 because OPNSense is tragically non-trivial and unstable to connect to DS-Lite).

And I have reproduced the issue using Unbound on a fresh Debian VM here too.

I have other non-OPNSense work-arounds to provide this site with DNS over HTTPS.

Including DNSCrypt if I replaced Unbound on port 53.

Although it is very strange that setting DNSCrypt as the forwarder on the same host produced the same error. No weird network environment should affect that. Should it?!

That thought makes wonder if giving up is wrong as DNSCrypt could resolve successfully but Unbound would not connect to DNSCrypt as forwarder!? That SHOULD work regardless of any filtering of 853 or 53, right?

DoH using port 443 would go right through any filtering of 853 or 53.

OPNSense's current Unbound is version 1.17.1 with libnghttp2 so it can do DoH downstream but it is not able to for forwarding alas.

It's frustrating as I have Unbound working with DoT on OPNSense at the site I manage for work.
And the reason I implemented OPNSense at home was to have a working analogue to that.
So each of these failures to be able to do basic things in OPNSense here make me disappointed in myself.

Thank you for taking the effort to chase this with me! You taught me to packet capture and check in wireshark too!

[cookiemonster]

Yes I can't see why it would fail but at least reproducing with a Debian VM running Unbound makes it easier to suspect some other element, like the ISP.
If you wanted too try something different just for kicks, you could try my overcomplicated method to implement DoT.
In summary I install a utility getdns/stubby on OPN. It runs listening on loopback and your choice of port. Then use the Custom Options in Unbound to forward to it. Similar to what you were trying to do with DNSCrypt, of which I have no experience by the way. I could share my stubby config for you to test.
I won't blame you if you instead chose to give up on this for now though.
I'm sorry I haven't been able to help you better.

[DanAnimal]

Don't be sorry, you've been great.

That method does sound just like I was doing with DNSCrypt (Except you are using DoT rather than DoH - which I increasingly suspect cannot function in my environment - I tested with kdig independent of Unbound with similar results).

Code Select Expand

root@unbound-test:~# kdig -d @1.1.1.1 +dnssec +tls-ca=/etc/ssl/certs/ca-certificates.crt +tls-host=cloudflare-dns.com home.cern
;; WARNING: can't setlocale, disabling IDN
;; DEBUG: Querying for owner(home.cern.), class(1), type(1), server(1.1.1.1), port(853), protocol(TCP)
;; DEBUG: TLS, imported 140 certificates from '/etc/ssl/certs/ca-certificates.crt'
;; WARNING: can't connect to 1.1.1.1@853(TCP)
;; ERROR: failed to query server 1.1.1.1@853(TCP)

So I had DNSCrypt listening on 5353 and resolving perfectly fine. Just like your getdns/stubby.

I could not get Unbound to forward to 127.0.0.1:5353?! (And thus my initial post).

Why are you needing to use Custom Options to forward to a port on localhost?

Shouldn't SERVICES: UNBOUND DNS: QUERY FORWARDING do what it says on the tin and forward requests rather than failing even to address another port on the same host?

[cookiemonster]

Why are you needing to use Custom Options to forward to a port on localhost?

Shouldn't SERVICES: UNBOUND DNS: QUERY FORWARDING do what it says on the tin and forward requests rather than failing even to address another port on the same host?

It might have been that the option wasn't available in the UI at the time I set this and now that I can see it there, I don't know if it does the same thing. It surely looks that way. When I get a chance to mess around I might try it and see to verify it creates the equivalent in the running config.
Anyhow, the contents of my Custom Options are these:

Code Select Expand

server:
do-not-query-localhost: no
forward-zone:
name: "."
forward-addr: 127.0.0.1@8053
And stubby is listening on that port of course.
Take care.

[DanAnimal]

Arigatou Monster san

So after writing my previous post I went back and re-set-up DNSCrypt on port 5353 and the forwarder to 127.0.0.1:5353 in SERVICES: UNBOUND DNS: QUERY FORWARDING (I was wondering about testing if it worked forwarding to the LAN IP)

And the blasted thing is working now!!!

Code Select Expand

root@uribou:~ # cat /usr/local/etc/unbound.opnsense.d/dot.conf
server:
  do-not-query-localhost: no

# Forward zones
forward-zone:
  name: "."
  forward-addr: 127.0.0.1@5353

Almost verbatim the same as yours bar the port number set through the GUI.
And the same as it was in my first post in this thread.

I have no idea what we did differently or changed (which is not great - I detest repairadoxes).

But it is working now, so thanks.

I'm going to go turn down all the log levels now.

[cookiemonster]

I'm glad to hear is now working. All a bit strange.
You might want to consider marking the original post subject to "SOLVED" :)

[CJ]

I'm still a bit confused about the original issue with Unbound and DoT necessitating DNSCrypt but glad it's working for you.