UnboundDNS (and dnsmasq) spontaneously stopped working today

[Eric Schoen]

I have a ProtectLi Intel Core i5 machine with 16 GiB of memory.  UnboundDNS spontaneously stopped responding to requests today.  I did not and had not in some weeks altered any settings (DNS or Firewall or Interface) on it.  I tried swapping dnsmasq for unbound, but get the same non responsiveness.

DNS requests using host/dig/nslookup time out, whether from on the opnsense machine itself or from a LAN host.  From a macOS LAN client, host -T fails immediately:

$ host -T btc.i2kconnect.com 192.168.0.1
;; communications error to 192.168.0.1#53: network down

But host -T from the opnsense machine times out.

unbound-control can't talk to it either, running from an opnsense-shell on the router and trying to access its control port 953 on its local IP address or on its loopback address 127.0.0.1.  I was running opnsense 23.7 when this happened, and upgraded to 24.1 in desperation but this made no difference.   I'm not seeing any packets dropped by the firewall. 

sockstat indicates that unbound is listening to port 53 for both TCP and UDP

root@btc-firewall:/var/log # sockstat -l -4 -p 53
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
unbound  unbound    25105 7  udp4   *:53                  *:*
unbound  unbound    25105 8  tcp4   *:53                  *:*
unbound  unbound    25105 11 udp4   *:53                  *:*
unbound  unbound    25105 12 tcp4   *:53                  *:*
unbound  unbound    25105 15 udp4   *:53                  *:*
unbound  unbound    25105 16 tcp4   *:53                  *:*
unbound  unbound    25105 19 udp4   *:53                  *:*
unbound  unbound    25105 20 tcp4   *:53                  *:*
root@btc-firewall:/var/log #

Once unbound starts up, there is no traffic in the unbound log either, as shown below.  Other than unbound/dnsmasq, the machine is routing as expected. 

Since the problem affects both dnsmasq and unbound, I suspect the problem is not the DNS services themselves, but I can't imagine what could be blocking the request traffic.  Any suggestions for how to proceed would be greatly welcomed.

2024-06-01T23:21:31   20   Notice   unbound   31787   Backgrounding unbound logging backend.   
2024-06-01T23:21:31   3   Informational   unbound   25105   [25105:0] info: dnsbl_module: updating blocklist.   
2024-06-01T23:21:30   20   Notice   unbound   29087   daemonize unbound dhcpd watcher.   
2024-06-01T23:21:30   3   Notice   unbound   25105   [25105:0] notice: init module 0: python   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: module config: "python iterator"   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 198.41.0.4 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2001:503:ba3e::2:30 port 53 (len 28)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 170.247.170.2 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2801:1b8:10::b port 53 (len 28)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 192.33.4.12 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2001:500:2::c port 53 (len 28)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 199.7.91.13 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2001:500:2d::d port 53 (len 28)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 192.203.230.10 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2001:500:a8::e port 53 (len 28)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 192.5.5.241 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2001:500:2f::f port 53 (len 28)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 192.112.36.4 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2001:500:12::d0d port 53 (len 28)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 198.97.190.53 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2001:500:1::53 port 53 (len 28)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 192.36.148.17 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2001:7fe::53 port 53 (len 28)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 192.58.128.30 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2001:503:c27::2:30 port 53 (len 28)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 193.0.14.129 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2001:7fd::1 port 53 (len 28)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 199.7.83.42 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2001:500:9f::42 port 53 (len 28)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 202.12.27.33 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip6 2001:dc3::35 port 53 (len 28)   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: A.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: B.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: C.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: D.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: E.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: F.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: G.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: H.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: I.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: J.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: K.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: L.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: M.ROOT-SERVERS.NET. * A AAAA   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: DelegationPoint<.>: 13 names (0 missing), 26 addrs (0 result, 26 avail) parentNS   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: Reading root hints from /root.hints   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 208.67.220.220 port 53 (len 16)   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ip4 208.67.222.222 port 53 (len 16)   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: DelegationPoint<.>: 0 names (0 missing), 2 addrs (0 result, 2 avail) parentNS   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: Forward zone server list:   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ignoring duplicate RR: 1.0.0.127.in-addr.arpa. PTR localhost   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ignoring duplicate RR: localhost A 127.0.0.1   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ignoring duplicate RR: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa. PTR localhost   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ignoring duplicate RR: localhost AAAA ::1   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ignoring duplicate RR: 1.0.168.192.in-addr.arpa. PTR btc-firewall.i2kconnect.com   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ignoring duplicate RR: btc-firewall.i2kconnect.com A 192.168.0.1   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ignoring duplicate RR: 200.0.168.192.in-addr.arpa. PTR btc.i2kconnect.com   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ignoring duplicate RR: btc.i2kconnect.com IN A 192.168.0.200   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ignoring duplicate RR: 201.0.168.192.in-addr.arpa. PTR btc-master.i2kconnect.com   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: ignoring duplicate RR: btc-master.i2kconnect.com IN A 192.168.0.201   
2024-06-01T23:21:30   3   Informational   unbound   25105   [25105:0] info: implicit transparent local-zone . TYPE0 IN   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: drop user privileges, run as unbound   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: chroot to /var/unbound   
2024-06-01T23:21:30   3   Debug   unbound   25105   [25105:0] debug: chdir to /var/unbound

[heaven73]

I had this behaviour with older Zenarmor version time ago when SSL Error pages (in beta) was active. deactivating that feature in Zenarmos solved my problem.

So could be another service crashing. i would check the system logs for errors

[Eric Schoen]

I didn't see any system errors in the log. 

For now, I've installed a new SSD, imaged a fresh 24.1 deployment, restored all but the Unbound DNS configuration settings, and then manually recreated the Unbound DNS settings I want.  This works.  But for what it's worth, I tried restoring the last full configuration backup that I took before I shut down the broken system into a Live CD session of 24.1. This produced exactly the same behavior as above.  I'm mystified, but happy to have a working DNS server in my network again.

[bestboy]

If you suspect a bad unbound config, did you run unbound-checkconf? (https://unbound.docs.nlnetlabs.nl/en/latest/manpages/unbound-checkconf.html)