UnboundDNS Returning OpenDNS Blocks Only With 1.1.1.1 DNS

[Mirmirmi]

I've been using OpnSense for a small one-site business for most of 2023 now and have been very happy with it. However, last week something strange started happening. For some seemingly random and specific websites (mail.google.com, reddit.com, backblaze.com, & youtube.com), Unbound DNS would take the queries for these sites from my LAN clients, forward them to one of my two configured DNS servers ( 1.1.1.1 & 8.8.8.8 ), and whenever 1.1.1.1 answered its request for one of these four websites, the reply would instead be for the IP 142.112.61.106 instead of the real IP. This is an OpenDNS IP meant for DNS filtering/blocking, from what I understand.

From the user's perspecitve, the browser page shows an impassable security error because the SSL certificate it got was for OpenDNS and not the site it tried to reach. When I turned the Unbound DNS logs up to the most verbose, I was able to see the OpenDNS IP in the reply only when queries were forwarded to 1.1.1.1, not when they went to 8.8.8.8 or any other DNS servers I later configured as the system DNS servers to resolve the issue.

I'm pretty confused becauase:

• We don't use OpenDNS web filtering
• OpenDNS isn't enabled or configured in OpnSense
• I don't use any Cloudflare DNS filtering and even if I did, they require you to use a different DNS server than 1.1.1.1.
• If I statically assigned 1.1.1.1 as a DNS server on a LAN client (bypassing Unbound DNS), the websites all worked normally.
• We have two different ISP's and the behavior occurred regardless of which one I turned on or off
• The behavior didn't occur if I plugged directly into an ISP's gateway, bypassing OpnSense
• I only have a Wireguard and Dynamic DNS plugin installed - no others
• We don't have any packet inspection configured
• The issue persisted even when I upgraded OpnSense from 23.3 to 23.7 and across all reboots

 

Because 1.1.1.1 replied normally when a LAN client queried directly, but replied with an OpenDNS block IP when OpnSense's Unbound DNS queried 1.1.1.1, and because it happens across two different ISPs, I'm led to believe something in OpnSense might be causing this. But I can't figure out what. Does anyone have any ideas?

Unbound DNS Log:

Code: [Select]

2023-10-19T09:25:59-05:00 Informational unbound [67904:0] info: response for reddit.com. A IN
2023-10-19T09:25:59-05:00 Informational unbound [67904:0] info: reply from <.> 1.1.1.1#53
2023-10-19T09:25:59-05:00 Informational unbound [67904:0] info: incoming scrubbed packet: ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 0
  ;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
  ;; QUESTION SECTION:
  reddit.com. IN A
  ;; ANSWER SECTION:
  reddit.com. 0 IN A 146.112.61.106
  ;; AUTHORITY SECTION:
  ;; ADDITIONAL SECTION:
  ;; MSG SIZE rcvd: 44


[cookiemonster]

Where do you have those two dns servers configured for OPN to use?
It'd be good to know what is configured for dns in System, Unbound and dhcp. If only to have a big picture.
Because in short, dhcp will tell clients what dns server to use, it should be unbound, and unbound in turn will only use what is configured as upstream otherwise goes to root dns servers. That's the general, normal flow, unless by configurations in different places that flow is altered.

[Mirmirmi]

Good question - I had DNS servers configured in the flow you described: System -> General -> DNS Servers. No custom forwarding configured in Unbound and it's using the system nameservers. DHCP indeed hands out only Unbound as the DNS server.

Our primary ISP is Comcast which has a "Security Edge" DNS filtering product which should be an obvious suspect. I called them immediately when this began happening and they said it wasn't on or enabled for our account. Even assuming they lied, I'm pretty sure that OpnSense got a reboot in during one of the spans when I had the Comcast gateway unplugged and was running off the secondary ISPs gateway so I wouldn't expect anything weird from that to be cached in Unbound (and would expect it to be flushed on reboot as I have "Flush DNS Cache during reload" checked). But I imagine the DNS log would have looked different if Unbound was replying with a cached result? It showed what looks like a normal reply from 1.1.1.1, not a cached response.

No Web proxy enabled either.

Thanks for reading.

[CJ]

Good question - I had DNS servers configured in the flow you described: System -> General -> DNS Servers. No custom forwarding configured in Unbound and it's using the system nameservers. DHCP indeed hands out only Unbound as the DNS server.

Do you have Use System Nameservers checked in either the Query Forwarding or DNS over TLS pages of Unbound?

Is allow DNS to be overridden by DHCP checked on the settings page?

[Mirmirmi]

Yes, "Use System Nameservers" is checked under Query Forwarding and DNS over TLS. the DNS over TLS page has nothing in the custom forwarding section either.

"Allow DNS server list to be overridden by DHCP/PPP on WAN" was checked when this issue started, but I found it early in troubleshooting, unchecked it, and the issue continued and persisted across reboots. It's unchecked now.

Other DNS settings I can think of:

• "Do not use the local DNS service as a nameserver for this system" is unchecked.
• "Disable DNS Rebinding Checks" is unchecked.
• "Register DHCP Leases" is checked.
• "Regisiter DHCP Static Mappings" is checked.
• The Unbound Blocklist is not enabled.
• My DHCPv4 page has nothing defined in the DNS servers section.
• There is one single Host Override in Unbound DNS for a local IP. No domain overrides.

I went to go try and trigger the behavior again by changing my system DNS servers to only 1.1.1.1 and restarting Unbound and it didn't happen (it did the same day the issue began, I could trigger it) - everything was normal & expected. So I guess that kinda leads the troubleshooting trail dead.

[cookiemonster]

I guess it does as normally there is no verbose logging enabled to look back due to storage requirements but from the history of it it makes me suspect it was the configured dns server at the time that was misconfigured momentarily. I read it that you saw the unexpected responses given to clients by unbound were what unbound received. So did as was told?