OPNsense handling DNS requests from clients when all DNS services are disabled

[dodgeboy]

I have been using PiHole for my local DNS server for years, but I'm considering switching to Unbound on my OPNsense box.  I started to tinker with it and I noticed something odd which is preventing me from proceeding.  OPNsense is answering (proxying?) DNS requests from devices on my LAN even when all DNS services are disabled.  Furthermore, if I enable Unbound, the overrides don't work.  A dig trace shows that my PiHole server is answering the request, even though I'm directing dig to OPNsense.  If I do an NSlookup and specify the server as OPNsense, OPNsense answers requests for anything on the internet but fails to answer for any hosts in the override list. 


Is there a setting buried somewhere which enables transparent DNS proxy/relay?  I tried deleting the DNS servers in the general settings, but it continues to process requests from clients.

[EricPerl]

You can check which process is handling DNS with

Code Select Expand

sockstat | grep :53
Are your host overrides enabled?

With regards to Unbound, if you don't have anything enabled in "Query Forwarding" or "DNS over TLS", Unbound will behave as a recursive resolver.
You can actually track what's going in or out of your firewall using the live view and filters (e.g. dst_port is 53, or 853 for DoT).

[verfluchten]

It's worth noting the version.
I'm on 21.7 and not seeing anything of the sort. Overrides work, and nothing bypasses the OS.

[dodgeboy]

You can check which process is handling DNS with

Code Select Expand

sockstat | grep :53
Are your host overrides enabled?

With regards to Unbound, if you don't have anything enabled in "Query Forwarding" or "DNS over TLS", Unbound will behave as a recursive resolver.
You can actually track what's going in or out of your firewall using the live view and filters (e.g. dst_port is 53, or 853 for DoT).

Thanks for the sockstat command.  When unbound is disabled, nothing is listening on port 53 (but the FW is still handling DNS requests). When unbound is enabled, there's a list of unbound services listening on port 53, as you'd expect.  However, the client behavior is the same. 

Query forwarding and DNS over TLS pages are both blank.

FW live log shows the request from the client to the FW hitting my inside-out rule, so it appears the FW is transparently proxying/redirecting my DNS request to the FW itself.  I have not knowingly configured any functionality like that, and I'm not even sure where/how it would be configured.

I am running an HA pair (CARP) with version 24.7.11_2.

[EricPerl]

I'm a little baffled.
As soon as I disable Unbound (double checking that the command yields nothing), dig & nslookup timeout (from a PC on the LAN side), as expected.
Even from within the firewall (Interfaces > Diagnostics), DNS lookups fail if Unbound is disabled (unless there are servers in System > General, explicitly or as override by DHCP on WAN. These only work internally by default, but Unbound can be configured to use them).

All the behaviors you're mentioning point to Unbound not being used... DNS via GW working from LAN while Unbound is disabled, overrides ignored.
For a given DNS request, can you show the frames in and out as shown in the firewall live view?

[dseven]

NAT port-forward on port 53 maybe?

[EricPerl]

That indeed would match the symptoms... I didn't think of that because I've never used port forwards on the LAN side.
I knew people do this to force them to use their DNS server though. It just didn't click.

If that was the cause, so much for not tinkering with DNS...

[dodgeboy]

I'm just as baffled as you.  The behavior makes no sense to me. 

I attached a screenshot of the live log of the DNS request with my static public IP obfuscated.  192.168.254.249 is the IP of OPNsense.  The transaction with the DNS server itself isn't recorded in the log. 

If I use the interface diagnostics to do a DNS lookup, it times out if I specify the FW as the DNS server, but it succeeds if I leave it blank, which uses the general system specified dns servers.  If I enable Unbound, it succeeds if I specify the FW as the server.  The fact that it fails when unbound is disabled is the first part of this scenario that has made sense to me.

I do have a NAT port forward for TCP/UDP 53 on the WAN, but not the LAN.  Disabling that port forwarding rule has no change on the behavior.  There's no other NAT port forwarding rules for port 53.

I feel like in this scenario I could benefit from something similar to the packet tracer feature on older Cisco ASAs.  You could simulate a packet and it would show every nat/fw rule it would hit in the firewall. 

[EricPerl]

I was unbaffled by dseven.
A port forwarding rule on the LAN side produces exactly your symptoms.
The port forward picks up DNS request (to any server) and sends it to the redirect.
In order of time, in the live view:

Code Select Expand

NAT   LAN in  client_IP:someport  any_DNS:53
Allow LAN in  client_IP:someport  redirect_IP:redirect_port=53
Allow WAN out routerWAN_IP:random redirect_IP:redirect_port=53
DNS inside the router is entirely bypassed. In the test above, Unbound was disabled.

Can you show your port forward rule?
If you have not done so, enable logging. It helps in locating the transformations...

What's the purpose of port forwarding 53 on the WAN side?
Did you notice that you're forwarding from WAN BACK IN to the LAN IP?

[dodgeboy]

I don't disagree that a NAT port forward on the LAN would produce identical symptoms, and that is exactly how my firewall is behaving.  However, I don't have a port forward rule on the LAN for port 53... at least not one that shows up in the NAT > Port Forward menu.  Is there another location for configuring port forwarding?

The purpose of my DNS port forward on the WAN interface is to redirect inbound DNS requests on the WAN VIP to a DNS server in my DMZ without consuming a public statis IP address for this purpose.  If I disable that rule it stops inbound DNS requests to that server and does not change the redirect behavior what we are troubleshooting on the LAN.  The WAN port forward rule does have logging enabled.

Here is my port forward rules:

Code Select Expand

Interface Proto       Address Ports Address Ports         IP         Ports         Description
WAN         TCP         * * [WAN VIP] 8989         [DMZ Host IP] 80 (HTTP)
LAN WAN UDP         * * [WAN VIP] 42000         [DMZ Host IP] 42000
WAN         TCP/UDP * * [WAN VIP] 53 (DNS) [DMZ Host IP] 53 (DNS) Backup public DNS
WAN         TCP         * * [WAN VIP] 33123         [DMZ Host IP] 33123
WAN         TCP         * * [WAN VIP] 33124         [DMZ Host IP] 33124

Is there a command, like sockstat, which shows all ports the firewall is redirecting?

[dseven]

I don't disagree that a NAT port forward on the LAN would produce identical symptoms, and that is exactly how my firewall is behaving.  However, I don't have a port forward rule on the LAN for port 53... at least not one that shows up in the NAT > Port Forward menu.  Is there another location for configuring port forwarding?

...

Is there a command, like sockstat, which shows all ports the firewall is redirecting?

If you have "Reflection" enabled (Firewall -> Settings -> Advanced), it'll create rules for your LAN(s) automatically.

You can see the them (rdr rules) under Firewall -> Diagnostics -> Statistics -> rules -> nat rules

[dodgeboy]

Thank you. Reflection is disabled, and after scanning through the NAT rules in the diagnostics section there is nothing there redirecting port 53. I really thought the diagnostic screen was going to show a translation that I didn't see in the configuration. The mystery continues.

[EricPerl]

AFAIK, these settings are only relevant if "NAT Reflection" is set to "Use System Default" in the port forwarding rule.
It's the default option but it could be set to Enable. This said, this is reflected (no pun intended) in the diagnostics...

I assume you have the port forward rule set to log.
Do you see it anything in the live view when making outbound DNS requests?

And you indicate your WAN rule points to a host in the DMZ.
Yet the log indicated a "forward" to the OPN LAN IP!!! I don't understand how this does not create an endless loop.

[dodgeboy]

Logging is enabled for every FW rule and every NAT/Port Forward rule until I get this figured out. 

I had a bit of a "breakthrough" this evening.  I created a new LAN FW rule and put it above my standard outbound permit rule so I would hit this specific FW rule when testing DNS.

Code Select Expand

Protocol Source         Port Destination Port         Gateway         Schedule        Description
IPv4 UDP 10.0.0.170 * This Firewall 53 (DNS) *                 * Laptop to FW DNS
IPv4            *         * *         *         Cable_and_Cellular * Inside out permit

With this rule in place, DNS behaves as it should.  It times out when unbound is disabled, and it resolves when it is enabled.  So I started looking into what was different between this specific rule and my normal inside out permit rule.  The difference, and the trigger for this condition, is the gateway.  If I have the gateway set to default (*), it behaves as it should.  If I set the gateway to my "Cable_and_Cellular" group, it does the weird redirect.  The gateway group exists so that my traffic will failover to my cellular gateway (external to OPNsense) if my main internet fails.  This works flawlessly.  I attached screenshots of my gateway and gateway group configuration.  What about this configuration would cause this behavior?  I don't see anything that would cause it. 

[EricPerl]

That's pretty much standard setup for multi-WAN per Multi WAN...
See step 5.
You can change the source to any.

Also, see first note at Policy based routing.