Opnsense keeps getting DNS rate limited by Pi-hole

[Mpegger]

I have a wierd issue on my network and I'm stumped as to the origin. Opnsense, for the 2nd time in a week, has gotten rate limited by Pi-Hole, even with the limit set to 6000 DNS quieres in a minute by a client. I wasn't able to see anything in the Pi-hole logs the last time it happened (I didnt noticed it before "old" logs were purged), but this time I caught it. Seems like Opnsense is making DNS requests for "quantumwarp-dot-com", almost 8k requests in less then 15 seconds. I don't have any logging on the Opnsense side that would show why that is happening, but from the looks of Pi-hole, it's originating from Opnsense, as all LAN DNS requests must go through Pi-hole first, even those from Opnsense, before going through Unbound (on Opnsense) and out to the WAN. The last time I had malformed DNS requests, it was because of a old alias that was trying to update a no-longer installed script in Opnsense. Currently, I have nothing custom installed in Opnsense, and it is up to date. I do have some "new" to my network docker apps running, but again, the origin is showing as Opnsense.

Anyone else seeing these requests?

How should I go about finding the origin of these requests? Just start logging all DNS requests from Unbound on Opnsense?

My LANs DNS flow is as follows: Unbound on Opnsense makes WAN DNS queries, and is firewalled to only accept DNS request from the 2 Pi-hole DNS servers on my LAN. All LAN devices via DHCP settings are directed to the Pi-holes, and any attempts by LAN devices to connect to WAN DNS servers via port 53, are redirected to the 2 Pi-hole DNS servers. All DNS request which go through Pi-hole, would show up in the Pi-hole logs with the LAN client origin, which means if it was some other device other the Opnsense making those 8k requests, the log would show which client. In this case, it shows that its Opnsense.

Just checked 2nd DNS server logs and that one had over 3k requests in the same time frame. So all in total, Opnsense is making 11k DNS requests in less then 15 seconds for that one domain.

[patient0]

You would have to turn up logging on OPNsense and also maybe enable logging on the DNS redirect rule on OPNsense.

All DNS request which go through Pi-hole, would show up in the Pi-hole logs with the LAN client origin, which means if it was some other device other the Opnsense making those 8k requests

If a client makes a DNS request to OPNsense instead of pi-hole, do you still see the client as the requester on pi-hole? Meaning: your pi-holes are on a different subnet and you NAT the requests?

[Mpegger]

If a client makes a DNS request to OPNsense instead of pi-hole, do you still see the client as the requester on pi-hole? Meaning: your pi-holes are on a different subnet and you NAT the requests?

My IPv4 LAN is on a single /24, no vLANs or seperate subnets. However, in your example, the origin should show as Opnsense, but again, I have Unbound (on Opnsense) behind a firewall rule. Only the 2 Pi-hole DNS servers can connect to it. Any other port 53 attempts (IPv4 & IPv6) through the gateway (Opnsense) get redirected (NAT) to the Pi-holes.

Also, I've never been to that site, or any of its related github links. I doubt very much any else in my household has visited that site either considering it's a technology related site. So I can only think of something running on a schedule.

What logging will I need to turn on? Just the logging in Unbound, or Unbound & some other logs in Opnsense? Also, if I'm logging all DNS queries in Unbound, what kind of log sizes can I expect? Opnsense is only using 10% of a 30GB VD, but I worry what would happen if the logs fill the available space and Opnsense started wigging out (as I've had happen in the past when the drive was full). The logging may have to good on for some days or even over a week, as the last time this happened was almost 1 week ago and it's been quiet till today.

PS; Hopefully I didnt mix up any terminology, but everything is on a single net of 192.168.2.0/24. IPv6 is via SLAAC and DHCPv6 with a /56 from the ISP. I've been playing around with assigning ULA, but that's only been for 2 devices for testing while I stilll try to learn DNSMasq and figure out how (or if) I will segment my network.

[patient0]

My IPv4 LAN is on a single /24, no vLANs or seperate subnets

That means if something (virus/malware/rough application/script) is trying to contact a public DNS directly, it will show up on the pi-holes as if the query came from OPNsense but it may really be originating from a client.

Enable "Log packets that are handled by that rule" for your DNS-redirected-to-pihole rule. If a clients makes these request you would see them.
You cannot view this attachment.

Also, I've never been to that site, or any of its related github links

There is not much going on on that site anyway, no reason to visit it at all, but maybe it's infected.

[Mpegger]

My IPv4 LAN is on a single /24, no vLANs or seperate subnets

That means if something (virus/malware/rough application/script) is trying to contact a public DNS directly, it will show up on the pi-holes as if the query came from OPNsense but it may really be originating from a client.

Is this true? With the firewall rules I have in place, I cannot directly query Unbound sinces it's on Opnsense, from any of my PCs/VMs. All DNS requests get redirected to the Pi-holes. Or at least thats how it appears to be working to me.

NAT >  Port Forward

Code Select Expand

LAN TCP/UDP ! DNS_IPv4 * * 53 (DNS) DNS_IPv4    53 (DNS) DNS redirect IPv4
LAN TCP/UDP ! DNS_IPv6 * * 53 (DNS) DNS_IPv6    53 (DNS) DNS redirect IPv6
Rules > LAN

Code Select Expand

In IPv4+6 TCP/UDP DNS_Server  * * 53 (DNS) * * Allow Pihole to contact Unbound
In IPv4 TCP/UDP ! DNS_IPv4  * DNS_IPv4  53 (DNS) * * DNS redirect IPv4
In IPv6 TCP/UDP ! DNS_IPv6  * DNS_IPv6  53 (DNS) * * DNS redirect IPv6
In IPv4+6 TCP/UDP * * * 53 (DNS) * * Reject all DNS to WAN and Unbound
Is this wrong? Or must the DNS server be on a seperate subnet (vlan?) for it to work properly?

[Edit] Aha! I think I understand now what you are saying. You mean if Opnsense gets a DNS query, because it's NATing it to Pi-hole, it will appear to Pi-hole as if it's originating from Opnsense, and not the original client that made the query? [/edit]

[Edit2] It just happened again twice in a row. Thousands of DNS queries in seconds showing as originating from Opnsense, except this time it was more familiar sites, including some of my own LAN only FQDNs. Internet access also started to go down as site were no longer resolving. Restarting Unbound in Opnsense fixed that. Something is either acting up in Opnsense, or theres something else amiss in my network. [/edit]

[Patrick M. Hausen]

Why are you redirecting the requests?

Give all your clients the Pihole as a DNS server via DHCP and block outbound DNS for all but the Pihole.

This way Pihole will see the client IP addresses.

[Mpegger]

Some IoT devices ignore the DHCP assigned DNS servers, or have hard coded DNS servers. I recall my living room TV was one such device, ignoring DHCP assigned DNS server, even though it was using DHCP for its IP address. Had to go digging deep in menus to disable it using its own "preferred" DNS servers. Now-a-days, you also have web browsers that will also ignore the assigned DNS servers in favor of using whatever DNS servers they want as well. It's also why I run with 2 DNS servers instead of 1, cause Android devices will automaticallly use a Google DNS server for the 2nd "backup" DNS server if only given 1 DNS server through DHCP (and tend to prefer the Google server over the primary assigned one). I have DNS over TLS port blocked as well, but clients trying to use that port will usually fall back to normal DNS, so that's still covered without any additional rules besides the block. Only thing I can't realiable block is DNS over HTTPS, since that would require blocking some commonly used websites.

[patient0]

You redirect does work, indeed.

If I remember correctly, in the case of a device trying to query (let's say) 1.1.1.1, on pi-hole the originating client is listed as OPNsense, since it is forwarded from OPNsense. Is that not so? That is why I thought it would be helpful to have logging for one or all of these rules enabled, temporarly.

If you now manually query 1.1.1.1 from a client, what does the pi-hole show as the client?

[Mpegger]

You redirect does work, indeed.

If I remember correctly, in the case of a device trying to query (let's say) 1.1.1.1, on pi-hole the originating client is listed as OPNsense, since it is forwarded from OPNsense. Is that not so? That is why I thought it would be helpful to have logging for one or all of these rules enabled, temporarly.

If you now manually query 1.1.1.1 from a client, what does the pi-hole show as the client?

It does show as originating from Opnsense when the request gets redirected (NAT).

Is it possible that Opnsense can get into a infinite loop with the DNS server(s)? I noticed that the Pi-holes sometimes use thier GUA address to make a DNS request, which is not one of the addresses I have in the "DNS_IPv6" alias in the firewall rules. The GUA address is being created using the ISP assigned /64 prefix, which is *not* fixed, via SLAAC, as well as creating another GUA address via DHCPv6. Neither of those IPv6 addresses are in the alias, so I can imagine a loop occuring, but the logs currently don't show that happening.

[patient0]

Is it possible that Opnsense can get into a infinite loop with the DNS server(s)?

I'm really not sure, in what scenario would you think that a loop be possible? The flow should be either 'client - pi-hole - unbound' or 'client - OPNsense - pi-hole - unbound'. Would there be a situation where unbound sends a request to pi-hole?

[meyergru]

There could be such situations, e.g. if "Services: Unbound DNS: Query Forwarding" has a DNS search domain forwarded to pi-hole.

We discovered, that with the DNSmasq/Unbound combination (where this is advised), you can see that Windows clients append the DNS search domains to any name and then ask the DNS forward server.

BTW: The domain in question has a few projects going, like three Joomla modules and a CRM system, all of which might also point to your "new" docker apps as being the culprit.

[Mpegger]

Is it possible that Opnsense can get into a infinite loop with the DNS server(s)?

I'm really not sure, in what scenario would you think that a loop be possible? The flow should be either 'client - pi-hole - unbound' or 'client - OPNsense - pi-hole - unbound'. Would there be a situation where unbound sends a request to pi-hole?

The alias that I use in the NAT rule that allows the Pi-hole servers to use the Unbound service on Opnsense, only has the IPv4 and LLA IPv6 of each Pi-hole server. If the Pi-hole server uses any of thier GUA IPv6 address to try to query the Unbound service on Opnsense, Opnsense would just forward (NAT) that request back to the Pi-hole server since the GUA is not in the alias of that NAT rule. I can see that happen on a regular basis in the logs, but again, I don't see (yet?) any appear to be a infinite loop. I've yet to see the issue crop up again so far since removing the ULA prefix (which the Pi-holes were also using) from my network.