Configuring a failover mechanism (firewall rules) in case my Pi-hole goes down

[New_User]

Hi all,

I need your help with configuring a failover mechanism (firewall rules) in case my Pi-hole goes down.

Background:

I outsourced all the DNS resolutions from my OPNSense to my Pi-hole (runs unbound).

I would like to setup a backup DNS solution if my Pi-hole goes down – whether configuring an unbound service on the OPNSense itself or just using an external DNS provider (cloudflare or similar).

Currently I have only the following rules:

i) rule no. 1 – redirects all DNS requests from port 53 on OPNsesne to Pi-hole (port 5335)
ii) rule no. 2 - intercepts any outgoing DNS queries and redirects them to Pi-hole (port 5335)

The issue is, that I do not know how (assuming it is possible) to use a conditions within FW rules (if IP1 is down, forward it to IP2).

Any help would be highly appreciated.

Thank you in advance!

[bartjsmit]

DNS is designed for failover. Build a second pi-hole and issue these DNS option 6 addresses via DHCP (or static configuration on non-DHCP hosts)

<pi-hole1.IP.addr>
<pi-hole2.IP.addr>
9.9.9.9

Bart...

[New_User]

Hi Bart,

Thank you for your response and sorry for not replying sooner.

I don't mind buying a second raspberry pi as a backup, but I'm not sure that I fully understood your suggestion.

It is my understanding that I have two places in OPNsense where I can configure DNS:

1) System --> General --> DNS serves + gateways

2) Services --> DHCPv4 --> DNS servers

You mentioned the second option, which I prefer, because it allows me to set a specific DNS for each VLAN.

However, under Services --> DHCPv4 --> DNS servers, there are only two fields to specify 2 DNS servers (not three).

Probably it was not clear from my post, so it is worth noting that the DHCP server runs on my OPNsense (an not on my pi-hole).

In any event, how do I force applications with hard-coded DNS addresses to go through specific DNS servers (in my case: pi-hole no. 1 & pi-hole no. 2, if the first one is down) without firewall rules? And if I need firewall rules for that, how do I set FW rules to redirect DNS queries to one of the two DNS servers (whichever is up)?

Thank you again!

[littlepepper]

In any event, how do I force applications with hard-coded DNS addresses to go through specific DNS servers (in my case: pi-hole no. 1 & pi-hole no. 2, if the first one is down) without firewall rules? And if I need firewall rules for that, how do I set FW rules to redirect DNS queries to one of the two DNS servers (whichever is up)?

Thank you again!

You can try this: https://labzilla.io/blog/force-dns-pihole (it is actually mostly NAT)

Also if you run more than 1 pihole consider gravity sync or other syncing utilities.

[gspannu]

Hi all,

I need your help with configuring a failover mechanism (firewall rules) in case my Pi-hole goes down.

Here is a solution... see if it works for you.

This will provide you with your primary resolver as PiHole and secondary resolver as Cloudflare (or Google or something similar). If your PiHole is accessible, it will always use PiHole, but if PiHole is down, it will use your secondary resolver. If I have read your post correctly, this is what you are intending to do...

1. Setup your PiHole (external) as a DNS resolver, you can use port 53 (the default).
Check that PiHole is resolving, issue something like nslookup google.com ip.of.pihole and test.

2. Create a custom dnsmasq conf file.
In OPNsense shell,

Code Select Expand

sudo nano /usr/local/etc/dnsmasq.conf.d/0-subnet-strict.conf
add the following 3 lines to the file

Code Select Expand

add-subnet=32
add-mac
strict-order

Save the file.

3. Goto System > Settings > General
DNS server settings
First entry: PiHole IP address (without any port numbers, as 53 is default)
Second entry: 1.1.1.1 or 8.8.8.8 or whatever you wish to choose
Third entry: Can be something similar (if you wish to have a 3rd)

4. Disable Unbound on OPNsense (we will use dnsmasq instead)

5. Enable dnsmasq instead, ensure that you use the default port 53.

6. Do not have any specific DNS servers in DHCP settings, leave blank... or set to OPNsense router address itself.

7. OPNsense shell, check the output of cat /etc/resolv.conf
This should now reflect your addresses of  Step 3.

8. Adjust your firewall rules.
- You only need your Rule 2, to force any errant devices to always use your OPNsense router address, port 53. Change from PiHole 5335 to OPNsense router address port 53.


Additional Info:

Step 2 explanation:
add-subnet=32: Thus passes your requesting clients IP address to PiHole, so you can see which client has requested the DNS resolution.
add-mac: Same as above, but sens clients MAC address. Remove line if not using.
strict-order: This is the important line. This forces dnsmasq to use the DNS servers in the order you have specified in Step 3. This is what forces OPNsense to use PiHole first, then 1.1.1.1 and so on...

DNS resolution order Test:
If DNS resolution works, now try unplugging your PiHole or shutting it down.... You will notice that DNS resolution still works...

Hope it helps...

[bartjsmit]

under Services --> DHCPv4 --> DNS servers, there are only two fields to specify 2 DNS servers (not three).

Two is better than one but three is the maximum.

Remember that certain clients (*cough Windows *cough) will only consider a DNS server unavailable when it stops pinging. A dual DNS server is more for updating one at a time without an outage. A load balancer will give you high availability for partial failures.

Probably it was not clear from my post, so it is worth noting that the DHCP server runs on my OPNsense (an not on my pi-hole).

There are good arguments for having separate servers for different services. The built-in DHCP/DNS/Squid in OPNsense are there to provide a drop-in replacement for ISP routers IMHO. There are many posts in this forum about on-board WiFi from that same perspective.

Ars did a good article about this recently https://arstechnica.com/information-technology/2024/02/doing-dns-and-dhcp-for-your-lan-the-old-way-the-way-that-works/ I reckon the pi-hole makes a better DHCP than OPNsense but a dedicated server (or container) trumps both. It is also a good learning experience.

In any event, how do I force applications with hard-coded DNS addresses to go through specific DNS servers (in my case: pi-hole no. 1 & pi-hole no. 2, if the first one is down) without firewall rules? And if I need firewall rules for that, how do I set FW rules to redirect DNS queries to one of the two DNS servers (whichever is up)?

Hard coding DNS in an application is horrible and should be a red flag about its quality. Firewall rules cannot take the place of a load balancer, since the firewall does not have the ability to check the health of each server in the pool. Horses for courses.

Bart...

[New_User]

You can try this: https://labzilla.io/blog/force-dns-pihole (it is actually mostly NAT)

Also if you run more than 1 pihole consider gravity sync or other syncing utilities.

Thank you! Actually I used the above page when I configured my setup. Using syncing utilities is my next step, but first I would like to have a stable setup, so I can take my time and learn how to improve it  :)

[New_User]

Remember that certain clients (*cough Windows *cough) will only consider a DNS server unavailable when it stops pinging. A dual DNS server is more for updating one at a time without an outage. A load balancer will give you high availability for partial failures.

Thanks. Noted.

There are good arguments for having separate servers for different services. The built-in DHCP/DNS/Squid in OPNsense are there to provide a drop-in replacement for ISP routers IMHO. There are many posts in this forum about on-board WiFi from that same perspective.

Ars did a good article about this recently https://arstechnica.com/information-technology/2024/02/doing-dns-and-dhcp-for-your-lan-the-old-way-the-way-that-works/ I reckon the pi-hole makes a better DHCP than OPNsense but a dedicated server (or container) trumps both. It is also a good learning experience.

I've heard arguments for having separate servers, but frankly, I'm not knowledgeable enough in this regard to make any smart decisions (save for the basic stuff) that will be appropriate in my use case. But I will do some research on this.

Thanks for the link! It seems like a very good and interesting article. I'll read it once I find some time.

Thank you.

[New_User]

Here is a solution... see if it works for you.

gspannu, many thanks for your detailed explanation! :)

I followed your instructions (to the best of my understanding), but it seems that the setup does not work as it supposed to :-/

I guess I missconfigured something..

Under System → Settings → General I set the following three addresses (with no gateways):

1. pi-hole-ip
2. 1.1.1.1 (Cloudflare)
3. 9.9.9.9 (Quad9)

Gateway switching - Allow default gateway switching – not checked.

I left only one FW rule that intercepts any outgoing (external) DNS queries from clients and redirects them to 127.0.0.1#53 (btw, is this the right method achieving this goal?)

Afterwards, I performed the following test:
I disconnected my pi-hole from the AP to force clients to use the second in line DNS server (1.1.1.1 in my example) and rebooted my OPNsense and my AP.

However, when I perform a test for DNS leaks via a browser, in the results I see two servers – one for Cloudflare and one for Quad9. Now, if the second DNS server (Cloudflare) is reachable and responsive, why the third server (Quad9) is being contacted? I repeated this test many times, and even when my pi-hole was up, the second and third DNS servers were used..

Another thing, according to the information available under Services → Dnsmasq DNS → Log File, it seems that all the DNS servers are used at the same moment:
- Informational – dnsmasq - using nameserver 9.9.9.9#53
- Informational – dnsmasq - using nameserver 1.1.1.1#53
- Informational – dnsmasq - using nameserver PI-HOLE-IP#53

In addition, according to the log file, I get warnings of three types:

a) ignoring nameserver 127.0.0.1 - local interface
b) LOUD WARNING: use --bind-dynamic rather than --bind-interfaces to avoid DNS amplification attacks via these interface(s);
c) LOUD WARNING: listening on MY-IP-ADDRESS may accept requests via interfaces other than pppoe0

Is it normal?

Again, thank you very much!

[bartjsmit]

DNS is not a security enforcement technique. It exists because wetware cannot remember IP addresses :)

Firewall (policy) sits in level-3 of your network - that means that everything is by IP address. If you want to restrict higher up in the stack, you need a proxy (like Squid for web access)

As a service to make life easier for your users, it's great. A dedicated user (e.g. a teenager) can by-pass your restrictions by SOCKS, DOH, VPN, etc. Since malware isn't (yet) at that level, you can reduce its impact with a filtering DNS server (like pi-hole).

Concentrate on making DNS fast (local) and available (resilience). DNS on your firewall is for your firewall to use. Give your users the servers they need by DHCP and allow those servers to resolve.

Remember the KISS principle ;)

Bart...

[New_User]

Just a quick update - under Services -->  Dnsmasq DNS --> Settings, I checked the "DNS Query Forwarding - Query DNS servers sequentially" checkbox, and now it seems that the "strict-order" command is obeyed :-)

With respect to the FW rule that intercepts unwanted DNS queries from clients, I'll need to explore this technique a bit further to make sure that it works properly (I'm a bit confused as to the device that eventually should be sending the "permitted" inquiries to external DNS servers (Cloudflare or Quad9 in my example) - whether it should be the OPNsense device itself or the client directly).

DNS is not a security enforcement technique. It exists because wetware cannot remember IP addresses :)

Firewall (policy) sits in level-3 of your network - that means that everything is by IP address. If you want to restrict higher up in the stack, you need a proxy (like Squid for web access)

As a service to make life easier for your users, it's great. A dedicated user (e.g. a teenager) can by-pass your restrictions by SOCKS, DOH, VPN, etc. Since malware isn't (yet) at that level, you can reduce its impact with a filtering DNS server (like pi-hole).

Concentrate on making DNS fast (local) and available (resilience). DNS on your firewall is for your firewall to use. Give your users the servers they need by DHCP and allow those servers to resolve.

Thank you. Eventually I will use DHCP to distribute the relevant DNS servers among certain groups of users (VLANs).

However, I still receive (in the Dnsmasq DNS - Log File) the below warnings:

a) ignoring nameserver 127.0.0.1 - local interface
b) LOUD WARNING: use --bind-dynamic rather than --bind-interfaces to avoid DNS amplification attacks via these interface(s);
c) LOUD WARNING: listening on MY-IP-ADDRESS may accept requests via interfaces other than pppoe0
d) possible DNS-rebind attack detected: pi.hole

Is there anything I can / should do to address these warnings?

Thank you all!