DNS, Forwarder, Unbound, wtf?

[etrask]

Okay guys.. I'm completely stumped. I've setup dozens of firewalls in my day, though admittedly I have never used OPNsense before. I am sure I am missing something completely stupid and obvious.

I have a very simple embedded PC I want to use as my perimeter router after I have confirmed functionality. It has 3 NICs - one will be LAN uplink, the other two bridged as a LAN network. I have already installed OPNsense 17.1.2 and configured this properly.

On the LAN side I want the firewall to do a bit of the networking infrastructure "heavy lifting" - I want it to run an NTP server, DHCP, DNS, IDS, and eventually a point to point VPN.

However I am running into a baffling problem with DNS I cannot seem to figure out. Under System -> Settings -> General I have specified two OpenNIC servers I want all the DHCP clients to use (as well as the firewall itself). These servers work, I have verified independently (52.175.214.157 and 45.32.230.225 for the curious). However, every once in a while, for no discernible reason, the firewall itself will stop being able to resolve anything, and ALL LAN clients will have their DNS requests, to ANY SERVER, blocked/rejected/filtered (can't quite tell which).

Right now I have DNS Resolver enabled, with "Enable DNSSEC support", "Enable forwarding mode", "Register DHCP leases in the DNS resolver", and "Register DHCP static mappings in the DNS resolver" set. But I should note, I get this wonky behavior if I switch to DNS Forwarder or disable both altogether.

Right now I have this OPNsense box daisy-chained under my perimeter router. So the WAN side is 192.168.1.x, the LAN subnet is 10.0.0.0/24. The clients on the LAN side are pulling DHCP leases, and getting a DNS assignment of 10.0.0.1 from this. However, when I nslookup from the LAN:

Code: [Select]

Default Server: <router host name>
Address: 10.0.0.1

> opnsense.org
Server: <router host name>
Address: 10.0.0.1

*** <router host name> can't find opnsense.org: Server Failed
> server 8.8.8.8
Default Server: [8.8.8.8]
Address: 8.8.8.8

> opnsense.org
Server: [8.8.8.8]
Address: 8.8.8.8

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds
DNS request timed out.
    timeout was 2 seconds.
*** Request to [8.8.8.8] timed-out
If I disable both DNS servers and get new DHCP leases on the LAN, the servers I specify under General get pushed as DNS servers on the leases, but I still get this same behavior.

I thought I had figured out what was going on - on a lark I decided to put this thing on my perimeter and see if maybe having internal networks on both sides of the firewall was confusing the poor thing. It actually worked for a while. I was even able to use the WAN-side IP as a DNS server (obviously I had intended to change this). But then I enabled IDS on the firewall, and the nonsensical DNS behavior started again - my firewall fell off the internet, nobody on LAN could get a UDP DNS request out... it was chaos.

I have precisely 4 rules setup for the LAN bridge: allow HTTPS to the firewall, allow SSH to the firewall, allow all IPv4, and allow all IPv6. These may be redundant but I wanted to make damn sure I could always get to the web interface since this box has no video out. On the WAN side I have rules to allow HTTPS and SSH into the firewall as well, which were working well until this latest DNS dump.

I am at a complete loss. Any ideas?

[etrask]

So I have a bit of an update... as with this person: https://forum.opnsense.org/index.php?topic=4643.0

...rebooting and leaving the device alone for a while seems to temporarily resolve the issue. DNS will come back up at some point, but it's impossible to predict. It's also impossible to predict how LONG it will stay up, as it will start failing again for no reason again.

Unfortunately I am using a NanoBSD install so I don't believe it keeps logs between power cycles. I will see if I can enable that (and maybe more verbose logging for some service?) and see if I can provide those details.

This is just a very odd problem to have, but of course completely killing all DNS is an absolute deal breaker for this otherwise stellar software

[Taomyn]

I think I had something similar a while back, I use the DNS Resolver, and if I enabled DNSSEC Support I too would see all DNS requests simply stop at random times. I thought I reported it, but perhaps I forgot.

[etrask]

A development of sorts: I had the router sitting my perimeter for about a week with no issues. Then earlier today it started exhibiting the same behavior as before - for no discernible reason, ALL DNS requests were blocked on the LAN. Didn't matter whether the lookup went to the router or to an external DNS - nothing at all.

Nothing had changed on the router for 20+ hours before this... it was sitting there with very little traffic (nobody home during the day). But when my roommate gets home he reports no DNS, etc.

Are there maybe some logs I can dig through and share here for more info? I really think there is a bug in here somewhere, and I'd like to help fix it any way I can!
(Difficulty: the thing has power-cycled since the last incident, not sure I even HAVE logs)