Messages

I upgraded 5 systems from 24.7.4_1 to 24.7.5.  The first three with Crowdsec installed.  All three with Crowdsec returned to the Lobby without rebooting.  On all three, I then initiated a reboot, two of which rebooted fine, the third hung up hard and had to be power cycled to recover it.

For the next two, I uninstalled the Crowdsec plugin, then initiated the upgrade.  On both of these systems, the upgrade went exactly as expected.  One of them did return to the Lobby briefly before rebooting, but both did reboot on their own withour issue.  I then re-installed the Crowdsec plugin withour issue.

This is certainly a Crowdsec issue.

I have five more system to upgrade, but I'll wait for a hotfix to test on them.

Upgrade to 24.7 went smoothly.  However, I did notice a cosmetic issue with the CPU Widget.  I have a Core i5-1135G7 CPU with 4 Cores, 8 threads.  The Widget is reporting it as 8 Cores, 4 threads.

[Flush DNS Cache during reload]

I've been troublshooting this issue for months, and didn't fully realize it was Unbound until reading this post.  Now when my internet goes down, TOP shows the unbound process at 100%.  I block DoH, and intercept all DNS and forward all Unbound request to DNSCrypt-Proxy, which means I have no fallback when Unbound goes stupid.  When this setup works, it works wonderfully.  However, when it stops, all DNS queries on my network go unanswered.  Even with this patch my unbound process would end up at 100% utilization nightly and I had to kill -9 <pid> it to get it to recover.

For now I've switched to using DNSCrypt-Proxy native, and I haven't seen a problem in almost two weeks.  However, I'm willing to go back to Unbound in the name of testing and troubleshooting for any developer that is willing to look into fixing this. I just need hints of what to collect for when it goes stupid.

I do have Flush DNS Cache during reload enabled, which I now wonder if that exacerbated this issue.

# configctl unbound check

Output would be good ;)


Cheers,
Franco

admin@RedactedOPNsense:~ % configctl unbound check
no errors in /var/unbound/unbound.conf
a

I'm more than willing to include any output requested, but I *think* I may have narrowed it down.  I have three "LAN" interfaces, I'll call [HOME], [GUEST], and [IOT].  I've grouped these three interfaces into one group called "LAN" and pretty much all my firewall rules for the three interfaces are managed in that single group, including allowing DNS to "This firewall".

Second, I have NAT snagging all TCP/UDP with destination port 53 being redirected to loopback (127.0.0.1 for IPv4 and ::1 for IPv6 - two different NAT rules).

After adding the internal Loopback interface to the LAN group (and removing all other changes, reverting back to how it was originally borked), things started to work again.

I can only speculate, but maybe the IPv6 NAT makes the request look like it is coming in on the Loopback interface, and that interface needs the allow rule for IPv6 based DNS traffic.  I've not seen this problem before, so it seems a recent change may have caused it, but it appears to be a firewall issue and not an Unbound issue.  Also, without the Loopback being added to the "LAN" group DNS queries to The IPv4 address always work, so it's specific to IPv6.

admin@RedactedOPNsense:~ % configctl unbound check
no errors in /var/unbound/unbound.conf

I went ahead and created a loopback for IPv6 DNS and assigned it:

fd00::1

I made sure my firewall rules allow DNS to this interface and then changed all the SERVICES: ROUTER ADVERTISEMENTS: [LAN] DNS entries to use this IP address, and it appears to be working.

Anything you would do differently?

Did you set the Unbound network interfaces to "All (recommended)"? Binding services to specific interfaces with dynamic addresses is discouraged.

Advertising LAN interface link-local addresses as DNS servers is a viable alternative. But you have to remember to change the RA / DHCPv6 settings when you change the interface assignments or replace a NIC.

My preferred configuration is an additional loopback interface which Unbound binds to. It has a ULA which is advertised as the DNS server address in all LANs.

Cheers
Maurice

Yes, I left the default at "All (recommended)"

I'm curious about your loopback idea as it seems like a much better solution, what settings did you use for that Interface?

You get that option as soon as you check "Interfaces: LAN" - "Manual configuration - Allow manual adjustment of DHCPv6 and Router Advertisements" and save.

Its the last option in "Track IPv6 Interface".

Thanks for the clarification.  I've set up all three LAN interfaces with their respective link-local addresses for DNS.  So far it appears to be working.  I'll follow up if I see any issues.

It seems odd that this isn't the default IPv6 DNS server advertised without changing to manual configuration.  It used to work fine with defaults, but hopefully this resolves the issue.

What plug-in are you using to get SERVICES: ROUTER-ADVERTISEMENTS:?  I don't have that on my router.

I'm sending the query to watever is being sent by OPNsense, not sure how my devices get IPv6 DNS servers using SLAAC, but it's not the Link-local IPv6 Address.  In my testing I did try to check the box in the Interface settings:  "Allow manual adjustment of DHCPv6 and Router Advertisements," and enabled DHCPv6 on my LAN interface and it assigned the same IPv6 address for DNS and then Unbound failed to respond over IPv6 on that address even if I only had IPv6 configured on the one interface.

I've noticed since upgrading to 23.7.6 that I would randomly lose Internet access.  It could have started earlier, but I've just noticed it in the last week or so.  After a boat load of troubleshooting, I seem to have narrowed it down to Unbound and IPv6 on multiple interfaces not working nicely together.  So, I'm getting IPv6 prefixes delegated to me by my ISP and using track-interface to assign the prefixes to each interface.  If I only assign/track IPv6 to one interface, I see no issues.  As soon as I assign/track IPv6 on additional interfaces then I can no longer do DNS queries to the IPv6 address on each respective subnet on the OPNsense router.  However, DNS queries to the IPv4 address continue to work fine.  For now I've disabled IPv6 on all but my primary LAN interface and it seems to be working,  It seems something was introduced recently in the code that started causing this problem as I haven't changed any configs in some time.

This is also fixed in 23.7.3

Thanks for the follow up Franco.  The patch you provided indeed fixes the issue on 23.7.2.

https://github.com/opnsense/core/commit/843c4d84

# opnsense-patch 843c4d84

Strange as there was a fix for this specifically... https://github.com/opnsense/core/commit/ac3b9740e2


Cheers,
Franco

I have eight OPNsense routers I manage, and they have all been updated to 23.7.2, and they are all still having this issue.  So, maybe that fix needs a fix?

I am on 23.7.2