OPNsense Forum

English Forums => 24.7, 24.10 Legacy Series => Topic started by: longtom on December 03, 2024, 02:32:42 PM

Title: 24.7.10 Unbound DNS: DNS over TLS NOK? -> OK with 24.7.10_1
Post by: longtom on December 03, 2024, 02:32:42 PM
My Unbound DNS over TLS is NOK with 24.7.10.
When i disable DNS over TLS it is OK again.

Edit: With todays version 24.7.10!
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: branbon on December 03, 2024, 03:06:48 PM
Also seeing similar issue with DNS over TLS. I had no System DNS servers set & was relying on Unbound to handle the resolution. (Adding a System DNS server remedied the issue for me for now)

Sample Unbound log:

2024-12-03T08:51:22-05:00 Notice unbound [57387:1] notice: ssl handshake failed 1.1.1.1 port 853
2024-12-03T08:51:22-05:00 Error unbound [57387:1] error: ssl handshake cert error: unable to get local issuer certificate
2024-12-03T08:51:22-05:00 Error unbound [57387:1] error: and additionally crypto error:0A000086:SSL routines::certificate verify failed
2024-12-03T08:51:22-05:00 Error unbound [57387:1] error: and additionally crypto error:80000002:system library::No such file or directory
2024-12-03T08:51:22-05:00 Error unbound [57387:1] error: and additionally crypto error:16000069:STORE routines::unregistered scheme
2024-12-03T08:51:22-05:00 Error unbound [57387:1] error: and additionally crypto error:80000002:system library::No such file or directory
2024-12-03T08:51:22-05:00 Error unbound [57387:1] error: and additionally crypto error:16000069:STORE routines::unregistered scheme
2024-12-03T08:51:22-05:00 Error unbound [57387:1] error: and additionally crypto error:80000002:system library::No such file or directory
2024-12-03T08:51:22-05:00 Error unbound [57387:1] error: ssl handshake failed crypto error:16000069:STORE routines::unregistered scheme


NTP was also unable to synchronize due to failed DNS lookups.
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: FullyBorked on December 03, 2024, 03:30:10 PM
Yup, add me to the list as well.  Took me a bit to figure out why DNS was borked after the update.  Disabling DNS over TLS resolves it.
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: dsh1705 on December 03, 2024, 03:37:36 PM
I am seeing the same behavior.
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: yuusou on December 03, 2024, 03:38:02 PM
Just posted on the other thread with the same issue.
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: KHE on December 03, 2024, 03:38:13 PM
From the release notes:
o system: remove the SSL bundles in default locations

Is this unbound still using these SSL bundles?
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: franco on December 03, 2024, 03:41:20 PM
Looks like an Unbound bug to me:

# opnsense-patch https://github.com/opnsense/core/commit/cdb8da72661

Patch, apply Unbound settings, test again. I can hotfix and see what fix upstream needs here.


Cheers,
Franco
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: franco on December 03, 2024, 03:47:04 PM
Apparently it's a feature they coined to be for "Windows" and default to off?

tls-win-cert: yes

instead of tls-cert-bundle... can anyone confirm?


Thanks,
Franco

https://nlnetlabs.nl/documentation/unbound/unbound.conf/
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: jphylips on December 03, 2024, 03:50:45 PM
Hi Franco,

The patch seems to work from my end.

Thanks a lot.
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: yuusou on December 03, 2024, 03:52:00 PM
How can I apply the patch and restart the service without overwriting the configuration? pluginctl overwrited dot.conf. *BSD is not my strong suit.

EDIT:
nevermind, realized opnsense uses template files after using my eyes and looking at the repo properly. All fixed now. Thank you!
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: gac on December 03, 2024, 03:53:58 PM
Patch works for me.
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: franco on December 03, 2024, 03:55:33 PM
Quote from: yuusou on December 03, 2024, 03:52:00 PM
How can I apply the patch and restart the service without overwriting the configuration? pluginctl overwrited dot.conf. *BSD is not my strong suit.

Just run the "opnsense-patch URL" command in the shell. It will do everything except hit apply for you.
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: FullyBorked on December 03, 2024, 03:57:47 PM
Quote from: franco on December 03, 2024, 03:47:04 PM
Apparently it's a feature they coined to be for "Windows" and default to off?

tls-win-cert: yes

instead of tls-cert-bundle... can anyone confirm?


Thanks,
Franco

https://nlnetlabs.nl/documentation/unbound/unbound.conf/

I don't see either of these entries in my unbound.conf file.  Should I check somewhere else? 
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: gac on December 03, 2024, 03:59:08 PM
Quote from: FullyBorked on December 03, 2024, 03:57:47 PM
Quote from: franco on December 03, 2024, 03:47:04 PM
Apparently it's a feature they coined to be for "Windows" and default to off?

tls-win-cert: yes

instead of tls-cert-bundle... can anyone confirm?


Thanks,
Franco

https://nlnetlabs.nl/documentation/unbound/unbound.conf/

I don't see either of these entries in my unbound.conf file.  Should I check somewhere else?
They would be in /var/unbound/etc/dot.conf
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: franco on December 03, 2024, 04:00:52 PM
No, /usr/local/opnsense/service/templates/OPNsense/Unbound/core/dot.conf otherwise it will be overwritten on apply.
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: FullyBorked on December 03, 2024, 04:02:03 PM
Quote from: gac on December 03, 2024, 03:59:08 PM
Quote from: FullyBorked on December 03, 2024, 03:57:47 PM
Quote from: franco on December 03, 2024, 03:47:04 PM
Apparently it's a feature they coined to be for "Windows" and default to off?

tls-win-cert: yes

instead of tls-cert-bundle... can anyone confirm?


Thanks,
Franco

https://nlnetlabs.nl/documentation/unbound/unbound.conf/

I don't see either of these entries in my unbound.conf file.  Should I check somewhere else?
They would be in /var/unbound/etc/dot.conf

Hmm, ok the link he quoted mentioned the unbound.conf.  My dot.conf file other than a single forwarding zone is empty. 
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: franco on December 03, 2024, 04:03:27 PM
24.7.10_1 is now live...
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: FullyBorked on December 03, 2024, 04:04:23 PM
Quote from: franco on December 03, 2024, 04:00:52 PM
No, /usr/local/opnsense/service/templates/OPNsense/Unbound/core/dot.conf otherwise it will be overwritten on apply.

Thanks, mine is currently un-patched, I show " tls-system-cert: yes". 
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: franco on December 03, 2024, 04:08:02 PM
> Thanks, mine is currently un-patched, I show " tls-system-cert: yes".

Can you add "tls-win-cert: yes" in the line below (with the same indent) and apply from GUI?

If that doesn't work "tls-cert-bundle: /usr/local/etc/ssl/cert.pem" and removing "tls-system-cert: yes" will do the trick.


Cheers,
Franco
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: longtom on December 03, 2024, 04:11:13 PM
Thanks a lot for the quick patch!  :)
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: FullyBorked on December 03, 2024, 04:14:42 PM
Quote from: franco on December 03, 2024, 04:08:02 PM
> Thanks, mine is currently un-patched, I show " tls-system-cert: yes".

Can you add "tls-win-cert: yes" in the line below (with the same indent) and apply from GUI?

If that doesn't work "tls-cert-bundle: /usr/local/etc/ssl/cert.pem" and removing "tls-system-cert: yes" will do the trick.


Cheers,
Franco

Adding "tls-win-cert" in the line below didn't fix it.  But replacing "tls-system-cert: yes" with "tls-cert-bundle: /usr/local/etc/ssl/cert.pem" did restore functionality. 

Do I need to leave the "tls-win-cert: yes" in place? 
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK? -> OK with 24.7.10_1
Post by: Wendigo on December 03, 2024, 04:15:27 PM
24.7.10_1 works fine for me. Thank you :)
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK?
Post by: gac on December 03, 2024, 04:20:08 PM
Quote from: FullyBorked on December 03, 2024, 04:02:03 PM
Quote from: gac on December 03, 2024, 03:59:08 PM
Quote from: FullyBorked on December 03, 2024, 03:57:47 PM
Quote from: franco on December 03, 2024, 03:47:04 PM
Apparently it's a feature they coined to be for "Windows" and default to off?

tls-win-cert: yes

instead of tls-cert-bundle... can anyone confirm?


Thanks,
Franco

https://nlnetlabs.nl/documentation/unbound/unbound.conf/

I don't see either of these entries in my unbound.conf file.  Should I check somewhere else?
They would be in /var/unbound/etc/dot.conf

Hmm, ok the link he quoted mentioned the unbound.conf.  My dot.conf file other than a single forwarding zone is empty.
The documentation for `unbound.conf` just shows every available option - Unbound is one of the (sensible) apps which allows for options to be spread across multiple configuration files, for example some provided by a package manager (eligible for overwriting) and some manually (which should not be overwritten). Or separated out by purpose/feature.

So `/var/unbound/etc/dot.conf` will contain a rendered config file with the configuration entries from the `unbound.conf` man page, which are relevant for DNS-over-TLS (or `dot`).
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK? -> OK with 24.7.10_1
Post by: franco on December 03, 2024, 04:46:37 PM
> Do I need to leave the "tls-win-cert: yes" in place?

No, apparently it is only an alias for tls-system-cert after all but there is a bug somewhere because it ignores the system directory location, which I haven't seen before. Things like this were tested to death in the last month in fetch, pkg and syslog-ng and they all worked as documented in OpenSSL.


Cheers,
Franco
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK? -> OK with 24.7.10_1
Post by: WhosTheBosch on December 30, 2024, 08:22:49 PM
I found a "bug" that I wanted to post here as it may be a use case that wasn't tested as it's a bit old. Hopefully it could also help someone else if they have the same issue. When the custom options for Unbound were removed in 21.7 I used a config file for NextDNS to be able to forward my Unbound queries to NextDNS as my upstream resolver. (If you read this thread you probably already saw the problem!)

router:/var/unbound/etc # more nextdns.conf
server:
  tls-cert-bundle: "/etc/ssl/cert.pem"
  forward-zone:
    name: "."
    forward-tls-upstream: yes
    forward-addr: IP1#MyConfigID.dns1.nextdns.io
    forward-addr: IP2#MyConfigID.dns2.nextdns.io

Yesterday I upgraded to OPNsense 24.7.11_2-amd64 and afterwards Unbound would not start. I want to say the upgrades thus far have always worked thanks for that! Upon inspection I saw the following error in Unbounds logs:

Error    unbound    Unable to open pipe. This is likely because Unbound isn't running.
So that line was unfortunately not too much help. Over the next few hours I thought it might be DNSBL due to the only other error I had below but that unfortunately wasn't it.

2024-12-30T06:30:50    Error    configd.py    [0b524d64-f2df-4652-b315-62c805b1db9a] Script action failed with Command '/usr/local/opnsense/scripts/unbound/wrapper.py -s ' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 78, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/unbound/wrapper.py -s ' returned non-zero exit status 1.
I also tried removing the Register DHCP Static Mappings but that didn't work either. I also did try to reinstall Unbound based on this post [SOLVED] Unbound not starting (https://forum.opnsense.org/index.php?topic=44468.msg222064#msg222064) which still gave me the same error. (Though I didn't move any config files)

After some research I found this thread which mentioned getting better error messages from the CLI. So I ssh'd there and found a much better error:

unbound -c /var/unbound/unbound.conf
[1735538100] unbound[82458:0] error: error in SSL_CTX verify crypto error:80000002:system library::No such file or directory
[1735538100] unbound[82458:0] error: and additionally crypto error:10000080:BIO routines::no such file
[1735538100] unbound[82458:0] error: and additionally crypto error:05880002:x509 certificate routines::system lib
[1735538100] unbound[82458:0] fatal error: could not set up connect SSL_CTX

I knew the custom forwarding was setup through a custom config file and thought that perhaps custom forwarding was no longer supported that way. So after a system restart with Unbound unable to start, I rm'd /var/unbound/etc/nextdns.conf successfully and was able to restart Unbound from the CLI successfully.

However, further troubleshooting found that nextdns.conf is continuously re-created somehow after the service restarting and/or system restarting. (It was late and I didn't track down the specifics for when it restarts.)

In my search I had read on this thread about the cert.pem file being moved and that was what I needed to figure a workaround out.

Quote from: KHE on December 03, 2024, 03:38:13 PMFrom the release notes:
o system: remove the SSL bundles in default locations

Is this unbound still using these SSL bundles?

With the nextdns.conf file being automatically re-created I couldn't update the location of the cert.pem file. So, as it was late I then figured I'd try the old school hack of copying the cert.pem file to the missing location from the nextdns.conf file:

cp /usr/local/etc/ssl/cert.pem /etc/ssl/cert.pem
Voila, Unbound will now start and I'm still able to use the custom forwarding I setup. Although I do realize I should update that to the fully supported way now. My questions are:

1. How can I remove the nextdns.conf file and why is it being created? I can "rm nextdns.conf" fine. However, about restart it appears again. I can't see it listed in the templates. I don't have anything mentioning nextdns in the following directories:
/usr/local/etc/unbound
/usr/local/opnsense/service/templates/OPNsense/Unbound/core/ (grep nextdns *.* returned nothing)
/var/unbound/conf.d/

2. Is the proper way to do custom fowarding for an upstream resolver then to use the Unbound DNS > DNS over TLS option?

3. Is there a way to get the errors that were seen running it in the CLI in the GUI? They were very helpful and yet I couldn't see them in the Unbound logs (log level 0), or in the General logs.
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK? -> OK with 24.7.10_1
Post by: WhosTheBosch on December 30, 2024, 09:10:06 PM
Unfortunately, the fix doesn't survive a reboot and for some reason the copied cert.pem file is removed from /etc/ssl/ (I'm not to familiar with FreeBSD). I also tried adding the following line to the crontab via crontab -e, however that won't work as it appears to be deleted from the crontab upon reboot as well. (it was added via root user)

@reboot cp /usr/local/etc/ssl/cert.pem /etc/ssl/cert.pem
I was thinking I could maybe put a startup script to run instead of adding it to the crontab but I'm not sure what would be deleted as well in that aspect. So for now having Unbound fail on restart / power on and then manually copying cert.pem will have to do until I'm able to remove the nexdns.conf file and set this up with the recommended method.
Title: Re: 24.7.10 Unbound DNS: DNS over TLS NOK? -> OK with 24.7.10_1
Post by: WhosTheBosch on December 30, 2024, 09:33:28 PM
Quote from: WhosTheBosch on December 30, 2024, 08:22:49 PMMy questions are:

1. How can I remove the nextdns.conf file and why is it being created? I can "rm nextdns.conf" fine. However, about restart it appears again. I can't see it listed in the templates. I don't have anything mentioning nextdns in the following directories:
/usr/local/etc/unbound
/usr/local/opnsense/service/templates/OPNsense/Unbound/core/ (grep nextdns *.* returned nothing)
/var/unbound/conf.d/

OK so I've found the configuration file in /usr/local/etc/unbound.opnsense.d/nextdns.conf - is the proper way to remove it from Unbound startup then to simply rm it from that directory?