Unbound DNS > DNS over TLS: SSL handshake failed error

[Aria]

Hello, I hope you are well.
I am using OPNsense 25.7.6 and I want to pass all the DNS traffic in my network over TLS using Unbound DNS, but I am facing problem.

The process that I did to achieve my goal:
1. Go to `System > Setting > General` menu.
2. Make sure all DNS server fields under Networking section are empty.
3. Uncheck the "Allow DNS server list to be overridden by DHCP/PPP on WAN" option.
4. Go to `Services > Unbound DNS > General`
5. Check the "Enable Unbound" option.
6. Check the "Enable DNSSEC Support" option.
7. Go to `Services > Unbound DNS > DNS over TLS`
8. Add some servers
9. Restart Unbound DNS service

Servers that I am using:
1.1.1.1:853 cloudflare-dns.com
1.0.0.1:853 cloudflare-dns.com
8.8.8.8:853 dns.google
8.8.4.4:853 dns.google
9.9.9.9:853 dns.quad9.net
149.112.112.112:853 dns.quad9.net

After these setups, I tried to resolve a DNS query in my machine:
```
❯ nslookup example.com
;; communications error to 192.168.10.1#53: timed out
;; communications error to 192.168.10.1#53: timed out
;; communications error to 192.168.10.1#53: timed out
;; no servers could be reached
```

I enable "Enables local gathering of statistics." In the "Unbound DNS reporting" section of the `Reporting > Settings` menu, and I see some `SERVFAIL` under `Details` tab of the `Reporting > Unbound DNS` menu. Then set the " Log Level Verbosity" to "Level 2" in the `Services > Unbound DNS > Advanced` and check "Log Queries" option. So I see these on log file:
```
2025-10-25T11:57:49 - Notice - unbound - [50971:0] notice: ssl handshake failed 1.1.1.1 port 853
2025-10-25T11:57:49 - Error - unbound - [50971:0] error: ssl handshake failed: channel closed
```

I tried to undo all the steps so:
1. Go to `System > Setting > General` menu.
2. Set three DNS server
2.1. 1.1.1.1
2.2. 8.8.8.8
2.3. 9.9.9.9
3. Check the "Allow DNS server list to be overridden by DHCP/PPP on WAN" option.
4. Go to `Services > Unbound DNS > General`
5. Uncheck the "Enable Unbound" option.
7. Go to `Services > Unbound DNS > DNS over TLS`
8. Remove all servers

But still no DNS query can be resolved on my network.
I set a temporary port forwarding rule to reroute all the incoming traffic from LAN to firewall at port 53(DNS) into 1.1.1.1:53

What is the reason of getting the "SSL handshake failed" error? What should I do to fix it?

thanks for your help.

[DEC670airp414user]

the error is coming from cloud flare.    id remove it and just add one server to verify its not something else causing it
9.9.9.9
por 853
verify CN = dns.quad9.net

you don't need to restart the service.  it does it itself

no need to check enable dnssec.  the majority do that already

[patient0]

1.1.1.1:853 cloudflare-dns.com
1.0.0.1:853 cloudflare-dns.com

https://developers.cloudflare.com/1.1.1.1/encryption/dns-over-tls/

Code Select Expand

dig one.one.one.one +short
1.0.0.1
1.1.1.1

If you do a DNS lookup for cloudflare-dns.com, you get different IPs:

Code Select Expand

dig  cloudflare-dns.com +short
104.16.249.249
104.16.248.249
I'd try either use the 104... IPs or use one.one.one.one for 1.1.1.1/1.0.0.1

[DEC670airp414user]

https://dnsprivacy.org/public_resolvers/#dns-over-tls-dot.    shows the cn the OP is using...
lots of confusion with cloudflare products
Why I don't use them

[Aria]

Thanks for your responses.

Based on your responses, I removed all the DNS servers on the "Services > Unbound DNS > DNS over TLS" menu and add the below servers one at the time to check if the problem is coming from Cloudflare or not.

The servers that I checked:
8.8.8.8:853 dns.google
8.8.4.4:853 dns.google
9.9.9.9:853 dns.quad9.net

The problem persist, and it seems that there is something wrong with my setup (higher possibility) or with OPNsense and Unbound (lower possibility)

First I thought that it could be a problem coming from the ISP something like blocking the 835 port for outgoing traffic.
So I tried to connect to the 1.1.1.1:853 via "Interfaces > Diagnostics > Port Probe" and it says "Connection to 1.1.1.1 853 port [tcp/domain-s] succeeded!"
I also test this with my machine:

Code Select Expand

❯ nc -v 1.1.1.1 853
Connection to 1.1.1.1 853 port [tcp/domain-s] succeeded!

These show that I can reach the port 853 of 1.1.1.1.

At the other hand, I saw some unresolving DNS queries for some NTP servers coming from the OPNsense Network Time service in the log files, so I thought that it could be a broken loop that Unbound fails to resolve NTP servers IPs because of the mismatch time and Network Time service also can not fix the time because of unresolved NTP servers.

So I tried to put some NTP server IPs in the "Time servers" section of the "Services > Network Time > General" menu.
Based on information in the "Services > Network Time > Status" menu, those NTP servers that had IP also had some information like Delay, Offset, and Jitter, but the problem with resolving DNS queries still persisted

I also put the log level of Unbound on the maximum (5) and let it log both Queries and Replies so it produces tons of logs, but I could not find the reason of the problem on the logs. I want to send the logs here so you guys can look at them, but I am nervous if there is any sensitive information.

[Aria]

I could not solve this issue yet. Someone help me to figure out the problem, please.

[meyergru]

Take a look at the dashboard, you can enable to see what services are running. It looks like Unbound does not start for some reason, as indicated by

Code Select Expand

❯ nslookup example.com
;; communications error to 192.168.10.1#53: timed out
;; communications error to 192.168.10.1#53: timed out
;; communications error to 192.168.10.1#53: timed out
;; no servers could be reached

I asuume 192.168.10.1 is the LAN IP of your OpnSense.

If the service does not start, probably there is a setup error that you can identify in the logs. Also, there are some more daemons that also use port 53 for DNS.

[Aria]

Thank you.
The status of `services` widget in the dashboard is:

Code Select Expand

System Configuration Daemon    Running
Cron                           Running
ISC DHCPv6 Server              Stopped
Dnsmasq DNS/DHCP               Running
Users and Groups               Running
Network Time Daemon            Running
Packet Filter                  Running
Router Advertisement Daemon    Running
System routing                 Running
System tunables                Running
Syslog-ng Daemon               Running
Unbound DNS                    Running
Web GUI                        Running
So I think the Unbound DNS is started and running.
I just saw some new records in the logs.

Code Select Expand

Notice     [3693:0] notice: init module 2: iterator
Notice     [3693:0] notice: init module 1: validator
Notice     [3693:0] notice: init module 0: python
Notice     Backgrounding unbound logging backend.
There are a lot of `SEREVFAIL` for different queries, for example:

Code Select Expand

Informational    [27402:1] info: 192.168.10.241 profile.accounts.firefox.com. A IN SERVFAIL 0.000000 0 46
Informational    [27402:1] info: 192.168.10.241 profile.accounts.firefox.com. A IN
Informational    [27402:0] info: 192.168.10.241 github.com. A IN SERVFAIL 0.000000 0 28
Informational    [27402:0] info: resolving github.com. A IN
Informational    [27402:0] info: 192.168.10.241 github.com. A IN
Informational    [27402:0] info: 127.0.0.1 0.opnsense.pool.ntp.org.Local. AAAA IN SERVFAIL 0.000000 1 47
Informational    [27402:1] info: 127.0.0.1 0.opnsense.pool.ntp.org.Local. A IN SERVFAIL 0.000000 1 47
Informational    [27402:0] info: 127.0.0.1 0.opnsense.pool.ntp.org. AAAA IN SERVFAIL 0.000000 1 41
Informational    [27402:0] info: 127.0.0.1 0.opnsense.pool.ntp.org. A IN SERVFAIL 0.000000 1 41
The output of the nslookup on my machine changed as well:

Code Select Expand

nslookup github.com
;; Got SERVFAIL reply from 192.168.10.1
Server:         192.168.10.1
Address:        192.168.10.1#53

** server can't find github.com: SERVFAIL
I didn't change any options in the settings, Just reboot it about 8 hours ago once.

[meyergru]

So Unbound is running. It must be either a configuration error, like using inoperative DNS forwarders for "." or defective DoT or bad routing/firewall rules that keep it from resolving names.

There are guides in the documentation on how to set up a valid combination of DHCP server and Unbound.

At this time, your OpnSense cannot resolve DNS names and in turn, cannot serve it to your network. Probably the only reason you did not notice is by the fact that most browsers use DoT or DoH themselves.