DNS-over-TLS - are ISPs interfering?

[chemlud]

Hi!

Have here two OPNsense latest (LibreSSL), both with DoT configured for longer time.

One simply fails to resolve from the beginning:

Code: [Select]

2021-09-30T17:59:02 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:02 unbound[30141] [30141:1] notice: ssl handshake failed 185.95.218.42 port 853
2021-09-30T17:59:02 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:01 unbound[30141] [30141:1] notice: ssl handshake failed 185.95.218.42 port 853
2021-09-30T17:59:01 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:01 unbound[30141] [30141:1] notice: ssl handshake failed 185.95.218.42 port 853
2021-09-30T17:59:01 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:01 unbound[30141] [30141:1] notice: ssl handshake failed 185.95.218.42 port 853
2021-09-30T17:59:01 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:01 unbound[30141] [30141:1] notice: ssl handshake failed 185.95.218.42 port 853
2021-09-30T17:59:01 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:01 unbound[30141] [30141:1] info: generate keytag query _ta-4f66. NULL IN
2021-09-30T17:59:01 unbound[30141] [30141:1] notice: ssl handshake failed 185.150.99.255 port 853
2021-09-30T17:59:01 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:01 unbound[30141] [30141:1] notice: ssl handshake failed 185.150.99.255 port 853
2021-09-30T17:59:01 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:01 unbound[30141] [30141:1] notice: ssl handshake failed 5.9.164.112 port 853
2021-09-30T17:59:01 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:01 unbound[30141] [30141:1] notice: ssl handshake failed 5.9.164.112 port 853
2021-09-30T17:59:01 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:00 unbound[30141] [30141:1] notice: ssl handshake failed 5.9.164.112 port 853
2021-09-30T17:59:00 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:00 unbound[30141] [30141:1] notice: ssl handshake failed 5.9.164.112 port 853
2021-09-30T17:59:00 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:00 unbound[30141] [30141:1] notice: ssl handshake failed 5.1.66.255 port 853
2021-09-30T17:59:00 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:00 unbound[30141] [30141:1] notice: ssl handshake failed 5.1.66.255 port 853
2021-09-30T17:59:00 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:00 unbound[30141] [30141:1] notice: ssl handshake failed 185.150.99.255 port 853
2021-09-30T17:59:00 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:00 unbound[30141] [30141:1] notice: ssl handshake failed 185.150.99.255 port 853
2021-09-30T17:59:00 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:00 unbound[30141] [30141:1] notice: ssl handshake failed 5.1.66.255 port 853
2021-09-30T17:59:00 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:59:00 unbound[30141] [30141:1] notice: ssl handshake failed 5.1.66.255 port 853
2021-09-30T17:59:00 unbound[30141] [30141:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T17:58:58 unbound[30141] [30141:0] info: start of service (unbound 1.13.2).
Therefore the other OPNsense is configured as DNS via a tunnel. This worked until today. Then this afternoon all of a sudden DNS failed on the other OPNsense completely, although 5 DoT servers are configured.

When I add 9.9.9.9 or mulvad DNS it starts to work again an the remote DNS (but not on the OPNsense initially failing during TLS handshake).

When I do a package capture on WAN (port 853) it looks like normal TLS 1.2 and 1.3 traffic with at least one remote IP (95.216.212.177), but DNSleaktest.com reports no functional DNS servers and in browser nothing loads (again: my usually working 5 DoT servers still configured).

The OPNsense is located in Northern Europe, most DNS servers I configured are located in Central Europe, but is there some kind of iron curtain for DoT troughout Europe?

WHY? Is the ISP interfering with DoT?

[chemlud]

It's getting even worse. On the initially failing OPNsense with 5 DoT servers I added 9.9.9.9 #853. Now I get DNS resolved in dnsleaktest.com, but on an obscure IP in the UK (see pic attached), but there is nothing in the unbound log:

Code: [Select]

2021-09-30T18:20:02 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:02 unbound[4249] [4249:1] notice: ssl handshake failed 185.150.99.255 port 853
2021-09-30T18:20:02 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:02 unbound[4249] [4249:1] notice: ssl handshake failed 185.95.218.42 port 853
2021-09-30T18:20:02 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:02 unbound[4249] [4249:1] notice: ssl handshake failed 185.95.218.42 port 853
2021-09-30T18:20:02 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:01 unbound[4249] [4249:1] info: generate keytag query _ta-4f66. NULL IN
2021-09-30T18:20:01 unbound[4249] [4249:1] notice: ssl handshake failed 185.150.99.255 port 853
2021-09-30T18:20:01 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:01 unbound[4249] [4249:1] notice: ssl handshake failed 185.150.99.255 port 853
2021-09-30T18:20:01 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:01 unbound[4249] [4249:1] notice: ssl handshake failed 5.9.164.112 port 853
2021-09-30T18:20:01 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:01 unbound[4249] [4249:1] notice: ssl handshake failed 5.9.164.112 port 853
2021-09-30T18:20:01 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:01 unbound[4249] [4249:1] notice: ssl handshake failed 5.9.164.112 port 853
2021-09-30T18:20:01 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:01 unbound[4249] [4249:1] notice: ssl handshake failed 5.9.164.112 port 853
2021-09-30T18:20:01 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:01 unbound[4249] [4249:1] notice: ssl handshake failed 185.95.218.42 port 853
2021-09-30T18:20:01 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:00 unbound[4249] [4249:1] notice: ssl handshake failed 185.95.218.42 port 853
2021-09-30T18:20:00 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:00 unbound[4249] [4249:1] notice: ssl handshake failed 5.1.66.255 port 853
2021-09-30T18:20:00 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:00 unbound[4249] [4249:1] notice: ssl handshake failed 5.1.66.255 port 853
2021-09-30T18:20:00 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:00 unbound[4249] [4249:1] notice: ssl handshake failed 5.1.66.255 port 853
2021-09-30T18:20:00 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:20:00 unbound[4249] [4249:1] notice: ssl handshake failed 5.1.66.255 port 853
2021-09-30T18:20:00 unbound[4249] [4249:1] error: ssl handshake failed crypto error:14FFF086:SSL routines:(UNKNOWN)SSL_internal:certificate verify failed
2021-09-30T18:19:55 unbound[4249] [4249:0] info: start of service (unbound 1.13.2).

[KHE]

They are using probably LE certs. OPNsense seems to have an issue with them since the DST Root CA X3
 expired today. Updating OPNsense also fails, if the mirror is using a LE certs. https://forum.opnsense.org/index.php?topic=24968.msg119835#msg119835
I checked unicast.censurfridns.dk and anycast.censurfridns.dk and they are failing for me and using LE certs. Seems like a pattern to me.

Just tested:

Code: [Select]

[admin@OPNsense ~]$ fetch -o mimugmail.conf https://www.routerperformance.net/mimugmail.conf
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
5843273977856:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: https://www.routerperformance.net/mimugmail.conf: Authentication error
A simple fetch also fails.

[chemlud]

hmmm, many thanks for reply! Then it would make sense... ;-)

How to end this catch 22 if no update (of certs) possible due to failing handshake? Any mirrors not using LE certs?

[dinguz]

Use http (not https) mirrors.

[KHE]

Hi,

let me guess, you updated your LE certs yesterday, before the ACME Client update was applied?
I had to delete all LE CA certs unter System: Trust: Authorities, even the new ones from yesterday.
I then reissued all LE certs on my OPNsense and the R3 (ACME Client) CA was added again and now my system works again as expected. Maybe a reboot also will work. My ensurfridns.dk are working again now.

KH

[chemlud]

Nope, no LE certs or authorities present on my boxes....

[KHE]

Ok, then you can try the solution from Felix: https://forum.opnsense.org/index.php?topic=24950.msg119873#msg119873

KH

[chemlud]

Hmm... I could update to 21.7.3_3 without any intervention from https update server...

Let's see what the DNS does over the day.