kubernetes ssl/tls alert handshake failure

[jenswe]

Hi!

My first post here. :)

I have been using opnsense for 2 years now and I am very happy so far.

But now I have a problem I cannot fix by myself. So I hope the community can help me.

I have Opnsense setup with 5 Vlans and one of those I call Lab, in this Vlan I want to run a k3s cluster (Kubernetes). But I am running in to a very strange behavior.

When I try to call for example GitHub.com from inside a pod in the k3s cluster I get:

Code Select Expand


curl -v https://github.com
* Host github.com:443 was resolved.
* IPv6: 2a06:98c1:3120::1, 2a06:98c1:3121::1
* IPv4: 188.114.96.1, 188.114.97.1
*   Trying [2a06:98c1:3120::1]:443...
* Immediate connect fail for 2a06:98c1:3120::1: Network unreachable
*   Trying [2a06:98c1:3121::1]:443...
* Immediate connect fail for 2a06:98c1:3121::1: Network unreachable
*   Trying 188.114.96.1:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /cacert.pem
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS alert, handshake failure (552):
* OpenSSL/3.3.2: error:0A000410:SSL routines::ssl/tls alert handshake failure
* closing connection #0
curl: (35) OpenSSL/3.3.2: error:0A000410:SSL routines::ssl/tls alert handshake failure


So my first thought was that something was wrong with k3s. But after a lot of web searching I could not find any similar problems. So then I thought it might be a problem with the MTU settings. But changing it to be 1450 on both the opnsense vlan and the Nic on linux Kubernetes machine (also setting flannel to use 1450) did not solve the problem.

So I then as a test connected an old Asus router I had around and boom it worked. Egress from within the pod worked perfectly.

So I guess my question is what setting have I missed on my opnsense router? Has anyone ever run in to this issue?

Super happy to get some help!

Thx!

[meyergru]

The output of your curl shows that you have no IPv6 connectivity and when you connect via IPv4, you get a TLS handshake error. This is most likely cause by the wrong CAfile (/cacert.pem).

When I do the same, I see:

Code Select Expand

* Connected to github.com (140.82.121.3) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3137 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [78 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]

Probably, your TLS configuration is broken. You can choose a CA path for curl specifically, but I guess this can be configured for the employed SSL library in some system configuration file. Since you use OpenSSL 3.3.2, it should probably reside in /etc/ssl/openssl.cnf.

[jenswe]

Thx for your reply.

But I do not think this is the issue since I have verified that it works with an other router.

I did some packet captures in Wireshark and what I could see is that from within the pod it sets the tls version to be only 1.0. But if I run the a test from the host it offers tls 1.3.
It is all so weird.

I found an other guy on reddit with the exact same issue:

https://www.reddit.com/r/kubernetes/comments/1c6qab6/tls_handshake_failure_in_kubernetes_pod/?rdt=36094

Also using opnsense.

[meyergru]

Do you happen to use any "transparent" proxy or component that intercepts traffic? In that case, read this, point 9.

Otherwise, other than any apparent MTU issues I do not see how OpnSense would interfere with a TLS connection.
You can check if MTU is an issue by using something like this: https://www.baeldung.com/linux/maximum-transmission-unit-mtu-ip and verify if you client's MTU is below or equal to the PMTU.