Caddy Layer4 Route TLS (SNI)

[alankeny]

Hello,

I've been using HA Proxy on OPNsense 24.7.4_1-amd64 to route HTTPS traffic to about a dozen internal web servers running different applications.  All of the internal servers run HTTPS with signed certs from Let's Encrypt.  I configured HA Proxy to route based on SNI, so it didn't need any certificates of its own.  The configuration was fairly convoluted, and it was difficult to add new servers.  I switched to Caddy, because it looks like it should be easier to setup and maintain.

I currently have Caddy running successfully with Domains and HTTP handlers configured for each server.  As I understand it Caddy is terminating TLS and establishing new TLS connections to each internal server.  I'd like to switch these to Layer4 routes using TLS (SNI), but I'm getting an error.

I've removed the domain and HTTP handler for a test site, and added it to Layer4 instead.  I've configured the domain, set the matcher to TLS (SNI), set the upstream domain, and set the upstream port.  In the log I get the following error:

Error caddy   "debug","ts":"2024-09-21T15:03:27Z","logger":"caddy.listeners.layer4", "msg":"matching","remote":"217.111.222.33:21026","error":"consumed all prefetched bytes","matcher":"layer4.matchers.tls","matched":false

After this it goes on to try to find a certificate for the website that's listed in the domain.  It doesn't find one, so the request fails.  Running tcpdump on the test server shows that Caddy never tries to establish a connection to it.

I haven't been able to find any mention of the error "consumed all prefetched bytes" online.  Since it works when Caddy terminates the TLS connection, I assume my firewall rules are OK.  Are there any other config changes I need to make to get the Layer4 TLS (SNI) matcher to work in Caddy?

Thanks,
Alan

[Monviech (Cedrik)]

Hey,

before we continue with this specific error, try to recreate the configuration from the second youtube video in here:

https://forum.opnsense.org/index.php?topic=38714.0

(The video is from a dev state, so SNI is TLS (SNI) now)

There shouldnt be any other changes to the config needed.

If it works with the OPNsense Webgui, we already know a lot more.

Additionally, please provide your current caddyfile. Remove your API keys if you use a DNS provider, and if you dont want to expose your domains replace the base domain with example.com

[alankeny]

Thanks for taking a look at this.  I'm getting the same error with the OPNsense Webgui following the example from the video..

I removed my DNS API key and disabled all of the domains except the OPNsense Webgui domain.  My current Caddy file is attached.

Thanks again,
Alan

[Monviech (Cedrik)]

Thanks for the config. Your config looks fine on first glance. Just one thing:

- you can remove 80 and 443 from the HTTP and HTTPS ports in General Settings, they're the default and do not have to be specified.

I have tried to reproduce it, but for me it always works. I can see these entries in my log:


2024-09-22T08:38:16   Debug   caddy   "debug","ts":"2024-09-22T06:38:16Z","logger":"layer4.matchers.tls","msg":"matched",
"remote":"172.16.1.105:57988","server_name":"vpn1.example.com"}

I have tried it with only layer 4 matchers, without any other domains/HTTP Handlers. And a mix of layer 4 matchers and domains and HTTP Handlers.

If you see the prefetched bytes message, it was a connection that did not match any of these matchers.

The only thing where it didnt work for me, was if the browser had previously connected to the same hostname via valid SSL certificate in Chrome. When I turned it into passthrough, Chrome showed "ERR_SSL_PROTOCOL_ERROR", because it didn't like going from valid to self signed certificate for the same hostname. Probably some security feature like HSTS.

--------------

Unrelated to that, I also found a small rendering bug in the template that would prevent the "Layer 4 HTTP and HTTPS" default ports to render in the reverse Proxy Configuration, it will be fixed in the next version. Was a small regression when os-caddy-1.7.0 was released. But since it renders for you, it means you are not on os-caddy-1.7.0_1 where the bug exists right now.

[alankeny]

SNI was working with HA Proxy, so my browser should be sending the SNI.  Caddy also seems to be getting the right name when using a domain + HTTP handler.  When I don't have the domain configured, I get log entries when it tries to match the domain with a certificate, and the domain matches what I've been using in layer4.

Is there any way to log what layer4 is attempting to match on like it does when Caddy searches for a certificate?

I don't know if this matters, but my OPNSense does list the os-caddy version as 1.7.0_1.

[Monviech (Cedrik)]

Ive talked with the Caddy devs and I will update the Layer4 module in the next release to latest. Maybe that fixes your problems.

Regarding the logs, just set it on debug and use the search field to search for your domain, it should show the layer 4 matcher logs like I posted above.

If nothing matched at all, try * as domain to match everything.

Also try out a different matcher like SSH too.

Its weird that it just works for me, but it doesnt work for you. Nobody else ever gave feedback on the module before. Youre the first one who gives any response on it in the scope of the Opnsense implementation of it.

Here is the github project page of it. It is a pretty new module so having weird stuff happening in some configurations is something that could happen.
https://github.com/mholt/caddy-l4

Reading the PRs might look like this:
https://github.com/mholt/caddy-l4/commit/22a03931ae46d0decc068e8d1ffe83e6187ed812

Dependency Update for module will be in next plugin version sometime next week: https://github.com/opnsense/tools/pull/433

[Monviech (Cedrik)]

Okay the root cause seemed to have been found with great debugging help from caddy community members after I got into conversation with them over this issue.

It had stopped working for me too now. It was browser related since under the hood a new festure was introduced in Chromium based browsers. Firefox was still working.

It was called TLS 1.3 hybridized Kyber support.

https://github.com/mholt/caddy-l4/commit/ea27408a3384a7f33b5d1cb49ec592cf76942be6

It caused the prefetched bytes error since the TLS handshake was increased in size with it.

I gonna test and deploy that patch in the next version of os-caddy, that will fix the TLS matching for all browsers.

[Monviech (Cedrik)]

Everything should be fixed now in the latest release.

The layer4 GUI has also been rewritten to allow way more powerful configurations.

If anybody wants to try it and give feedback, be my guest.

[alankeny]

Sorry for the delay getting back to this thread.

I tested with the latest release today, and everything is working as expected.  I have a dozen layer4 routes configured that all connect successfully to their correct sites.  Another layer4 route passes through an SSH connection.  I also added a domain & HTTP handler combo for one specific site, so I could add basic authentication in front of it, but only when it's accessed remotely.

Thanks for all of your work on this.  I really appreciate it!
Alan

[Monviech (Cedrik)]

  Awesome thanks for the feedback. It was pretty tricky to figure this out but Im happy it had the desired result.