Caddy Layer4 Configuration

[halpdesk]

I have three servers running separate instances of Caddy for separate services that I'm trying to make externally available. To do so, I've deployed the Caddy plugin and have enabled Layer4 proxying.

I think the configuration is correct (TCP, TLS SNI) along with my basic Caddy setups on the other servers (with LE auto HTTPS) - but it only works intermittently.

By that, I mean some of the services can sometimes be accessed externally, but a few minutes later might become unavailable, and so on with other services.

Are there any watch-outs I should be on the lookout for to resolve this?

[Monviech (Cedrik)]

You should enable debug logs on all of these servers and check them when things become unavailable. There are no general pitfalls here, its probably something that needs logs to debug.

Also say the browser you are using. Also try using curl instead when the issue happens.


-----

As alternative you can use the normal reverse proxy with this option:

https://docs.opnsense.org/manual/how-tos/caddy.html#redirect-acme-http-01-challenge

That way your backend caddys and frontend caddy will be able to issue lets encrypt at the same time.
Just make sure your backend caddys use the trusted proxies option.

[halpdesk]

Thanks for the quick response! I've attached the most recent lines from my Caddy log, where you can see I'm externally navigating to several subdomains unsuccessfully (nvr, plan, and sso).

The server I'm forwarding them to has a hostname of debian and Caddy exposed via port 443, which is also managing TLS via HTTPS.

Something that might be important to note: I've disabled Unbound overrides/split DNS while testing these subdomains. When I navigate to plan.halp.app from inside my network, the layer4 proxy seems to work fine. When I use my cell phone to navigate to plan.halp.app from outside my network, it does not work.

I've tried this on both Edge and Firefox mobile.

[Monviech (Cedrik)]

Is that a real domain name? Cause I get nxdomain when looking it up.

If there's no public DNS record then it will not work in the WAN. The DNS record (A and/or AAAA record) has to point to the public IP address of the OPNsense that runs Caddy.

[halpdesk]

It's not - I should have elaborated that I masked my real domain in the uploaded file for privacy reasons. I can confirm the domain and subdomains resolve correctly internally and externally - I've used the command line and various DNS lookup tools to ensure everything is pointing to my public IP address.

Within OPNsense, I also have a NAT port forward rule that takes HTTPS traffic with a destination of the WAN address and forwards it to OPNsense Caddy (192.168.1.1, port 443). In OPNsense Caddy settings, I've configured the port to 443 (it defaulted to 20443). My OPNsense install lives on port 440, not 443 - so I don't think that's the issue.

In a nutshell, this is what I'm experiencing:

I have three servers that serve different applications -

Server 1 - A, B, C
Server 2 - D, E, F
Server 3 - G, H I

When I restart Caddy and then externally visit one of Server 1's apps, they all work - but the services on Servers 2 and 3 do not load.

If I restart Caddy again and then try to access a service on Server 2, all of the apps on Server 2 now work - but the services on Servers 1 and 3 do not load.

The same thing happens if I restart and then start by loading a Server 3 app.

I've shared my Caddy config with masked domains if it helps.

[Monviech (Cedrik)]

That does sound like a really strange issue.

Your caddyfile looks fine, its what I run as well when using the layer4 proxy.

One thing, you do not need a Port Forward.

If Caddy listens on 443, and your WAN has 443 free, a firewall rule is enough.

Please check out the documentation:

https://docs.opnsense.org/manual/how-tos/caddy.html#installation
https://docs.opnsense.org/manual/how-tos/caddy.html#faq

[halpdesk]

Thanks, I just updated OPNsense to remove the port forward and configure the firewall rules.

Attached is an updated log file. It looks like the Layer4 proxy is working in that it's interpreting plan.halp.app and sending it to the right server - but nothing is loading in the browser. This is consistent across services when trying to access them externally.

Is there something special I need to enable on the internal Caddy instances on Servers 1, 2, and 3 to accept the layer4 proxy? It's almost like the traffic is being load balanced that only services on a single server work at a time.

[Monviech (Cedrik)]

I am sadly totally unsure here as I have no experience with that configuration.

I think it would be best to go to https://caddy.community and share your setup, debug logs of the layer 4 caddy and the backend caddys, and ask for assistence.

If there is some sort of configuration issue, or bug, please post here again so I can include it either in the documentation, or as a patch.

You can reference this thread or the reddit thread you made as well.

Thank you.

[Monviech (Cedrik)]

I tested this configuration and it worked for me:

Code Select Expand

2024-11-25T16:35:39 Debug caddy "debug","ts":"2024-11-25T15:35:39Z","logger":"layer4.matchers.tls","msg":"matched","remote":"176.3.82.66:62279","server_name":"opn-cp-be-02.pischem.com"}
2024-11-25T16:35:38 Debug caddy "debug","ts":"2024-11-25T15:35:38Z","logger":"layer4.matchers.tls","msg":"matched","remote":"176.3.82.66:62451","server_name":"opn-cp-be-02.pischem.com"}
2024-11-25T16:34:56 Debug caddy "debug","ts":"2024-11-25T15:34:56Z","logger":"layer4.matchers.tls","msg":"matched","remote":"176.3.82.66:62470","server_name":"opn-cp-be-01.pischem.com"}
2024-11-25T16:34:56 Debug caddy "debug","ts":"2024-11-25T15:34:56Z","logger":"layer4.matchers.tls","msg":"matched","remote":"176.3.82.66:62274","server_name":"opn-cp-be-01.pischem.com"}

Code Select Expand


servers {
protocols h1 h2 h3
listener_wrappers {
layer4 {
import /usr/local/etc/caddy/caddy.d/*.layer4listener

@a1bec573-cedd-495a-b356-922fd681697f tls sni opn-cp-be-01.pischem.com

route @a1bec573-cedd-495a-b356-922fd681697f {
proxy tcp/172.16.1.121:443 {
}
}
@c777fc6a-ce6f-48d3-b5c1-83820945e247 tls sni opn-cp-be-02.pischem.com

route @c777fc6a-ce6f-48d3-b5c1-83820945e247 {
proxy tcp/172.16.1.122:443 {
}
}
}
tls
}
}


Maybe try if one hostname per route makes a difference for your case.

I sadly can not do more right now, this has to be troubleshooted on your side.

[halpdesk]

No worries! I really appreciate the help you've given me so far.

I have since discovered that some of the subdomains still had Unbound overrides, which I've disabled. I'm now at a point where each of the subdomains tries to load externally, but comes up with a blank page (as opposed to throwing errors before).

I'm using a custom DNS module and a wildcard cert on the other Caddy instances, so I'm assuming I just need to configure something to allow OPNsense Caddy and them to speak properly.

[halpdesk]

I tested this configuration and it worked for me:

Is there a chance it's because I'm forwarding subdomains to Caddy instances that are set up for wildcard domains?

[Monviech (Cedrik)]

Your configuration doesnt even need the layer 4 proxy.

You can do everything using the normal reverse proxy.

You just have to put trusted proxies into your backend caddies.
https://caddyserver.com/docs/caddyfile/options#trusted-proxies

If your backend caddies use automatic lets encrypt, add this into your frontend caddy in each domain:
https://docs.opnsense.org/manual/how-tos/caddy.html#redirect-acme-http-01-challenge
You dont need that if you use the dns challenge everywhere.

The handler is totally easy, its just the domainname, and https://upstreamip:port and thats it.
https://docs.opnsense.org/manual/how-tos/caddy.html#creating-a-simple-reverse-proxy


If you still have issues please try the caddy community for more specific help.

[halpdesk]

Following up on this for anyone who may stumble on it in the future. The issue was not anything to do with the Layer4 configuration, but that I was redirecting traffic to three different servers all using their own certificates for the same wildcard domain, which caused browser issues when going back and forth across servers.

Everything works now that I've specified each individual subdomain in my Caddyfile vs using *.halp.app.

[Monviech (Cedrik)]

Hey that is quite interesting. I'm glad you found that out, good job and thanks for following up on this.