Tutorial: Caddy (Reverse Proxy) + Let's Encrypt Certificates + Dynamic DNS

[Patrick M. Hausen]

@meyergru exactly the situation with my home lab vs. our data centre operation  ;)

Thanks for the concise summary.

[Monviech (Cedrik)]

I just knew the target audience of this. Caddy filled the niche pretty well imo for home setups.

[meyergru]

No complaints for the right target audience, Cedrik - I just heard it was much easier and tried it.

Even if you would add some features, you would only end up at something inherently more complex. Plus, some restrictions seem to be hard-wired (like the TLS 1.3 ciphers imposed by Go).

[Monviech (Cedrik)]

The Reverse Proxy is pretty much finished, but the layer 4 module will get more features over time.

Next thing is an OpenVPN matcher where you can multiplex on the same port via tls static keys for example.

Theres some neat Layer 7 matchers coming.

https://github.com/mholt/caddy-l4/pull/251

[yarcod]

Hi!

I tried setting up the Caddy plugin but ran into some issues. Just setting up a basic reverse proxy does not seem to work for me when following the guide. It doesn't give me any errors from the GUI and the Caddy service appears to be running properly, but trying to run

Code Select Expand

caddy reload --config /usr/local/etc/caddy/Caddyfile results in an error not displayed elsewhere:

Code Select Expand


Error: sending configuration to instance: performing request: Post "http://127.0.0.1:2019/load": dial tcp 127.0.0.1:2019: connect: connection refused

Maybe it is not possible/supported to reload caddy this way? Even though the service appears to be up, it does not seem able to connect properly, given:

Code Select Expand


> curl -vL 127.0.0.1:80
*   Trying 127.0.0.1:80...
* Immediate connect fail for 127.0.0.1: Connection refused
* Failed to connect to 127.0.0.1 port 80 after 0 ms: Could not connect to server
* closing connection #0
curl: (7) Failed to connect to 127.0.0.1 port 80 after 0 ms: Could not connect to server

What could I be missing? Thanks in advance!

[Monviech (Cedrik)]

sockstat -l | grep -i 80
sockstat -l | grep -i 443

Make sure no other services use port 80 and 443.

service caddy status

Check if Caddy is running

service caddy restart
service caddy reload

Restart Caddy if needed

[yarcod]

sockstat -l | grep -i 80
sockstat -l | grep -i 443

I can see that Caddy has managed to bind to port 443, but not 80. AFAICT no other service has taken port 80 either: Doing curl on localhost:443 results in "error:0A000438:SSL routines::tlsv1 alert internal error". Output of sockstat for port 80:

Code Select Expand


# sockstat -l | grep -i 80
root     lighttpd   30649 4   tcp4   127.0.0.1:43580       *:*
root     lighttpd   30649 5   tcp6   ::1:43580             *:*
root     ntpd       10336 23  udp6   fe80::e01d:f0ff:fe2a:e3c2%vtnet0:123 *:*
root     ntpd       10336 24  udp6   fe80::b8a0:afff:fe98:12fe%vtnet1:123 *:*
root     ntpd       10336 27  udp6   fe80::1%lo0:123       *:*
root     php-cgi    38055 0   stream /tmp/php-fastcgi.socket-0
root     php-cgi    38040 0   stream /tmp/php-fastcgi.socket-0
root     sshd       11827 8   tcp6   fe80::1%lo0:2223      *:*


service caddy status

Caddy appears to be running just fine: "caddy is running as pid 72385."

service caddy restart
service caddy reload

This did perform a clean restart and reload of Caddy, as opposed to asking Caddy to reload itself. So that's nice to know! Still, it unfortunately did not make any difference.

I noticed that I had used an older version of the plugin, so just to be safe I also did reset most settings and applied it again to see if the update had fixed something related to what I attempted to setup. But still no success.

[Patrick M. Hausen]

I can see that Caddy has managed to bind to port 443, but not 80. AFAICT no other service has taken port 80 either: [...]

Make sure System > Settings > Administration > HTTP Redirect - Disable web GUI redirect rule

is checked.

[Monviech (Cedrik)]

I could say more if I can see the Caddyfile. But right now no idea really.

[yarcod]

Thanks -- I ensured that it was still checked.

I could say more if I can see the Caddyfile. But right now no idea really.

The Caddyfile looks like this (did not find any spoiler tag to compress my message):

Code Select Expand


{
   log {
      output net unixgram//var/run/caddy/log.sock {
      }
      format json {
         time_format rfc3339
      }
      level DEBUG
   }

   servers {
      protocols h1 h2 h3
   }

   auto_https off
   grace_period 10s
   import /usr/local/etc/caddy/caddy.d/*.global
}

# Reverse Proxy Configuration


# Reverse Proxy Domain: "62300f65-2a08-47ef-b1f7-d0e31bbd30c4"
*.edholm.cc {
   tls /var/db/caddy/data/caddy/certificates/temp/64c1e555e19da.pem /var/db/caddy/data/caddy/certificates/temp/64c1e555e19da.key

   @07356a3a-8362-4a3e-afd3-7f0b521a44e9 {
      host jelly
   }
   handle @07356a3a-8362-4a3e-afd3-7f0b521a44e9 {
      handle {
         reverse_proxy 192.168.1.101:8096 {
         }
      }
   }
}

import /usr/local/etc/caddy/caddy.d/*.conf


[Monviech (Cedrik)]

Auto HTTPS is off thats why there is no port 80.

Change the setting in general settings.

It needs to be on for the automatic http to https redirect even if you dont use lets encrypt.

[Monviech (Cedrik)]

@Meyergru:

Thanks for trying it out and your review. I can add some comments to your points:

Today I took the opportunity to try out Caddy reverse proxy instead of HAproxy, mostly because of a very specific problem with HAproxy...

I must say I reverted after trying it thouroughly. My 2cents on this are as follows:

- Caddy is suited to home setups and inexperienced users. HAproxy is much more complex.
- For example, the certificate setup is much easier, because you just have to specify the domain and it just works (tm).

-> Perfect, that was the target audience. So I guess I hit the right spot if that was noticable. ;D They like how neatly integrated features like dynamic dns, dns providers and the dns01 challenge is for their certificates. Creating a new domain just takes a few clicks and it just worksTM. These features are mostly unnecessary for enterprises. Steps to set things up are kept to a minimum with minimal abstraction.

- However, if you have more than just one domain, Caddy setup gets a little tedious:
* you have to create one domain/certificate plus a http backend for any domain, which includes creating different ones for www.domain.de and domain.de. You cannot combine certificates for multiple domains unless they are subdomains.

-> Thats the tradeoff of less abstraction. It means experts will find limitations here.

* You do not have much control over what type of certificate(s) are created - you cannot specifiy strength or ECC vs. RSA (much less both) and I have not found a means to control if ZeroSSL vs. LetsEncrypt is used.
* The ciphers being employed cannot be controlled easily - or, for TLS 1.3, at all. That results in an ssllabs.com score which is suboptimal, becaus 128bit ciphers are allowed. This cannot be changed because of Go limitations.

-> That one is indeed for the Go developers to battle out with their userbase: https://github.com/golang/go/issues/29349

Though the defaults are okay, and an online test that gives a score can be misleading in this case. It probably comes down to company policies. Most of the discussion here always revolves about compliance.

* You cannot use more than one type of DNS-01 verification if you use wildcard domains.

-> Limitation of the plugin, not of caddy. The plugin combines the same DNS provider for dynamic dns (which only allows one provider) and the dns01 challenge (which allows multiple providers). To make it simple and less confusing, the lowest denominator was chosen. So it is more suited for home/homelab environments or small businesses here that have one provider for their domains. I know of some people who use it to reverse proxy exchange servers for example, that works well even with outlook etc...

* The Auto HTTPS feature looks nice first, but indeed it uses a 308 instead of a 301 code, which breaks some monitoring and can only be modified via custom include files.

-> That sounds like an easy fix if you can tell me what is needed there to change the default if needed.

So, if you just want to reverse-proxy some services in your home network, go with Caddy. For an OpnSense guarding your internet site with several services/domains, stay with HAproxy.

-> That is a very sane conclusion and I mostly agree with it. Though the way the layer4/7 proxy and matcher ecosystem evolves makes it pretty powerful in its own way, if not only looking at the reverse proxy.

[meyergru]

* The Auto HTTPS feature looks nice first, but indeed it uses a 308 instead of a 301 code, which breaks some monitoring and can only be modified via custom include files.

-> That sounds like an easy fix if you can tell me what is needed there to change the default if needed.

That can be done via something like:

Code Select Expand


/usr/local/etc/caddy/caddy.d/redirect.conf:

    http:// {
        redir https://{hostport}{uri} 301
    }


You may also need this to avoid conflicts:

Code Select Expand


/usr/local/etc/caddy/caddy.d/redirect.global:

    auto_https disable_redirects


Would be nice to have a switch for this, because the custom files are not part of the saved configuration.

So, if you just want to reverse-proxy some services in your home network, go with Caddy. For an OpnSense guarding your internet site with several services/domains, stay with HAproxy.

-> That is a very sane conclusion and I mostly agree with it. Though the way the layer4/7 proxy and matcher ecosystem evolves makes it pretty powerful in its own way, if not only looking at the reverse proxy.

HAproxy can do layer 4 proxying as well. I use this to translate IPv6 to IPv4 backends all the time, for example to access intranet docker services behind dynamic IPv6 prefixes (as docker is not easy to set up with IPv6, even less so with dynamic IPs).

[Monviech (Cedrik)]

I have asked a caddy maintainer about the redirect and since there is no global setting that changes the way "Auto HTTPS" creates them, I will not offer it.

It would also conflict with current features like the http01 challenge redirection that create a http domain when enabled.

https://github.com/opnsense/plugins/blob/5c4e3a231f6a46cef9d9ae9dc87d92b12ad97fb9/www/caddy/src/opnsense/service/templates/OPNsense/Caddy/Caddyfile#L280

Also it would need deduplication if multiple domains with the same hostname and different ports are created.

It would sadly get a little messy and convoluted in my opinion.

[yarcod]

Auto HTTPS is off thats why there is no port 80.

Change the setting in general settings.

It needs to be on for the automatic http to https redirect even if you dont use lets encrypt.

Ah, thanks! That explains the lack of port 80! And indeed, checking that box immediately redirected http to https. However, perhaps the help text for this option should be slightly updated? It currently states:

Select the Auto HTTPS option. "On (default)" creates automatic certificates using "Let's Encrypt" or "ZeroSSL". "Off" turns all automatic certificate requests off.

Something mentioning the redirect could be added?

Regardless, I am still running into the issue I had when prodding https before:

Code Select Expand


# curl -vL https://127.0.0.1:443
*   Trying 127.0.0.1:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* OpenSSL/3.0.15: error:0A000438:SSL routines::tlsv1 alert internal error
* closing connection #0
curl: (35) OpenSSL/3.0.15: error:0A000438:SSL routines::tlsv1 alert internal error


The Let's encrypt TLS cert is setup inside OPNsense (this guide is probably the closest to what I have: https://sysadmin102.com/2023/05/create-lets-encrypt-wildcard-certificates-on-opnsense-with-acme-client/) in order to re-use the certificate elsewhere in the network. However, despite the plugin being setup to use this certificate it does not seem able to serve it:

Code Select Expand


<15>1 2024-10-28T23:20:18+01:00 OPNsense.edholm.cc caddy - - [meta sequenceId="11"] "debug","ts":"2024-10-28T22:20:18Z","logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"90.229.225.174","remote_port":"31331","server_name":"edholm.cc","remote":"90.229.225.174:31331","identifier":"edholm.cc","cipher_suites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"cert_cache_fill":0.0001,"load_or_obtain_if_necessary":true,"on_demand":false}
<11>1 2024-10-28T23:20:18+01:00 OPNsense.edholm.cc caddy - - [meta sequenceId="12"] "debug","ts":"2024-10-28T22:20:18Z","logger":"http.stdlib","msg":"http: TLS handshake error from 90.229.225.174:31331: no certificate available for 'edholm.cc'"}
<15>1 2024-10-28T23:27:38+01:00 OPNsense.edholm.cc caddy - - [meta sequenceId="1"] "debug","ts":"2024-10-28T22:27:38Z","logger":"events","msg":"event","name":"tls_get_certificate","id":"c99f387a-0cb1-4de9-af74-f764659607dd","origin":"tls","data":{"client_hello":{"CipherSuites":[52393,52392,49195,49199,49196,49200,49161,49171,49162,49172,156,157,47,53,49170,10,4867,4865,4866],"ServerName":"","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":null,"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"45.79.163.72","Port":65403,"Zone":""},"LocalAddr":{"IP":"192.168.1.1","Port":443,"Zone":""}}}}
<15>1 2024-10-28T23:27:38+01:00 OPNsense.edholm.cc caddy - - [meta sequenceId="2"] "debug","ts":"2024-10-28T22:27:38Z","logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"192.168.1.1"}
<15>1 2024-10-28T23:27:38+01:00 OPNsense.edholm.cc caddy - - [meta sequenceId="3"] "debug","ts":"2024-10-28T22:27:38Z","logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"45.79.163.72","remote_port":"65403","server_name":"","remote":"45.79.163.72:65403","identifier":"192.168.1.1","cipher_suites":[52393,52392,49195,49199,49196,49200,49161,49171,49162,49172,156,157,47,53,49170,10,4867,4865,4866],"cert_cache_fill":0.0001,"load_or_obtain_if_necessary":true,"on_demand":false}
<11>1 2024-10-28T23:27:38+01:00 OPNsense.edholm.cc caddy - - [meta sequenceId="4"] "debug","ts":"2024-10-28T22:27:38Z","logger":"http.stdlib","msg":"http: TLS handshake error from 45.79.163.72:65403: no certificate available for '192.168.1.1'"}


Does the logs above tell you anything? I don't get why it would not find a matching certificate, because the one I have is setup for "*.edholm.cc". What could I be missing for that not to work?