Tutorial: Caddy (Reverse Proxy) + Let's Encrypt Certificates + Dynamic DNS

[morik_opnsense]

@Cedrik, First off, thank you so very much for creating this (and other) wonder plugins for OPNsense. You make our lives so convenient.

I've read through plugin configuration documentation and through this thread. But, an answer didn't jump out. So, I hope you could excuse me for bothering you with a little issue of mine. TLDR: If os-caddy plugin acts as an ACME server for a site, then a) what would be the ACME url? b) will the location of root ca be /var/db/caddy/data/caddy/certificates/local/<site>?

Presently, I use caddy cascaded in my internal network with a made-up domain name TLS'ed by Caddy. I did so prior to having an externally valid domain name - which I now do. Previously working caddy setup at a high-level was:

Code Select Expand

Caddy master instance (say on internal1.domain)
 {
   acme_server
   tls_internal
  }

Caddy slaves instances (say on internal2.domain)
 https://internal2.domain{
   tls {
     ca https://<internal1.domain>/acme/local/directory
     ca_root <path_to_caddy_master_ca>.pem
   }
  }

Caddy slaves instances (say on internal3.domain)
 https://internal3.domain{
   tls {
     ca https://<internal1.domain>/acme/local/directory
     ca_root <path_to_caddy_master_ca>.pem
   }
  }
... and so on


Now, with os-caddy on Opnsense being available, I'd like to rid of Caddy master on <instance1.domain> and utilize Opnsense caddy plug-in instead. GUI doesn't provide an option to declare acme_server, tls_internal. But inclusion of *.conf on a per-site basis is allowed. So, I added it like so: (reverse proxied external domain to internal2.domain running caddy slave1)

Code Select Expand

# Reverse Proxy Domain: "104d6421-2b1f-407e-af83-f087021ee1b1"
valid.domain {
log {
output file /var/log/caddy/access/104d6421-2b1f-407e-af83-f087021ee1b1.log {
roll_keep_for 10d
}
}
acme_server
tls internal


@08831d42-ad4c-40dc-a7d5-53af52ac6490_validdomain {
not client_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
}
handle @08831d42-ad4c-40dc-a7d5-53af52ac6490_validdomain {
abort
}

handle {
reverse_proxy internal2.domain {
                     transport http {
tls_insecure_skip_verify
}
     header_up Host {http.reverse_proxy.upstream.hostport}
  }
}
}

Then Caddy slaves instances's (internal2.domain) config is changed as:

Code Select Expand

https://internal2.domain{
   tls {
     ca https://<external.domain>/acme/local/directory
     ca_root <path_to_caddy_master_ca copied from /var/db/caddy/data/caddy/certificates/local/internal1.domain>.crt
   }
  }


Unbound resolves external.domain to Opsense's address (192.168.0.1) via host overrides. This is to prevent local request for external domain being handled and served locally without having to go out into the interwebs.

Because Opnsense webGUI is running on 8443, I also tried https://<192.168.0.1>:8443/acme/local/directory but to no avail. ACME server's url appears invalid. I'm unsure how to proceed. Your guidance would be much appreciated.

[Monviech (Cedrik)]

You could use the HTTP01 challenge redirection on the Caddy Server of OPNsense to the cascaded other Caddy Servers. All Caddy Servers listen per default on 80/443 so nothing speciaö needs to be configured. Its what I do.

No need for ACME Server directive.

https://docs.opnsense.org/manual/how-tos/caddy.html#redirect-acme-http-01-challenge

In each domain the IP for the Challenge redirection wouöd be the backend Caddy responsible for it.

[morik_opnsense]

You could use the HTTP01 challenge redirection on the Caddy Server of OPNsense to the cascaded other Caddy Servers.

I see. I was under the impression that the challenges on cascades servers would work because they get their certs (and therefore a hierarchy) from cascade master. Meaning, Caddy opnsense has, for a given external site, a valid cert. If acme challenge is sent to an internal (hosting an internal site) caddy instance, then the request would not have the same valid verification chain on client-side? Please grant me follow-ups in case I get stuck.


An other unrelated questions for your guidance:
GitHub code for this plugin indicates a custom caddy build (with DNS providers, L4 etc) is being used (120 standard modules, 80 or so optional modules). Would there be steps on how to add modules of interest eg crowdsec? I have used xcaddy in the past to generate custom images on Linux but not on freebsd. I can update this post with a working caddy config where crowdsec together with rate L5-7 rate limiters can take L4-7 information (not just L3 - which is what I think present crowdsec integration provides) to block unwanted behaviors.

[Monviech (Cedrik)]

If Caddy(Main) and Caddy(Sub) are in a trusted network, you can reverse_proxy between these Caddies without using TLS.

TLS is only important on the way through untrusted networks, e.g. from Client on the Internet to your Caddy(Main).

Caddy(Main) will hold all certificates and terminate tls. The other Caddies would not need to issue any certificates.

So you do not really need an ACME Server or a ACME Challenge redirection. Just use Plaintext.

Regarding the build:

https://caddyserver.com/docs/command-line#caddy-add-package

You can use this to add any package you want from the command line. It will not be persistent though. If the opnsense repo pushes an update at some point you must do it again.

Currently the plugin is rather finished and very specific or overly complicated things will most likely not be added to prevent feature creep.

[morik_opnsense]

If Caddy(Main) and Caddy(Sub) are in a trusted network, you can reverse_proxy between these Caddies without using TLS.

TLS is only important on the way through untrusted networks, e.g. from Client on the Internet to your Caddy(Main).

Caddy(Main) will hold all certificates and terminate tls. The other Caddies would not need to issue any certificates.

So you do not really need an ACME Server or a ACME Challenge redirection. Just use Plaintext.

I understand. Perhaps a better qualification of one of my use-case is in order.

• Telegraf, Influx and Grafana stacks are employed for telemetry. Latter two are web-based interfaces which require explicit certificate configuration in order to use http(s). Telegraf client configurations need root_ca for http(s) and host-specific keys if TLS verification option is enabled.
• Now, first-order question I asked myself was: Do I really need to protect telemetry information from telegram clients to influx db server? Not really : is the honest answer. That said, telemetry information reveals plenty about topology. So, having telemetry data streamed using TLS protection does carry benefits
• The certs (for external domain) will now reside on Caddy(master). With ACME server enabled on Caddy, I can have certbot request the necessary certs and auto-provision (e.g. during cert renewal) certs onto appropriate roles

Reference telegraf client information required:

Code Select Expand

  ## Optional TLS Config for use on HTTP connections.
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

Similar information is required for influx and Grafana.

Regarding the build:

https://caddyserver.com/docs/command-line#caddy-add-package

You can use this to add any package you want from the command line. It will not be persistent though. If the opnsense repo pushes an update at some point you must do it again.

Thank you for the reference. I understand.

The crowdsec usage I referred to earlier is as follows:

Code Select Expand


--> Global block of Caddyfile (info generated using cscli bouncer add
{
crowdsec {
api_url http://<Opnsense_fw>:8080
api_key <valid_key>
ticker_interval 15s
}
}

--> In the site-block of clients
{
route {
# crowdsec based filtering
crowdsec
                ... whatever logic is necessary ...
        }

}


Currently the plugin is rather finished and very specific or overly complicated things will most likely not be added to prevent feature creep.

Thank you for sharing the plugin status and roadmap. I only asked about crowdsec modules (which can be used like above) because a log-based integration of crowdsec is already implemented on your plugin for Opnsense. Addition of the few extra parameters required may increase the hardening posture.[/list]

[awshirley]

Hi,

I'm currently using the Nginx Proxy Manager and I'm trying to switch to the Caddy plug-in.

Originally I was getting errors with Let's Encrypt and fixed that issue but I'm still unable to access anything behind the reverse proxy.  All I'm seeing in the logs is this:

"info","ts":"2025-01-12T06:13:00Z","logger":"http.log.access","msg":"handled request","request":{"remote_ip":"127.0.0.1","remote_port":"12807","client_ip":"127.0.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost","uri":"/scrape.php?v=6&url=https://www.spamhaus.org/drop/asndrop.json","headers":{"User-Agent":["python-requests/2.32.3"],"Accept-Encoding":["gzip, deflate"],"Accept":["*/*"],"Connection":["keep-alive"]}},"bytes_read":0,"user_id":"","duration":0.000013514,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://localhost/scrape.php?v=6&url=https://www.spamhaus.org/drop/asndrop.json"],"Content-Type":[]}}

I'm not exactly sure what the problem is.  All suggestions are welcome.

Thank you!

[Monviech (Cedrik)]

That looks like a configuration issue. localhost tries to reach spamhaus on lovalhost? Really weird. Without Caddyfile I dont know whats going on.

[awshirley]

Here's my caddy file.  I've removed the personal info.  I hope this helps.

# DO NOT EDIT THIS FILE -- OPNsense auto-generated file


# caddy_user=root

# Global Options
{
   log {
      output net unixgram//var/run/caddy/log.sock {
      }
      format json {
         time_format rfc3339
      }
   }

   servers {
      protocols h1 h2 h3
      log_credentials
   }

   dynamic_dns {
      provider duckdns xxx-xxxxx-xxxx
      domains {
         mysite.duckdns.org *
         mysite.duckdns.org ab
      }
      versions ipv4
      update_only
   }

   email myemail@domain.com
   grace_period 10s
   import /usr/local/etc/caddy/caddy.d/*.global
}

# Reverse Proxy Configuration


# Reverse Proxy Domain: "xxx-xxx-xxxxx"
*.mysite.duckdns.org {
   tls {
      issuer acme {
         dns duckdns {
            api_token xxx-xxx-xxxxx
         }
      }
   }

   @xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx {
      host ab.mysite.duckdns.org
   }
   handle @xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx {
      handle {
         reverse_proxy 192.168.x.xxx {
         }
      }
   }
}

import /usr/local/etc/caddy/caddy.d/*.conf

[Patrick M. Hausen]

@awshirley please place code inside code tags like so:

Code Select Expand

This
  is
code

[Monviech (Cedrik)]

I see nothing wrong with the config at first glance so it remains a mystery that must be solved in the infrastructure it happens.

Maybe check the opnsense docs for the troubleshooting guide for caddy or open a new thread in the caddy community or here in the forum so somebody can pick it up. Its probably an infrastructure issue in some way.

[awshirley]

Thank you for reviewing my caddy file.  You pointed me in the right direction, and infrastructure issue.  I had port forwarding turned on for 40 and 443 to point to NPM.  I turned it off and Caddy now has the ports.

Thanks again!