Tutorial: Caddy (Reverse Proxy) + Let's Encrypt Certificates + Dynamic DNS

[OPNenthu]

Thanks- I deleted the file and don't see the error now after bouncing the service.

Here's the 'before' and 'after' state:

Code Select Expand

root@firewall:~ # cd /var/run/caddy/
root@firewall:/var/run/caddy # ls -l
total 2
-rw-rw----  1 root www 6 Mar 14 23:00 caddy.pid
s-w--w----  1 root www 0 Mar 14 23:00 caddy.sock
srw-rw-rw-  1 root www 0 Mar 14 22:57 log.sock
root@firewall:/var/run/caddy #
root@firewall:/var/run/caddy # rm caddy.sock

(bounced the service here)

Code Select Expand

root@firewall:/var/run/caddy # ls -l
total 2
-rw-------  1 www  www 6 Mar 15 07:32 caddy.pid
s-w--w----  1 www  www 0 Mar 15 07:32 caddy.sock
srw-rw-rw-  1 root www 0 Mar 14 22:57 log.sock


[awshirley]

I'm getting an error in the Caddy log when trying to use the reverse proxy on a Plex instance.  The log shows:

"error","ts":"2025-03-19T23:36:47Z","logger":"http.log.error","msg":"EOF","request":{"remote_ip":"192.168.x.xxx","remote_port":"50589","client_ip":"192.168.x.xxx","proto":"HTTP/2.0","method":"GET","host":"plexsub.mydomain.com","uri":"/media/providers?X-Plex-Product=Plex%20Web&X-Plex-Version=4.145.1&X-Plex-Client-Identifier=y1574g5pgysu0b7435g9qsqd&X-Plex-Platform=Firefox&X-Plex-Platform-Version=136.0&X-Plex-Features=external-media%2Cindirect-media%2Chub-style-list&X-Plex-Model=bundled&X-Plex-Device=Windows&X-Plex-Device-Name=Firefox&X-Plex-Device-Screen-Resolution=1536x731%2C1536x864&X-Plex-Token=TWeNgtGispep-E4RBR1m&X-Plex-Language=en&X-Plex-Session-Id=72ff17fc-21db-4b3b-8437-9194ca66bd7d","headers":{"Referer":["http://192.168.x.xxx:32400/"],"Accept-Encoding":["gzip, deflate, br, zstd"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:136.0) Gecko/20100101 Firefox/136.0"],"Accept-Language":["en"],"Dnt":["1"],"Sec-Fetch-Site":["cross-site"],"Accept":["application/json"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["cors"],"Sec-Gpc":["1"],"Te":["trailers"],"Origin":["http://192.168.x.xxx:32400"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"plexsub.mydomain.com"}},"duration":0.000950754,"status":502,"err_id":"kr1iycyqd","err_trace":"reverseproxy.statusError (reverseproxy.go:1373)"}

Plex is stating that remote access through the reverse proxy doesn't work.  Is this something easily fixed?

[Monviech (Cedrik)]

TWIMC Just warning here that DNS Providers might get a bit more inconvenient soon.

https://github.com/opnsense/plugins/issues/4643

Only cloudflare will remain default compiled in as it is maintained directly by the caddy organization. All other providers will be optionally installable via CLI with e.g.

Code Select Expand

caddy add-package github.com/caddy-dns/duckdns

https://caddyserver.com/docs/command-line#caddy-add-package

If they won't compile after the caddy binary is updated to caddy-v2.10.0 please reach out to their maintainers via https://github.com/caddy-dns

This had happened once already and I don't want to run after 40 repos with for something I don't even use personally (I dont use dns-challenge or dynamic-dns, I maintain this in my free time)

[OPNenthu]

JFYI- the earlier issue about caddy.sock file being owned by root and causing errors when using the 'www' user in Caddy can still be seen.  Yesterday I re-installed OPNsense and imported my config, then installed the Caddy plugin.  Because of my configs the user was preset to 'www' but the sock file was owned by root and needed to be deleted manually.

[Monviech (Cedrik)]

Im a little surprised, can you find out the exact spot where it happens so it can be reproduced?

Please take this as reference what was implemented to mitigate it and the discussion:

https://github.com/opnsense/plugins/pull/4403

If there is still an issue please open one.

[OPNenthu]

I tried to reproduce on a test install of OPNsense in Proxmox, but could not.  After changing the user to 'www' and default ports to 8080/8443 the service started fine.

Looking at the GitHub ticket, the fix was merged in December.  I started using OPNsense around 24.7 so I'm wondering if there's something in my main router config file from before the fix that is getting carried over and causing an issue?  I know it sounds crazy but I can't imagine what else it could be. I blew away my previous root filesystem when I did the fresh install yesterday using the 25.1 installer, so literally the only difference between it and the test instance in Proxmox would be the config import.

The socket file does have the correct ownership & permissions (root:www, 0220), and I confirmed that the 'www' user is in the 'www' group as well, so there shouldn't be an issue with that.

I'll post back if I notice anything in my config.xml file, but since this is not reproducible on a clean install then I don't think it warrants a ticket.

[morik_opnsense]

Hello,

I recently upgraded from Opnsense (FreeBSD 14.2-RELEASE-p2) Business Edition 24.10 --> 25.4, and caddy server stopped working.

Code Select Expand

caddy --version
v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

I have crowdsec plugin on caddy which I use in Caddyfile for integration w/ crowdsec. So, usually, after major upgrades on Opnsense I end up doing the following:

Code Select Expand

caddy add-package github.com/hslatman/caddy-crowdsec-bouncer github.com/caddyserver/transform-encoder
and off to the races I go. However, this time around, I'm getting a weird 400 error.

Code Select Expand

caddy add-package github.com/hslatman/caddy-crowdsec-bouncer github.com/caddyserver/transform-encoder
2025/05/01 01:17:17.322 INFO this executable will be replaced {"path": "/usr/local/bin/caddy"}
2025/05/01 01:17:17.322 INFO requesting build {"os": "freebsd", "arch": "amd64", "packages": ["github.com/caddy-dns/desec@v0.0.0-20240526070323-822a6a2014b2", "github.com/caddy-dns/scaleway@v0.0.0-20231227190624-561fd7f77b1b", "github.com/caddyserver/transform-encoder", "github.com/caddy-dns/namedotcom@v0.1.3-0.20231028060845-b9fae156cd97", "github.com/caddy-dns/ovh@v0.0.3", "github.com/caddy-dns/porkbun@v0.2.1", "github.com/caddy-dns/rfc2136@v0.1.1", "github.com/hslatman/caddy-crowdsec-bouncer", "github.com/caddy-dns/netcup@v0.1.1", "github.com/caddy-dns/vultr@v0.0.0-20230331143537-35618104157e", "github.com/caddyserver/ntlm-transport@v0.1.3-0.20230224201505-e0c1e46a3009", "github.com/caddy-dns/cloudflare@v0.0.0-20240703190432-89f16b99c18e", "github.com/caddy-dns/directadmin@v0.3.1", "github.com/caddy-dns/hetzner@v0.0.2-0.20240820184004-23343c04385f", "github.com/caddy-dns/linode@v0.7.2", "github.com/caddy-dns/namecheap@v0.0.0-20240114194457-7095083a3538", "github.com/caddy-dns/acmedns@v0.3.0", "github.com/caddy-dns/bunny@v0.1.1-0.20240209091254-71ced26b4224", "github.com/caddy-dns/acmeproxy@v1.0.6", "github.com/caddy-dns/infomaniak@v1.0.1", "github.com/caddy-dns/inwx@v0.3.1", "github.com/caddy-dns/mailinabox@v0.0.2-0.20240829173454-39d0e3ce8e25", "github.com/caddy-dns/powerdns@v1.0.1", "github.com/caddy-dns/azure@v0.5.0", "github.com/caddy-dns/gandi@v1.0.4-0.20240531160843-d814cce86812", "github.com/caddy-dns/hexonet@v0.1.0", "github.com/mholt/caddy-ratelimit@v0.1.0", "github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead", "github.com/caddy-dns/dnsmadeeasy@v1.1.3", "github.com/mholt/caddy-dynamicdns@v0.0.0-20241025234131-7c818ab3fc34", "github.com/caddy-dns/duckdns@v0.4.0", "github.com/caddy-dns/ionos@v1.1.0"]}
Error: download failed: download failed: HTTP 400: unable to fulfill download request (id=43358b0e-5041-4319-adac-d96d6a1e570e)


Code Select Expand

caddy upgrade
2025/05/01 01:24:33.309 INFO this executable will be replaced {"path": "/usr/local/bin/caddy"}
2025/05/01 01:24:33.309 INFO requesting build {"os": "freebsd", "arch": "amd64", "packages": ["github.com/caddy-dns/cloudflare@v0.0.0-20240703190432-89f16b99c18e", "github.com/caddy-dns/gandi@v1.0.4-0.20240531160843-d814cce86812", "github.com/caddy-dns/inwx@v0.3.1", "github.com/caddy-dns/acmeproxy@v1.0.6", "github.com/caddy-dns/dnsmadeeasy@v1.1.3", "github.com/caddy-dns/duckdns@v0.4.0", "github.com/caddy-dns/hetzner@v0.0.2-0.20240820184004-23343c04385f", "github.com/caddy-dns/mailinabox@v0.0.2-0.20240829173454-39d0e3ce8e25", "github.com/caddy-dns/namecheap@v0.0.0-20240114194457-7095083a3538", "github.com/mholt/caddy-ratelimit@v0.1.0", "github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead", "github.com/caddy-dns/bunny@v0.1.1-0.20240209091254-71ced26b4224", "github.com/caddy-dns/directadmin@v0.3.1", "github.com/caddy-dns/linode@v0.7.2", "github.com/caddy-dns/infomaniak@v1.0.1", "github.com/caddy-dns/netcup@v0.1.1", "github.com/caddy-dns/vultr@v0.0.0-20230331143537-35618104157e", "github.com/caddy-dns/acmedns@v0.3.0", "github.com/caddy-dns/azure@v0.5.0", "github.com/caddy-dns/desec@v0.0.0-20240526070323-822a6a2014b2", "github.com/caddy-dns/ovh@v0.0.3", "github.com/caddy-dns/porkbun@v0.2.1", "github.com/caddy-dns/scaleway@v0.0.0-20231227190624-561fd7f77b1b", "github.com/caddyserver/ntlm-transport@v0.1.3-0.20230224201505-e0c1e46a3009", "github.com/caddy-dns/rfc2136@v0.1.1", "github.com/caddy-dns/hexonet@v0.1.0", "github.com/caddy-dns/ionos@v1.1.0", "github.com/caddy-dns/namedotcom@v0.1.3-0.20231028060845-b9fae156cd97", "github.com/caddy-dns/powerdns@v1.0.1", "github.com/mholt/caddy-dynamicdns@v0.0.0-20241025234131-7c818ab3fc34"]}
Error: download failed: download failed: HTTP 400: unable to fulfill download request (id=704dc2db-afa9-4ee4-953a-6ba7ffec9803)

Firewall is not blocking either DNS translation of GitHub or ip connectivity to it. I am wondering if anyone else is having this issue?

[Monviech (Cedrik)]

caddy add-package uses the build servers of the caddy project to supply you with a binary.

They have no SLA and always serve the latest version of caddy.

Right now the latest version probably has build incompatabilities with the build you are requesting.

Try using xcaddy instead for your personal build.

[morik_opnsense]

Thank you, Cedrik.

Try using xcaddy instead for your personal build.

I presume this'd mean I'd issue xcaddy with all the package names indicated in the previously failed command, like:

Code Select Expand

xcaddy build \
 --with github.com/caddy-dns/scaleway \
 --with github.com/caddy-dns/desec \
 ...
 (+ packages of my interest e.g. crowdsec)

/r
morik
PS: Your efforts in maintaining os-caddy port for OPNsense and other software are very much appreciated!

[Monviech (Cedrik)]

You only need to specify the packets you want to use. The base build uses the top 4 packages in this list and then just the single DNS provider you need + your custom packages.

https://github.com/opnsense/tools/blob/31002815e15e1f50cc7ab0af5c3f1cd155878926/config/25.1/make.conf#L100

If you cannot build caddy 2.10.0 try to pass version 2.9.1 and use the same commit hashes as I do in the link.

[morik_opnsense]

Thank you, Cedrik.

OPNSense: You want to do what?
Me: pkg install xcaddy
OPNSense: Did you take me for a fool?
OPNSense:

Code Select Expand

pkg install xcaddy
Updating OPNsense repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%  249 KiB 255.0kB/s    00:01
Processing entries: 100%
OPNsense repository update completed. 870 packages processed.
All repositories are up to date.
pkg: No packages available to install matching 'xcaddy' have been found in the repositories

Me:

Code Select Expand

sed -in 's/no/yes/'  /usr/local/etc/pkg/repos/FreeBSD.conf
FreeBSD: { enabled: yes }


pkg install xcaddy
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating OPNsense repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%  249 KiB 255.0kB/s    00:01
Processing entries: 100%
OPNsense repository update completed. 870 packages processed.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
pkg: 1.19.2_5 -> 2.1.2 [FreeBSD]

Number of packages to be upgraded: 1

The process will require 31 MiB more space.
12 MiB to be downloaded.

Proceed with this action? [y/N]: N


OPNSense: Told you, I'll win.
Me: I give up

Also, few minutes later,
Me: having never done ports, why don't we hose our system...

Code Select Expand

pkg install git
...
git clone --depth=1 https://git.FreeBSD.org/ports.git /usr/ports
cd /usr/ports/www/xcaddy
make install clean
....

mkdir -p ~/caddy_build && cd ~/caddy_build
xcaddy build \
  --with github.com/caddyserver/ntlm-transport \
  --with github.com/mholt/caddy-dynamicdns \
  --with github.com/mholt/caddy-l4 \
  --with github.com/mholt/caddy-ratelimit \
  --with github.com/hslatman/caddy-crowdsec-bouncer \
  --with github.com/caddyserver/transform-encoder
...
...
././caddy version
v2.10.0 h1:fonubSaQKF1YANl8TXqGcn4IbIRUDdfAkpcsfI/vX5U=

<< make my changes on crowdsec >>
configctl caddy restart
OK

Phew.. No idea what else did I break, but the feature which I wanted works now. Of course, I do not know how future os-caddy updates will behave. Life is indeed an adventure :-)

[Monviech (Cedrik)]

You probably did not break anything, I plan to evaluate to make it more straight forward in the future, maybe.

https://github.com/opnsense/plugins/issues/4668

[morik_opnsense]

https://github.com/opnsense/plugins/issues/4668

Thank you!

[Monviech (Cedrik)]

I evaluated it and its possible but very brittle. So its not going to be included in the plugin.

In your case you built caddy with go121 which might be an issue since go124 is required for all features to work correctly.

The build will be thinned out soon to only include cloudflare, which will make caddy add-package less prone to fail.

It failed because there was a breaking libdns change again and it includes a couple of dns providers that are dependent on it, thus it failed.

For xcaddy, using a separate freebsd vm where you can tightly control the build environment might be the best choice.

All in all, go limits the flexibility of caddy here, comparing it to dynamic modules in nginx or apache.