Caddy plugin stopping

[collinsit]

Good morning, I have recently migrated from PFSense to OpnSense and am really liking it. I currently run an NGINX box on a Linux VM which works well but I like the idea of handling all the filtering through the firewall directly. I tested out the NGINX plugin on the firewall but couldn't get certain ports to stream properly like they do on my Linux NGINX install and while I was troubleshooting, I stumbled upon Caddy which seemed pretty lightweight and easy to install.

I had already setup the ACME plugin to generate certificates and was using it with NGINX so I setup my caddy reverse proxy rules and my layer 4 streaming rules and was shocked how easily they all worked and everything seemed great. I used the certificate that the ACME plugin generated for all the HTTPS rules and that worked fine. The issue is, every few hours, the plugin will just stop. The way I know is that I get a bunch of messages about systems being down. I can login to the firewall and restart it and it all comes back up but it will do it again in a few hours.

I have gone through the logs but since I am new to Caddy I am not sure what to really look at and googling the various errors hasn't helped. I could use Caddy if I can get this to be stable or go back to NGINX and NAT the specific ports I was having issues with streaming but for some reason, the NATTING of the ports doesn't seem to take effect if the NGINX plugin is running. If I don't use it, and NAT those ports directly to the server, it all works fine.

Thanks for any suggestions or help here.

[goobs]

Not to jump on your thread but I have a similar experience with the Caddy plugin.

Similarly I have been using NGINX Proxy Manager but on a Proxmox server to proxy/redirect incoming web host traffic, without issue but wanted to see if my firewall could do the job too.

I tried the Caddy plugin nearly two weeks ago with the latest OPNsense 24.7 but got the memory issues and Caddy dying on me.

This week I updated to 25.1, then removed a bunch of SNORT IDS rules to get my memory usage down from 59% to 49% of the 8GB I have on my firewall, but Caddy died within a few hours of running.

I see the following in the System/General log files :

2025-02-11T13:10:43   Notice   kernel   <3>pid 47033 (caddy), jid 0, uid 0, was killed: failed to reclaim memory   
2025-01-30T15:34:06   Notice   kernel   <3>pid 87676 (caddy), jid 0, uid 0, was killed: failed to reclaim memory

I can start Caddy up again with Monit but it will still die later.

[collinsit]

That's the strange part, I am not seeing any logs saying it is running out of memory. I did read other posts online saying that people had experienced a memory leak with it but I haven't seen anything saying i'm running out of memory.

I have 16GB of ram in my firewall and don't have a ton of heavy usage plugins running so I don't really suspect memory is the actual issue but I can't figure out what is going on. I have reverted back to using my Linux VM for the time being as it is at least stable and works well.

[Monviech (Cedrik)]

(Im the plugin maintainer)

If Caddy stops with a panic it will log it. So look for panics in the logfiles in /var/log/caddy/caddy.log and if you find one post it here.

Then we can see which module causes it.

Depending on the module we can then open an issue upstream.

E.g. there is a report here so you can see what a panic looks like:

https://github.com/mholt/caddy-l4/issues/295

[collinsit]

Thank you for the response Monviech, see below for the entries from that logfile. I have currently disabled the plugin so it is running through my Linux NGINX box but hopefully there will be something helpful in these entries. I am not sure if these are directly related to the crash or not so if you want me to re-enable it again and see if I get another crash I can try that tonight.

{"level":"info","ts":1738974488.0721657,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974488.0725987,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974488.07268,"msg":"No files matching import glob patte
rn","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974488.0727434,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738974488.0728152,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738974488.0762215,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738974488.078006,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=78285) - Caddy is running in the background
{"level":"info","ts":1738974734.3310537,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974734.331348,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974734.3313804,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974734.331417,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}

[Monviech (Cedrik)]

I need something that says panic, it will look the same as in the issue I have linked above.

If there are no panics, check how the RAM usage of the caddy binary evolves, if it steadily increases until it reaches out of memory, eg multiple gigabytes of RAM used.

Thanks for your help.

[collinsit]

I can test that again tonight when the servers aren't heavily used. It generally happens within a few hours so it shouldn't take a long time to see it happen.

Is there a good way to monitor how much ram the process is using? I apologize if that is something I should know already as I am relatively new to the system.

Thanks

[Monviech (Cedrik)]

Executing this in the root shell will show all processes sorted by RAM and updates every second:

top -o res -s 1

Its also interesting if there's a difference if the layer 4 module is enabled or disabled.

[collinsit]

I ran Caddy again tonight and it worked fine but an hour or so and then stopped. Below is what I see in that logfile but nothing from what I can tell about any specific panic.


The resources on the server sat pretty consistent for the whole time and didn't seem to creep up much. I'm attaching a copy of the top view taken shortly before the service stopped. It was pretty consistent throughout the whole time things were running. I am going to re-enable my NGINX server but if there is anything else you want me to test, let me know.

{"level":"info","ts":1738974488.0721657,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974488.0725987,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974488.07268,"msg":"No files matching import glob patte
rn","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974488.0727434,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738974488.0728152,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738974488.0762215,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738974488.078006,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=78285) - Caddy is running in the background
{"level":"info","ts":1738974734.3310537,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974734.331348,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974734.3313804,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974734.331417,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738974734.331456,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738974734.3334653,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738974734.3345199,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=60153) - Caddy is running in the background
{"level":"info","ts":1738975322.6449044,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738975322.6451797,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738975322.6452117,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738975322.645248,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738975322.6452894,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738975322.6472473,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738975322.6482174,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=44523) - Caddy is running in the background
{"level":"info","ts":1738975422.0715666,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738975422.0718377,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738975422.0718775,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738975422.071914,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738975422.0719538,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738975422.0739639,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738975422.075407,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=74543) - Caddy is running in the background
panic: send on closed channel

goroutine 10782 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x86c02f1f0, {0x15aa82a44
7b8, 0x86c3b0bc0})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 10726
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739023449.4873703,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739023449.487672,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739023449.4877079,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739023449.4877467,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739023449.4877892,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739023449.489998,"msg":"adapted config to JSON","adapter":
"caddyfile"}
{"level":"info","ts":1739023449.4909961,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=18512) - Caddy is running in the background
panic: send on closed channel

goroutine 92 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x86c5a0cb0, {0x5c74f68f4
c0, 0x86c2ddd20})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739053775.421173,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739053775.4214938,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739053775.4215293,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739053775.421567,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739053775.4216142,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739053775.4244912,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739053775.4255404,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=50585) - Caddy is running in the background
panic: send on closed channel

goroutine 92 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x870134620, {0x18816c0d8
90, 0x870523ee0})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739062426.591729,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739062426.5920196,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739062426.5920596,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739062426.592098,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739062426.5921452,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739062426.5943375,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739062426.5956733,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=85549) - Caddy is running in the background
panic: send on closed channel

goroutine 93 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x87029af50, {0x2b544c41b
4e8, 0x8702b1700})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739065245.8650196,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739065245.8653393,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739065245.8653727,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739065245.8654163,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739065245.8654664,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739065245.8683712,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739065245.8694892,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=79758) - Caddy is running in the background
{"level":"info","ts":1739072460.8001187,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739072460.800456,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739072460.8004968,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739072460.8005433,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739072460.8006308,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739072460.8029695,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739072460.8044856,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=98895) - Caddy is running in the background
{"level":"info","ts":1739073385.0745974,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739073385.074918,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739073385.07495,"msg":"No files matching import glob patte
rn","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739073385.0749958,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739073385.075042,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739073385.0771258,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739073385.078136,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=14780) - Caddy is running in the background
{"level":"info","ts":1739073912.8176806,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739073912.8179882,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739073912.8180237,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739073912.8180597,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739073912.8181174,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739073912.8201737,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739073912.8214817,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=8396) - Caddy is running in the background
panic: send on closed channel

goroutine 115 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8704471f0, {0x2e80efd9c
828, 0x870477a40})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739156722.0015497,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739156722.0018783,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739156722.0019114,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739156722.001972,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739156722.0020287,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739156722.0042887,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739156722.005349,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=43620) - Caddy is running in the background
{"level":"info","ts":1739199652.7526066,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739199652.7529013,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739199652.752934,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739199652.7529862,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739199652.753033,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739199652.755767,"msg":"adapted config to JSON","adapter":
"caddyfile"}
{"level":"info","ts":1739199652.756928,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=51524) - Caddy is running in the background
panic: send on closed channel

goroutine 45339 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x870260c40, {0x69f8ca8fc
20, 0x8702ac500})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 45299
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739213533.973209,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739213533.9735048,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739213533.973533,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739213533.9735708,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739213533.9736202,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739213533.9765584,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739213533.9776092,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=57180) - Caddy is running in the background
panic: send on closed channel

goroutine 9269 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x870295500, {0x187e44bed
ab8, 0x87045a2a0})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 9149
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739224498.843148,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739224498.843496,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739224498.843529,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739224498.8435707,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739224498.8436186,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739224498.8467724,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739224498.8479614,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=56747) - Caddy is running in the background
panic: send on closed channel

goroutine 12802 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8700df9d0, {0x1f32a37fc
b90, 0x87040f3a0})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 12292
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739323557.2691522,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739323557.2695,"msg":"No files matching import glob patter
n","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739323557.2695348,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739323557.2695723,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739323557.2696235,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739323557.2721164,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739323557.2736979,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=36570) - Caddy is running in the background
{"level":"info","ts":1739324268.177545,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739324268.1778708,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739324268.1778998,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739324268.1779385,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739324268.177986,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739324268.1809487,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739324268.182059,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=52550) - Caddy is running in the background
panic: send on closed channel

goroutine 3991 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8700bfdc0, {0x32f2b9e37
280, 0x87040f420})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 3843
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.

[Monviech (Cedrik)]

Thank you a lot, it looks like its also a layer 4 module issue on your side.

I gonna write in the existing ticket. Can you provide more information like the Caddyfile you were using? (you can anonymize thibgs like api keys and domains)

[collinsit]

Of course, attached is the caddy file as I figured it would be easier to read as an attachment instead of inline in the post.

If there is anything you notice that you would like me to try, let me know. Since you said it appears to be a layer 4 issue I could potentially just NAT those ports as the reverse proxy is really only required for 80 and 443 but it would be really nice to have it all working through Caddy as it is a pretty nice setup.

[Monviech (Cedrik)]

Thanks, for what I got as information upstream its mostly related to UDP streaming. Try to see if the crashes still happen if you don't use any UDP in the layer 4 proxy.

[collinsit]

I can do that. It looks like I only have one layer 4 UDP connection so I can move that to a NAT and re-enable the plugin.

I'll update you later if that works.

[Monviech (Cedrik)]

Thanks for helping :)

[collinsit]

Of course, I know it is hard to troubleshoot issues if people won't test. I have switched over the config, after removing the UDP layer 4 so i'll let you know what happens. If there are going to be issues I would expect it in the next couple hours.

While we are waiting for this to happen I was wondering, is there a way to add other DNS providers to the plugin? I am currently using the ACME plugin for my certificate enrollment as they support my provider, ClouDNS but it doesn't look like the Caddy plugin supports that one. The ACME plugin is setup and working fine so not sure there is any need to change but it could be one less thing to manage if it was in Caddy too.