Good morning, I have recently migrated from PFSense to OpnSense and am really liking it. I currently run an NGINX box on a Linux VM which works well but I like the idea of handling all the filtering through the firewall directly. I tested out the NGINX plugin on the firewall but couldn't get certain ports to stream properly like they do on my Linux NGINX install and while I was troubleshooting, I stumbled upon Caddy which seemed pretty lightweight and easy to install.
I had already setup the ACME plugin to generate certificates and was using it with NGINX so I setup my caddy reverse proxy rules and my layer 4 streaming rules and was shocked how easily they all worked and everything seemed great. I used the certificate that the ACME plugin generated for all the HTTPS rules and that worked fine. The issue is, every few hours, the plugin will just stop. The way I know is that I get a bunch of messages about systems being down. I can login to the firewall and restart it and it all comes back up but it will do it again in a few hours.
I have gone through the logs but since I am new to Caddy I am not sure what to really look at and googling the various errors hasn't helped. I could use Caddy if I can get this to be stable or go back to NGINX and NAT the specific ports I was having issues with streaming but for some reason, the NATTING of the ports doesn't seem to take effect if the NGINX plugin is running. If I don't use it, and NAT those ports directly to the server, it all works fine.
Thanks for any suggestions or help here.
Not to jump on your thread but I have a similar experience with the Caddy plugin.
Similarly I have been using NGINX Proxy Manager but on a Proxmox server to proxy/redirect incoming web host traffic, without issue but wanted to see if my firewall could do the job too.
I tried the Caddy plugin nearly two weeks ago with the latest OPNsense 24.7 but got the memory issues and Caddy dying on me.
This week I updated to 25.1, then removed a bunch of SNORT IDS rules to get my memory usage down from 59% to 49% of the 8GB I have on my firewall, but Caddy died within a few hours of running.
I see the following in the System/General log files :
2025-02-11T13:10:43 Notice kernel <3>pid 47033 (caddy), jid 0, uid 0, was killed: failed to reclaim memory
2025-01-30T15:34:06 Notice kernel <3>pid 87676 (caddy), jid 0, uid 0, was killed: failed to reclaim memory
I can start Caddy up again with Monit but it will still die later.
That's the strange part, I am not seeing any logs saying it is running out of memory. I did read other posts online saying that people had experienced a memory leak with it but I haven't seen anything saying i'm running out of memory.
I have 16GB of ram in my firewall and don't have a ton of heavy usage plugins running so I don't really suspect memory is the actual issue but I can't figure out what is going on. I have reverted back to using my Linux VM for the time being as it is at least stable and works well.
(Im the plugin maintainer)
If Caddy stops with a panic it will log it. So look for panics in the logfiles in /var/log/caddy/caddy.log and if you find one post it here.
Then we can see which module causes it.
Depending on the module we can then open an issue upstream.
E.g. there is a report here so you can see what a panic looks like:
https://github.com/mholt/caddy-l4/issues/295
Thank you for the response Monviech, see below for the entries from that logfile. I have currently disabled the plugin so it is running through my Linux NGINX box but hopefully there will be something helpful in these entries. I am not sure if these are directly related to the crash or not so if you want me to re-enable it again and see if I get another crash I can try that tonight.
{"level":"info","ts":1738974488.0721657,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974488.0725987,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974488.07268,"msg":"No files matching import glob patte
rn","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974488.0727434,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738974488.0728152,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738974488.0762215,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738974488.078006,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=78285) - Caddy is running in the background
{"level":"info","ts":1738974734.3310537,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974734.331348,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974734.3313804,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974734.331417,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
I need something that says panic, it will look the same as in the issue I have linked above.
If there are no panics, check how the RAM usage of the caddy binary evolves, if it steadily increases until it reaches out of memory, eg multiple gigabytes of RAM used.
Thanks for your help.
I can test that again tonight when the servers aren't heavily used. It generally happens within a few hours so it shouldn't take a long time to see it happen.
Is there a good way to monitor how much ram the process is using? I apologize if that is something I should know already as I am relatively new to the system.
Thanks
Executing this in the root shell will show all processes sorted by RAM and updates every second:
top -o res -s 1
Its also interesting if there's a difference if the layer 4 module is enabled or disabled.
I ran Caddy again tonight and it worked fine but an hour or so and then stopped. Below is what I see in that logfile but nothing from what I can tell about any specific panic.
The resources on the server sat pretty consistent for the whole time and didn't seem to creep up much. I'm attaching a copy of the top view taken shortly before the service stopped. It was pretty consistent throughout the whole time things were running. I am going to re-enable my NGINX server but if there is anything else you want me to test, let me know.
{"level":"info","ts":1738974488.0721657,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974488.0725987,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974488.07268,"msg":"No files matching import glob patte
rn","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974488.0727434,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738974488.0728152,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738974488.0762215,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738974488.078006,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=78285) - Caddy is running in the background
{"level":"info","ts":1738974734.3310537,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974734.331348,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974734.3313804,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974734.331417,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738974734.331456,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738974734.3334653,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738974734.3345199,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=60153) - Caddy is running in the background
{"level":"info","ts":1738975322.6449044,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738975322.6451797,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738975322.6452117,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738975322.645248,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738975322.6452894,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738975322.6472473,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738975322.6482174,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=44523) - Caddy is running in the background
{"level":"info","ts":1738975422.0715666,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738975422.0718377,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738975422.0718775,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738975422.071914,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738975422.0719538,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738975422.0739639,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738975422.075407,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=74543) - Caddy is running in the background
panic: send on closed channel
goroutine 10782 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x86c02f1f0, {0x15aa82a44
7b8, 0x86c3b0bc0})
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 10726
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739023449.4873703,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739023449.487672,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739023449.4877079,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739023449.4877467,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739023449.4877892,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739023449.489998,"msg":"adapted config to JSON","adapter":
"caddyfile"}
{"level":"info","ts":1739023449.4909961,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=18512) - Caddy is running in the background
panic: send on closed channel
goroutine 92 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x86c5a0cb0, {0x5c74f68f4
c0, 0x86c2ddd20})
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739053775.421173,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739053775.4214938,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739053775.4215293,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739053775.421567,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739053775.4216142,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739053775.4244912,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739053775.4255404,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=50585) - Caddy is running in the background
panic: send on closed channel
goroutine 92 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x870134620, {0x18816c0d8
90, 0x870523ee0})
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739062426.591729,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739062426.5920196,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739062426.5920596,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739062426.592098,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739062426.5921452,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739062426.5943375,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739062426.5956733,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=85549) - Caddy is running in the background
panic: send on closed channel
goroutine 93 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x87029af50, {0x2b544c41b
4e8, 0x8702b1700})
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739065245.8650196,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739065245.8653393,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739065245.8653727,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739065245.8654163,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739065245.8654664,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739065245.8683712,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739065245.8694892,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=79758) - Caddy is running in the background
{"level":"info","ts":1739072460.8001187,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739072460.800456,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739072460.8004968,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739072460.8005433,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739072460.8006308,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739072460.8029695,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739072460.8044856,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=98895) - Caddy is running in the background
{"level":"info","ts":1739073385.0745974,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739073385.074918,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739073385.07495,"msg":"No files matching import glob patte
rn","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739073385.0749958,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739073385.075042,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739073385.0771258,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739073385.078136,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=14780) - Caddy is running in the background
{"level":"info","ts":1739073912.8176806,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739073912.8179882,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739073912.8180237,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739073912.8180597,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739073912.8181174,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739073912.8201737,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739073912.8214817,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=8396) - Caddy is running in the background
panic: send on closed channel
goroutine 115 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8704471f0, {0x2e80efd9c
828, 0x870477a40})
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739156722.0015497,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739156722.0018783,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739156722.0019114,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739156722.001972,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739156722.0020287,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739156722.0042887,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739156722.005349,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=43620) - Caddy is running in the background
{"level":"info","ts":1739199652.7526066,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739199652.7529013,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739199652.752934,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739199652.7529862,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739199652.753033,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739199652.755767,"msg":"adapted config to JSON","adapter":
"caddyfile"}
{"level":"info","ts":1739199652.756928,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=51524) - Caddy is running in the background
panic: send on closed channel
goroutine 45339 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x870260c40, {0x69f8ca8fc
20, 0x8702ac500})
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 45299
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739213533.973209,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739213533.9735048,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739213533.973533,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739213533.9735708,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739213533.9736202,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739213533.9765584,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739213533.9776092,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=57180) - Caddy is running in the background
panic: send on closed channel
goroutine 9269 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x870295500, {0x187e44bed
ab8, 0x87045a2a0})
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 9149
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739224498.843148,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739224498.843496,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739224498.843529,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739224498.8435707,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739224498.8436186,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739224498.8467724,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739224498.8479614,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=56747) - Caddy is running in the background
panic: send on closed channel
goroutine 12802 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8700df9d0, {0x1f32a37fc
b90, 0x87040f3a0})
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 12292
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739323557.2691522,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739323557.2695,"msg":"No files matching import glob patter
n","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739323557.2695348,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739323557.2695723,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739323557.2696235,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739323557.2721164,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739323557.2736979,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=36570) - Caddy is running in the background
{"level":"info","ts":1739324268.177545,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739324268.1778708,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739324268.1778998,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739324268.1779385,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739324268.177986,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739324268.1809487,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739324268.182059,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=52550) - Caddy is running in the background
panic: send on closed channel
goroutine 3991 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8700bfdc0, {0x32f2b9e37
280, 0x87040f420})
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 3843
github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
Thank you a lot, it looks like its also a layer 4 module issue on your side.
I gonna write in the existing ticket. Can you provide more information like the Caddyfile you were using? (you can anonymize thibgs like api keys and domains)
Of course, attached is the caddy file as I figured it would be easier to read as an attachment instead of inline in the post.
If there is anything you notice that you would like me to try, let me know. Since you said it appears to be a layer 4 issue I could potentially just NAT those ports as the reverse proxy is really only required for 80 and 443 but it would be really nice to have it all working through Caddy as it is a pretty nice setup.
Thanks, for what I got as information upstream its mostly related to UDP streaming. Try to see if the crashes still happen if you don't use any UDP in the layer 4 proxy.
I can do that. It looks like I only have one layer 4 UDP connection so I can move that to a NAT and re-enable the plugin.
I'll update you later if that works.
Thanks for helping :)
Of course, I know it is hard to troubleshoot issues if people won't test. I have switched over the config, after removing the UDP layer 4 so i'll let you know what happens. If there are going to be issues I would expect it in the next couple hours.
While we are waiting for this to happen I was wondering, is there a way to add other DNS providers to the plugin? I am currently using the ACME plugin for my certificate enrollment as they support my provider, ClouDNS but it doesn't look like the Caddy plugin supports that one. The ACME plugin is setup and working fine so not sure there is any need to change but it could be one less thing to manage if it was in Caddy too.
That provider was released today per random change. So if you update to 25.1.1 you'll get a new caddy plugin version that supports it.
https://github.com/opnsense/plugins/pull/4507
Though you have to add it manually from the shell like this though, so that the caddy binary compiles the provider in.
https://caddyserver.com/docs/command-line#caddy-add-package
caddy add-package github.com/caddy-dns/cloudns
That's great you added it. I just did the update and it is showing in the list now and I ran the command to add the module too. Not sure i'll move off ACME right now as it is working well and it will be nice to have the option to use the certificate outside of Caddy if I decide to do that as well. I am currently using it for the web interface of the firewall which is nice.
So far no crashes with the server since I made the change so that is promising. In the event that this turns out to be the issue, is there a possible fix to it? My ideal plan is I would like to be able to use the Caddy server to route to the servers internally and externally so I don't have to maintain certificates in two places. Right now I am using my internal NGINX server to handle the internal communications and the Caddy to handle the external ones but it is just more infrastructure to maintain so being able to get rid of the NGINX server would be nice but I obviously can't do that if I can't get the UDP ports to route properly.
Just check the github issue I have linked earlier. If there is progress a fix will emerge. Best subscribe to the github issue or comment in it so you get notifications from it.
https://github.com/mholt/caddy-l4/issues/295
When the issue has been found and patched upstream, the patch will be downstreamed over time and end up in a new release here.
I have a similar issue and have updated to the latest OPnsense 25.1.1 and Caddy plugin 1.8.2 but still get the memory issue after running for a day:
2025-02-14T07:52:15 Notice kernel <3>pid 87856 (caddy), jid 0, uid 0, was killed: failed to reclaim memory
2025-02-13T13:18:23 Notice kernel <118>Log: /var/log/caddy/caddy.log
2025-02-13T13:18:23 Notice kernel <118>Starting caddy... done
2025-02-11T13:10:43 Notice kernel <3>pid 47033 (caddy), jid 0, uid 0, was killed: failed to reclaim memory
2025-01-30T15:34:06 Notice kernel <3>pid 87676 (caddy), jid 0, uid 0, was killed: failed to reclaim memory
No panic messages in caddy logs
Have you checked "/var/log/caddy/caddy.log" specifically? It's not exposed in the GUI.
Thank you for posting the link to the issue Cedrik, you're right, it sounds exactly like what I am running into and since removing that section of the config, the system has been stable.
In your experience, how long does it generally take for bugs to get fixed. I am liking the simplicity of Caddy but wondering if I should try to figure out the issues I was having with NGINX on OpnSense and try to get it to work as it is very reliable and stable once properly setup. It also has a pretty neat WAF filtering options which would be nice to implement if I can get the rest of it working.
Thanks for all your help.
Thanks for confirming.
Sorry I cannot answer that specifically, it depends on the scope of the underlying issue.
As with everything, use what works best for you. :)
I appreciate that thanks Cedrik. I know how hard it can be to maintain stuff, especially when a lot of it is out of your control. I just wasn't sure how long bugs normally take to fix. The main reason I left PFSense to come to OpnSense is they released updates every year or even less and for a firewall product, that seemed way to long between updates so I like that OpnSense does it more frequently.
I will probably investigate the PFSense issues I am having a bit more but the simplicity of Caddy is really nice and (other than this bug) it just works which is great.
Quote from: Monviech (Cedrik) on February 14, 2025, 12:54:08 PMHave you checked "/var/log/caddy/caddy.log" specifically? It's not exposed in the GUI.
Yes I checked all the logs from that folder.
I do not have Layer4 setup and only two Reverse Proxies. With Caddy disabled the memory usage sits at 3.1/8GB RAM, With Caddy running it is initially 3.2/8GB then after a day 4/8GB so not running low on RAM.
I do appreciate that with issues you cannot reproduce or have any meaningful errors in the logs it is a bit difficult to progress a solution.
My system does seem to have had remnants of ZenArmor from previous setups that have not quite removed cleanly so it could be something there but everything else seems ok and I am reluctant to wipe my firewall to try Caddy on a clean machine so will leave without it for now.
Thanks
Using something else is a valid solution too. Sometimes things just dont work out :)
Hi Cedrik
I am still using Caddy but have found that the memory issues I have are also affecting Unbound DNS.
My inactive memory still creeps up and up even without Caddy installed so it's not the plugin's issue.
Thanks for your help.
Are you sure its not the ARC cache of zfs that eats up your RAM slowly over time? Cause that's expected and wanted.
No, I am not using zfs just ufs.