Messages

Fair enough, i'll leave them unless they are broken. My headers all connect to the upstream servers by IP though as I didn't want to deal with DNS issues or certificate issues on the upstreams so all sites are using the skip tls insecure but this is the only one that seemed to break.

Thanks again for the quick help.

You are a rockstar, that worked great. Not sure why it only affected one of the sites I have hosted though.

Should I change that on all the handlers or is it one of those things that if it is working, don't bother messing with it?

Good morning, I just did an update to my Opnsense firewall. Everything seems ok but I have one site that redirects through the Caddy plugin that is now failing. It was working previously and just stopped after the update. I looked at the updates that were done and it looks like Caddy updated to 2.1.0 as part of the update. From what I can tell, all the other sites are working fine, it is just this one which is a pretty standard HTTPS site so I am not sure why it has stopped.

I changed the DNS records so the connection points directly to the server, not going through Caddy and it works fine that way but as soon as I point it to Caddy again, it stops. I have looked at the logs on the firewall and am not seeing any real errors related to this either.

The error I am seeing in the web browser is.

Not Found
HTTP Error 404. The requested resource is not found.

Any suggestions?

Thanks

Neal

Hi Cedrik, I installed the new package a few days ago and re-enabled the layer 4 transport that had been failing. Things seem to have been working fine since then with no crashes of Caddy so that is great, thank you for getting that fixed.

I wanted to ask another, slightly unrelated question. Do you just maintain Caddy on OpnSense or do you assist with the regular Caddy as well? I have been looking at implementing a WAF firewall on my OpnSense box and there are a few ways to do it with ModSecurity and NGINX or HaProxy. I looked up whether it can be done through Caddy and they explicitly say not to run Caddy and ModSecurity on the same box and if I am going to use ModSecurity I should have it go to an external Caddy box which I was thinking I could possibly build on a Linux VM as I do like Caddy now that I am getting used to it. I just wasn't sure if you did anything with Caddy outside of OpnSense.

Thank you again for fixing the Layer 4 transport issue.

Great, thank you for the steps. I'll give it a try and see if it still causes issues. I only have one UDP stream on that firewall that seemed to cause the issue as it stopped crashing as soon as I disabled that.

Just to clarify Cedrik, is this patch for the UDP streaming? The thread took a bit of a turn from when I posted about that issue so just not sure which issue this patch is supposed to fix.

Is the patch not released through the regular OpnSense update or does it have to be done manually? I can test it on my system if you can tell me how it needs to be applied.

I currently run NGINX on a Linux VM and have been using it that way for years and it works well. I just moved from PFSense to OpnSense and really like OpnSense better and love the fact that there is an NGINX plugin for it, there wasn't in PFSense.

I have worked on moving my setup from the Linux VM to the OpnSense plugin and got most things working ok. There were some challenges as I had options setup on the VM in the config files that I couldn't seem to duplicate in the GUI of OpnSense so not sure how necessary they will all be. The main issue I have run into is with the streaming module and one particular service.

The setup is pretty basic, it has a web interface and then a couple other TCP ports for transferring data. I setup the web port no problem and when I enable the other ports on the streaming module they seem fine but aren't working properly. The strange part is I can telnet to these ports remotely and it connects but when I try to programmatically connect through them, it doesn't work. Another strange part is, if I leave the web portion enabled and just try to directly NAT these ports, ideally not going through the NGINX it still doesn't work. If I disable NGINX and do it all with direct NAT or through the Linux NGINX it works perfectly.

I am not sure if the NGINX box is intercepting all traffic on that hostname regardless of the port and that is why direct NAT isn't working with it enabled and not sure why telnet works to those ports but the communication doesn't properly.

Does anyone have any suggestions at what the issue might be or how to even start troubleshooting it? It would be nice to terminate all these connections at the firewall and direct them appropriately from there but I am not sure why this part isn't working. I have a few other services running over the streaming module on different servers and they all work fine, it is just this one that isn't.

Any help would be really appreciated.

I appreciate that thanks Cedrik. I know how hard it can be to maintain stuff, especially when a lot of it is out of your control. I just wasn't sure how long bugs normally take to fix. The main reason I left PFSense to come to OpnSense is they released updates every year or even less and for a firewall product, that seemed way to long between updates so I like that OpnSense does it more frequently.

I will probably investigate the PFSense issues I am having a bit more but the simplicity of Caddy is really nice and (other than this bug) it just works which is great.

Thank you for posting the link to the issue Cedrik, you're right, it sounds exactly like what I am running into and since removing that section of the config, the system has been stable.

In your experience, how long does it generally take for bugs to get fixed. I am liking the simplicity of Caddy but wondering if I should try to figure out the issues I was having with NGINX on OpnSense and try to get it to work as it is very reliable and stable once properly setup. It also has a pretty neat WAF filtering options which would be nice to implement if I can get the rest of it working.

Thanks for all your help.

That's great you added it. I just did the update and it is showing in the list now and I ran the command to add the module too. Not sure i'll move off ACME right now as it is working well and it will be nice to have the option to use the certificate outside of Caddy if I decide to do that as well. I am currently using it for the web interface of the firewall which is nice.

So far no crashes with the server since I made the change so that is promising. In the event that this turns out to be the issue, is there a possible fix to it? My ideal plan is I would like to be able to use the Caddy server to route to the servers internally and externally so I don't have to maintain certificates in two places. Right now I am using my internal NGINX server to handle the internal communications and the Caddy to handle the external ones but it is just more infrastructure to maintain so being able to get rid of the NGINX server would be nice but I obviously can't do that if I can't get the UDP ports to route properly.

Of course, I know it is hard to troubleshoot issues if people won't test. I have switched over the config, after removing the UDP layer 4 so i'll let you know what happens. If there are going to be issues I would expect it in the next couple hours.

While we are waiting for this to happen I was wondering, is there a way to add other DNS providers to the plugin? I am currently using the ACME plugin for my certificate enrollment as they support my provider, ClouDNS but it doesn't look like the Caddy plugin supports that one. The ACME plugin is setup and working fine so not sure there is any need to change but it could be one less thing to manage if it was in Caddy too.

I can do that. It looks like I only have one layer 4 UDP connection so I can move that to a NAT and re-enable the plugin.

I'll update you later if that works.

Of course, attached is the caddy file as I figured it would be easier to read as an attachment instead of inline in the post.

If there is anything you notice that you would like me to try, let me know. Since you said it appears to be a layer 4 issue I could potentially just NAT those ports as the reverse proxy is really only required for 80 and 443 but it would be really nice to have it all working through Caddy as it is a pretty nice setup.

I ran Caddy again tonight and it worked fine but an hour or so and then stopped. Below is what I see in that logfile but nothing from what I can tell about any specific panic.


The resources on the server sat pretty consistent for the whole time and didn't seem to creep up much. I'm attaching a copy of the top view taken shortly before the service stopped. It was pretty consistent throughout the whole time things were running. I am going to re-enable my NGINX server but if there is anything else you want me to test, let me know.

{"level":"info","ts":1738974488.0721657,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974488.0725987,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974488.07268,"msg":"No files matching import glob patte
rn","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974488.0727434,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738974488.0728152,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738974488.0762215,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738974488.078006,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=78285) - Caddy is running in the background
{"level":"info","ts":1738974734.3310537,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974734.331348,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974734.3313804,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974734.331417,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738974734.331456,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738974734.3334653,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738974734.3345199,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=60153) - Caddy is running in the background
{"level":"info","ts":1738975322.6449044,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738975322.6451797,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738975322.6452117,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738975322.645248,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738975322.6452894,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738975322.6472473,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738975322.6482174,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=44523) - Caddy is running in the background
{"level":"info","ts":1738975422.0715666,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738975422.0718377,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738975422.0718775,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738975422.071914,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738975422.0719538,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738975422.0739639,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738975422.075407,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=74543) - Caddy is running in the background
panic: send on closed channel

goroutine 10782 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x86c02f1f0, {0x15aa82a44
7b8, 0x86c3b0bc0})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 10726
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739023449.4873703,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739023449.487672,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739023449.4877079,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739023449.4877467,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739023449.4877892,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739023449.489998,"msg":"adapted config to JSON","adapter":
"caddyfile"}
{"level":"info","ts":1739023449.4909961,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=18512) - Caddy is running in the background
panic: send on closed channel

goroutine 92 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x86c5a0cb0, {0x5c74f68f4
c0, 0x86c2ddd20})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739053775.421173,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739053775.4214938,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739053775.4215293,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739053775.421567,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739053775.4216142,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739053775.4244912,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739053775.4255404,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=50585) - Caddy is running in the background
panic: send on closed channel

goroutine 92 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x870134620, {0x18816c0d8
90, 0x870523ee0})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739062426.591729,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739062426.5920196,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739062426.5920596,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739062426.592098,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739062426.5921452,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739062426.5943375,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739062426.5956733,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=85549) - Caddy is running in the background
panic: send on closed channel

goroutine 93 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x87029af50, {0x2b544c41b
4e8, 0x8702b1700})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739065245.8650196,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739065245.8653393,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739065245.8653727,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739065245.8654163,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739065245.8654664,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739065245.8683712,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739065245.8694892,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=79758) - Caddy is running in the background
{"level":"info","ts":1739072460.8001187,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739072460.800456,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739072460.8004968,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739072460.8005433,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739072460.8006308,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739072460.8029695,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739072460.8044856,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=98895) - Caddy is running in the background
{"level":"info","ts":1739073385.0745974,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739073385.074918,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739073385.07495,"msg":"No files matching import glob patte
rn","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739073385.0749958,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739073385.075042,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739073385.0771258,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739073385.078136,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=14780) - Caddy is running in the background
{"level":"info","ts":1739073912.8176806,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739073912.8179882,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739073912.8180237,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739073912.8180597,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739073912.8181174,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739073912.8201737,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739073912.8214817,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=8396) - Caddy is running in the background
panic: send on closed channel

goroutine 115 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8704471f0, {0x2e80efd9c
828, 0x870477a40})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739156722.0015497,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739156722.0018783,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739156722.0019114,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739156722.001972,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739156722.0020287,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739156722.0042887,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739156722.005349,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=43620) - Caddy is running in the background
{"level":"info","ts":1739199652.7526066,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739199652.7529013,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739199652.752934,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739199652.7529862,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739199652.753033,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739199652.755767,"msg":"adapted config to JSON","adapter":
"caddyfile"}
{"level":"info","ts":1739199652.756928,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=51524) - Caddy is running in the background
panic: send on closed channel

goroutine 45339 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x870260c40, {0x69f8ca8fc
20, 0x8702ac500})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 45299
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739213533.973209,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739213533.9735048,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739213533.973533,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739213533.9735708,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739213533.9736202,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739213533.9765584,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739213533.9776092,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=57180) - Caddy is running in the background
panic: send on closed channel

goroutine 9269 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x870295500, {0x187e44bed
ab8, 0x87045a2a0})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 9149
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739224498.843148,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739224498.843496,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739224498.843529,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739224498.8435707,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739224498.8436186,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739224498.8467724,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739224498.8479614,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=56747) - Caddy is running in the background
panic: send on closed channel

goroutine 12802 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8700df9d0, {0x1f32a37fc
b90, 0x87040f3a0})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 12292
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739323557.2691522,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739323557.2695,"msg":"No files matching import glob patter
n","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739323557.2695348,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739323557.2695723,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739323557.2696235,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739323557.2721164,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739323557.2736979,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=36570) - Caddy is running in the background
{"level":"info","ts":1739324268.177545,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739324268.1778708,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739324268.1778998,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739324268.1779385,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739324268.177986,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739324268.1809487,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739324268.182059,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=52550) - Caddy is running in the background
panic: send on closed channel

goroutine 3991 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8700bfdc0, {0x32f2b9e37
280, 0x87040f420})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 3843
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.

I can test that again tonight when the servers aren't heavily used. It generally happens within a few hours so it shouldn't take a long time to see it happen.

Is there a good way to monitor how much ram the process is using? I apologize if that is something I should know already as I am relatively new to the system.

Thanks