Messages

I currently run NGINX on a Linux VM and have been using it that way for years and it works well. I just moved from PFSense to OpnSense and really like OpnSense better and love the fact that there is an NGINX plugin for it, there wasn't in PFSense.

I have worked on moving my setup from the Linux VM to the OpnSense plugin and got most things working ok. There were some challenges as I had options setup on the VM in the config files that I couldn't seem to duplicate in the GUI of OpnSense so not sure how necessary they will all be. The main issue I have run into is with the streaming module and one particular service.

The setup is pretty basic, it has a web interface and then a couple other TCP ports for transferring data. I setup the web port no problem and when I enable the other ports on the streaming module they seem fine but aren't working properly. The strange part is I can telnet to these ports remotely and it connects but when I try to programmatically connect through them, it doesn't work. Another strange part is, if I leave the web portion enabled and just try to directly NAT these ports, ideally not going through the NGINX it still doesn't work. If I disable NGINX and do it all with direct NAT or through the Linux NGINX it works perfectly.

I am not sure if the NGINX box is intercepting all traffic on that hostname regardless of the port and that is why direct NAT isn't working with it enabled and not sure why telnet works to those ports but the communication doesn't properly.

Does anyone have any suggestions at what the issue might be or how to even start troubleshooting it? It would be nice to terminate all these connections at the firewall and direct them appropriately from there but I am not sure why this part isn't working. I have a few other services running over the streaming module on different servers and they all work fine, it is just this one that isn't.

Any help would be really appreciated.

I appreciate that thanks Cedrik. I know how hard it can be to maintain stuff, especially when a lot of it is out of your control. I just wasn't sure how long bugs normally take to fix. The main reason I left PFSense to come to OpnSense is they released updates every year or even less and for a firewall product, that seemed way to long between updates so I like that OpnSense does it more frequently.

I will probably investigate the PFSense issues I am having a bit more but the simplicity of Caddy is really nice and (other than this bug) it just works which is great.

Thank you for posting the link to the issue Cedrik, you're right, it sounds exactly like what I am running into and since removing that section of the config, the system has been stable.

In your experience, how long does it generally take for bugs to get fixed. I am liking the simplicity of Caddy but wondering if I should try to figure out the issues I was having with NGINX on OpnSense and try to get it to work as it is very reliable and stable once properly setup. It also has a pretty neat WAF filtering options which would be nice to implement if I can get the rest of it working.

Thanks for all your help.

That's great you added it. I just did the update and it is showing in the list now and I ran the command to add the module too. Not sure i'll move off ACME right now as it is working well and it will be nice to have the option to use the certificate outside of Caddy if I decide to do that as well. I am currently using it for the web interface of the firewall which is nice.

So far no crashes with the server since I made the change so that is promising. In the event that this turns out to be the issue, is there a possible fix to it? My ideal plan is I would like to be able to use the Caddy server to route to the servers internally and externally so I don't have to maintain certificates in two places. Right now I am using my internal NGINX server to handle the internal communications and the Caddy to handle the external ones but it is just more infrastructure to maintain so being able to get rid of the NGINX server would be nice but I obviously can't do that if I can't get the UDP ports to route properly.

Of course, I know it is hard to troubleshoot issues if people won't test. I have switched over the config, after removing the UDP layer 4 so i'll let you know what happens. If there are going to be issues I would expect it in the next couple hours.

While we are waiting for this to happen I was wondering, is there a way to add other DNS providers to the plugin? I am currently using the ACME plugin for my certificate enrollment as they support my provider, ClouDNS but it doesn't look like the Caddy plugin supports that one. The ACME plugin is setup and working fine so not sure there is any need to change but it could be one less thing to manage if it was in Caddy too.

I can do that. It looks like I only have one layer 4 UDP connection so I can move that to a NAT and re-enable the plugin.

I'll update you later if that works.

Of course, attached is the caddy file as I figured it would be easier to read as an attachment instead of inline in the post.

If there is anything you notice that you would like me to try, let me know. Since you said it appears to be a layer 4 issue I could potentially just NAT those ports as the reverse proxy is really only required for 80 and 443 but it would be really nice to have it all working through Caddy as it is a pretty nice setup.

I ran Caddy again tonight and it worked fine but an hour or so and then stopped. Below is what I see in that logfile but nothing from what I can tell about any specific panic.


The resources on the server sat pretty consistent for the whole time and didn't seem to creep up much. I'm attaching a copy of the top view taken shortly before the service stopped. It was pretty consistent throughout the whole time things were running. I am going to re-enable my NGINX server but if there is anything else you want me to test, let me know.

{"level":"info","ts":1738974488.0721657,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974488.0725987,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974488.07268,"msg":"No files matching import glob patte
rn","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974488.0727434,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738974488.0728152,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738974488.0762215,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738974488.078006,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=78285) - Caddy is running in the background
{"level":"info","ts":1738974734.3310537,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974734.331348,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974734.3313804,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974734.331417,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738974734.331456,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738974734.3334653,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738974734.3345199,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=60153) - Caddy is running in the background
{"level":"info","ts":1738975322.6449044,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738975322.6451797,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738975322.6452117,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738975322.645248,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738975322.6452894,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738975322.6472473,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738975322.6482174,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=44523) - Caddy is running in the background
{"level":"info","ts":1738975422.0715666,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738975422.0718377,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738975422.0718775,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738975422.071914,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738975422.0719538,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738975422.0739639,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738975422.075407,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=74543) - Caddy is running in the background
panic: send on closed channel

goroutine 10782 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x86c02f1f0, {0x15aa82a44
7b8, 0x86c3b0bc0})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 10726
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739023449.4873703,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739023449.487672,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739023449.4877079,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739023449.4877467,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739023449.4877892,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739023449.489998,"msg":"adapted config to JSON","adapter":
"caddyfile"}
{"level":"info","ts":1739023449.4909961,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=18512) - Caddy is running in the background
panic: send on closed channel

goroutine 92 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x86c5a0cb0, {0x5c74f68f4
c0, 0x86c2ddd20})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739053775.421173,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739053775.4214938,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739053775.4215293,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739053775.421567,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739053775.4216142,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739053775.4244912,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739053775.4255404,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=50585) - Caddy is running in the background
panic: send on closed channel

goroutine 92 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x870134620, {0x18816c0d8
90, 0x870523ee0})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739062426.591729,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739062426.5920196,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739062426.5920596,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739062426.592098,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739062426.5921452,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739062426.5943375,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739062426.5956733,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=85549) - Caddy is running in the background
panic: send on closed channel

goroutine 93 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x87029af50, {0x2b544c41b
4e8, 0x8702b1700})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739065245.8650196,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739065245.8653393,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739065245.8653727,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739065245.8654163,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739065245.8654664,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739065245.8683712,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739065245.8694892,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=79758) - Caddy is running in the background
{"level":"info","ts":1739072460.8001187,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739072460.800456,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739072460.8004968,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739072460.8005433,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739072460.8006308,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739072460.8029695,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739072460.8044856,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=98895) - Caddy is running in the background
{"level":"info","ts":1739073385.0745974,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739073385.074918,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739073385.07495,"msg":"No files matching import glob patte
rn","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739073385.0749958,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739073385.075042,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739073385.0771258,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739073385.078136,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=14780) - Caddy is running in the background
{"level":"info","ts":1739073912.8176806,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739073912.8179882,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739073912.8180237,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739073912.8180597,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739073912.8181174,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739073912.8201737,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739073912.8214817,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=8396) - Caddy is running in the background
panic: send on closed channel

goroutine 115 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8704471f0, {0x2e80efd9c
828, 0x870477a40})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739156722.0015497,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739156722.0018783,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739156722.0019114,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739156722.001972,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739156722.0020287,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739156722.0042887,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739156722.005349,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=43620) - Caddy is running in the background
{"level":"info","ts":1739199652.7526066,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739199652.7529013,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739199652.752934,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739199652.7529862,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739199652.753033,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739199652.755767,"msg":"adapted config to JSON","adapter":
"caddyfile"}
{"level":"info","ts":1739199652.756928,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=51524) - Caddy is running in the background
panic: send on closed channel

goroutine 45339 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x870260c40, {0x69f8ca8fc
20, 0x8702ac500})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 45299
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739213533.973209,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739213533.9735048,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739213533.973533,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739213533.9735708,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739213533.9736202,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739213533.9765584,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739213533.9776092,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=57180) - Caddy is running in the background
panic: send on closed channel

goroutine 9269 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x870295500, {0x187e44bed
ab8, 0x87045a2a0})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 9149
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739224498.843148,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739224498.843496,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739224498.843529,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739224498.8435707,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739224498.8436186,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739224498.8467724,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739224498.8479614,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=56747) - Caddy is running in the background
panic: send on closed channel

goroutine 12802 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8700df9d0, {0x1f32a37fc
b90, 0x87040f3a0})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 12292
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:85 +0x6c5
{"level":"info","ts":1739323557.2691522,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739323557.2695,"msg":"No files matching import glob patter
n","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739323557.2695348,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739323557.2695723,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739323557.2696235,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739323557.2721164,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739323557.2736979,"msg":"redirected default logger","from"
:"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=36570) - Caddy is running in the background
{"level":"info","ts":1739324268.177545,"msg":"using config from file","file":"/u
sr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1739324268.1778708,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1739324268.1778998,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1739324268.1779385,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1739324268.177986,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1739324268.1809487,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1739324268.182059,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=52550) - Caddy is running in the background
panic: send on closed channel

goroutine 3991 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8700bfdc0, {0x32f2b9e37
280, 0x87040f420})
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/serv
er.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.
go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 3843
        github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/layer4/app.

I can test that again tonight when the servers aren't heavily used. It generally happens within a few hours so it shouldn't take a long time to see it happen.

Is there a good way to monitor how much ram the process is using? I apologize if that is something I should know already as I am relatively new to the system.

Thanks

Thank you for the response Monviech, see below for the entries from that logfile. I have currently disabled the plugin so it is running through my Linux NGINX box but hopefully there will be something helpful in these entries. I am not sure if these are directly related to the crash or not so if you want me to re-enable it again and see if I get another crash I can try that tonight.

{"level":"info","ts":1738974488.0721657,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974488.0725987,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974488.07268,"msg":"No files matching import glob patte
rn","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974488.0727434,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738974488.0728152,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738974488.0762215,"msg":"adapted config to JSON","adapter"
:"caddyfile"}
{"level":"info","ts":1738974488.078006,"msg":"redirected default logger","from":
"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=78285) - Caddy is running in the background
{"level":"info","ts":1738974734.3310537,"msg":"using config from file","file":"/
usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738974734.331348,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738974734.3313804,"msg":"No files matching import glob pat
tern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738974734.331417,"msg":"No files matching import glob patt
ern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}

That's the strange part, I am not seeing any logs saying it is running out of memory. I did read other posts online saying that people had experienced a memory leak with it but I haven't seen anything saying i'm running out of memory.

I have 16GB of ram in my firewall and don't have a ton of heavy usage plugins running so I don't really suspect memory is the actual issue but I can't figure out what is going on. I have reverted back to using my Linux VM for the time being as it is at least stable and works well.

Good morning, I have recently migrated from PFSense to OpnSense and am really liking it. I currently run an NGINX box on a Linux VM which works well but I like the idea of handling all the filtering through the firewall directly. I tested out the NGINX plugin on the firewall but couldn't get certain ports to stream properly like they do on my Linux NGINX install and while I was troubleshooting, I stumbled upon Caddy which seemed pretty lightweight and easy to install.

I had already setup the ACME plugin to generate certificates and was using it with NGINX so I setup my caddy reverse proxy rules and my layer 4 streaming rules and was shocked how easily they all worked and everything seemed great. I used the certificate that the ACME plugin generated for all the HTTPS rules and that worked fine. The issue is, every few hours, the plugin will just stop. The way I know is that I get a bunch of messages about systems being down. I can login to the firewall and restart it and it all comes back up but it will do it again in a few hours.

I have gone through the logs but since I am new to Caddy I am not sure what to really look at and googling the various errors hasn't helped. I could use Caddy if I can get this to be stable or go back to NGINX and NAT the specific ports I was having issues with streaming but for some reason, the NATTING of the ports doesn't seem to take effect if the NGINX plugin is running. If I don't use it, and NAT those ports directly to the server, it all works fine.

Thanks for any suggestions or help here.