Caddy responds properly to LAN NET addresses but not WAN NET

[suttjr]

Hi all,
  I have an pair of OPNSense VMs (25.1.8_1) configured with CARP for a LAN and WAN addresses. Due to the nature of this network segmentation, both devices on the WAN and LAN side are using the OPNsense appliances for DNS. In this case I'm fairly certain it's not DNS (since ya know, "it's always DNS"). I have all DNS records that caddy will servicing pointing to the WAN CARP address (I have tried the WAN IP of the appliance too).  I am brand new to caddy so I really am not sure if I've missed something in the documentation or if CARP requires a special consideration I am unaware of.

The issue:

When a client on the LAN NET hits the WAN IP address (CARP or device) as published in DNS, caddy responds as expected.  When a client on the WAN NET the WAN IP Address (CARP or device), the connection times out.  Strangely though, I can see the caddy https rule being hit and allowing the traffic through. I do get a redirect when I hit http://caddyendpoint.domain.com as I would expect.

I had figured this was some sort of NAT issue because I have NAT rules set as Hybrid for proper CARP behavior.  But the same behavior occurs when hitting the WAN IP of an OPNSense appliance.  I can see the session in the state table (ESTABLISHED:FIN_WAIT_2), so the WAN firewall rule is working as expected.  The connection times out, and there doesn't appear to be any relevant errors in caddy or on the OPNsense that I can find.    I have tried access lists including the WAN NET for the domains in the caddy config.  I have tried adding overlapping firewall rules that call out the static source and destination.  I have even created an ANY/ANY firewall rule for WAN NET to This firewall".  No change in the behavior or logging.  I am sure I have missed a tick box or something simple, but I can't seem to find it.

I am hoping someone can point me in the right direction. Just finding a proper error would be good enough to keep moving forward.

Any help would be greatly appreciated.

Thanks!

[Monviech (Cedrik)]

Best way to troubleshoot would be enabling the HTTP access log in a domain, setting the log level to debug, and then see if requests get through from WAN and caddy receives them. You can see quite some information in these access debug logs.

[suttjr]

Thanks @Monviech!
    I have enabled the http access logs on all my domains.  It looks like I'm getting a "status:308" - Permanent Redirect, on all WAN NET requests.

I also for fun tested with VPN clients, which also work successfully like LAN NET clients, though on a different subnet.  Some requests from the LAN NET receive status:304 but requests from VPN NET appear to have no redirects at all.

I do not understand this behavior, but speculating wildly tells me the 308 - Permanent Redirect is causing an issue in some way. 

Any thoughts on where I might look to resolve that?

Your help is greatly appreciated!

thanks!

[Monviech (Cedrik)]

Try to disable all handlers and just leave the domains enabled.

You will be served an empty site with a HTTP 200 code when you connect on 443 and can inspect if caddy serves a certificate in your browser.

If that works from WAN, you know its something about your backend servers you proxy to.

EDIT: Also permanent redirects get cached by the client, try to use curl for your tests or private browswr windows.

[suttjr]

I disabled all of the handlers and tried each one through curl.

Strangely enough, I get nothing in the caddy log and a connection reset on the WAN NET client.

curl -L https://images.*****.info
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to images.********.info:443


Let me know if there other information I can provide.

[Monviech (Cedrik)]

Maybe you have some Port Forward or 1:1 NAT rules on WAN that send requests to the wrong destination?

[suttjr]

I disabled all the 1:1 NAT rules and removed any port forward rules, except anti lockout.


This is the only log I can find.

"info","ts":"2025-06-20T13:00:51Z","logger":"http.log.access.1dc5ae0b-e131-4fa4-bf1b-1f456e91fb68","msg":"handled request","request":{"remote_ip":"192.168.**.***","remote_port":"53560","client_ip":"192.168.**.***","proto":"HTTP/1.1","method":"GET","host":"images.********.info","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0"],"Accept-Language":["en-US,en;q=0.9"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Accept-Encoding":["gzip, deflate"],"Connection":["keep-alive"],"Cache-Control":["max-age=0"],"Dnt":["1"]}},"bytes_read":0,"user_id":"","duration":0.000639833,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://images.********.info/"],"Content-Type":[]}}

I didn't realize before the 308 was a redirect to https.  It appears then caddy doesn't want to serve HTTPS to the WAN NET clients. 

Not sure what that means from a caddy perspective. 

Any other thoughts?  If not, I may just have to have caddy live somewhere else.

Thanks!

[Monviech (Cedrik)]

Nope no idea right now.

Either try the caddy community, or follow your plan to use it on a separate instance with NAT. Sometimes the most pragmatic choice is the best.

[suttjr]

I appreciate the help!  I'll see what the Caddy community has to say.

Thanks!