Like this plugin? Consider donating to me. 8)
(https://upload.wikimedia.org/wikipedia/de/thumb/5/5f/PayPal_2023_logo.svg/1920px-PayPal_2023_logo.svg.png) (https://www.paypal.com/paypalme/pischem)
This plugin is simple to use and very easy to configure. Yet, it also offers plenty of advanced options for more complicated usecases at the same time.
- For Reverse Proxy + automatic Let's Encrypt Certificates follow these steps:
1. Installation (https://docs.opnsense.org/manual/how-tos/caddy.html#installation)
2. Prepare OPNsense for Caddy after installation (https://docs.opnsense.org/manual/how-tos/caddy.html#prepare-opnsense-for-caddy-after-installation)
3. Creating a simple reverse proxy (https://docs.opnsense.org/manual/how-tos/caddy.html#creating-a-simple-reverse-proxy)
(Please note that the docs have been updated for 24.7, so there might be different terminology at a few steps.)
- For Dynamic DNS follow this additional step:
4. Dynamic DNS (https://docs.opnsense.org/manual/how-tos/caddy.html#dynamic-dns)
Layer 4 module:
https://docs.opnsense.org/manual/how-tos/caddy.html#caddy-layer4-proxy
If you have questions or find an issue, please ask here or post on Github, I will answer them and fix problems as soon as possible.
Thankyou. A very well written piece of software, a great plugin that makes Caddy a breeze to use.
Great work done... hope to see this integrated into the official OPNsense library someday.
Appreciate the work you've done and the help on github.
I have several handlers working now for my domain. Only one is accessible externally (Internet) and the rest are all available internally only on my LAN or VPN via the Access lists functions. This was far easier than HAProxy or nginx for my needs. I've actually disabled the configs I had there and migrated them to Caddy since my use cases are straightforward.
In an effort to try and give something back, I've front-ended my Unifi console with this Caddy plugin and wish to share a quick tutorial here. There are many ways to do this (e.g. update the cert for Unifi itself to a Trusted Cert). However, this method is potentially an easier way where we will just trust the Unifi cert. Every 2 years or so, this cert will need to be updated.
Step 1 - Get the Unifi CA cert: Many ways to do this, but opted for lazy way. Navigate to your Unifi console in any browser. Click the cert icon in the address bar (most likely will say "Not secure"). Then click the "cert is not valid" link or the link your browser has to show the cert. Go to the details tab and find "Export". Export the cert and save it to a location with a name you'll recognize (e.g. Unifi.crt).
Step 2 - Get the cert text: Right click on the "Unifi.crt" or whatever you named it and open it with notepad or notepad++ or vi or nano or your text editor of choice. Copy the details to your clipboard:
At time of this writing for my version of the Unifi console the text for the Unifi.crt is/was:
-----BEGIN CERTIFICATE-----
MIIDfTCCAmWgAwIBAgIEZVl1bjANBgkqhkiG9w0BAQsFADBrMQswCQYDVQQGEwJV
UzERMA8GA1UECAwITmV3IFlvcmsxETAPBgNVBAcMCE5ldyBZb3JrMRYwFAYDVQQK
DA1VYmlxdWl0aSBJbmMuMQ4wDAYDVQQLDAVVbmlGaTEOMAwGA1UEAwwFVW5pRmkw
HhcNMjMxMTE5MDIzOTQyWhcNMjYwMjIxMDIzOTQyWjBrMQswCQYDVQQGEwJVUzER
MA8GA1UECAwITmV3IFlvcmsxETAPBgNVBAcMCE5ldyBZb3JrMRYwFAYDVQQKDA1V
YmlxdWl0aSBJbmMuMQ4wDAYDVQQLDAVVbmlGaTEOMAwGA1UEAwwFVW5pRmkwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCaxheTfuaMfIltQuageZS7T55k
cs9xGlvgP3QGGpqeEpk43lHJBZaJmhUtI91dFkfBxyMUBe5AcSWoazRpLYmIuDqI
6T1GDzwe38Hjsy+sCl/CLtjBqga4FCVhqDwMppqqgyBAU6eSLdHr3mTNQk21X5VS
rgU5/zNx9Yon9ChhIxxukqCqMneVXDr8gSGWB3n9AxwTeVm8xVEWWxd+1ho0KP+T
6mxy3UhygoKbgJgxGH8rXgoAwxV8J7YG/soJ5SrJBHm3mZEquUZkFnjmd7Vd0Sl0
6y+44mkxOa0f0evh9LYpvCfXT7Q/705Ucqn9u0lSK86yZN2hjf3GUU5koiOJAgMB
AAGjKTAnMBMGA1UdJQQMMAoGCCsGAQUFBwMBMBAGA1UdEQQJMAeCBVVuaUZpMA0G
CSqGSIb3DQEBCwUAA4IBAQCS9KXCOjT7tqplCrMR3CyeYQAOJsyr1fAGK6AF9jHO
KD7CidfR7ZJi/ocZABr7i+F2zSaPXw9PwHrbRF44uiTxHt5jlXLDhYNI/MqI2Yx/
yx1ddT1UiJF3MM4v8xx7CBpvBKYl6Wr8D561wT7ywb/PcGZ/N7TZbT3yqEr/W3CB
5LhDitlf+DZ0F/SmDFvmFdf0Y9vQuqrylYnTwTr4di7UUoPYbBFf3w4Uu4DAfROX
MoB9Jr0jBi7+nH+MtUHFtn1vcrRU8Q8O4TCq0S+WLGhbrILzBbrX6hCbsrzxEYzW
tzDGic7FQrRpqGfwfeCHsEU42LpBPpaIGKZXt3EhzYo9
-----END CERTIFICATE-----
Step 3 - Add cert to OPNsense trusted store: Login to OPNsense console and go to System-> Trust -> Authorities. Click the + to add a Trust Authority.
Descriptive name : Unifi's Self-Signed Console CA
Method: Import an existing Certificate Authority
Certificate data: paste the full text from Step 2
Click Save
(We will need to edit this trusted cert once it expires and replace it with the newly issued one. with my current Unifi console version, that will be February 20th, 2026. If upgrading the console version, the cert may change and need to be updated as well - depends on what Ubiquiti does with the Unifi Console)
Step 3 - Create the Unifi handler: Assuming domains, subdomains, etc. are all configured via other tutorials.
Create a "handler" in Caddy "Handlers" as normal like you would for any other http site with the backend server domain and port for the Unifi console in your infrastructure. Unifi's console requires https, so to avoid the 502 and similar errors, we need to configure Caddy to "handle" the https. This is completed by supplying the Subject Alternative Name (SAN) value from the Unifi.crt which is DNS Name=Unifi, the CA we added to the trusted authorities for OPNsense, and utilizing TLS between caddy and the Unifi console.
Handle type: handle
handle path: (blank).
backend server domain: your unifi's IP / hostname
backend server port: your unifi's port (usually 8443)
TLS: "checked"
TLS Trust CA Certificate: select the item named from Step 2 (e.g. Unifi's Self-Signed Console CA).
TLS Server Name: Unifi
Add a description, save and apply. Navigate to the handler for Unifi's console and your connection should now be encrypted and trusted: https -> Caddy -> https to Unifi server.
Thank you a lot I have added your post to the main post as additional tutorial. ^^
My utmost respect. This is a very successful plugin. Simple, direct and, above all, easy to use for inexperienced users.
The alternative HAProxy/Nginx are really not characterized by their user-friendliness and simplicity.
WHERE CAN I BUY YOU A BEER OR COFFEE? would like to support your work!
Perhaps it would be possible to add a few more aspects, pitfalls and tips to the above tutorial.
- Specific syntax of the API key for the DNS challenge
- Where to find Subject Alternative Name (SAN)
- Instructions for e.g. Nextcloud etc.
What I have not yet managed to get right.
That the OPNsense internal calls are forwarded directly to the backend server and certificates are still issued by Caddy. All connections only succeed via HTTP without certificates. Could you help with an example or give me a hint? I have now tried it with Vaultwarden and Sterling PDF (both in a docker container), resolution to external works without problems even with wildcard, but to internal only as described above.
Many thanks for the great work and good luck with the integration in OPNsense!
Hey, thanks for the praise. Right now I don't have any donations set up, I'm doing this mostly because I use this plugin myself excessively everywhere.
Regarding TLS Certificates, it's a good idea to follow this tutorial for a successful backend TLS connection. When you get this to work you can also get all other TLS examples to work: https://github.com/Monviech/os-caddy-plugin#how-to-create-a-handle-with-tls-and-a-trusted-self-signed-certificate
In the coming version the API Key stuff is documented a bit better, also with a reference to the repository where Caddy stores all DNS Providers: https://github.com/caddy-dns (Here you can find the docs for each dns provider module).
I don't have instructions for Nextcloud, but you can always check the /usr/local/etc/caddy/Caddyfile and also browse the https://caddy.community where there are a lot of Caddyfile examples for specific setups like Nextcloud.
Many thanks for your advice. I followed your tutorials - it works perfectly.
(Self-signed) certificates from OPNsense, Unifi, Proxmox TLS works between proxy and backend.
However, it is unclear to me how this should work for virtual machines, containers - which do not yet have a certificate.
Was under the assumption that the certificate that Caddy issues externally can also be used for the connection between ReversProxy and the backend.
Or should I simply create an override in Unbound?
Like:
host.example.com ---- unbound----opnsense 127.0.0.1-----caddy----TLS----backend
Somehow I can't see the way right now.
TLS is not a requirement to create a connection between Caddy and your Backend Servers. The standard is to use HTTP without encryption (which is also the Caddy default).
That is called TLS Termination https://en.m.wikipedia.org/wiki/TLS_termination_proxy in Reverse Proxy Jargon. Just leave the port in the Handler empty, and it will use Port 80 and HTTP. Or use any other port (other than 443) and leave TLS unchecked to use an unencrypted connection per default.
Encrypting the connection in a trusted network (aka not internet) is unneeded most of the time.
You also use Caddy internally if you want to access these VMs with a TLS connection. Since you have made an A-Record in your authorative nameserver that points to the external IP of the OPNsense, all internal requests to this A-Record will reach the OPNsense per default (with HTTPS), and Caddy will reverse proxy them back to your backend servers (with HTTP). You dont need any special NAT rules or any unbound overrides or anything.
Hello - I recently upgraded and am wondering if something isn't quite working as expected.
Short version -
1) upgraded OPNsense to 24.1.3_1
2) I had two new subdomains I wanted to add (in addition to the ones I've already created). However, I am unable to get these to work. Other/previous subdomains are still working without issue.
3) To troubleshoot, I enabled "http access log" for my domain and also enabled Log HTTP Access in JSON Format.
4) Have hit apply, save, and even restarted the Caddy service a few times.
No logs in
:/var/log/caddy/access # ls
:/var/log/caddy/access #
nothing is showing in the caddy.log either for access
The Caddyfile looks correct with the new host and handle
The autosave.json in /usr/local/etc/caddy/.config/caddy does not have the new host and handle (don't think that is an issue, but just mentioning it).
Any ideas to tell me what I am doing wrong?
edit: To troubleshoot further, I modified an existing working "Handle" via the Caddy UI. I changed the IP and port to be of the settings for one of the new domains. After saving and applying, I am still presented the original configuration in the browser when I access that Handler subdomain. Seems like something isn't being persisted properly based on this test; at worst, I should receive an error, but instead things still work, albeit for the original subdomain configuration.
edit2: Since the plugin is close to being or is already available natively, do I need to do some cleanup steps with the repo since I installed the plugin prior to the native access?
I would suggest you remove the plugin one time, restart the firewall, and then reinstall it. Your config won't be lost. There has been a big cleanup due to the code review and a lot of things changed. The .config folder isnt used anymore now either.
The autoconf.json is somewhere in "/var/db/caddy/config/caddy" now cause the standard rc.d file of freebsd is used now "/usr/local/etc/rc.d/caddy". It should define that autosave path as ${caddy_directory:="/var/db/${name}"}.
The plugin has been merged but it will be available natively in a future version, probably during the next OPNsense update. After that you can remove my repo. So far my repo serves the plugin, and the actual caddy binary already comes from the OPNsense Repo now.
"caddy-custom-2.7.6.3.0.3.5.3_XX.pkg" (Check here: https://pkg.opnsense.org/FreeBSD:13:amd64/snapshots/latest/All/)
(I had to remove and reinstall the plugin too, I'm using it on a few firewalls myself)
Sorry that you're having trouble, an integration like this and changing a lot of things is pretty hard without having some weird things happen.
Disregard - uninstall, reboot, reinstall worked.
Thank you!
This is a great plug-in and I appreciate your efforts. Weird things always happen - appreciate the help
The new handlers are working as expected - one with TLS and the other clear. Straightforward and works well.
Awesome. Good to know. ^^
I have problems with "Timeout during connect (likely firewall problem)". Which rules have to be set on Port 80 and 443 to "ThisFirewall"? Is it correct to setup a IN-Rule for the LAN to "ThisFirewall" or should "ThisFirewall" the Source?
I do not receive a certificate and wonder what to do. In your FAQ is no need for additional rules.
Great plugin. Thank you very much for your efforts. It should make a Reverse proxy setup much easier.
Hello.
The rule on WAN/LAN/ other interfaces should be:
Direction: In
TCP/IP Version: IPv4/IPv6
Protocol: TCP
Source: Any
Destination: This Firewall
Destination Port Range: HTTP
A second rule with HTTPS should be made too.
The WAN rule makes sure external clients can connect to your domains, and that Let's Encrypt can issue the certificate.
The same rules on LAN allows your internal clients to connect to the same domains.
If you want to restrict access afterwards while retaining the Lets Encrypt functionality you can use basic auth or access lists (build into the plugin). Dont use Firewall rules for that.
If you dont get a certificate then, check that your FQDN resolves to the external IP Adress of your Firewall (A-Record).
Also, make sure you have the GUI redirect rule disabled, and have the WEB UI listen on an alternate port.
nice plugin... super simple !
Is it possible to defer certs to the opnsense trust store ?
I already have LE generating certs there... and would like to use those, rather than have caddy own the process of creating/renewing the cert ?
------------ answer ------
I can see now, that you can select other cert if you use advanced option for the domain.
Glad you could find the option you need. I think the ACME Plugin and Caddy can run at the same time and issue certificates too, I don't think there are regressions, but I don't know.
Its interesting to use the build in certificate generation of caddy because it also does automatic ocsp stapling.
Also, make sure you create an automation that restarts caddy when the Lets Encrypt certificates are renewed by the ACME Plugin if you continue using it. Otherwise the certs wont be reloaded if theyre reissued.
I'll check if I can create a pull request to add that as automation like nginx and haproxy.
EDIT: https://github.com/opnsense/plugins/pull/3877
Thanks monviech... I gave caddy another try... I currently run HAProxy, but dont really need load balancing for the home network, caddy is simpler.
My results were uneven... thus far. Here is what I did....
- Turned off ddns as relying on opnsense for that
- Gave the domain a custom cert located in the opnsense trust store.
- Gave the domain a custom port of 30000, as haproxy is currently binding to 443 and 80.
- With this approach, caddy does not terminate the connection. Seems to work however if I give it default 443
- Further to this... I disabled haproxy, and enabled caddy
- created a brand new domain and opnsense LE cert.
- bound caddy to 443 and seemed to work ok
- Home assistant loaded fine, the backend is unencrypted
- when backend was encrypted however, I checked the tls box for the backend, but alas failed to certify
- this was the opnsense gui... which I put on a different port (41443)
- Gui failed to load.
- Similar approach seems to work in haproxy... where you check tls but dont bother to certify.
I will try again in a few days... to see if I can work around some of these things...
best regards,
Caddy has port 80 and 443 as a requirement for itself. Running it at the same time as other services that use it is not supported.
When using the build in certificate generation, any port on the Domain works, even ports like 30000 etc. I know that cause a small project uses this plugin where they have the same domain from 30000 to 30050 listening on the front end, reverse proxying each port to a different handler. (Reverse Proxying a lot of stable diffusion instances for the API.)
For the "check a box that just skips TLS verification" there is a new feature for that coming in the next version that allows that.
Otherwise the docs have examples how it works with the OPNsense GUI right now.
Thanks for trying the plugin. ^^
Hi,
I would like to install this plug in but can't find it in the Plugins list under firmware.
Opnsense version 24.1.b_130
Any suggestions?
Thanks!
It's in 24.1.4
Has anyone had issues with Google home devices (not the routers but the hubs or other display devices) losing functionality after implementing caddy directly on the firewall? I previously had nginx proxy manager and currently used zoraxy. Both of which i had a nat rule setup and everything works fine. I'd like to use caddy and have everything setup directly on the firewall but that's a deal breaker. I setup the wan/lan rules and removed the nat as noted in the guide. Everything works great with the exception of it breaking functionality of those devices
I'm using Google Home Minis and also Chromecasts. I didn't experience any breakage in functionality after implementing Caddy on my firewall.
I can't imagine what the problem should be, maybe a configuration problem of the Firewall or NAT rules.
Please check the firewall live log what happens when you voice command your google devices. Check if DNS fails (most probable cause), or packets get blocked.
that's the thing...I won't even load...it appears to be a dns issue. Because it can't contact google, it just hangs on the either the loading screen or it displays the clock but the clock never updates because it can't contact google. My tv's which have google built in work fine, but just an older lenovo google display device craps the bed whenever i've tried.
There's probably not much I can do about that without being able to know what exactly is going wrong, and how to reproduce it.
It's highly unlikely it has to do with running Caddy. Its more likely to be a firewall configuration issue.
I tend to agree. Just wanted to ask in case you'd heard anything like that before. I appreciate it
Hi,
how can i add a wildcard DNS entry with Ionos as DNS provider? The plugin is only creating the "@" entry
Hello,
I am using this module and the configuration examples from it: https://github.com/mholt/caddy-dynamicdns
If there is something my template does wrong, please give me a caddyfile configuration example, or ask in the issues of the plugin maintainer if you can update a full wildcard domain.
So far, the plugin either updates a base domain with @, or subdomains. I dont think *.example.com will be updated, but I dont know for sure since I programmed the template only with the given examples.
I also inquired further here before making the template: https://caddy.community/t/dynamic-dns-module-question-about-domain-configuration/22291
Hi,
i've addad a ddns.global file in /usr/local/etc/caddy/caddy.d with
dynamic_dns {
provider ionos xyz-api-key
domains {
domain.tld * @
}
check_interval 5m
versions ipv4
ttl 1h
}
an with this it's creating the @ and * DNS entry (at least for Ionos).
So a extra field in the config GUI would be nice to fill some extra DNS entrys and just use @ if nothing is enterd there :)
Looks good, thanks for testing. I will adjust the template to turn '@' into '*' if it is a wildcard domain like *.example.com, but leave it as '@' when it is a base domain like example.com.
A simple fix in the template logic should do it I think? Since *.example.com and example.com need to coexist anyway in the GUI when wildcard and base domain are both needed, since they don't include each other.
would be awesome :D waiting for the update.
I'll test it and report back (if i can remember). If i forget to report back then you can assume that it's working ;)
@Aergernis:
https://github.com/opnsense/plugins/pull/3989
Hi! Thank you for this plugin! I just moved from the "other sense" so i'm a newb here....
I tried to configure caddy but it won't even start. It gave me these error:2024-05-23T21:31:23 Informational caddy "info","ts":"2024-05-23T19:31:23Z","logger":"tls","msg":"finished cleaning storage units"}
2024-05-23T21:31:23 Error caddy "error","ts":"2024-05-23T19:31:23Z","logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
2024-05-23T21:31:23 Informational caddy "info","ts":"2024-05-23T19:31:23Z","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x87024b400"}
2024-05-23T21:31:23 Informational caddy "info","ts":"2024-05-23T19:31:23Z","logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
2024-05-23T21:31:23 Informational caddy "info","ts":"2024-05-23T19:31:23Z","logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
2024-05-23T21:31:23 Informational caddy "info","ts":"2024-05-23T19:31:23Z","logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2024-05-23T21:31:23 Informational caddy "info","ts":"2024-05-23T19:31:23Z","logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
2024-05-23T21:31:23 Informational caddy "info","ts":"2024-05-23T19:31:23Z","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x87024b400"}
2024-05-23T21:31:23 Informational caddy "info","ts":"2024-05-23T19:31:23Z","logger":"admin","msg":"admin endpoint started","address":"unix//var/run/caddy/caddy.sock","enforce_origin":false,"origins":["//::1","","//127.0.0.1"]}
2024-05-23T21:30:58 Informational caddy "info","ts":"2024-05-23T19:30:58Z","logger":"tls","msg":"finished cleaning storage units"}
2024-05-23T21:30:58 Error caddy "error","ts":"2024-05-23T19:30:58Z","logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
2024-05-23T21:30:58 Informational caddy "info","ts":"2024-05-23T19:30:58Z","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x870496380"}
If i press play it won't start at all. I already tried to uninstall, reboot and reinstall.
problem seems to be Auto HTTPS "On". If i turn off or any other one it starts automatically
Hey, did you follow the docs? If that happens it sounds like you didnt disable the WebGui redirect rule of the OPNsense. AutoHTTPs enables port 80, if thats already used then Caddy cant bind to it and wont start.
https://docs.opnsense.org/manual/how-tos/caddy.html#prepare-opnsense-for-caddy-after-installation
Quote from: Monviech on May 21, 2024, 04:14:31 PM
@Aergernis:
https://github.com/opnsense/plugins/pull/3989
is this already< included in 1.5.5_1? If so, it's not working. The only difference is that there are now 2 @ entries for the domain
dynamic_dns {
provider ionos *****
domains {
domian.tld @
domian.tld @
}
Oh no its not, you have to use opnsense-patch to include it:
opnsense-patch -c plugins f3532fc9d878e1f8b13dd0b6242f2ee6918b9b72
Its gonna be in 1.5.6.
Docs about opnsense-patch: https://docs.opnsense.org/manual/opnsense_tools.html#id2
Quote from: Monviech on May 25, 2024, 01:44:55 PM
Oh no its not, you have to use opnsense-patch to include it:
opnsense-patch -c plugins f3532fc9d878e1f8b13dd0b6242f2ee6918b9b72
Its gonna be in 1.5.6.
Docs about opnsense-patch: https://docs.opnsense.org/manual/opnsense_tools.html#id2
Perfect. Working now, thanks :D
Awesome, thank you for testing and creating this issue. Your efforts helped to make the plugin better. ^^
Quote from: Monviech on May 24, 2024, 06:14:03 AM
Hey, did you follow the docs? If that happens it sounds like you didnt disable the WebGui redirect rule of the OPNsense. AutoHTTPs enables port 80, if thats already used then Caddy cant bind to it and wont start.
https://docs.opnsense.org/manual/how-tos/caddy.html#prepare-opnsense-for-caddy-after-installation
Sorry for my late reply.
You're right, i didn't disable the webgui redirect :( , my fault, after that the plugin works perfectly!
Thank you again for your work!!
Thanks for the feedback. Im working on adding a validation so it won't happen again to others.
https://github.com/opnsense/plugins/pull/3999
You said, in another thread this :"I just dislike the idea that it will be an easy way out and people will use it for all scenarios where they could use proper certificate handling instead..." regarding the TLS insecure skip verify.
At the moment the only way to make nextcloud and plex work behind caddy is thanks to this check.
If i'd like uncheck this, could be a good way follow this approach ? https://docs.opnsense.org/manual/how-tos/caddy.html#reverse-proxy-the-opnsense-webui
I completely understand if you don't want explaining something that may seem trivial to you, no problem at all ;)
And sorry if i made a dumb question, again... :-[
Yes this approach is the right one.
You have to build trust between Caddy (which is the client) communicating with plex or nextcloud (which is the server)
To establish the connection over TLS, the certificates have to be trusted.
When your PC is the client towards Caddy, they trust each other because your PC has a root certificate for Lets Encrypt installed automatically.
When Caddy is the Client towards a server with a self signed certificate (like Nextcloud), it doesnt have any root certificate stored for that. Thats why it needs to be explicitely added, and the SAN name of the certificate has to match.
Though, if you don't get it to work, there is no shame using the Skip Verify, if your upstream is in your private LAN. Its mostly important if your upstream is in the internet again.
If the upstream is in the same private infrastructure as Caddy or any other reverse proxy a perfectly valid option in my opinion is to drop TLS for the backend connection altogether.
That's how I run Confluence and all that other Java/Tomcat based stuff because managing certificates in Java just sucks.
-///-
So I am a little confused on this - forgive me, new to Caddy and setting up reverse proxies.
First off - having issues with Firewall Rules after implementing this, and I am already weak with them, but no matter what I'm doing with rules, I can't ever seem to get them working with Caddy.
I know in the documentation it says:
Port Forwards, NAT Reflection, Split Horizon DNS or DNS Overrides in Unbound are not required. Only create Firewall rules that allow traffic to the default ports of Caddy.
So are we creating rules that just point to 80 and 443? I tried that, didn't go so well, lol.
Second thing, under Creating the Simple Proxy section - for the Upstream Domain IP: Is this our firewall IP, or our hosted DNS name? The documentation doesn't do a good job for me on this. Currently, I just have my domain name there, but at first believed it would have been my OPN.
I recently updated that section, its not live yet.
https://github.com/opnsense/docs/blob/master/source/manual/how-tos/caddy.rst#creating-a-simple-reverse-proxy
Also, the firewall rules as stated at the beginning of these docs are all that are needed.
Thanks for the updated notes.
So, for the Upstream Domain in your notes: 192.168.10.1
Is this the IP of OPN you are using, or the ISP router IP?
A reverse proxy works like this:
It is a webserver, that is the frontend part. You input your domains there, and it listens for these domain names.
At the same time it is also a client, like a browser. It needs to know where to send traffic to. This is the Upstream part, it will connect to an internal hosted service like a Nextcloud.
It combines both, to intercept all requests, and sending them to a different destination. Its a proxy from the internet to your internal services. (reverse proxy)
e.g.
nextcloud.example.com -> A Record points to external IP address (e.g. 1.1.1.1) of your OPNsenses WAN interface where Caddy listens.
1. Client (Your phone) tries to connect to --> example.com
2. It reaches the Opnsense, on which Caddy listens for the Domain nextcloud.example.com
3. Caddy then dials the Upstream (E.g. the internal IP address (192.168.1.1) of the Nextcloud in your private network behind the OPNsense)
4. The Nextcloud server receives the request from caddy, and sends it back to caddy, and caddy to your phone.
1.1.1.1 192.168.1.1
Phone Browser <-> Caddy <-> Nextcloud
can i bind caddy on specific interface?
i have two wan interface and run nginx proxy on one. i want use caddy for the second, to test , but i see no options to bind caddy on a specific interface.
Hello, there is no implementation for that in the GUI.
https://caddyserver.com/docs/caddyfile/directives/bind#syntax
It uses the standard wildcard interface. And it can only bind to a specific IP address, or hostname.
Since the directive "bind" does only accept "network addresses", including that is a little unreliable on a firewall (especially when the IP can vanish since its dynamic) I expect caddy to just crash then.
I mean potentially it could be there in global options, but I really don't know if I want to offer it. You could always create a custom configuration though in the import folders that includes this directive.
https://caddyserver.com/docs/caddyfile/options#default-bind
You can create these two files to override the interface globally for whole caddy:
/usr/local/etc/caddy/caddy.d/defaultbind.global
default_bind 1.1.1.1
/usr/local/etc/caddy/caddy.d/defaultbind.conf
http:// {
bind 1.1.1.1
}
The http block is explained in the documentation about default_bind.
That will be imported into the Caddyfile and Caddy will bind globally to that IP. Now you can freely use the GUI to test out Caddy.
nice , i will try.
then i have no conflicts with the nginx plugin , what used port 80/443 on the first wan port?
If nginx binds to e.g.:
1.1.1.1:80 and 1.1.1.1:443
And Caddy will bind to
1.1.1.2:80 and 1.1.1.2:443
there will be no socket overlap and no conflicts.
i have it done, on first look, looks good.
but why are run caddy on root user? should it not www?
root caddy 56705 8 tcp4 ip:443
Ah yes, its possible but not easy. I wanted to implement it but I don't know what this kind of stuff will influence on the firewall.
It would have to use an /old/ subsystem in FreeBSD called "mac_portacl".
https://github.com/opnsense/core/issues/7419
https://github.com/opnsense/plugins/issues/3946
Hello,
I have installed Caddy with LE, and have no issues with Opnsense access.
For some reason my Synology NAS worked fine for a while and then I am getting the error message.
This page isn't working nas.domain.com redirected you too many times.
Try deleting your cookies.
ERR_TOO_MANY_REDIRECTS.
Also I am not able. to setup reverse proxy for Portainer and Couchdb installed on docker in Synology NAS. I have used the same settings for all of the them. And only Opnsense is working for me.
Can someone help me to fix the issue and I can share more information if needed to understand the issue.
Thank you in advance.
I think Synology has a forced redirect from port 80/443 to port 5000/5001 that has to be disabled to prevent a redirection loop.
https://forum.opnsense.org/index.php?topic=39942.msg195706#msg195706
I tried changing the redirect for https tp port 4443, but still getting the same error.
This page isn't working synology.domain.com redirected you too many times.
Try deleting your cookies.
ERR_TOO_MANY_REDIRECTS
Because some questions have already arisen here regarding a Nextcloud installation behind an opnsense with reversproxy caddy plugin. A small guide:
1. Follow the Documentation of this great plugin of Monviech [do it exactly as described]
https://docs.opnsense.org/manual/how-tos/caddy.html#how-to-install (https://docs.opnsense.org/manual/how-tos/caddy.html#how-to-install)
- 1. Installation
- 2. Prepare OPNsense for Caddy after installation
2. Create an A-Record with an external DNS Provider that points to the external IP Address of the OPNsense
3. Create a VM/SERVER/LXC/CONTAINER on your favorite hypervisor
- must be accessible from the opnsense via a static ip
- For example 192.168.10.1
4. Create a simple-reverse-proxy for nextcloud
https://docs.opnsense.org/manual/how-tos/caddy.html#creating-a-simple-reverse-proxy (https://docs.opnsense.org/manual/how-tos/caddy.html#creating-a-simple-reverse-proxy)
For example:
[FRONTEND]
Domain: nextcloud.yourdomain.eu
Port: Leave empty to use port 443 with automatic redirection from port 80
Description: nextcloud.yourdomain.eu - frontend
[BACKEND]
Domain: nextcloud.yourdomain.eu
Description: nextcloud.yourdomain.eu - backend
Upstream Domain:192.168.10.1
Upstream Port:
11000 [IMPORTANT - you need to reach the apache web server in the nextcloud instance]
DONT FORGET TO APPLY
5. Run a shell in the VM/SERVER/LXC/CONTAINER and prepare the nextcloud installation
sudo apt update && apt upgrade && apt-get install unattended-upgrades && apt install curl -y
curl -fsSL https://get.docker.com | sudo sh
docker version
mkdir /nextcloud
mkdir /mnt/data6. Create a docker-compose.yml file for the nextcloud container
Quote
nano /nextcloud/docker-compose.yml
[PASTE]
services:
nextcloud-aio-mastercontainer:
image: nextcloud/all-in-one:latest
init: true
restart: always
container_name: nextcloud-aio-mastercontainer
volumes:
- nextcloud_aio_mastercontainer:/mnt/docker-aio-config
- /var/run/docker.sock:/var/run/docker.sock:ro
ports:
- 8080:8080
environment:
AIO_DISABLE_BACKUP_SECTION: false
APACHE_PORT: 11000
APACHE_IP_BINDING: 0.0.0.0
NEXTCLOUD_DATADIR: /mnt/data
NEXTCLOUD_MOUNT: /mnt/
NEXTCLOUD_UPLOAD_LIMIT: 20G
NEXTCLOUD_MAX_TIME: 7200
NEXTCLOUD_MEMORY_LIMIT: 4096M
NEXTCLOUD_TRUSTED_CACERTS_DIR: /path/to/my/cacerts
NEXTCLOUD_STARTUP_APPS: deck twofactor_totp tasks calendar contacts notes
TALK_PORT: 3478
WATCHTOWER_DOCKER_SOCKET_PATH: /var/run/docker.sock
volumes:
nextcloud_aio_mastercontainer:
name: nextcloud_aio_mastercontainer
7. Go to https://192.168.10.1:8080
- Login the AIO installer with pw seed
- Enter nextcloud.yourdomain.eu
- Follow the Nextcloud AIO installer as shown
8. Go to https://nextcloud.yourdomain.eu
- reverse proxy and ssl-cert via caddy plugin
- If you wish restrict access to only internal IPs
- https://scan.nextcloud.com
- https://www.ssllabs.com/ssltest/
For further reading: https://github.com/nextcloud/all-in-one (https://github.com/nextcloud/all-in-one)
Quote from: wirehire on June 04, 2024, 02:57:13 PM
i have it done, on first look, looks good.
but why are run caddy on root user? should it not www?
root caddy 56705 8 tcp4 ip:443
Soon it can be enabled optionally to run as www user, with restriction to upper-ports.
https://github.com/opnsense/plugins/pull/4081
This means, you can run caddy on any upper ports, e.g. 8080 and 8443, and use the port forwarding of PF to bind port 80 and 443, using Port address translation to send them to 8080 and 8443.
Hey, first of all: thank you for this plugin!
I switched from deSEC to Cloudflare and now I'm getting Error's and it stopped working. I configured Cloudflare's API as mentioned in the caddy docs (https://github.com/caddy-dns/cloudflare). It looked like an API problem, but the API is working correctly. Anyone an Idea what I'm missing here? Drives me crazy, because it worked fine through deSEC...
2024-07-11T11:20:20 Debug caddy "debug","ts":"2024-07-11T09:20:20Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/NPCUGkSt_wD2IWe237P8Ug","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["449"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:20 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["RFqgHTYGSgLDQn5zyQus3TlVAA0dV_AgfEfYCSqlEbE"],"Retry-After":["86400"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
2024-07-11T11:20:17 Debug caddy "debug","ts":"2024-07-11T09:20:17Z","logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"vault.domain.xyz","challenge_type":"http-01"}
2024-07-11T11:20:17 Debug caddy "debug","ts":"2024-07-11T09:20:17Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/chall/klxrvi7gfralxj_sO71jWg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["164"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:17 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90/authz/NPCUGkSt_wD2IWe237P8Ug>;rel=\"up\""],"Replay-Nonce":["6E5wD7Bn28fzZFdRIhxY-kHq1V7Pib5lfX1hjTipyjI"],"Retry-After":["60"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
2024-07-11T11:20:15 Debug caddy "debug","ts":"2024-07-11T09:20:15Z","logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"vault.domain.xyz","challenge_type":"http-01"}
2024-07-11T11:20:15 Debug caddy "debug","ts":"2024-07-11T09:20:15Z","logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"vault.domain.xyz","challenge_type":"http-01"}
2024-07-11T11:20:15 Informational caddy "info","ts":"2024-07-11T09:20:15Z","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"vault.domain.xyz","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
2024-07-11T11:20:15 Debug caddy "debug","ts":"2024-07-11T09:20:15Z","logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
2024-07-11T11:20:15 Debug caddy "debug","ts":"2024-07-11T09:20:15Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/NPCUGkSt_wD2IWe237P8Ug","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["449"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:15 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["mc4smWXSmABMoANYjQXXXzYPNBKrPnIS8Mg6fx_0J6Y"],"Retry-After":["86400"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
2024-07-11T11:20:12 Debug caddy "debug","ts":"2024-07-11T09:20:12Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Length":["278"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:12 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/ePdppdFxHGxCGbOdAvumEQ"],"Replay-Nonce":["N0RQNWJzpxpg9jw-t0w10AV1et-TIz90-K-awUNs-2A"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":201}
2024-07-11T11:20:11 Debug caddy "debug","ts":"2024-07-11T09:20:11Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store"],"Content-Type":["application/octet-stream"],"Date":["Thu, 11 Jul 2024 09:20:11 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["dubyvT4QrKuMzUWj3QzTladPHhmL43wwc9PWuhz7z4U"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
2024-07-11T11:20:10 Debug caddy "debug","ts":"2024-07-11T09:20:10Z","logger":"tls.issuance.acme.acme_client","msg":"creating order","account":"https://acme.zerossl.com/v2/DV90/account/zI17MJiuzZy2KCCeoxuxow","identifiers":["vault.domain.xyz"]}
2024-07-11T11:20:10 Debug caddy "debug","ts":"2024-07-11T09:20:10Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme.zerossl.com/v2/DV90","headers":{"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Content-Length":["645"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:10 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15724800; includeSubDomains"]},"status_code":200}
2024-07-11T11:20:10 Informational caddy "info","ts":"2024-07-11T09:20:10Z","logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme.zerossl.com/v2/DV90/account/zI17MJiuzZy2KCCeoxuxow","account_contact":["mailto:abc@abc.de"]}
2024-07-11T11:20:10 Informational caddy "info","ts":"2024-07-11T09:20:10Z","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["vault.domain.xyz"],"ca":"https://acme.zerossl.com/v2/DV90","account":"abc@abc.de"}
2024-07-11T11:20:10 Informational caddy "info","ts":"2024-07-11T09:20:10Z","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["vault.domain.xyz"],"ca":"https://acme.zerossl.com/v2/DV90","account":"abc@abc.de"}
2024-07-11T11:20:10 Debug caddy "debug","ts":"2024-07-11T09:20:10Z","logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
2024-07-11T11:20:10 Error caddy "error","ts":"2024-07-11T09:20:10Z","logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"vault.domain.xyz","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge"}
2024-07-11T11:20:10 Error caddy "error","ts":"2024-07-11T09:20:10Z","logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"vault.domain.xyz","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1830756237/286284483307","attempt":2,"max_attempts":3}
2024-07-11T11:20:10 Error caddy "error","ts":"2024-07-11T09:20:10Z","logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"vault.domain.xyz","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]}}
2024-07-11T11:20:10 Debug caddy "debug","ts":"2024-07-11T09:20:10Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540185137","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["979"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:10 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["zgT1AlV-e62lhajWnK7NFktNLv_vxDvmyIHriusfwdEbgwDHJBI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:10 Debug caddy "debug","ts":"2024-07-11T09:20:10Z","logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"vault.domain.xyz","challenge_type":"tls-alpn-01"}
2024-07-11T11:20:10 Debug caddy "debug","ts":"2024-07-11T09:20:10Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/375540185137/6dCOjw","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["191"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:10 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540185137>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/375540185137/6dCOjw"],"Replay-Nonce":["zgT1AlV-ZNjqD20ClwybZ8eDxXrUqmgEm1TZobqpM9QB218Jj-w"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:09 Error caddy "debug","ts":"2024-07-11T09:20:09Z","logger":"http.stdlib","msg":"http: TLS handshake error from 127.0.0.1:23624: EOF"}
2024-07-11T11:20:09 Debug caddy "debug","ts":"2024-07-11T09:20:09Z","logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"vault.domain.xyz","challenge_type":"tls-alpn-01"}
2024-07-11T11:20:09 Debug caddy "debug","ts":"2024-07-11T09:20:09Z","logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"vault.domain.xyz","challenge_type":"tls-alpn-01"}
2024-07-11T11:20:09 Informational caddy "info","ts":"2024-07-11T09:20:09Z","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"vault.domain.xyz","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
2024-07-11T11:20:09 Debug caddy "debug","ts":"2024-07-11T09:20:09Z","logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
2024-07-11T11:20:09 Debug caddy "debug","ts":"2024-07-11T09:20:09Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540185137","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:09 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["eFSVlf8U4xWcZeKJWc2ZPIjm64cHusyQhzX7_35gDM3LNRN3kJE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:09 Debug caddy "debug","ts":"2024-07-11T09:20:09Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["342"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:09 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1830756237/286284483307"],"Replay-Nonce":["zgT1AlV-BI6xmFywkTmeWStBmnzrHpVzGovRTMo3kXra6Adr0ak"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
2024-07-11T11:20:09 Debug caddy "debug","ts":"2024-07-11T09:20:09Z","logger":"tls.issuance.acme.acme_client","msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/1830756237","identifiers":["vault.domain.xyz"]}
2024-07-11T11:20:08 Error caddy "error","ts":"2024-07-11T09:20:08Z","logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"vault.domain.xyz","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2606:4700:3036::ac43:d41a: Invalid response from http://vault.domain.xyz/.well-known/acme-challenge/7l0JbJU_ZyHGmhvKl75evbljzzBgdrwE6H7OWNDGReo: 521","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1830756237/286284439187","attempt":1,"max_attempts":3}
2024-07-11T11:20:08 Error caddy "error","ts":"2024-07-11T09:20:08Z","logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"vault.domain.xyz","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2606:4700:3036::ac43:d41a: Invalid response from http://vault.domain.xyz/.well-known/acme-challenge/7l0JbJU_ZyHGmhvKl75evbljzzBgdrwE6H7OWNDGReo: 521","instance":"","subproblems":[]}}
2024-07-11T11:20:08 Debug caddy "debug","ts":"2024-07-11T09:20:08Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1166"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:08 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["eFSVlf8U7eKR1AlXtU9lRLDgHnVy6EN2gqEZVm9KRytegcxucrg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:08 Debug caddy "debug","ts":"2024-07-11T09:20:08Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:08 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["zgT1AlV-euTwn1vhs0TH98VKYpMMKLgHg3M6TgZdhDenSu9HmYg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:07 Debug caddy "debug","ts":"2024-07-11T09:20:07Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:07 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["eFSVlf8UYhJhC44nClV9RV6nvIIzaK2DAjImU5JGJdz44gWUncU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:07 Debug caddy "debug","ts":"2024-07-11T09:20:07Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:07 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["eFSVlf8USQKy9NDO6JcHhf_MBteZI361LL4GYaDlA5z_9xyAuFM"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:06 Debug caddy "debug","ts":"2024-07-11T09:20:06Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:06 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["eFSVlf8UxmzVHyJhwPe9eEQ6tvQV01TrVht81-oIAqP7mhai5OM"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:06 Debug caddy "debug","ts":"2024-07-11T09:20:06Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:06 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["zgT1AlV-OJT-LyggcLxiAMMSFuyoLzmCBEOLBXPQxBeybPGmESo"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:06 Debug caddy "debug","ts":"2024-07-11T09:20:06Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:05 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["eFSVlf8U6obW-QWTqVmyaxPKEVACLYUXFXFxCHYe-O4PAFHjYkY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:05 Debug caddy "debug","ts":"2024-07-11T09:20:05Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:05 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["eFSVlf8USJMpZQkltYIVjT5-CjgsbvQkBKO0KvsY626muUBfs6s"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:05 Informational caddy "info","ts":"2024-07-11T09:20:05Z","logger":"dynamic_dns","msg":"finished updating DNS","current_ips":["IPv4","IPv6"]}
2024-07-11T11:20:05 Error caddy "error","ts":"2024-07-11T09:20:05Z","logger":"dynamic_dns","msg":"failed setting DNS record(s) with new IP address(es)","zone":"vault.domain.xyz","error":"expected 1 zone, got 0 for vault.domain.xyz"}
2024-07-11T11:20:05 Debug caddy "debug","ts":"2024-07-11T09:20:05Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:05 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["zgT1AlV-CvYAXwpXuF1shAoALOl7uw1a2tfIsDvEFPZQXx47cdY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:04 Debug caddy "debug","ts":"2024-07-11T09:20:04Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:04 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["eFSVlf8UY_XLOY2Bx8vljY4TUUxGIWBp6I1TPViNFndKjcC-4Ck"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:04 Debug caddy "debug","ts":"2024-07-11T09:20:04Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:04 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["zgT1AlV-LYsOEdE5lzwAGFIFauFJxas65qd56rV33pJfPFQIves"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:03 Debug caddy "debug","ts":"2024-07-11T09:20:03Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:03 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["zgT1AlV-xLtdkY8POAtrKQ9a4t3v_xHSQNG3UhG6BnZnfADaK1M"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:03 Informational caddy "info","ts":"2024-07-11T09:20:03Z","logger":"dynamic_dns","msg":"updating DNS record","zone":"vault.domain.xyz","type":"AAAA","name":"@","value":"IPv6","ttl":0}
2024-07-11T11:20:03 Informational caddy "info","ts":"2024-07-11T09:20:03Z","logger":"dynamic_dns","msg":"updating DNS record","zone":"vault.domain.xyz","type":"A","name":"@","value":"IPv4","ttl":0}
2024-07-11T11:20:03 Debug caddy "debug","ts":"2024-07-11T09:20:03Z","logger":"dynamic_dns.ip_sources.simple_http","msg":"lookup","type":"IPv6","endpoint":"https://api64.ipify.org","ip":"IPv6"}
2024-07-11T11:20:03 Debug caddy "debug","ts":"2024-07-11T09:20:03Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:03 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["eFSVlf8Ush8AvtRobCEsVGwqCUm2GexI1Hp2bWusYFHkTRgjsuw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:03 Debug caddy "debug","ts":"2024-07-11T09:20:03Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:02 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["zgT1AlV-8ayFaFcJYlewB8vBEfzHuzQBzzf2uB0mGWAAgIX_qRY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:03 Debug caddy "debug","ts":"2024-07-11T09:20:03Z","logger":"dynamic_dns.ip_sources.simple_http","msg":"lookup","type":"IPv4","endpoint":"https://api64.ipify.org","ip":"IPv4"}
2024-07-11T11:20:02 Debug caddy "debug","ts":"2024-07-11T09:20:02Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:02 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["zgT1AlV-iPhTWKwNrOl_pJclv7FEUfeIEqzZ5H_5hZ7OHWz4RbI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:02 Debug caddy "debug","ts":"2024-07-11T09:20:02Z","logger":"dynamic_dns","msg":"looked up current IPs from DNS","lastIPs":null}
2024-07-11T11:20:02 Error caddy "error","ts":"2024-07-11T09:20:02Z","logger":"dynamic_dns","msg":"unable to lookup current IPs from DNS records","error":"expected 1 zone, got 0 for vault.domain.xyz"}
2024-07-11T11:20:02 Debug caddy "debug","ts":"2024-07-11T09:20:02Z","logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"vault.domain.xyz","challenge_type":"http-01"}
2024-07-11T11:20:02 Debug caddy "debug","ts":"2024-07-11T09:20:02Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/375540126317/1Lewag","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["187"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:02 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/375540126317/1Lewag"],"Replay-Nonce":["eFSVlf8UyZXdEZd8mwkJB6yFDyoZ-wVUZbRYH-OsfEukg-tTT9I"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:02 Debug caddy "debug","ts":"2024-07-11T09:20:02Z","logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"vault.domain.xyz","challenge_type":"http-01"}
2024-07-11T11:20:02 Debug caddy "debug","ts":"2024-07-11T09:20:02Z","logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"vault.domain.xyz","challenge_type":"http-01"}
2024-07-11T11:20:02 Informational caddy "info","ts":"2024-07-11T09:20:02Z","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"vault.domain.xyz","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
2024-07-11T11:20:02 Debug caddy "debug","ts":"2024-07-11T09:20:02Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/375540126317","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["800"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:01 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["eFSVlf8UnTzZAxtF3xpZJCTUeU2Ps5MRqmWxZClrlIGYYwpLiyI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:01 Debug caddy "debug","ts":"2024-07-11T09:20:01Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Boulder-Requester":["1830756237"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["342"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:01 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1830756237/286284439187"],"Replay-Nonce":["eFSVlf8UgxC3Ov5ci2luZFH8tZxr_XJq2m-T3zKw4ZccQBsd0PI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
2024-07-11T11:20:01 Debug caddy "debug","ts":"2024-07-11T09:20:01Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Thu, 11 Jul 2024 09:20:01 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["eFSVlf8UldoMJHBdgupStVclatMJ6jwCSZ6H_08oajcJghaPbxY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:01 Debug caddy "debug","ts":"2024-07-11T09:20:01Z","logger":"tls.issuance.acme.acme_client","msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/1830756237","identifiers":["vault.domain.xyz"]}
2024-07-11T11:20:01 Debug caddy "debug","ts":"2024-07-11T09:20:01Z","logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.8.4 CertMagic acmez (freebsd; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["746"],"Content-Type":["application/json"],"Date":["Thu, 11 Jul 2024 09:20:01 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/1830756237","account_contact":["mailto:abc@abc.de"]}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["vault.domain.xyz"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"abc@abc.de"}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["vault.domain.xyz"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"abc@abc.de"}
2024-07-11T11:20:00 Debug caddy "debug","ts":"2024-07-11T09:20:00Z","logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
2024-07-11T11:20:00 Debug caddy "debug","ts":"2024-07-11T09:20:00Z","logger":"events","msg":"event","name":"cert_obtaining","id":"a3ed979c-3d86-499c-92ad-5ae7deba6b1e","origin":"tls","data":{"identifier":"vault.domain.xyz"}}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"tls.obtain","msg":"obtaining certificate","identifier":"vault.domain.xyz"}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"tls.obtain","msg":"lock acquired","identifier":"vault.domain.xyz"}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"tls","msg":"finished cleaning storage units"}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/db/caddy/data/caddy","instance":"68f4b45b-f584-42d9-bafa-99c122f2bda3","try_again":"2024-07-12T09:20:00Z","try_again_in":86399.999997449}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"tls.obtain","msg":"acquiring lock","identifier":"vault.domain.xyz"}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","msg":"serving initial configuration"}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","msg":"autosaved config (load with --resume flag)","file":"/var/db/caddy/config/caddy/autosave.json"}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"http","msg":"enabling automatic TLS certificate management","domains":["vault.domain.xyz"]}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
2024-07-11T11:20:00 Debug caddy "debug","ts":"2024-07-11T09:20:00Z","logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
2024-07-11T11:20:00 Debug caddy "debug","ts":"2024-07-11T09:20:00Z","logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
2024-07-11T11:20:00 Debug caddy "debug","ts":"2024-07-11T09:20:00Z","logger":"dynamic_dns","msg":"beginning IP address check"}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
2024-07-11T11:20:00 Debug caddy "debug","ts":"2024-07-11T09:20:00Z","logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["vault.domain.xyz"]},{}]}},"http":{"grace_period":10000000000,"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.178.6:8000"}]}]}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x870981180"}
2024-07-11T11:20:00 Informational caddy "info","ts":"2024-07-11T09:20:00Z","logger":"admin","msg":"admin endpoint started","address":"unix//var/run/caddy/caddy.sock","enforce_origin":false,"origins":["","//127.0.0.1","//::1"]}
Caddy-Config:
# Global Options
{
log {
output net unixgram//var/caddy/var/run/log {
}
format json {
time_format rfc3339
}
level DEBUG
}
dynamic_dns {
provider cloudflare API-Token
domains {
vault.domain.xyz @
}
}
email abc@abc.com
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}
# Reverse Proxy Configuration
# Reverse Proxy Domain: "8e333c2b-cff5-465f-b899-d89f446438c5"
vault.domain.xyz {
handle {
reverse_proxy 192.168.178.6:8000 {
}
}
}
import /usr/local/etc/caddy/caddy.d/*.conf
Thanks in advance, hope its not a totally dumb mistake ::) ;D
Hello, I don't see the TLS block in your domain. It should look like this:
# Reverse Proxy Domain: "d5c1169f-8f95-4091-823b-7095c15fbb5f"
example.com {
tls {
dns cloudflare secretapikeyhere
}
handle {
reverse_proxy 172.16.100.1 {
}
}
}
Did you check the Checkbox "Dns-01 Challenge" in your domain in "Trust"?
Puh, thanks alot! :-[ - got a certificate now:
2024-07-11T22:11:37 Informational caddy "info","ts":"2024-07-11T20:11:37Z","logger":"dynamic_dns","msg":"updating DNS record","zone":"vault.domain.xyz","type":"AAAA","name":"@","value":"IPv6","ttl":0}
2024-07-11T22:11:37 Informational caddy "info","ts":"2024-07-11T20:11:37Z","logger":"dynamic_dns","msg":"updating DNS record","zone":"vault.domain.xyz","type":"A","name":"@","value":"IPv4","ttl":0}
2024-07-11T22:11:35 Debug caddy "debug","ts":"2024-07-11T20:11:35Z","logger":"events","msg":"event","name":"cached_managed_cert","id":"30f5dd13-a0ea-4f72-8ab9-ef83302c2b13","origin":"tls","data":{"sans":["vault.domain.xyz"]}}
2024-07-11T22:11:35 Debug caddy "debug","ts":"2024-07-11T20:11:35Z","logger":"tls.cache","msg":"added certificate to cache","subjects":["vault.domain.xyz"],"expiration":"2024-10-09T09:55:23Z","managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"51760c73851d473ec28884675ecde4e5413d434e12f04760093aecb819909f51","cache_size":1,"cache_capacity":10000}
I already thought it was such a small thing...
Now of course I have the next problem, namely that I can't reach the domain and get an error at Cloudflare (error code 521). Host is reachable, CNAME is also configured. Is there anything else I need to consider for Cloudflare?
If Cloudflare is only your DNS Proviser and nothing more (no CDN or Cloudflare tunnels etc), then nothing else has to be considered there.
Now the issue should be your upstream. If you get a blank page + certificate in the browser, then there is a connection issue to the upstream (so your internal service+port).
The most likely cause is that the internal service listens on https instead of http, so try to enable "TLS Insecure Skip Verify" in the handler and see if it works then.
You are the man! Thank you so much for the great support! Everything is now working as it should. :D :D :D
Hello,
may i ask for your help @Monviech. i would like to install stalwart mailserver behind the caddy plugin. according to the documentation https://stalw.art/docs/server/reverse-proxy/caddy/ (https://stalw.art/docs/server/reverse-proxy/caddy/), the following must be done to enable Proxy Protocol support directly within Caddy.
It is mentioned that the plugin called proxy_protocol should be used. As I understand it this is already integrated in Caddy.
Caddyfile example
mail.example.com {
redir https://example.com{uri}
}
example.com {
# Set this path to your site's directory.
root * /usr/share/caddy
# Enable the static file server.
file_server
}
mail.example.com {
reverse_proxy 127.0.0.1:8080
}
In addition, crontab must be created in order to automate copying the certificates obtained by Caddy
0 3 * * * cat /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/example.com/example.com.crt > /opt/stalwart-mail/cert/example.com.pem
0 3 * * * cat /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/example.com/example.com.key > /opt/stalwart-mail/cert/example.com.priv.pem
My questions:
1. can this be implemented with the plugin and the GUI or do I have to use Custom Configuration File
2. How do I create the automated copying oft the certificates obtained by Caddy to the stalwart container
Thank you very much for your help and for your great plugin. It has helped me on many levels and also given me a lot of insight.
Hello, anything could be implemented into the plugin. The question is if it makes sense. For edge cases like these the custom configuration files are the best choice.
I don't understand why they need a fileserver and a root web directory. It's not needed for the reverse proxy. If this is a requirement, please /don't/ set it up on the OPNsense.
Also this cronjob can break since Caddy creates more than just Lets Encrypt folder, it also creates a ZeroSSL folder.
Hi,
would love to see MultiWAN support for dynamic DNS so that all public IPs get a DNS entry (in my case 2)
If you leave the checks empty, Caddy will check your current IP with a default HTTP check, and update your A and AAAA Records accordingly. If the default route of the OPNsense to the internet should change, Caddy will update your entries with the other WAN IP at the next check interval.
Great plugin however when i set it up and try to navigate to the domain i've configured (as per tutorial )
subdomain.domain.com uses an unsupported protocol.
ERR_SSL_VERSION_OR_CIPHER_MISMATCH
I'm using the dns challenge to generate the certificate, any ideas why this would cause a problem?
Hey, to help you I need your Caddyfile (please redact any api keys or passwords).
And I need some debug logs. Put your log settings to debug, do the request that fails and post them here.
Thanks, I've attached the caddyfile and log and the letsencrypt challenge works successfully so it has been able to request a certificate, its when i navigate to the page I get that error in my browser
Which application do you reverse proxy? Does it listen internally on HTTP or HTTPS?
For HTTPS enable "TLS Insecure Skip Verify" in the handler.
Or better, use HTTP internally.
Sadly the log doesnt show anything.
i tried both sonarr and portainer (one with and without ssl) and neither are working.
its worth mentioning I only forwarded the internal 80/443 on my LAN rather than exposing on the WAN - I assume this is fine if I don't want any of my servers to be public facing.
any other idea's on what to test? my opnsense is more or less out of the box, I'm not doing anything particularly interesting with my configuration outside of this.
I have no experience with the configuration you are doing.
I am always using the public domain name pointing to the public IP address of the OPNsense, and use Access Lists to restrict access to internal networks.
Reference: https://docs.opnsense.org/manual/how-tos/caddy.html#restrict-access-to-internal-ips
Another reference where I explained my preferred setup in detail: https://www.reddit.com/r/opnsense/comments/1dwbr88/issue_using_oscaddy_to_generate_wildcard_cert/
In your case, you might have to use Split DNS, to avoid NAT Reflection/Hairpinning problems. (https://docs.opnsense.org/manual/how-tos/nat_reflection.html)
Just FYI for all os-caddy users.
Version os-caddy-1.6.1 which will be part of 24.7 will be the point the plugin stays feature wise for now. New feature requests will be weighted heavily for benefit vs. making the UI more cluttered due to being an edge case.
It can do a lot of things, it can fit many usecases, and it is still pretty simple to configure. I think this is the right point to stop active developement and go into full maintaining mode.
That means:
- Fixing Bugs
- Maintaining the code base
- Maintaining/Updating the build of the caddy binary and dns providers
The point where active developement would start again is when new features are introduced, like Layer 4 proxy support in the Caddyfile, which could happen somewhere next year.
Thank you all ^^
I made a video to show off the new widgets, and how fast a reverse proxy is set up with this plugin.
https://www.youtube.com/watch?v=6ip8Sx4zcDA
Out of nowhere, the Caddyfile Layer 4 support came.
I have implemented TLS SNI matching of hostnames, and streaming this traffic to an upstream without tls termination:
https://github.com/opnsense/plugins/pull/4112
If anybody wants to test it, I would love some feedback, since it is a complicated new feature (still tried to keep the scope small and make it very easy to configure)
https://cloud.pischem.com/index.php/s/rw8Z86AzkEx3673
- Install the .pkg and replace the caddy binary with the new one
- Go to "General Settings" - Enable the advanced options - Enable "Enable Layer4"
- Go to the "Layer4 Routes" Tab, and create a route for a domain. All matched TLS traffic will then be sent to an upstream without being terminated.
- If you don't want the Layer4 Support anymore, just deactivate the option and it will be gone completely.
At the same time, the normal Reverse Proxy continues to work. Since this is all based on SNI, both the Layer4 Routes and the HTTP Reverse Proxy work on the same ports, giving maximum flexibility how your traffic is handled.
EDIT:
Already rewriting the feature slightly. I want to allow multiple domains per matcher, and also a not matcher.
EDIT2:
New version is uploaded that can do more stuff. Read the PR for examples.
It worked once yesterday, then the DHCP went haywire and I had to reset the config. Now a day later, updated caddy to 1.6.0 and re-entered the exact same information, it doesn't want to anymore.
The goal was to make my Unify controller reachable on controller.example.com within my LAN. So it is not accessable from outside. I am on IONOS, so I have set up the following accordingly:
General Settings
- ACME Email: my address from IONOS
- DNS Provider: IONOS
- DNS API Standard Field: my API Key
Reverse Proxy
Domains
- example.com
- DNS: Dynamic DNS [ x ]
This means that I can at least see the current WAN IP in my IONOS account, as my WAN IP is dynamic. Wonderful then. I create a wildcard domain:
Domains
- *.example.com
- DNS-01 Challenge [ x ]
Subdomains
- controller.example.com
- Domain: *.example.com
- Subdomain: controller.example.com
- Access List: private_ipv4
Handlers
- controller.example.com
- Domain: *.example.com
- Subdomain: controller.example.com
- Upstream domain 192.168.10.10
- Upstream Port: 8443
I don't see any errors in the Caddy log and on the other hand, my browser just says "Page could not be found".
Shouldnt dynamic DNS be activated on the subdomain too? Otherwise it won't update "controller.example.com" at ionos.
Please post your Caddyfile for further analysis.
I tried that now, but it won't work. My browser still complains, that the site could not be found. However, at IONOS, an entry for the subdomain was added. However, I set this subdomain in caddy, to be only accessible from LAN 192.168.0.0/16. I have the feeling, that caddy won't update the local side..
Just to inform you, I'm going to sleep now, but I'll be back tomorrow.
I have checked your Caddyfile and it looks fine.
The only thing I can see is that "Dynamic DNS" is activated for the wildcard domain "*.example.com" too. That means, all subdomains under the wildcard domain are matched automatically by the dynamic DNS entry for the wildcard domain.
Otherwise, I don't see any mistakes here, so I am at a loss without some logs. If you don't see any logs, it must mean that Caddy does not receive anything.
Check the HTTP Access logs for your domains in the filesystem if your requests get through to Caddy.
After some time had passed and it still wasn't working, I decided to take a very radical step. I reinstalled OPNsense completely using a previous config that I created before I started to use Caddy. Then I followed the same procedure as before. So one domain for example.com and one for *.example.com. I have activated DynDNS for both. Even if I wasn't sure - especially after your statement for the wildcard. However, I absolutely wanted to avoid a limitation by LetsEncrypt. It then successfully created a certificate for both domains. I then created a subdomain unify.example.com. I specified the wildcard domain as the domain. I wonder under what circumstances I would select example.com as the domain if I were to create a subdomain.
I have given the handler 192.168.10.10 with port 8443. TLS and Skip Verify are checked.
The only explanation I can come up with is that I was either limited and that's why it didn't work, or that something was wrong with the configuration of Caddy.
By the way. OPNsense with the widgets is a real cool and helpful bonus.
Im happy it works again, and you like the widgets. Maybe something went wrong somewhere.
Also, you /only/ need to add "example.com" if you want to route traffic directed /exactly/ to that domain.
For example you have:
example.com - Your website is hosted there.
cloud.example.com - Your nextcloud is there.
*.example.com would /not/ match example.com.
So if you don't have any use for example.com cause nothing matches that domain exactly, you can remove or deactivate it.
Thank you. Is it correct, that if I only want to have nextcloud.example.com available internally, that I still need to activate DynDNS?
If you have activated the DNS-01 challenge for a domain, Caddy does not necessarily have to be opened to the outside and does not really need to have Dynamic DNS activated.
You could just as well work with an internal DNS Server (Split DNS Zone) that points "nextcloud.example.com" to an internal IP address like "192.168.1.1".
How you set it all up depends on your usecase. It's very flexible, but when you stray too far from the documentation I wrote, things might get harder to set up and maintain.
Are there any more information to NAT reflection, related to Caddy? I found some forum posts, issues from users and blog entries related to NGINX. However, it would be nice to have it on a similar easy level, as it is on the Caddy OPNsense documentation.
In addition to that, I would be interested in any advantages or disadvantages to a override in UBound..
You do not have any NAT when using Caddy in any scenario.
Caddy listens on "any" interface.
That means, if it receives a request from the internet, e.g. from 1.1.1.1, that targets the external IP of the OPNsense, e.g. 2.2.2.2 (DNS record of nextcloud.example.com), the request will be taken by Caddy, and then proxied to an internal IP, e.g. 192.168.1.2 (actual IP of nextcloud server). No NAT here.
Same happens when Caddy receives something from an internal IP address. E.g., there is an Unbound DNS override that points the nextcloud.example.com to an interface IP address of the OPNsense, e.g. 192.168.1.1. When one of your internal clients, e.g. 192.168.1.10 sends a request to nextcloud.example.com, it will hit 192.168.1.1 where Caddy listens. Cady proxies that request to 192.168.1.2 (where the actual nextcloud server listens.) Also no NAT.
I think you got me wrong. Similar to the documentation "Prepare OPNsense for Caddy After Installation" https://docs.opnsense.org/manual/how-tos/caddy.html#id4, I would like to know, how to setup NAT reflection or a Domain override in OPNsense.
One side note, am I correct that if I use Caddy for a subdomain that is actually only accessible from LAN (set access to LAN), I at least get the benefit of a valid certificate?
You understood me wrong. Why do you want NAT Reflection if there is no NAT to begin with?
I explained that in my last post.
If you meant you search for tutorials because you want to NAT other services, here you go: https://docs.opnsense.org/manual/how-tos/nat_reflection.html
Hi, i'm trying to switch from a docker install of NGINX Proxy Manager behind the opnsense firewall to this caddy plugin right on the firewall.
I got a couple of internal address i use that can be accessed only on the local network.
There's a wildcard setup with a proper certificate ( *local.domain.tld ) and it works on NGINX.
After following the guide at the first page, it does not seem to work on caddy.
I got 2 domains : 1 with the main domain.tld, one with the wildcard.
Under subdomains, i've setup the first one there under the wildcard *.local.domain.tld
I got 2 handlers, 1 with the main domain link to localhost under upstream, 1 with the wildcard+subdomain linked to a local lan address with a specified port.
The main domain link works and gets a proper page/certificate
The subdomain is unreachable/can't connect
What am i missing?!
Hello, please share your Caddyfile and point to the spot that does not work. Otherwise it will be hard to help you. You can find the Caddyfile in "Diagnostics". Make sure you omit sensitive information (like API keys etc...)
There you go!
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file
# caddy_user=root
# Global Options
{
log {
include http.log.access.6a100fb9-863d-4a8e-a6dc-6aaad5598184
output net unixgram//var/run/caddy/log.sock {
}
format json {
time_format rfc3339
}
}
email *email*
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}
# Reverse Proxy Configuration
# Reverse Proxy Domain: "ef64738b-136e-4bba-b267-f6307990db7b"
domain.tld {
tls /var/db/caddy/data/caddy/certificates/temp/66a3c470808dc.pem /var/db/caddy/data/caddy/certificates/temp/66a3c470808dc.key
handle {
reverse_proxy 127.0.0.1 {
transport http {
tls
}
}
}
abort
}
[u][b]
This section does not work[/b][/u]
# Reverse Proxy Domain: "6a100fb9-863d-4a8e-a6dc-6aaad5598184"
*.local.domain.tld {
log 6a100fb9-863d-4a8e-a6dc-6aaad5598184
tls /var/db/caddy/data/caddy/certificates/temp/66a26c37d9228.pem /var/db/caddy/data/caddy/certificates/temp/66a26c37d9228.key
@febd140e-6307-4080-8419-d1de0c6a23b2 {
host *.local.domain.tld
}
handle @febd140e-6307-4080-8419-d1de0c6a23b2 {
handle {
reverse_proxy *local 192 address*:port {
transport http {
tls
tls_server_name *.local.domain.tld
}
}
}
abort
}
@dbd15585-f172-4fbd-8524-13d6dcd351af {
client_ip local 192 address
}
handle @dbd15585-f172-4fbd-8524-13d6dcd351af {
}
abort
}
*.local.domain.tld {
log 6a100fb9-863d-4a8e-a6dc-6aaad5598184
tls /var/db/caddy/data/caddy/certificates/temp/66a26c37d9228.pem /var/db/caddy/data/caddy/certificates/temp/66a26c37d9228.key
@febd140e-6307-4080-8419-d1de0c6a23b2 {
host *.local.domain.tld
}
This does not make sense. You defined a wildcard domain, and then put the same wildcard domain as its subdomain?
If "*.local.domain.tld" is set up, it should then be something like "sub1.local.domain.tld" that matches below it.
It is explained here in the docs.
https://docs.opnsense.org/manual/how-tos/caddy.html#wildcard-domain-with-subdomains
If your intention was to send all subdomains to that target, regardless of what they are, put the handler directly on the wildcard domain, without choosing any subdomain for it.
'ive redacted wrongly sorry :
*.local.domain.tld {
log 6a100fb9-863d-4a8e-a6dc-6aaad5598184
@febd140e-6307-4080-8419-d1de0c6a23b2 {
host sub1.local.domain.tld
The Caddyfile does validate under diagnostics, but still won't proxy to the local server.
You have enabled TLS, does that mean your internal service has a globally trusted certificate? Because if not, you need to make sure Caddy trusts the certificate.
Check this out, it explains it: https://docs.opnsense.org/manual/how-tos/caddy.html#reverse-proxy-the-opnsense-webgui
Otherwise, disable both TLS options you have set, and enable "TLS Insecure Skip Verify", it will skip certifocate handling and the internal HTTPS connection will "just work".
Hey Monviech, I'm sorry to have to ask you again. I think I've understood it now and the tutorials I've found give me a better feeling about going in the right direction. However, what I still don't understand is that the examples for a DNS override assume that Caddy is on a system outside the firewall/OPNsense. I'm alluding to the fact that you specify an IP for an override.... What kind of IP do I specify if the respective gateway, i.e. Caddy, is to be addressed from the respective subnet? I have several VLANS for which I would like to set the override. Presumably there is no such thing as ThisFirewall for this...
It does not matter for which of the interface IP addresses you set the DNS Override. In each of the VLANs, the OPNsense is the default gateway. It receives all packets by default.
That means, regardless of which IP you take for your single override, as long as it points to an interface the OPNsense has, the traffic will reach Caddy.
Just make sure you have a Firewall rule on each VLAN that allows HTTP and HTTPS to "This Firewall" and each VLAN uses Unbound as their DNS server.
Oh. I assumed that I have to redirect e.g. nextcloud.example.com to 192.168.1.1 LAN GW so that it arrives at the firewall and thus at Caddy and Caddy forwards it to the Nextcloud 192.168.10.10.. So I just have to directly name the destination on which the Nextcloud is running? For example, 192.168.10.10? That's crazy. Just crazy!
Just skip DNS overrides altogether and use the external address of your firewall even from internal networks. Caddy is layer 7 proxying, so neither NAT nor NAT reflection is necessary.
Quote from: Baender on July 26, 2024, 10:29:25 PM
Oh. I assumed that I have to redirect e.g. nextcloud.example.com to 192.168.1.1 LAN GW so that it arrives at the firewall and thus at Caddy and Caddy forwards it to the Nextcloud 192.168.10.10.
That assumption is right, I also explained that.
But I especially didn't write dns overrides into the Caddy docs since its just kinda wonky in practice. Patrick is right here.
I guess I need more knowledge about these layers.. I just ask, because I have noticed, that every call of my subdomain, from my local network, has the destination of my WAN address. So I assumed, that the request for an internal service is send to the Unbound DNS, then to IONOS and then back to the FW/Caddy.. That's the story behind my questions..
No, it is not sent across the internet. Read what I wrote again earlier. Since the OPNsense also has your WAN address, Caddy listens on there and will just bounce it back into your network.
Also its no shame to ask, networking is pretty complex, so you should definitely read some more and try stuff out. ;)
Quote from: Monviech on July 26, 2024, 10:06:16 PM
You have enabled TLS, does that mean your internal service has a globally trusted certificate? Because if not, you need to make sure Caddy trusts the certificate.
Check this out, it explains it: https://docs.opnsense.org/manual/how-tos/caddy.html#reverse-proxy-the-opnsense-webgui
Otherwise, disable both TLS options you have set, and enable "TLS Insecure Skip Verify", it will skip certifocate handling and the internal HTTPS connection will "just work".
Disabling evrything TLS made no change.
Sorry I don't know then. Without some debug logs its uncertain what happens there. I need some info that is not anynomized so theres no mistakes due to wrong omissions.
- Check your DNS, does "nslookup yoursubdomainname" really resolve to the IP address of the OPNsense?
- If Yes, Whats the output of "curl -v subdomainname"
- What do the debug logs show when you try to reach it?
- Which kind of application is listening there? Is it a HTTP or HTTPS application.
- If the application demands a HTTPS connection, did you enable "TLS Insecure Skip Verify" like I asked?
- When you deactivate the handler for the subdomain AND disable "abort", do you at least see an empty webpage and the certificate?
If its a very complex issue, you can also go to https://caddy.community and fill out their help template. Show your old nginx configuration, and your current Caddyfile. That way they can see if theres a mistake.
Since there are a lot of troubleshooting requests, I have expanded the documentation:
https://github.com/Monviech/opnsense-docs/blob/caddy-layer4/source/manual/how-tos/caddy.rst#help-nothing-works
It is not live in the main docs yet, but it can be read here.
Is it correct, that if I add rules to the LAN interface, that I can replace any for source with LAN net? I ask, because the instruction states to use the same rules for WAN for LAN.
Yes you can if you want. Though it does not really matter since only the LAN interface receives pakets from the LAN anyway already.
Quote from: Monviech on July 27, 2024, 07:21:45 AM
Sorry I don't know then. Without some debug logs its uncertain what happens there. I need some info that is not anynomized so theres no mistakes due to wrong omissions.
- Check your DNS, does "nslookup yoursubdomainname" really resolve to the IP address of the OPNsense?
- If Yes, Whats the output of "curl -v subdomainname"
- What do the debug logs show when you try to reach it?
- Which kind of application is listening there? Is it a HTTP or HTTPS application.
- If the application demands a HTTPS connection, did you enable "TLS Insecure Skip Verify" like I asked?
- When you deactivate the handler for the subdomain AND disable "abort", do you at least see an empty webpage and the certificate?
If its a very complex issue, you can also go to https://caddy.community and fill out their help template. Show your old nginx configuration, and your current Caddyfile. That way they can see if theres a mistake.
So, thanks for the heads up! I finally managed to make it work.
As every time something ain't working in networking... its always dns !
For DNS resolving, i'm using pi-hole > unbound > DoT
Pi-hole was not resolving the "local" domain and was throwing "non-existing domain"
Adding a local dns record thru pihole, pointing to the firewall made it resolve.
Now works as intended !
Good job, yeah DNS is tricky sometimes. Great it works :D
Would you happen to know where the www folder for caddy is located?
I'd like the main domain to point to a single html file inside caddy
something like this :
<!DOCTYPE html>
<html>
<body style="overflow:hidden; margin:0; text-align:center;">
<img src="image.jpg" style="height:100vh; max-width:100%; object-fit: contain;">
</body>
</html>
The plugin is not set up for it. It only serves as reverse proxy.
Though, you can read the documentation here:
https://caddyserver.com/docs/caddyfile/directives/file_server
You would need to set up a "root path". I would suggest to put it somewhere inside "/var/db/caddy/data/caddy/..."
To configure the file_server, you could then use the import statements. The *.conf file can set it up inside the site block. https://docs.opnsense.org/manual/how-tos/caddy.html#custom-configuration-files
I actually never used Caddy as a static web server before, only as reverse proxy. So, I'm not much of help here. Good luck though, shouldn't be too hard.
Also, I do not recommend to serve files from the firewall itself. ;)
Here is a new video that shows off the Layer4 routing of the next plugin version os-caddy-1.6.2
It remains very simple to configure, which is the main focus of this plugin.
https://www.youtube.com/watch?v=1IykZemclVA
The first version of the Layer4 feature is now complete and will be in the next plugin version. Additional Traffic matchers have been added.
https://github.com/opnsense/plugins/pull/4133
This means as an example:
The firewall only has port 443 open and Caddy listens there with its new Layer 4 module.
- SSH can be sent from port 443 to 192.168.1.1:22
- At the same time, RDP can be sent from 443 to 172.16.100.99:3389
- A TLS connection for OpenConnect VPN matched by SNI "vpn.example.com" can be sent to 127.0.0.1:8443
- The normal reverse Proxy can continue to work and serve any number of HTTPS Domains.
Since this is quite new in this form in Caddy, feedback is valuable.
Hi Monviech, thank you very much for this great plugin. I would have a suggestion for improvement for the documentation "Reverse Proxy the OPNsense WebGUI". I followed the description to secure my subdomain. Therefore, I followed the description from the point of view that it is not about the OPNsense WebGUI for me: "The same approach can be used for any upstream destination using TLS and a self-signed certificate." From my point of view, it was difficult for me to get started in the section "Add a new HTTP Handler with the following options" because at first I didn't realize that "TLS Server Name: OPNsense.localdomain" should also be entered here for my case.
Background why I wanted to do this: I have the Unify Controller running via Docker in a DMZ and wanted to access it via the URL unify.example.com.
In Caddy I created
- a wildcard domain: "*.example.com"
- a subdomain "unify.example.com" with access list at 192.168.1.0/24
- a handler
- with Upsteam domain and port to the Docker host of the Unify Controller
- set "TLS Insecure Skip Verify" [ x ], otherwise the subdomain would not work
- activated TLS under "Trust"
So today I went to unify.example.com and was greeted with a different interface. The interface belonged to a "securepoint utm v12". That set my alarm bells ringing. With "nslookup unify.example.com" I saw that numerous IP addresses lead to the subdomain, but only one of the listed ones was the current IPv4 of my Internet connection. I don't know if the settings for Unbound could have anything to do with it, so I suspect that I was allowed to see a live application of a man-in-the-middle attack.
So I followed the documentation. I went to the IP address of my Unify controller, clicked on the 🔒 lock, which was crossed out, and obtained the .pem file. I added the contents of the file under System ‣ Trust ‣ Authorities as "Import an existing Certificate Authority" in Certificate data. Including "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----". As I didn't have any "Private key data" and didn't know what to enter for "Serial for next certificate", I left this blank.
Under the handler, I adjusted the settings under "Trust", for "TLS Trust Pool" my created "Authorities" and under "TLS Server Name: OPNsense.localdomain". I would have thought that I could uncheck "TLS Insecure Skip Verify", but then the page remains blank/white.
The steps led to the Unify controller actually being displayed again.
https://forum.opnsense.org/index.php?topic=38714.msg190745#msg190745
The tls server name is not always "opnsense.local". It is actually the SAN (subject alternative name) of the certificate you add into system - trust - authorities.
You have to check the certificate in the browser to find it out.
If both the certificate and the name are correct, then the connection works without tls insecure skip verify.
If you want something improved, feel free to open an issue here, or even better create a pull request and improve the section in the docs.
https://github.com/opnsense/docs
Thank you, for the explenation and the link. I edited the handler, and now it works without "TLS Insecure Skip Verify".
QuoteThe tls server name is not always "opnsense.local". It is actually the SAN (subject alternative name) of the certificate you add into system - trust - authorities.
Personally, I would be satisfied with the information or the sentence in the tip box.
I did not find the word SAN explicitly in Firefox, but only under "Security" "Validated by: CN=unifi" and under "Show certificate" only the owner or issuer name unify. I therefore did not see the connection to SAN here. If you could improve this part and put it in the green tip box, I'd be fine.
how do I enable http.realip for domains behind cloudflare proxy?
https://caddy.its-em.ma/v1/docs/http.realip
I think it happens automatically as soon as you configure "trusted proxies" in the General Settings (advanced view)
This looks fantastic, but somehow it has managed to feel more complicated to setup than just a small VM with Caddy on it lol. Followed the instructions for the firewall and everything else for the initial setup, added my DNS provider with all the keys. I did turn off the ability to automatically create records, as many of my services I absolutely don't want accessible from off the network.
Despite having a functional A record in my DNS, I see a bunch of errors from Let's Encrypt claiming that there are no valid A or AAAA records for the domain I had entered. It actually errored enough times before I noticed that I got locked out.
Also, I've definitely never needed an email for Let's Encrypt before, not sure what that's about if I'm being honest, but whatever. Really I'd like to figure out why they're claiming there are no valid records despite having been perfectly fine on my VM that I paused 10 minutes before setting up.
The email has been changed to required since there are two issuers inside. ZeroSSL requires an Email, Let's Encrypt does not. If one fails the other is tried automatically.
To keep it simple I required the Email. Just put in whatever ;D
Quote from: Monviech on August 07, 2024, 11:46:44 AM
The email has been changed to required since there are two issuers inside. ZeroSSL requires an Email, Let's Encrypt does not. If one fails the other is tried automatically.
To keep it simple I required the Email. Just put in whatever ;D
Ah, that makes sense. Any idea why Let's Encrypt would be claiming I have no valid A or AAAA records? They're definitely there.
Sorry I'm not sure here, not so good with Lets Encrypt Troubleshooting. Maybe related to this?
https://github.com/opnsense/plugins/issues/4161
If in doubt, its /always DNS/™
If not maybe ask in the https://caddy.community and show your debug logs. They know this plugin exists.
I have a problem with a dockerized Unify Controller. I was able to set a subdomain to the web GUI of Unify (Port 8443) with Cuddy. The subdomain is unify.example.com. There is an option for the AP, to set a domain instead of an IP, for the guest hotspot. Without the domain option active, connecting to the Guest WLAN opens 192.168.1.10:8080 (http). It is possible to enable https, however without a proper certificate, this will a valid option.
So I thought, it would be a good option to use Caddy and activate the domain option in the AP. The problem is, that the hotspot tries to resolve to https://unify.example.com:8843 for https and :8080 for http.
I guess, that this will not work with caddy, because it only listens on 80 and 443, right?
Is it possible, to use Caddy here? Do I need to add FW rules for WAN, that allow 8843 or 8080?
You can add the same domains multiple times, with different ports.
e.g.
example.com
example.com:8080
example.com:8443
and give each of them the right handler.
Thats a supported configuration.
And yes of course they need firewall rules.
Do I need a specific configuration for that? I mean in the WebGUI of OPNsense, I set a domain example.com and a wildcard domain *.example.com. All actual subdomains are a result of the wildcard domain. I ask, because the description of the port states, that all subdomains will listen on the same port as the domain..
You should be able to add the same wildcard domain with different ports multiple times, and then select the other wildcard domain with the different port in a handler in addition to the same already created subdomain.
Essentially I think that should work, I have never tested it before though.
I don't think you have to duplicate the subdomains in the subdomain tab. I think it should just work to add the existing ones with the different wildcard domain in a handler, since they will get added to it.
Just try it out and if you get strange results show me your Caddyfile.
Is it required to adjust the firewall rules? I ask, because the Caddy documentation assume, that Caddy uses Port 443 and 80 and so the firewall rules. If I add a domain with port 8843, is it required to add a corresponding firewall rule as well?
Yes you have to open all ports in firewall rules that Caddy uses. It does not open any ports for you automatically.
Hi,
Caddy is always adding new A records for * and @ even when update only is checked in settings
It depends on the provider, all of them are different, and all of the modules are written by different people so they do not all share the same featureset.
Best go to https://github.com/caddy-dns and find the provider you are using and open an issue where you also share part of your caddyfile.
This explains a lot. I noticed, that my A-records increased with old IP addresses, since I use Caddy. I deleted old entries by hand on the IONOS dashboard.
Quote from: Monviech on August 14, 2024, 06:30:11 AM
It depends on the provider, all of them are different, and all of the modules are written by different people so they do not all share the same featureset.
Best go to https://github.com/caddy-dns and find the provider you are using and open an issue where you also share part of your caddyfile.
Thanks for Info.
Quote from: Baender on August 14, 2024, 10:52:46 AM
This explains a lot. I noticed, that my A-records increased with old IP addresses, since I use Caddy. I deleted old entries by hand on the IONOS dashboard.
It's the same for me with IONOS
Here is the issue where I implemented the "Update Only" checkbox. It seems like it was broken for Ghandi too with duplicate A records being created.
https://github.com/opnsense/plugins/issues/4036
https://github.com/caddy-dns/gandi/issues/9
I do not doubt that other modules suffer from the same jank sometimes. The Dynamic DNS is not a "core" module of Caddy. So only the DNS Challenge will always work correctly since that seems to be the main usecase.
I will open an issue for IONOS and see if it can be closed. In the meantime, I'll check whether setting the Update Only setting brings any improvement. It hasn't been active for me yet.
Btw. it would also be great if there was a link to the repository (https://github.com/caddy-dns) next to "DNS Provider". Because either I overlooked the solution or I couldn't find the way to enter the API key for IONOS straight away. In the end, I did find the solution, but it probably took me 15 minutes. A quick click on the repository now shows me that it's right there.
It is actually baked into the very helptext inside the plugin.
Reading the help text... helps. :)
https://github.com/opnsense/plugins/blob/f9610f33c559498a1354cb6ba3154aa26a828b21/www/caddy/src/opnsense/mvc/app/controllers/OPNsense/Caddy/forms/dnsprovider.xml#L6
To be able to read it means to have found it. In other words, to have seen it. I guess I haven't. So everything is fine. Thank you!
Hi Monvlech
New user to OPNsense here. I followed the installation guide and setup the firewall rules. However something is still missing when I attempt to create a new domain and it doesn't end up working.
For starters on my Caddy Certificates widget it says "Caddy does not manage any automatic certificates" and I get the following error from my logs
2024-08-15T16:10:33-05:00 Error caddy "error","ts":"2024-08-15T21:10:33Z","logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"opnsense-test.marquez.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge"}
2024-08-15T16:10:33-05:00 Error caddy "error","ts":"2024-08-15T21:10:33Z","logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"opnsense-test.marquez.com","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1893209516/296576201246","attempt":2,"max_attempts":3}
2024-08-15T16:10:33-05:00 Error caddy "error","ts":"2024-08-15T21:10:33Z","logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"opnsense-test.marquez.com","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]}}
Would appreciate any help trying to figure out what I haven't configured correctly. Appreciate the help!
The error means that the TLS-ALPN-01 challenge fails.
Without knowing more of the infrastructure its hard to help.
Make sure Caddy can receive traffic directed to it for your Domain Name, on IPv4 and (if available) IPv6. That means the A and AAAA records have to point to the external IP address of the OPNsense, or to the external IP of the router you use that forwards or DNATs this traffic to the OPNsense with Caddy.
If you can't open Firewall for ACME, consider using the DNS-01 challenge with a DNS Provider to receive certificatea without Firewall issues.
Thanks for confirming. I was researching for a couple of hours and finally realized my issue. I need a domain first and to point it in the right spot. I was so new and your tutorial made it seem so simple that that part never registered. Thank you, I think once I sort that out I should be good to go.
Yeah the tutorial does imply some knowledge about how the domain name system works, and that you need your own domains and stuff.
But it's hard to make it clearer cause at some point you just have to assume a certain level of knowledge, or you have to start with adam and eve in these tutorials.
Happy you understood whats not working and how to fix it. :)
Quote from: Monviech on August 14, 2024, 06:30:11 AM
It depends on the provider, all of them are different, and all of the modules are written by different people so they do not all share the same featureset.
Best go to https://github.com/caddy-dns and find the provider you are using and open an issue where you also share part of your caddyfile.
The problem with the IONOS DNS Plugin is fixed and working as it should when "Update only" is selected. See Issue: https://github.com/caddy-dns/ionos/issues/7 (https://github.com/caddy-dns/ionos/issues/7)
Will caddy will be updated automaticly in near future by OPNSense or do i have to stick with my own build? Not sure how it works.
Thanks for taking care of it.
I'll update the build for the next os-caddy version (1.6.3).
The build works like this:
https://github.com/opnsense/tools/blob/a1c883deffd29c15588e852f7a143bea04d7214a/config/24.7/make.conf#L97
The commit hashes are updated manually every once in a while to keep the build reproducible.
Im still waiting for another module to get an update so I won't update the dependencies for this minor version since I want to do it in one swoop.
I gonna try again for os-caddy-1.6.4.
no one cares...
<delete me>
Quote from: Monviech on August 20, 2024, 07:02:26 PM
I gonna try again for os-caddy-1.6.4.
Is the IONOS Patch included in 1.6.4?Never mind. Thought i've seen 1.6.4 but it's 1.6.3_1
@Aergernis
I have updated the build though, so just go to packages and press "reinstall" on "caddy-custom". The ionos patch should be included then.
https://github.com/opnsense/tools/pull/429
@W0nderW0lf
I did not know an answer there, best go to https://caddy.community.
I have allowed ports 80 and 443 on both WAN and LAN for Caddy. If I now set a subdomain in Caddy for a LAN application: https://foo.example.com:8843, do I then have to allow the port on WAN and on LAN or is LAN sufficient? The application should not be accessible from the Internet.
Just LAN will be enough I think.
Do you have any advice on how I can get wordpress working. I have Caddy working with these domains except wp.domain.org:
kuma.domain.org
unraid.domain.org
proxmox.domain.org
portainer.domain.org
wp.domain.org
All of them Kuma, unRaid, Proxmox, and Portainer come up with no problem. Certificates working as well. Except for Wordpress. Kuma, Portainer and Wordpress are all docker containers. When I try to load Wordpress I get a, "Bad gateway Error code 502" from Cloudflare. I just don't understand why all the others work, but WordPress doesn't. Any advise would be greatly appreciated.
I get these two errors from Caddy. I took out my IP's and replaced it with DOMAIN for this message:
"error","ts":"2024-09-10T20:59:47Z","logger":"http.log.access","msg":"handled request","request":{"remote_ip":"DOMAIN","remote_port":"62058","client_ip":"DOMAIN","proto":"HTTP/2.0","method":"GET","host":"DOMAIN","uri":"/","headers":{"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"Cf-Ipcountry":["US"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8"],"X-Forwarded-For":["DOMAIN"],"Priority":["u=0, i"],"Accept-Encoding":["gzip, br"],"Sec-Fetch-Site":["none"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Upgrade-Insecure-Requests":["1"],"Cdn-Loop":["cloudflare; loops=1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Cf-Ray":["8c1257f16b642863"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Sec-Gpc":["1"],"Cf-Connecting-Ip":["DOMAIN"],"Dnt":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"DOMAIN"}},"bytes_read":0,"user_id":"","duration":3.011785027,"size":0,"status":502,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}
"error","ts":"2024-09-10T20:59:47Z","logger":"http.log.error","msg":"dial tcp 10.10.72.10:8189: i/o timeout","request":{"remote_ip":"DOMAIN","remote_port":"62058","client_ip":"DOMAIN","proto":"HTTP/2.0","method":"GET","host":"DOMAIN","uri":"/","headers":{"X-Forwarded-Proto":["https"],"Cf-Ipcountry":["US"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-User":["?1"],"X-Forwarded-For":["DOMAIN"],"Priority":["u=0, i"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Upgrade-Insecure-Requests":["1"],"Cdn-Loop":["cloudflare; loops=1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0"],"Accept-Encoding":["gzip, br"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Dest":["document"],"Sec-Gpc":["1"],"Cf-Connecting-Ip":["DOMAIN"],"Dnt":["1"],"Cf-Ray":["8c1257f16b642863"],"Sec-Fetch-Mode":["navigate"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"DOMAIN"}},"duration":3.011785027,"status":502,"err_id":"hydwhx7fz","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
Hey. The log states that caddy is unable to connect to
dial tcp 10.10.72.10:8189: i/o timeout
Its a Layer 3 Problem. Maybe a typo? Or the host has a firewall rule that denies access from the IP Caddy comes from? It must be a network problem.
https://caddy.community/t/log-i-o-timeout-meaning/20048
I configured my firewall to allow everything in and out for testing purposes, but it still doesn't work so it has to be something else. I'm going to try other CMS's like Joomla and Drupal to see if they work. Everything else I've tried like Kuma, unRaid, Proxmox, and Portainer all work just fine.
I didnt mean the firewall of the opnsense. I mean maybe theres a firewall on the host itself that serves wordpress, and it blocks the incoming connection. Maybe theres fail2ban or something on that host too. I would check anything that blocks IP addresses. I tried with my own wordpress and it "justworks TM".
I apologize I didn't understand what you meant. However since you showed me it could be done I went back and figured it out. I was using a template in Portainer. When I redeployed using stack method it worked. Thank you very much for your plugin. I couldn't get nginx or haproxy to work because they are too complicated for me.
Some weeks ago, there was an issue with the dynamic DNS for IONOS, where the API added new records instead of updating them. The issue was fixed, but I wonder when it will come to OPNsense caddy.
Today :)
If the caddy-custom is not updated automatically, just go to
/ui/core/firmware#packages
search the caddy-custom package and press "reinstall"
Nice. It became a very annoying routine, in the last few weeks, to clean old records. I am looking forward to the update.
I mean its in here: https://forum.opnsense.org/index.php?topic=42787
So already available.
This plug in looks great - but also a bit over my paygrade.
I've read thru the documentation, and this post, and have a request ...
Would it be possible to put together a youtube video showing a setup walk through for the following scenario (which I suspect would be beneficial for many homelab users):
Assuming the user already has a registered domain name ( for example: example.com )
Assuming they are already set up with a Cloudflare account
The video to show what would be required in OPNSense / the caddy plug in to:
set up to have a certificate that automatically renews associated with example.com
set up to have caddy used to securely reference specific internal addresses such as:
opnsense.example.com
homeassistant.example.com
openmediavault.example.com
jellyfin.example.com
etc.
(on which ever ports are required)
setup to ensure none of the above can be accessed externally from the internet (i.e. access only applies to internal accesses)
setup to have the external ip address of example.com updated via Cloudflare when it changes.
In addition to Caddy on the OPNsense, I set up a Caddy proxy in a subnet 192.168.50.10. In that Caddy file, I would like to add the global trusted_proxies directive:
QuoteEnabling this causes trusted requests to have the real client IP parsed from HTTP headers (by default,
Is the IP address of the trusted proxy, the gateway of the subnet? 192.168.50.1? So to say, the Caddy proxy of the OPNsense is a trusted source?
I feel like I should add this here as well:
How to enable HSTS for the "official" Nextcloud docker.
Go to Caddy --> Reverse Proxy --> HTTP Headers and add a new one.
Header: header_down
Header Type: +Strict-Transport-Security
Header Value: max-age=31536000;
Save and add it to your Nextcloud http handler. Save and apply.
Heya,
Thanks a lot for your work and for the tutorial. It really helped me out a lot.
There is one small issue that I am having with Caddy and my nextcloud (of course) container.
I can access it using my domain from outside my local network, but when trying to connect it through my local network, I keep getting ERR_SOCKET_NOT_CONNECTED or PR_END_OF_FILE_ERROR issues. I did not mess with NAT reflection nor with Unbound DNS as the tutorial says that there won't be a need for it, but I still cannot get access locally.
My caddyfile is as follows:
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file
# caddy_user=root
# Global Options
{
log {
output net unixgram//var/run/caddy/log.sock {
}
format json {
time_format rfc3339
}
level DEBUG
}
dynamic_dns {
provider cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
domains {
example.com *
example.com nc
}
}
email example@mail.com
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}
# Reverse Proxy Configuration
# Reverse Proxy Domain: "a46e07f0-97d6-40ee-a4ba-c219beee103f"
*.example.com {
tls {
issuer acme {
dns cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
}
}
@f601ec75-1d72-4165-a41c-34322ad8a17a {
host nc.example.com
}
handle @f601ec75-1d72-4165-a41c-34322ad8a17a {
handle {
reverse_proxy 10.150.0.10:8666 {
header_down +Strict-Transport-Security "max-age=31536000;"
}
}
}
}
import /usr/local/etc/caddy/caddy.d/*.conf
Unfortunately no logs from caddy as nothing shows up when trying to access it internally. When using nslookup on the domain, I get my opnsense ip.
Server: firewall.home.lan
Address: 10.150.0.1
Name: nc.example.com
Address: 10.150.0.1
Any help would be appreciated!!
Did you create the same Firewall rule for WAN also for LAN/other interfaces you use?
You shouldn't use Split DNS Zones with Caddy, just use the external IP address from internally and externally. I can see your nextcloud resolves to the internal IP of your OPNsense. (Technically its possible and people do it but it complicates some firewall rules and stuff)
https://docs.opnsense.org/manual/how-tos/caddy.html#caddy-troubleshooting
If nothing of these hints work please post some debug logs where caddy logged the error you see happening.
Quote from: Monviech on September 26, 2024, 03:34:23 PM
Did you create the same Firewall rule for WAN also for LAN/other interfaces you use?
You shouldn't use Split DNS Zones with Caddy, just use the external IP address from internally and externally. I can see your nextcloud resolves to the internal IP of your OPNsense. (Technically its possible and people do it but it complicates some firewall rules and stuff)
https://docs.opnsense.org/manual/how-tos/caddy.html#caddy-troubleshooting
If nothing of these hints work please post some debug logs where caddy logged the error you see happening.
The Split DNS setup in Unbound was remaining config from my old HAProxy setup, that's fixed now. But it did not solve my issue. The actual problem I found was that I was checking the Dynamic DNS box in both Domains and Subdomains, and I have some misconfiguration as well on the Cloudflare dashboard. Since cleaning all that up, it has been working perfectly.
Thanks a lot!
Edit: Quick question, is there any way to use other caddy directives in the GUI e.g. php_fastcgi and fileserver? It would be an improvement for me to use caddy instead of using Nextcloud with its own embedded Apache web server.
Great that you figured it out.
No you can only use the directives in the GUI that you can see in the GUI.
For custom stuff you can always use custom configuration files:
https://docs.opnsense.org/manual/how-tos/caddy.html#custom-configuration-files
I advice against it though. Use it only as reverse proxy. Do not serve static files with it (on the OPNsense).
I can't add multiple ports for load balancing? Am I missing something?
No the GUI only allows for the same port being used on all backend webservers in the same loadbalancing group.
Whats the usecase for different ports there?
Since all webservers that load balance should be configured the same way, why serve them on different ports?
Multiple instances per machine/GPU for the service.
Looking at the caddy config I did wonder why it was not just one field with ip:port entries so you could do any combination as you can in the config itself.
It has grown historically while building the plugin and now its hard to change it without breaking existing setups.
Its one of these things.
There are some validations attached tp the port field too, since when you change to the www user it gets validated extensively.
Im sure it could all be somehow resolved with migrations and different fieldtypes but the usecase is very small so somebody who needs it would have to invest time there.
Is what it is, I'll just use something else for now.
It's a pain when you have things tied up in validation and existing configs.
It's very cool to have the additional functions of caddy available on opnsense.
well you could work around it by having multiple virtual IPs on that host and bind one GPU instance per virtual IP on the same port for each. Then each socket on the same host would also be unique even with the same port.
But yeah this wont be resolved anytime soon.
Or write your own config file for that one usecase. You can still use the GUI for all other things.
https://docs.opnsense.org/manual/how-tos/caddy.html#custom-configuration-files
Hi,
I install Caddy and configure follow the tutorial but I have error:
"error","ts":"2024-10-11T07:26:56Z","logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"toto.pequod.sokil.fr","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:connection - 89.219.181.98: Timeout during connect (likely firewall problem)"}
I really don't know where to start
I also on freeBSD and debian install caddy to test with the same error.
I have another site with OPNsense and caddy on debian behind without error, I miss something but what ?
Well I get no connection to your IP either. So its either a firewall problem, the IP is a CGNAT IP, you have to troubleshoot that with curl for example:
curl -v 89.219.181.98
* Trying 89.219.181.98:80...
^C
curl -v 89.219.181.98:443
* Trying 89.219.181.98:443...
^C
See theres nothing, no response. So Let's Encrypt can not connect either.
https://imgur.com/a/y2YyIJN
I created nat port forward to my server directly with the port 80 and everything work.
If I forward 443 to 80, can be considerate as a good test to check if ISP block something ?
https://imgur.com/n3XyyQt I reach the webserver
Maybe the This Firewall alias does not work for you for some reason.
Try to disable the port forward rule.
Set the rules on WAN to "WAN address" instead.
I have a similar problem. From time to time, my domains are not reachable. I restricted them to the LAN network. Some services and my vacuum robot. The only thing that helps, is to perform a restart of the OPNsense and to get a new IP and new Records for the domains. A restart of caddy won't work. The services are reachable by their IP, when the problem occurs.
Would it help to set caddy to debug and send the log from the beginning, when it's working until the moment it fails? It could be a lot of log data, because I don't know when it will happen. Moreover I think, that this is related to a problem with IPv6 prefix delegation. In general, would it be a good idea, to combine the caddy logs, with the logs of the OPNSENSE system? Is this only manually possible?
I do think such a problem is out of scope for me to troubleshoot. Sorry.
I m agree with you, it's firewall problem config but witch checkbox ;D
I continue to search
Hey there, thank you very much, for bringing Caddy to OPNsense. It is a real pleasure, to such wonderful tool on the sense. I even like it that much, that I use Caddy as a webserver or reverse proxy in other projects as well.
I would like to add a TURN and STUN Server (Coturn: https://github.com/coturn/coturn (https://github.com/coturn/coturn)) to my infrastructure. It is a requirement for my Nextcloud. At first I tried to an additional subdomain to Caddy, that extents to my nextcloud.example.com.
- nextcloud.example.com
- nextcloud.example.com:3478
However, that required to have a wildcard domain with the port 3478 and the actual subdomain
nextcloud.example.com:3478. Not to mention opening another port on the firewall.
Would it be possible to use the
Caddy: Layer4 Routes feature for my project? Is it an TLS (SNI) type then and moreover, is it still required to open the port 3478 for it?
Using Nextcloud Talk is notoriously difficult.
I would rather use IPv6 where a stun and turn server is not needed, because webrtc can create a direct socket from endpoint to endpoint.
Quote from: Baender on October 18, 2024, 08:12:13 AM
Would it be possible to use the Caddy: Layer4 Routes feature for my project? Is it an TLS (SNI) type then and moreover, is it still required to open the port 3478 for it?
STUN and TURN is UDP, so no. You need inbound NAT port forwarding for that. Which answers the port question, I guess :)
Actually the Layer 4 feature in Caddy had a big overhaul, so you can use it to stream UDP traffic too, from any port to any port. And you can even use layer 7 matchers additionally inside the layer 4 matchers to selectively match and route traffic.
Check it out :)
Quote from: Monviech on October 18, 2024, 09:03:15 AM
Actually the Layer 4 feature in Caddy had a big overhaul, so you can use it to stream UDP traffic too, from any port to any port.
TIL - thanks :)
@Patrick
E.g. you can match and route wireguard which is UDP
https://github.com/opnsense/plugins/issues/4201#issuecomment-2405077886
Thank you for the explanation.
I have noticed, that since I enabled IPv6 and using Track Interface, most of my subdomains, that are restricted to local IPs, won't work. This is because I only set IPv4 addresses to the access list. How can I allow "local" IPv6 addresses?
You just have to add your GUA to that access list.
e.g.
2001:db8:1234:5600::/56
If its not static you are a little out of luck here since the access list can not be updated with a dynamic prefix. In that casw it would be easier to only have A Records and no AAAA records for your domains.
If you use the dynamic dns feature set it to ipv4 only.
Oh, no. Please don't tell me that.. Yes, it is dynamic.. Could I override the Caddyfile then? I know, that for Caddy there might be such Plugin, but if I would write a script, that checks the current IP? Would that help?
Just a side note from my side: with one of the recent caddy releases on OPNsense, the description for domains and subdomains got a bit out of control. I personally find it problematic, to show the domain and the description, that results in a lot of text, in the selection GUI and on the overview. Do you know what I mean?
The issue here is how to tell caddy that the IP has to change in the access list, and then also reload the service so it is indeed loaded. This can be rather complex, since there is no general default access list into which this could be implemented genericly. Everybody manages their own lists that are saved in their own configuration.
There is no official module for this: https://github.com/caddyserver/caddy/issues/5813
And regarding the description, I had to add it to the subdomain dropdown since otherwise there would be no way to keep different subdomains apart, that have the same subdomain name, but are added under multiple wildcard domains with different ports. So we have to life with that now.
If you want to write a script it has to be a php script that targets the API that the framework of the caddy plugin provides. You could update your specific access list witj your correct IPv6 network when it changes and then use the service API to trigger a reconfigure act.
https://github.com/opnsense/plugins/tree/master/www/caddy/src/opnsense/mvc/app/controllers/OPNsense/Caddy/Api
All descriptions in domains/subdomains etc... are optional btw so delete the ones you dont need.
Okay a really complex topic then. Would it be (in theory) possible, to let Caddy read the aliases of the firewall? Would it be a polling, or as it might be in the memory, on the fly change, if I would add the IPv6 address as an alias?
I guess, still out of scope. However, I am just curious.
Edit:
Would it be possible, to set dynamic DNS setting on the domain/subdomain? I mean if it should pass a IPV6 and/or IPv4 address? Maybe the module for dynDNS is not structured for this..
I have caddy configured as a reverse proxy for the OPNsense GUI and as a SSH multiplexer, more or less according to the docs found here: https://github.com/opnsense/docs/blob/11e66816989bb12633e01e144ebf42b11508755a/source/manual/how-tos/caddy.rst (https://github.com/opnsense/docs/blob/11e66816989bb12633e01e144ebf42b11508755a/source/manual/how-tos/caddy.rst)
Unfortunately, I am having an issue where it prevents system reboots, both from the GUI and from the CLI menu (option 6). The rebooting hangs on shutting down caddy, which apparently never happens. The only thing that currently works is issue a 'shutdown -r now' from the CLI. The log files aren't giving me much direction how to go about investigating this. Any tips or advice?
@dinguz
I have to ask upstream for this. I'll come back to this when I know more. What I can say is I do sometimes fight with the shutdown behavior of caddy. When it has open tcp/udp connections it can wait indefinitely until they are closed in certain configurations.
Thats why the service script now has sigkill implemented. But that will not happen on "service caddy stop" (which is used during reboot), only on "configctl caddy stop".
https://github.com/opnsense/plugins/pull/4261
@Baender
The dynamic dns module only has global settings. So set Ipv4 only globally if you need access lists with rfc1918 addresses. Otherwise look into forward auth or basic auth to secure access if you rely on dynamic ipv6. The way access lists are handled will not be changed in the current plugin implementation.
@dinguz
https://github.com/mholt/caddy-l4/issues/258
For now either close the ssh connection after initiating a restart, or use "configctl caddy stop" since it has a mechanism that will kill caddy after 20 seconds.
Quote from: Monviech on October 25, 2024, 01:58:40 PM
For now, either close the ssh connection after initiating a restart, or use "configctl caddy stop" since it has a mechanism that will kill caddy after 20 seconds.
Many thanks!
Today I took the opportunity to try out Caddy reverse proxy instead of HAproxy, mostly because of a very specific problem with HAproxy...
I must say I reverted after trying it thouroughly. My 2cents on this are as follows:
- Caddy is suited to home setups and inexperienced users. HAproxy is much more complex.
- For example, the certificate setup is much easier, because you just have to specify the domain and it just works (tm).
- However, if you have more than just one domain, Caddy setup gets a little tedious:
* you have to create one domain/certificate plus a http backend for any domain, which includes creating different ones for www.domain.de and domain.de. You cannot combine certificates for multiple domains unless they are subdomains.
* You do not have much control over what type of certificate(s) are created - you cannot specifiy strength or ECC vs. RSA (much less both) and I have not found a means to control if ZeroSSL vs. LetsEncrypt is used.
* The ciphers being employed cannot be controlled easily - or, for TLS 1.3, at all. That results in an ssllabs.com score which is suboptimal, becaus 128bit ciphers are allowed. This cannot be changed because of Go limitations (https://caddy.community/t/specify-tlsv1-3-cipher-suites/13058/3).
* You cannot use more than one type of DNS-01 verification if you use wildcard domains.
* The Auto HTTPS feature looks nice first, but indeed it uses a 308 instead of a 301 code, which breaks some monitoring and can only be modified via custom include files.
So, if you just want to reverse-proxy some services in your home network, go with Caddy. For an OpnSense guarding your internet site with several services/domains, stay with HAproxy.
@meyergru exactly the situation with my home lab vs. our data centre operation ;)
Thanks for the concise summary.
I just knew the target audience of this. Caddy filled the niche pretty well imo for home setups.
No complaints for the right target audience, Cedrik - I just heard it was much easier and tried it.
Even if you would add some features, you would only end up at something inherently more complex. Plus, some restrictions seem to be hard-wired (like the TLS 1.3 ciphers imposed by Go).
The Reverse Proxy is pretty much finished, but the layer 4 module will get more features over time.
Next thing is an OpenVPN matcher where you can multiplex on the same port via tls static keys for example.
Theres some neat Layer 7 matchers coming.
https://github.com/mholt/caddy-l4/pull/251
Hi!
I tried setting up the Caddy plugin but ran into some issues. Just setting up a basic reverse proxy does not seem to work for me when following the guide. It doesn't give me any errors from the GUI and the Caddy service appears to be running properly, but trying to run caddy reload --config /usr/local/etc/caddy/Caddyfile results in an error not displayed elsewhere:
Error: sending configuration to instance: performing request: Post "http://127.0.0.1:2019/load": dial tcp 127.0.0.1:2019: connect: connection refused
Maybe it is not possible/supported to reload caddy this way? Even though the service appears to be up, it does not seem able to connect properly, given:
> curl -vL 127.0.0.1:80
* Trying 127.0.0.1:80...
* Immediate connect fail for 127.0.0.1: Connection refused
* Failed to connect to 127.0.0.1 port 80 after 0 ms: Could not connect to server
* closing connection #0
curl: (7) Failed to connect to 127.0.0.1 port 80 after 0 ms: Could not connect to server
What could I be missing? Thanks in advance!
sockstat -l | grep -i 80
sockstat -l | grep -i 443
Make sure no other services use port 80 and 443.
service caddy status
Check if Caddy is running
service caddy restart
service caddy reload
Restart Caddy if needed
Quote from: Monviech on October 27, 2024, 05:42:48 AM
sockstat -l | grep -i 80
sockstat -l | grep -i 443
I can see that Caddy has managed to bind to port 443, but not 80. AFAICT no other service has taken port 80 either: Doing curl on localhost:443 results in "error:0A000438:SSL routines::tlsv1 alert internal error". Output of sockstat for port 80:
# sockstat -l | grep -i 80
root lighttpd 30649 4 tcp4 127.0.0.1:43580 *:*
root lighttpd 30649 5 tcp6 ::1:43580 *:*
root ntpd 10336 23 udp6 fe80::e01d:f0ff:fe2a:e3c2%vtnet0:123 *:*
root ntpd 10336 24 udp6 fe80::b8a0:afff:fe98:12fe%vtnet1:123 *:*
root ntpd 10336 27 udp6 fe80::1%lo0:123 *:*
root php-cgi 38055 0 stream /tmp/php-fastcgi.socket-0
root php-cgi 38040 0 stream /tmp/php-fastcgi.socket-0
root sshd 11827 8 tcp6 fe80::1%lo0:2223 *:*
Quote from: Monviech on October 27, 2024, 05:42:48 AM
service caddy status
Caddy appears to be running just fine: "caddy is running as pid 72385."
Quote from: Monviech on October 27, 2024, 05:42:48 AM
service caddy restart
service caddy reload
This did perform a clean restart and reload of Caddy, as opposed to asking Caddy to reload itself. So that's nice to know! Still, it unfortunately did not make any difference.
I noticed that I had used an older version of the plugin, so just to be safe I also did reset most settings and applied it again to see if the update had fixed something related to what I attempted to setup. But still no success.
Quote from: yarcod on October 27, 2024, 10:06:40 AM
I can see that Caddy has managed to bind to port 443, but not 80. AFAICT no other service has taken port 80 either: [...]
Make sure System > Settings > Administration > HTTP Redirect - Disable web GUI redirect rule
is checked.
I could say more if I can see the Caddyfile. But right now no idea really.
Quote from: Patrick M. Hausen on October 27, 2024, 01:31:02 PM
Thanks -- I ensured that it was still checked.
Quote from: Monviech on October 27, 2024, 04:35:12 PM
I could say more if I can see the Caddyfile. But right now no idea really.
The Caddyfile looks like this (did not find any spoiler tag to compress my message):
{
log {
output net unixgram//var/run/caddy/log.sock {
}
format json {
time_format rfc3339
}
level DEBUG
}
servers {
protocols h1 h2 h3
}
auto_https off
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}
# Reverse Proxy Configuration
# Reverse Proxy Domain: "62300f65-2a08-47ef-b1f7-d0e31bbd30c4"
*.edholm.cc {
tls /var/db/caddy/data/caddy/certificates/temp/64c1e555e19da.pem /var/db/caddy/data/caddy/certificates/temp/64c1e555e19da.key
@07356a3a-8362-4a3e-afd3-7f0b521a44e9 {
host jelly
}
handle @07356a3a-8362-4a3e-afd3-7f0b521a44e9 {
handle {
reverse_proxy 192.168.1.101:8096 {
}
}
}
}
import /usr/local/etc/caddy/caddy.d/*.conf
Auto HTTPS is off thats why there is no port 80.
Change the setting in general settings.
It needs to be on for the automatic http to https redirect even if you dont use lets encrypt.
@Meyergru:
Thanks for trying it out and your review. I can add some comments to your points:
QuoteToday I took the opportunity to try out Caddy reverse proxy instead of HAproxy, mostly because of a very specific problem with HAproxy...
I must say I reverted after trying it thouroughly. My 2cents on this are as follows:
Quote- Caddy is suited to home setups and inexperienced users. HAproxy is much more complex.
- For example, the certificate setup is much easier, because you just have to specify the domain and it just works (tm).
-> Perfect, that was the target audience. So I guess I hit the right spot if that was noticable. ;D They like how neatly integrated features like dynamic dns, dns providers and the dns01 challenge is for their certificates. Creating a new domain just takes a few clicks and it just worksTM. These features are mostly unnecessary for enterprises. Steps to set things up are kept to a minimum with minimal abstraction.
Quote- However, if you have more than just one domain, Caddy setup gets a little tedious:
* you have to create one domain/certificate plus a http backend for any domain, which includes creating different ones for www.domain.de and domain.de. You cannot combine certificates for multiple domains unless they are subdomains.
-> Thats the tradeoff of less abstraction. It means experts will find limitations here.
Quote* You do not have much control over what type of certificate(s) are created - you cannot specifiy strength or ECC vs. RSA (much less both) and I have not found a means to control if ZeroSSL vs. LetsEncrypt is used.
* The ciphers being employed cannot be controlled easily - or, for TLS 1.3, at all. That results in an ssllabs.com score which is suboptimal, becaus 128bit ciphers are allowed. This cannot be changed because of Go limitations.
-> That one is indeed for the Go developers to battle out with their userbase: https://github.com/golang/go/issues/29349
Though the defaults are okay, and an online test that gives a score can be misleading in this case. It probably comes down to company policies. Most of the discussion here always revolves about compliance.
Quote* You cannot use more than one type of DNS-01 verification if you use wildcard domains.
-> Limitation of the plugin, not of caddy. The plugin combines the same DNS provider for dynamic dns (which only allows one provider) and the dns01 challenge (which allows multiple providers). To make it simple and less confusing, the lowest denominator was chosen. So it is more suited for home/homelab environments or small businesses here that have one provider for their domains. I know of some people who use it to reverse proxy exchange servers for example, that works well even with outlook etc...
Quote* The Auto HTTPS feature looks nice first, but indeed it uses a 308 instead of a 301 code, which breaks some monitoring and can only be modified via custom include files.
-> That sounds like an easy fix if you can tell me what is needed there to change the default if needed.
QuoteSo, if you just want to reverse-proxy some services in your home network, go with Caddy. For an OpnSense guarding your internet site with several services/domains, stay with HAproxy.
-> That is a very sane conclusion and I mostly agree with it. Though the way the layer4/7 proxy and matcher ecosystem evolves makes it pretty powerful in its own way, if not only looking at the reverse proxy.
Quote from: Monviech on October 27, 2024, 08:29:56 PM
Quote* The Auto HTTPS feature looks nice first, but indeed it uses a 308 instead of a 301 code, which breaks some monitoring and can only be modified via custom include files.
-> That sounds like an easy fix if you can tell me what is needed there to change the default if needed.
That can be done via something like:
/usr/local/etc/caddy/caddy.d/redirect.conf:
http:// {
redir https://{hostport}{uri} 301
}
You may also need this to avoid conflicts:
/usr/local/etc/caddy/caddy.d/redirect.global:
auto_https disable_redirects
Would be nice to have a switch for this, because the custom files are not part of the saved configuration.
Quote from: Monviech on October 27, 2024, 08:29:56 PM
QuoteSo, if you just want to reverse-proxy some services in your home network, go with Caddy. For an OpnSense guarding your internet site with several services/domains, stay with HAproxy.
-> That is a very sane conclusion and I mostly agree with it. Though the way the layer4/7 proxy and matcher ecosystem evolves makes it pretty powerful in its own way, if not only looking at the reverse proxy.
HAproxy can do layer 4 proxying as well. I use this to translate IPv6 to IPv4 backends all the time, for example to access intranet docker services behind dynamic IPv6 prefixes (as docker is not easy to set up with IPv6, even less so with dynamic IPs).
I have asked a caddy maintainer about the redirect and since there is no global setting that changes the way "Auto HTTPS" creates them, I will not offer it.
It would also conflict with current features like the http01 challenge redirection that create a http domain when enabled.
https://github.com/opnsense/plugins/blob/5c4e3a231f6a46cef9d9ae9dc87d92b12ad97fb9/www/caddy/src/opnsense/service/templates/OPNsense/Caddy/Caddyfile#L280
Also it would need deduplication if multiple domains with the same hostname and different ports are created.
It would sadly get a little messy and convoluted in my opinion.
Quote from: Monviech on October 27, 2024, 06:56:52 PM
Auto HTTPS is off thats why there is no port 80.
Change the setting in general settings.
It needs to be on for the automatic http to https redirect even if you dont use lets encrypt.
Ah, thanks! That explains the lack of port 80! And indeed, checking that box immediately redirected http to https. However, perhaps the help text for this option should be slightly updated? It currently states:
QuoteSelect the Auto HTTPS option. "On (default)" creates automatic certificates using "Let's Encrypt" or "ZeroSSL". "Off" turns all automatic certificate requests off.
Something mentioning the redirect could be added?
Regardless, I am still running into the issue I had when prodding https before:
# curl -vL https://127.0.0.1:443
* Trying 127.0.0.1:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* OpenSSL/3.0.15: error:0A000438:SSL routines::tlsv1 alert internal error
* closing connection #0
curl: (35) OpenSSL/3.0.15: error:0A000438:SSL routines::tlsv1 alert internal error
The Let's encrypt TLS cert is setup inside OPNsense (this guide is probably the closest to what I have: https://sysadmin102.com/2023/05/create-lets-encrypt-wildcard-certificates-on-opnsense-with-acme-client/) in order to re-use the certificate elsewhere in the network. However, despite the plugin being setup to use this certificate it does not seem able to serve it:
<15>1 2024-10-28T23:20:18+01:00 OPNsense.edholm.cc caddy - - [meta sequenceId="11"] "debug","ts":"2024-10-28T22:20:18Z","logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"90.229.225.174","remote_port":"31331","server_name":"edholm.cc","remote":"90.229.225.174:31331","identifier":"edholm.cc","cipher_suites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"cert_cache_fill":0.0001,"load_or_obtain_if_necessary":true,"on_demand":false}
<11>1 2024-10-28T23:20:18+01:00 OPNsense.edholm.cc caddy - - [meta sequenceId="12"] "debug","ts":"2024-10-28T22:20:18Z","logger":"http.stdlib","msg":"http: TLS handshake error from 90.229.225.174:31331: no certificate available for 'edholm.cc'"}
<15>1 2024-10-28T23:27:38+01:00 OPNsense.edholm.cc caddy - - [meta sequenceId="1"] "debug","ts":"2024-10-28T22:27:38Z","logger":"events","msg":"event","name":"tls_get_certificate","id":"c99f387a-0cb1-4de9-af74-f764659607dd","origin":"tls","data":{"client_hello":{"CipherSuites":[52393,52392,49195,49199,49196,49200,49161,49171,49162,49172,156,157,47,53,49170,10,4867,4865,4866],"ServerName":"","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":null,"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"45.79.163.72","Port":65403,"Zone":""},"LocalAddr":{"IP":"192.168.1.1","Port":443,"Zone":""}}}}
<15>1 2024-10-28T23:27:38+01:00 OPNsense.edholm.cc caddy - - [meta sequenceId="2"] "debug","ts":"2024-10-28T22:27:38Z","logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"192.168.1.1"}
<15>1 2024-10-28T23:27:38+01:00 OPNsense.edholm.cc caddy - - [meta sequenceId="3"] "debug","ts":"2024-10-28T22:27:38Z","logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"45.79.163.72","remote_port":"65403","server_name":"","remote":"45.79.163.72:65403","identifier":"192.168.1.1","cipher_suites":[52393,52392,49195,49199,49196,49200,49161,49171,49162,49172,156,157,47,53,49170,10,4867,4865,4866],"cert_cache_fill":0.0001,"load_or_obtain_if_necessary":true,"on_demand":false}
<11>1 2024-10-28T23:27:38+01:00 OPNsense.edholm.cc caddy - - [meta sequenceId="4"] "debug","ts":"2024-10-28T22:27:38Z","logger":"http.stdlib","msg":"http: TLS handshake error from 45.79.163.72:65403: no certificate available for '192.168.1.1'"}
Does the logs above tell you anything? I don't get why it would not find a matching certificate, because the one I have is setup for "*.edholm.cc". What could I be missing for that not to work?
Another question for that matter, would it possible to add an option to include files under "caddy.d" inside a domain matcher? In my case it would create a Caddyfile like this:
*.edholm.cc {
# Options
import /usr/local/etc/caddy/caddy.d/*.conf
# Or perhaps lookup files with the configured domain as a prefix, like *.edholm.cc.conf
}
Mainly this is to make external automations able to add individual handlers as separate files for an existing domain. Benefit is that the plugin keeps track of the TLS cert and some other FW configured things, and, in my case, Ansible could still add configurations without requiring opening the webui manually. This should not break any existing plugin functionality, I think, except if you add files which handles the same subdomains as the plugin is set up to handle already.
Your *.edholm.cc certificate probably doesn't have a SAN for exactly edholm.cc.
Its only valid for subdomains under edholm.cc but not for the domain exactly.
Im not sure why it doesnt work for your setup though, it looks correct in the caddyfile if the cert is really for *.example.com. All manual wildcards I have used before, and also ones generated by Caddy itself always worked. If you have s supported DNS Provider just let caddy handle the wildcard domain generation.
------
If you want to change options remotely, use the REST API the plugin provides. Most modern OPNsense plugins and core features have this API.
e.g., in a browser call /api/caddy/reverse_proxy/get
https://docs.opnsense.org/development/api.html
Main advantage, everything you do will be validated and serialized into the /conf/config.xml so OPNsense backups contain it.
I'm working on getting Caddy set up on my OPNSense box, and running into some issues that are probably stupid user error from someone who's brand new to the featureset of OPNSense. Hoping someone can help.
My situation: I got Caddy set up with cloudflare pointing to my domain. Was able to obtain a wildcart cert from cloudflare. Set up my subdomains for Plex, Sonarr, HomeAssistant, and router GUI.
I'm getting 502 errors on all of them from the WAN, but can access Plex from the local network (the rest give 502). I can't play anything on Plex, but can see and interact with my dashboard.
As far as my setup goes, here's what I have:
Cloudflare DNS records:
ha.example.us. 1 IN A <my WAN IP>
plex.example.us. 1 IN A <my WAN IP>
*.example.us. 1 IN A <my WAN IP>
example.us. 1 IN A <my WAN IP>
router.example.us. 1 IN A <my WAN IP>
sonarr.example.us. 1 IN A <my WAN IP>
I set up the firewall rules as directed in the tutorial, with HTTP and HTTPS rules on both LAN and WAN.
Domain:
Protocol: https://
domain: *.example.us
Cert: ACME
DNS-01 Challenge: checked
Dynamic DNS: unchecked
Subdomains:
Domain: *.example.us
Subdomain: plex.example.us
Dynamic DNS: checked
Other services are set up the same.
HTTP Handlers:
Domain: *.example.us
Subdomain: plex.example.us
directive: reverse_proxy
Protocol: Https://
Upstream Domain: plex server's local IP
Upstream Port: 32400
TLS Insecure Skip Verify: checked
Other services are set up the same, but with correct subdomain, local IP, and port.
Here's a my caddyfile:
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file
# caddy_user=root
# Global Options
{
log {
output net unixgram//var/run/caddy/log.sock {
}
format json {
time_format rfc3339
}
}
servers {
protocols h1 h2 h3
}
dynamic_dns {
provider cloudflare <my token>
domains {
example.us plex
example.us router
example.us sonarr
example.us ha
}
}
email myemail@gmail.com
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}
# Reverse Proxy Configuration
# Reverse Proxy Domain: "14b010dd-6f6d-4ba8-94df-898d341059b8"
*.example.us {
tls {
issuer acme {
dns cloudflare <my token>
}
}
@85082da4-6d7a-410d-9955-d9c114e40692 {
host plex.example.us
}
handle @85082da4-6d7a-410d-9955-d9c114e40692 {
handle {
reverse_proxy 192.168.1.137:32400 {
transport http {
tls
tls_insecure_skip_verify
}
}
}
}
@fc7226be-505c-4b24-b95b-e01cb5e11a32 {
host router.example.us
}
handle @fc7226be-505c-4b24-b95b-e01cb5e11a32 {
handle {
reverse_proxy 192.168.1.1:8443 {
}
}
}
@e857ccbe-2f6a-4d4d-86f5-f8a5ab908e70 {
host sonarr.example.us
}
handle @e857ccbe-2f6a-4d4d-86f5-f8a5ab908e70 {
handle {
reverse_proxy 192.168.1.137:8989 {
transport http {
tls
tls_insecure_skip_verify
}
}
}
}
@fa0581ec-66ca-4526-ae6b-6a059d16f73c {
host ha.example.us
}
handle @fa0581ec-66ca-4526-ae6b-6a059d16f73c {
handle {
reverse_proxy 192.168.1.138:8123 {
}
}
}
}
import /usr/local/etc/caddy/caddy.d/*.conf
Any help here? Do I need to do any setup on my Plex/Sonarr/HA box? Forward ports 443/80? Is unbound DNS tripping me up somehow? Or did I just miss something else?
Did you move the OPNsense UI to a different port and disable the HTTP --> HTTPS redirect?
Yes I did. Sorry for forgetting to mention that
I think you have to configure "trusted proxies" to a list of cloudflare IPs in General Settings. (if you use it as CDN and not only as DNS provider)
There can also be cloudflare specific settings to be done at cloudflare itself I do not know about. I dont use it sorry. I think Cloudflare can itself be tje reverse proxy entry point for domains configured on it.
Check out what
curl -v example.com
returns from the outside.
Quote from: Gautier on October 11, 2024, 09:48:21 AM
Hi,
I install Caddy and configure follow the tutorial but I have error:
"error","ts":"2024-10-11T07:26:56Z","logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"toto.pequod.sokil.fr","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:connection - 89.219.181.98: Timeout during connect (likely firewall problem)"}
I really don't know where to start
I also on freeBSD and debian install caddy to test with the same error.
I have another site with OPNsense and caddy on debian behind without error, I miss something but what ?
Hi,
Just to say I just install opnsense with caddy on another ISP (another country) and it's work like a charm. I ask mu isp to help me but they send a pastry chef so ...
Anyway just to say thank you for your help.
Regards
Hey thanks for checking back in. Yeah ISPs are a big factor, some use CGNAT, some block port 80/443 etc..
Quote from: Monviech on November 04, 2024, 06:12:19 AM
I think you have to configure "trusted proxies" to a list of cloudflare IPs in General Settings. (if you use it as CDN and not only as DNS provider)
There can also be cloudflare specific settings to be done at cloudflare itself I do not know about. I dont use it sorry. I think Cloudflare can itself be tje reverse proxy entry point for domains configured on it.
Check out what
curl -v example.com
returns from the outside.
I'm only using cloudflare for DNS to access my home network. My server is inside the network. As far as cloudflare specific settings, to my knowledge there aren't any. I watched several youtube tutorials on setting up cloudflare as DDNS with OPNSense, but unfortunately, I couldn't find anything tied to your plugin (there's really next to no info I've found outside of this thread and a handful of posts you've replied to on Reddit)
Here's the result of the curl entry...
* Host sonarr.mydomain.us:80 was resolved.
* IPv6: (none)
* IPv4: <mywanip>
* Trying <mywanip>:80...
* Connected to sonarr.mydomain.us (<mywanip>) port 80
> GET / HTTP/1.1
> Host: sonarr.mydomain.us
> User-Agent: curl/8.9.1
> Accept: */*
>
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://sonarr.mydomain.us/
< Server: Caddy
< Date: Mon, 04 Nov 2024 17:52:26 GMT
< Content-Length: 0
<
* we are done reading and this is set to close, stop send
* shutting down connection #0
Well that looks good. Port 80 is caddy answering. Do the same curl for 443 too.
curl -v example.com:443
Yeah the plugin is still less than a year old so I dont expect any other user tutorials yet. Might take some more time. Thats why I wrote these big docs maybe somebody can PR a cloudflare tutorial sometime.
* Host sonarr.mydomain.us:80 was resolved.
* IPv6: (none)
* IPv4: <wanip>
* Trying <wanip>:80...
* Connected to sonarr.mydomain.us (<wanip>) port 80
> GET / HTTP/1.1
> Host: sonarr.mydomain.us
> User-Agent: curl/8.9.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://sonarr.mydomain.us/
< Server: Caddy
< Date: Mon, 04 Nov 2024 19:32:57 GMT
< Content-Length: 0
<
* shutting down connection #0
Looks the same as what I got on port 80, other than the "HTTP/1.1 308 Permanent Redirect" line. So if cloudflare is getting through to Caddy, any suggestions as far as what's blocking the request from getting through the firewall?
As for setting up cloudflare, assuming my current issues are on the firewall side, there's nothing special about the setup. You just have to create an A record for your root domain, or using * to get a wildcard cert. I didn't even create the DNS entries for my subdomains, cloudflare did it dynamically.
I think you have to curl -v https://example.com
So far we have just seen port 80 open but never port 443. If connections time out externally its 443 that could be the issue.
If 443 is indeed open and caddy answers and there are no timeouts then its weird.
Apparently my brain isn't turned on yet today. I didn't even notice that both tests were for port 80. Here's 443
curl -v sonarr.example.us:443
* Host sonarr.example.us:443 was resolved.
* IPv6: (none)
* IPv4: <wanip>
* Trying <wanip>:443...
* Connected to sonarr.example.us (<wanip>) port 443
> GET / HTTP/1.1
> Host: sonarr.example.us:443
> User-Agent: curl/8.9.1
> Accept: */*
>
* Request completely sent off
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad Request
<
Client sent an HTTP request to an HTTPS server.
* shutting down connection #0
Yeah well that works too, I would have tested with https://example.com instead of only using example.com:443 but this shows the port is open and theres at least some answer. It is not clear what answers though.
Just try to fiddle a bit around and see what responds. Im unable to do this step by step here in this thread. Also check out the tutorial section here that explains what should happen if things work.
Your Caddyfile looks totally fine so it has to be some networking issue.
https://docs.opnsense.org/manual/how-tos/caddy.html#help-nothing-works
I ran the curl command a few more ways and got some results that seem mildly interesting. Figured I'd repost them in case they're telling to you...
C:\Users\me>curl -v mydomain.us
* Host mydomain.us:80 was resolved.
* IPv6: (none)
* IPv4: <wanIP>
* Trying <wanIP>:80...
* Connected to mydomain.us (<wanIP>) port 80
> GET / HTTP/1.1
> Host: mydomain.us
> User-Agent: curl/8.9.1
> Accept: */*
>
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://mydomain.us/
< Server: Caddy
< Date: Mon, 04 Nov 2024 21:16:01 GMT
< Content-Length: 0
<
* we are done reading and this is set to close, stop send
* shutting down connection #0
C:\Users\me>curl -v https://mydomain.us
* Host mydomain.us:443 was resolved.
* IPv6: (none)
* IPv4: <wanIP>
* Trying <wanIP>:443...
* Connected to mydomain.us (<wanIP>) port 443
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.
* closing connection #0
curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.
C:\Users\me>curl -v https://www.mydomain.us
* Host www.mydomain.us:443 was resolved.
* IPv6: (none)
* IPv4: <wanIP>
* Trying <wanIP>:443...
* Connected to www.mydomain.us (<wanIP>) port 443
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.x
> GET / HTTP/1.1
> Host: www.mydomain.us
> User-Agent: curl/8.9.1
> Accept: */*
>
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
< HTTP/1.1 200 OK
< Alt-Svc: h3=":443"; ma=2592000
< Server: Caddy
< Date: Mon, 04 Nov 2024 21:19:35 GMT
< Content-Length: 0
<
* Connection #0 to host www.mydomain.us left intact
So it looks like a request to the bare domain responds over http/port 80, a bare request trying to force https results in a TLS handshake error, but a https request to www works. That's particularly interesting to me since I didn't set up a www entry in my DNS record.
Is there anything noteworthy you're seeing from those results?
I'm running through the troubleshooting steps and reviewing firewall rules etc to see if I can find something on the firewall end that's stopping the request from passing through to the client...
It really doesnt tell me mich, Ive never seen errors like these before.
Maybe Caddy Debug logs of the failed requests or HTTP access logs can show whats happening. Right now Im just as much in the dark as you though, sorry.
It looks like a weirder problem, maybe https://caddy.community can help better.
I am a noob and try to play arround with caddy in opnsense'
i have a question regarding certificates that are showing up in the lobby dashboard but not needed anymore. Is there a way to delete those certificates?
thanks in advance
Hello, these certificates should be eventually cleaned up by caddy's storage cleanup routine. But that can take till past expiry or longer.
You can delete them manually in the filesystem.
/var/db/caddy/data/caddy/certificates/
Hello, caddy doesn't seem to like the firewall live log view. Auto-refresh will not stay enabled. I don't have this problem when viewing logs via direct IP.
Caddy logs show this "context canceled" error when it happens:
"debug","ts":"2024-11-18T13:32:10Z","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.5.1:444","duration":0.001754159,"request":{"remote_ip":"192.168.5.4","remote_port":"59054","client_ip":"192.168.5.4","proto":"HTTP/2.0","method":"GET","host":"opn.example.com","uri":"/api/diagnostics/firewall/log/?digest=91a55b1d9ceb232a54b94da9ad86d84e&limit=1000","headers":{"Accept":["application/json, text/javascript, */*; q=0.01"],"Sec-Gpc":["1"],"Sec-Fetch-Mode":["cors"],"Accept-Language":["en-US,en;q=0.5"],"X-Csrftoken":["EzCqIYHuVYedZX-dW038qA"],"Sec-Fetch-Dest":["empty"],"Cookie":["REDACTED"],"X-Forwarded-For":["192.168.5.4"],"Referer":["https://opn.example.com/ui/diagnostics/firewall/log"],"Sec-Fetch-Site":["same-origin"],"Te":["trailers"],"Dnt":["1"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0"],"X-Requested-With":["XMLHttpRequest"],"Content-Type":["application/json"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["opn.example.com"],"Accept-Encoding":["gzip, deflate, br, zstd"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"opn.example.com"}},"error":"context canceled"}
What could be the problem?
Hello, I just tested the firewall live log with both the layer 4 proxy and the reverse proxy and auto refresh always worked for me.
My WebGUI runs on HTTPS and 4444.
Here is my Caddyfile section:
vpn1.example.com {
handle {
reverse_proxy 127.0.0.1:4444 {
transport http {
tls
tls_insecure_skip_verify
}
}
}
}
I wonder what I'm doing wrong then. My section looks similar to yours, with the addition of the certificate, I just have the WebUI bound to a specific interface:
handle {
reverse_proxy 192.168.5.1:444 {
transport http {
tls
tls_trust_pool file /var/db/caddy/data/caddy/certificates/temp/670931cbb863a.pem
tls_server_name OPNsense.localdomain
}
}
}
Can you try if the same issue happens with the layer4 proxy?
It's really easy, just add this to "Layer4 Proxy". You do not even have to disable your Reverse Proxy for the OPNsense Webgui configuration since Layer 4 will match first:
Services: Caddy Web Server: Layer4 Proxy
Edit Layer4 Route
Description: OPNsense WebGUI
Matchers: TLS (SNI Client Hello)
Domain: opnsense.example.com (Your opnsense domain name)
Upstream Domain: 127.0.0.1 (Or your IP address if the WebGUI does not listen on ALL interfaces)
Upstream Port: 444
Save, Apply.
Reopen the WebGUI and it should serve the self signed certificate through Caddy instead of the Lets Encrypt one. Test if the LiveLog still doesn't work.
Live logs work with that method. So is the issue somehow with the LE cert? I'd much prefer to use my LE cert for this.
Edit: it seems to be a browser issue. Live logs work fine with my original set up in Brave, but not librewolf. Strange.
It's the combination of the ResistFingerprinting setting and Caddy with LE certs that seems to be the problem, for me at least.
Never mind, was looking via direct IP and didn't notice.
Oh okay so things work now with a different browser? I tested with Blink based one, Chrome on Mac.
They do. I tried in normal Firefox and it also works fine there. So I'm guessing it's some hardening setting in Librewolf. Thanks for helping me narrow this down! I'll keep digging into the settings of LW.
I need some help with this great plugin. I want to use Caddy, as reverse proxy. I have in my local network, a immich-server running.
If I make the configuration with only http:// everthing works perfect. But i want to use it with https://.
I always get this error in the browser.
Fehlercode: SSL_ERROR_INTERNAL_ERROR_ALERT
Where do you want to use https?
Domain (Frontend) or HTTP Handler (Upstream to Backend)?
Hi all I need some help with the following:
I configured caddy with subdomains and using cloudflare api.
The problem is i get a lots of errors in my log ( see example and no its not my real ip)
"debug","ts":"2024-12-02T03:28:36Z","logger":"http.stdlib","msg":"http: TLS handshake error from 34.38.48.249:37638: no certificate available for '22.200.100.002'"}
Can someone point me in the rightdirection to block this?
Thank in advance
Any request that hits caddy on its ports will trigger an evaluation if it should be further processed or not. So these are not warning or errors, but a debug message that the frontend received a connection request that could not be mapped to any configured hostname (thus no available certificate).
If you think its an attack block the requesting IP via Crowdsec or Firewall rules.
Quote from: Monviech (Cedrik) on December 01, 2024, 12:49:11 PM
Where do you want to use https?
Domain (Frontend) or HTTP Handler (Upstream to Backend)?
Domain (Fronted), the backend has only http.
Have you checked the Caddy Certificate widget (dashboard) if there is a certificate when you enable https for the frontend? If not check the caddy logs why it fails to issue one.
Actually I tried to split port 443 in HAProxy. I couldn't find a working solution for my setup. While struggling around Cedrik gave me the hint to try Caddy. The idea behind HAProxy was to restrict access to the LAN and to present all certs to any clients or applications in the LAN. Connections from outside are only allowed through VPN. In this setup there's only a minimum of ports at the WAN interface open.
The main reason for port-sharing is that more and more wifi's in hotels or airports have only two ports open. As long as there's no deep packet inspection, one could use port 443 for openvpn. In other enviroments wireguard maybe a good choice.
Here are the steps sharing the port 443 between openvpn and a web application running on https, which are working for me. As pre-requisites there are (up-to-date)
OPNsense 24.7.9_1-amd64
FreeBSD 14.1-RELEASE-p6
OpenSSL 3.0.15
- all DNS records setup at the ISP/DNS registrar
- all (let's encrypt) certificates are stored at the correct local places and up-to-date
- there's a user created for openvpn
- local certificates have been created for the vpn-server and the vpn-client (user)
- there's a VPN instance up and running bound to 127.0.0.1 on port 1194
a) in Caddy - general settings - enable caddy and layer4 proxy. Advanced, Log, DNS, etc. are left on default.
b) in reverse proxy - http access - create your acl. I allow access only to LAN and VPN. HTTP response code for me is 403, the message is "HTTP 403 - Forbidden"
c) in reverse proxy - Domains - create your web-application on port 443 (https). Don't forget the corresponding certificate and the access list to this application.
d) in reverse proxy - http handlers - create the web-application which belongs to step c). Handler is "handle", leave path to "any", directive is "reverse_proxy", leave http version on default, protocol is "https", define your upstream domain/IP on the upstream port 443. Leave upstream path empty. Change the TLS server name that matches the SAN "Subject Alternative Name" of the offered upstream certificate.
e) in layer4 proxy - leave/change routing type "listener_wrappers", protocol is TCP, local port leave empty, matchers is "openvpn", mode and key is "any", upstream domain is "127.0.0.1", upstream port is 1194. Leave the rest empty/on default.
Connect your roadwarrior through port 443 to the openvpn instance. I used for client export "file only".
That's it. Working at least for me. If there are questions with this setup, I'll try to help. I had to start over for a second try. The first approach didn't work as expected. While re-installing (I removed every leftover from caddy via cli) it worked in the way I described. This time I was better prepared and didn't change or alter any setting while configuring caddy. Be sure to have all pre-requisites working as they should. Then start configuring caddy.
I can't push the DNS through the linux client (not working with WIN-clients), access to the LAN apps works only with IP's. Or connecting via vnc to a machine in the LAN. I can live with that. Or maybe someone is able to rule this out.
regards,
stefan
P.S. thank's to cedrik - all credits to him
Hello,
I'm quite new to OPNsense. I installed it on N100 box. It replaces an unifi udr.
So far I managed to replace udr successfully even for Wireguard server and a site2site openvpn. The worst part had been to transfer unifi settings from udr to network controller as it's unsupported by unifi.
And I'm very happy with that. My connection is much more stable and consistent. I let you guess when the replacement occurred on the screenshot
Now I'd like to replace my nginx proxy manager (hosted on an Unraid docker container) with caddy on opnsense.
Does caddy support reverse proxying hosts using web sockets like home assistant, jeedom or Plex ?
Yes, definitely.
Thanks. I'll try that.
I see that acme plugin is not necessary. I already setup acme plugin for issuing a certificate for opnsense webgui. It's not publicly available so I guess it's normal to not have record in my dns zone (domain is registered at ovh and so is the zone)
Can it live along with caddy plugin ?
if I use caddy with dns challenge and all keys correctly filled in does it take care of issuing certificates as well as creating cname records ?
Can I delete cname records already made for nginx prox manager ?
i have a domain and use subdomains for publishing apps like home assistant.
Ovh dynamic dns client doesn't handle ipv6 is there a way to make apps available also in ipv6?
edit: quick followup
it's setup. Certificates are issued and the widget on dashboard is very convenient.
I hadn't payed attention to dyndns but in my case not very useful for caddy. I use it for my domain ip and all my published apps are subdomains so setting up a cname record in my provider webui is more convenient than adding a dyndns for each subdomain.
For the moment I can't use ipv6. I suppose if I want to I would have to transfer the zone to Cloudflare.
Hi everyone,
I just setup caddy and it works totally fine - for one domains.
The second domain always fails on the external interface.
I have two domains:
image.example1[.]com
image.example2[.]com
Both point to the same reverse proxy (caddy).
image.example1[.]com - works totally fine. EDIT: Doesnt work anymore. Only when this DNS resolves to internal interface of FW
image.example2[.]com - results in a server connect error.
If I test it from my internal network, setting up a local DNS resolver pointing image.example2[.]com to the internal interface of my FW, it works totally fine.
So the firewall accepts the requests for this hostname / domain. Internally it works. Externally it fails.
Duplicating the caddy configuration from example2 domain to example1 (which works with with multiple sites) it works totally fine.
There must be something wrong with the domain. But I have no idea why. I verified it multiple times in my local DNS, my public DNS etc.
Any input is highly appreciated! I am totally lost on this one.
Tom
EDIT: Now the same hostname on both different domains doesnt work externally.
having DNS for image.example1[.]com and image.example2[.]com pointing to internal interface of FW - All good!
having DNS for image.example1[.]com and image.example2[.]com pointing to external interface of FW - Cant connect to server!
This is the config.
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file
# caddy_user=root
# Global Options
{
log {
output net unixgram//var/run/caddy/log.sock {
}
format json {
time_format rfc3339
}
}
servers {
protocols h1 h2 h3
}
email security@example-1.com
auto_https off
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}
# Reverse Proxy Configuration
# Reverse Proxy Domain: "79179cc0-6ab4-4a49-9d79-5d58cf46062a"
drive.example-1.com {
tls /var/db/caddy/data/caddy/certificates/temp/65d767a6b7244.pem /var/db/caddy/data/caddy/certificates/temp/65d767a6b7244.key
handle {
reverse_proxy 192.168.1.110:10003 {
transport http {
tls
tls_insecure_skip_verify
}
}
}
}
# Reverse Proxy Domain: "cfb6ccd1-1006-453b-92f6-1d449b125935"
data.example-1.com {
tls /var/db/caddy/data/caddy/certificates/temp/65d767a6b7244.pem /var/db/caddy/data/caddy/certificates/temp/65d767a6b7244.key
handle {
reverse_proxy 192.168.1.110:5001 {
transport http {
tls
tls_insecure_skip_verify
}
}
}
}
# Reverse Proxy Domain: "03f9ea48-f45d-4609-9df8-01bdba6806ed"
file.example-1.com {
tls /var/db/caddy/data/caddy/certificates/temp/65d767a6b7244.pem /var/db/caddy/data/caddy/certificates/temp/65d767a6b7244.key
handle {
reverse_proxy 192.168.1.110:7001 {
transport http {
tls
tls_insecure_skip_verify
}
}
}
}
# Reverse Proxy Domain: "0a5d6560-cd3b-44a3-8d34-fd86f6e7b37e"
photos.example-1.com {
tls /var/db/caddy/data/caddy/certificates/temp/65d767a6b7244.pem /var/db/caddy/data/caddy/certificates/temp/65d767a6b7244.key
handle {
reverse_proxy 192.168.1.110:5003 {
transport http {
tls
tls_insecure_skip_verify
}
}
}
}
# Reverse Proxy Domain: "a76af138-0c5c-453d-80d0-e21bd176672a"
photos.example-2.my {
tls /var/db/caddy/data/caddy/certificates/temp/65d767a6b7244.pem /var/db/caddy/data/caddy/certificates/temp/65d767a6b7244.key
handle {
reverse_proxy 192.168.1.110:5003 {
transport http {
tls
tls_insecure_skip_verify
}
}
}
}
import /usr/local/etc/caddy/caddy.d/*.conf
If they all use the same wildcard certificate you must use one wildcard domain with the certificate and the rest subdomains.
https://docs.opnsense.org/manual/how-tos/caddy.html#wildcard-domain-with-subdomains
In your case ignore DNS Provider and DNS challenge in that document.
Please ignore the below: I removed all configuration and recreated from scratch - now both certificates is present
Hi, I am new to this plugin but succeeded to setup a test domain yesterday. After testing I disabled the http and https fw port rules. Today I wanted to add a subdomain but forgot to enable access to the web ports. After reopening I have tried to restart caddy, delete and recreate several subdomains but no joy. I seems that the the certificate is still from yesterday and has no subdomains.
Is there anyway to trigger a new attempt?
I currently use a DNS provider that is not supported by your plug-in (I'm not sure if my provider have any API support at all), so I guess that it not possible to use DNS Challenge?
Up to now (using certbot) I have just added all my subdomains to Subject Alt names on the main domain. Is it possible to use this plugin with Subject Alt names, or do I have to create a "Domain" for every sub domain and ignore the subdomain section in the user interface?
Hi,
I am fairly new to this topic and I only need caddy as a reverse proxy. My Dyn-DNS Provider (ipv64.net) is not listed in the app and so I really don't know what to do. I've tried the acme-plugin and I was able to get a proper wildcard certificate for my website with the dns challenge.
I can choose this certificate under certificate in the reverse proxy tab. But on the dashborad there is a message that there is no certificate handled by caddy. So is this okay?
I think it would be better to get rid of the acme-plugin and manged everything with the caddy plugin, but how can I configure this?
hi all,
Can somebody help me with the following. I want to understand if its possible within caddy to restrict/block a certain url like e.g. abc.def.com/xyz and leave abc.def.com accassible. what would be the best way to config this.
thanks in advance
This here explains it. Enable advanced mode in a handler to see all options like "path"
https://docs.opnsense.org/manual/how-tos/caddy.html#multiple-http-handlers-for-the-same-domain
@Cedrik, First off, thank you so very much for creating this (and other) wonder plugins for OPNsense. You make our lives so convenient.
I've read through plugin configuration documentation (https://docs.opnsense.org/manual/how-tos/caddy.html) and through this thread. But, an answer didn't jump out. So, I hope you could excuse me for bothering you with a little issue of mine. TLDR: If os-caddy plugin acts as an ACME server for a site, then a) what would be the ACME url? b) will the location of root ca be /var/db/caddy/data/caddy/certificates/local/<site>?
Presently, I use caddy cascaded (https://caddy.community/t/use-caddy-for-local-https-tls-between-front-end-reverse-proxy-and-lan-hosts/11650) in my internal network with a made-up domain name TLS'ed by Caddy. I did so prior to having an externally valid domain name - which I now do. Previously working caddy setup at a high-level was:
Caddy master instance (say on internal1.domain)
{
acme_server
tls_internal
}
Caddy slaves instances (say on internal2.domain)
https://internal2.domain{
tls {
ca https://<internal1.domain>/acme/local/directory
ca_root <path_to_caddy_master_ca>.pem
}
}
Caddy slaves instances (say on internal3.domain)
https://internal3.domain{
tls {
ca https://<internal1.domain>/acme/local/directory
ca_root <path_to_caddy_master_ca>.pem
}
}
... and so on
Now, with os-caddy on Opnsense being available, I'd like to rid of Caddy master on <instance1.domain> and utilize Opnsense caddy plug-in instead. GUI doesn't provide an option to declare acme_server, tls_internal. But inclusion of *.conf on a per-site basis is allowed. So, I added it like so: (reverse proxied external domain to internal2.domain running caddy slave1)
# Reverse Proxy Domain: "104d6421-2b1f-407e-af83-f087021ee1b1"
valid.domain {
log {
output file /var/log/caddy/access/104d6421-2b1f-407e-af83-f087021ee1b1.log {
roll_keep_for 10d
}
}
acme_server
tls internal
@08831d42-ad4c-40dc-a7d5-53af52ac6490_validdomain {
not client_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
}
handle @08831d42-ad4c-40dc-a7d5-53af52ac6490_validdomain {
abort
}
handle {
reverse_proxy internal2.domain {
transport http {
tls_insecure_skip_verify
}
header_up Host {http.reverse_proxy.upstream.hostport}
}
}
}
Then Caddy slaves instances's (internal2.domain) config is changed as:
https://internal2.domain{
tls {
ca https://<external.domain>/acme/local/directory
ca_root <path_to_caddy_master_ca copied from /var/db/caddy/data/caddy/certificates/local/internal1.domain>.crt
}
}
Unbound resolves external.domain to Opsense's address (192.168.0.1) via host overrides. This is to prevent local request for external domain being handled and served locally without having to go out into the interwebs.
Because Opnsense webGUI is running on 8443, I also tried https://<192.168.0.1>:8443/acme/local/directory but to no avail. ACME server's url appears invalid. I'm unsure how to proceed. Your guidance would be much appreciated.
You could use the HTTP01 challenge redirection on the Caddy Server of OPNsense to the cascaded other Caddy Servers. All Caddy Servers listen per default on 80/443 so nothing speciaö needs to be configured. Its what I do.
No need for ACME Server directive.
https://docs.opnsense.org/manual/how-tos/caddy.html#redirect-acme-http-01-challenge
In each domain the IP for the Challenge redirection wouöd be the backend Caddy responsible for it.
Quote from: Monviech (Cedrik) on January 07, 2025, 07:20:58 AMYou could use the HTTP01 challenge redirection on the Caddy Server of OPNsense to the cascaded other Caddy Servers.
I see. I was under the impression that the challenges on cascades servers would work because they get their certs (and therefore a hierarchy) from cascade master. Meaning, Caddy opnsense has, for a given external site, a valid cert. If acme challenge is sent to an internal (hosting an internal site) caddy instance, then the request would not have the same valid verification chain on client-side? Please grant me follow-ups in case I get stuck.
An other unrelated questions for your guidance:
GitHub code for this plugin indicates a custom caddy build (with DNS providers, L4 etc) is being used (120 standard modules, 80 or so optional modules). Would there be steps on how to add modules of interest eg crowdsec? I have used xcaddy in the past to generate custom images on Linux but not on freebsd. I can update this post with a working caddy config where crowdsec together with rate L5-7 rate limiters can take L4-7 information (not just L3 - which is what I think present crowdsec integration provides) to block unwanted behaviors.
If Caddy(Main) and Caddy(Sub) are in a trusted network, you can reverse_proxy between these Caddies without using TLS.
TLS is only important on the way through untrusted networks, e.g. from Client on the Internet to your Caddy(Main).
Caddy(Main) will hold all certificates and terminate tls. The other Caddies would not need to issue any certificates.
So you do not really need an ACME Server or a ACME Challenge redirection. Just use Plaintext.
Regarding the build:
https://caddyserver.com/docs/command-line#caddy-add-package
You can use this to add any package you want from the command line. It will not be persistent though. If the opnsense repo pushes an update at some point you must do it again.
Currently the plugin is rather finished and very specific or overly complicated things will most likely not be added to prevent feature creep.
Quote from: Monviech (Cedrik) on January 07, 2025, 05:52:10 PMIf Caddy(Main) and Caddy(Sub) are in a trusted network, you can reverse_proxy between these Caddies without using TLS.
TLS is only important on the way through untrusted networks, e.g. from Client on the Internet to your Caddy(Main).
Caddy(Main) will hold all certificates and terminate tls. The other Caddies would not need to issue any certificates.
So you do not really need an ACME Server or a ACME Challenge redirection. Just use Plaintext.
I understand. Perhaps a better qualification of one of my use-case is in order.
- Telegraf, Influx and Grafana stacks are employed for telemetry. Latter two are web-based interfaces which require explicit certificate configuration in order to use http(s). Telegraf client configurations need root_ca for http(s) and host-specific keys if TLS verification option is enabled.
- Now, first-order question I asked myself was: Do I really need to protect telemetry information from telegram clients to influx db server? Not really : is the honest answer. That said, telemetry information reveals plenty about topology. So, having telemetry data streamed using TLS protection does carry benefits
- The certs (for external domain) will now reside on Caddy(master). With ACME server enabled on Caddy, I can have certbot request the necessary certs and auto-provision (e.g. during cert renewal) certs onto appropriate roles
Reference telegraf (https://github.com/influxdata/telegraf/blob/master/plugins/outputs/influxdb_v2/README.md) client information required:
## Optional TLS Config for use on HTTP connections.
# tls_ca = "/etc/telegraf/ca.pem"
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
## Use TLS but skip chain & host verification
# insecure_skip_verify = false
Similar information is required for influx and Grafana.
Quote from: Monviech (Cedrik) on January 07, 2025, 05:52:10 PMRegarding the build:
https://caddyserver.com/docs/command-line#caddy-add-package
You can use this to add any package you want from the command line. It will not be persistent though. If the opnsense repo pushes an update at some point you must do it again.
Thank you for the reference. I understand.
The
crowdsec usage I referred to earlier is as follows:
--> Global block of Caddyfile (info generated using cscli bouncer add
{
crowdsec {
api_url http://<Opnsense_fw>:8080
api_key <valid_key>
ticker_interval 15s
}
}
--> In the site-block of clients
{
route {
# crowdsec based filtering
crowdsec
... whatever logic is necessary ...
}
}
Quote from: Monviech (Cedrik) on January 07, 2025, 05:52:10 PMCurrently the plugin is rather finished and very specific or overly complicated things will most likely not be added to prevent feature creep.
Thank you for sharing the plugin status and roadmap. I only asked about crowdsec (https://caddyserver.com/docs/modules/crowdsec) modules (which can be used like above) because a log-based integration of crowdsec is already implemented on your plugin for Opnsense. Addition of the few extra parameters required may increase the hardening posture.[/list]
Hi,
I'm currently using the Nginx Proxy Manager and I'm trying to switch to the Caddy plug-in.
Originally I was getting errors with Let's Encrypt and fixed that issue but I'm still unable to access anything behind the reverse proxy. All I'm seeing in the logs is this:
"info","ts":"2025-01-12T06:13:00Z","logger":"http.log.access","msg":"handled request","request":{"remote_ip":"127.0.0.1","remote_port":"12807","client_ip":"127.0.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost","uri":"/scrape.php?v=6&url=https://www.spamhaus.org/drop/asndrop.json","headers":{"User-Agent":["python-requests/2.32.3"],"Accept-Encoding":["gzip, deflate"],"Accept":["*/*"],"Connection":["keep-alive"]}},"bytes_read":0,"user_id":"","duration":0.000013514,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://localhost/scrape.php?v=6&url=https://www.spamhaus.org/drop/asndrop.json"],"Content-Type":[]}}
I'm not exactly sure what the problem is. All suggestions are welcome.
Thank you!
That looks like a configuration issue. localhost tries to reach spamhaus on lovalhost? Really weird. Without Caddyfile I dont know whats going on.
Here's my caddy file. I've removed the personal info. I hope this helps.
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file
# caddy_user=root
# Global Options
{
log {
output net unixgram//var/run/caddy/log.sock {
}
format json {
time_format rfc3339
}
}
servers {
protocols h1 h2 h3
log_credentials
}
dynamic_dns {
provider duckdns xxx-xxxxx-xxxx
domains {
mysite.duckdns.org *
mysite.duckdns.org ab
}
versions ipv4
update_only
}
email myemail@domain.com
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}
# Reverse Proxy Configuration
# Reverse Proxy Domain: "xxx-xxx-xxxxx"
*.mysite.duckdns.org {
tls {
issuer acme {
dns duckdns {
api_token xxx-xxx-xxxxx
}
}
}
@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx {
host ab.mysite.duckdns.org
}
handle @xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx {
handle {
reverse_proxy 192.168.x.xxx {
}
}
}
}
import /usr/local/etc/caddy/caddy.d/*.conf
@awshirley please place code inside code tags like so:
This
is
code
I see nothing wrong with the config at first glance so it remains a mystery that must be solved in the infrastructure it happens.
Maybe check the opnsense docs for the troubleshooting guide for caddy or open a new thread in the caddy community or here in the forum so somebody can pick it up. Its probably an infrastructure issue in some way.
Thank you for reviewing my caddy file. You pointed me in the right direction, and infrastructure issue. I had port forwarding turned on for 40 and 443 to point to NPM. I turned it off and Caddy now has the ports.
Thanks again!
I'm so stoked for this plugin! I'm migrating from pfSense and caddy running in docker and I currently rely pretty heavily on the caddy-security, https://github.com/greenpau/caddy-security, plugin. Would you consider adding this module in the caddy build included here? Not sure if there is a formal request route for this or not so let me know. I've added the caddy-security module myself w/ the add-package param but I'm guessing it will get nuked with the next update.
Also, I'm doing some snippet imports in the global block due to my use of caddy-security and there is no method to import custom configs before the global block. I added an 'import /usr/local/etc/caddy/caddy.d/*.snippet' line above the global block in the auto generated "DO NOT EDIT THIS FILE" Caddyfile to work around this but I suspect that will get nuked at some point as well. I'm betting I'm probably no where near best practice here so no sweat if it doesn't make sense but if there are use cases to load custom configs/snippets ahead of the global block it would be cool to have an import line included in the auto generated Caddyfile.
Can't express how grateful I am for this... awesome work!
Hey there,
I will not add the caddy security package or make it configurable in the GUI. I suggest you use forward_auth instead with the supported Auth Providers in the plugin.
https://docs.opnsense.org/manual/how-tos/caddy.html#forward-auth
This method is more lightweight and flexible and there are no known issues.
Quote from: Monviech (Cedrik) on January 19, 2025, 06:29:36 AMHey there,
I will not add the caddy security package or make it configurable in the GUI. I suggest you use forward_auth instead with the supported Auth Providers in the plugin.
https://docs.opnsense.org/manual/how-tos/caddy.html#forward-auth
This method is more lightweight and flexible and there are no known issues.
All good, I'm currently using Organizr and the caddy-security plugin for my forward_auth needs so I figured it wouldn't hurt to ask. I've got Authentik running now and although it's a bit overkill for my homelab needs it does the trick. Thanks again for making this happen!
Hey good to know you got it to work with Authentik.
Yeah I just got to be careful with how much to include and caddy-security seems to have some history.
Quote from: Monviech (Cedrik) on January 31, 2025, 05:15:27 PMHey good to know you got it to work with Authentik.
Yeah I just got to be careful with how much to include and caddy-security seems to have some history.
Totally understand! Admittedly, I was not aware of the caddy-security plugin history.
Quote from: Monviech (Cedrik) on January 31, 2025, 05:15:27 PMHey good to know you got it to work with Authentik.
Yeah I just got to be careful with how much to include and caddy-security seems to have some history.
Admitedly, I was not aware of the history with the caddy-security plugin but dug this up... https://github.com/greenpau/caddy-security/issues/349. Dropping it here for future reference/context. Totally get the need to keep the risk profile low w/ the included caddy build. All good!
Quote from: smoofus on January 31, 2025, 04:31:27 PMQuote from: Monviech (Cedrik) on January 19, 2025, 06:29:36 AMHey there,
I will not add the caddy security package or make it configurable in the GUI. I suggest you use forward_auth instead with the supported Auth Providers in the plugin.
https://docs.opnsense.org/manual/how-tos/caddy.html#forward-auth
This method is more lightweight and flexible and there are no known issues.
All good, I'm currently using Organizr and the caddy-security plugin for my forward_auth needs so I figured it wouldn't hurt to ask. I've got Authentik running now and although it's a bit overkill for my homelab needs it does the trick. Thanks again for making this happen!
Hey,
sorry for dropping in, but could you give me more Feedback on how you got Authentik running with Caddy ? I'm struggeling to get it running with setting Authentik as Forward Auth.
br
Schubdog
Quote from: schubdog on February 09, 2025, 08:39:16 PMHey,
sorry for dropping in, but could you give me more Feedback on how you got Authentik running with Caddy ? I'm struggeling to get it running with setting Authentik as Forward Auth.
br
Schubdog
Happy to help if I can. Are you having issues with the authentik config or the caddy Auth Provider config?
Caddy Auth Provider config:
Forward Auth Provider: Authentik
Protocol: http://
Forward Auth Domain: ip or domain of host running authentik
Forward Auth Port: 9000
Forward Auth URI: /outpost.goauthentik.io/auth/caddy
Copy Headers: Select any headers you need to forward. Most my stuff needs the X-Authentik-Username, X-Authentik-Groups and X-Authentik-Email headers
I think the auth port and uri are default and should work unless you changed them when configuring authentik. You can verify by going to Applications -> Providers in authentik, select the forward auth provider for your app and then click on the "Caddy(Standalone)" section under the setup section. You will see the # forward authentication to outpost snippet with the needed port and uri. Mine looks like this...
# forward authentication to outpost
forward_auth http://outpost.company:9000 {
uri /outpost.goauthentik.io/auth/caddy
High level Authentik config:
Create a new application and proxy provider (I typically use the wizard and then tweak if needed after)
Application:
Nothing of note to call out here
Provider:
Select 'Proxy Provider'
Choose the 'implicit' Authorization flow
Select the "Forward auth (single application) box
Add the url of the app you want to forward auth to
Outposts:
Edit the authentik Embedded Outpost
Pick your application on the left and add it to the outpost with the >
Save
Caddy Handler:
When configuring the handler enable advanced mode, upper left, and tick the 'Forward Auth' setting
copy headers can be left empty, the needed headers are added automatically by the template generation. Its just for additional headers like "Authorization" when Authentik sends a Basic Auth Header for example.
Hey,
thank your for the reply. I got it now worky partly, so i can access my proxied Domains secure by Authentik from outside my lan.
But there are two things i'm still struggeling.
1. Bypass with API. I'm used to work with Authelia and was able to set bypass rules with API Keys for Vaultwarden and notifiarr but i don't get it working with Caddy and Authentik
2. How can i access my proxied domain in Lan ?
br
Schubdog
Quote from: schubdog on February 13, 2025, 10:14:29 PMHey,
thank your for the reply. I got it now worky partly, so i can access my proxied Domains secure by Authentik from outside my lan.
But there are two things i'm still struggeling.
1. Bypass with API. I'm used to work with Authelia and was able to set bypass rules with API Keys for Vaultwarden and notifiarr but i don't get it working with Caddy and Authentik
2. How can i access my proxied domain in Lan ?
br
Schubdog
For #1, go to Applications -> Providers -> (edit the application proxy prorivider for the app you want to bypass api) -> Advanced protocol settings -> Unauthenticated Paths, and then add the api path to the "Unauthenticated Paths" field.
For example:
Unauthenticated Paths: */api/.*
For #2, not sure I follow 100% here. Maybe add local dns entries to point to your wan ip?
Great plugin (once I figured it out!)
I'm not a networking expert, more of a homelabber. I have OPNSense setup with Adguard Home and Unbound with DNS over TLS.
I was having some trouble getting the Caddy access lists working to restrict some services to my LAN IPs only. To get this all working I had to setup overrides in Unbound that point these URLs back to my Caddy when on my LAN. i.e. I setup example.website.com in Caddy and then in Unbound I had to setup an override to point this URL back to 192.168.1.1 (where Caddy is running on my opnsense router).
My assumption was that because they were encrypted with DNS over TLS that the Caddy reverse proxy can't intercept them?
I think the only other way to get Caddy working with this setup was using the layer4 proxy? I took a look but the options and setup was just a bit too confusing for me. Downside is I just need to setup an override for every service but it really isn't that bad.
Just posting in case anyone has feedback or other ideas here. I was hoping to not require setting these up but it works now.
If there is any feedback on the layer4 proxy with my setup or another way to avoid the overrides in Unbound, I'd love to hear it!
I need assistance with two Caddy issues
1. I started using Caddy for a reverse proxy last week. Once I got it work, I started having issues when it had been running for 6 - 12 hours. The reverse proxy was working and then it would stop. I didn't see anything in the logs that indicated what the problem is. I'm not using a dyndns service.
2. I followed the instructions to use Caddy as a Layer 4/7 proxy for SSH. When trying to SSH in, all I got was a message stating the connection had been reset. I couldn't log into SSH. I didn't see anything unusual in the logs.
I can post any config or logs that are needed.
Thanks!
If you use UDP (only UDP matters) in the layer 4 proxy right now there is an open issue that can make it crash after a while:
https://github.com/mholt/caddy-l4/issues/295
For SSH, last time I tested it was working. Don't know what could be the issue if its like in the docs.
Thank you for the info on the UDP bug. I don't recall seeing and option to turn UDP off for Layer 4 to work properly.
You just don't define any UDP rules in the layer4 proxy and you should be fine (for now until the bug is fixed upstream).
Though you can take a look at /var/log/caddy/caddy.log to see if you can find any panics in there.
Hi Cedrik, apologies if this is answered elsewhere (I didn't find it)-
I've changed the Caddy user to 'www' from 'root' and set the ports to 8080 and 8443 respectively. Issue now is that I am getting permission errors in the log. Example:
2025-03-14T23:03:05-04:00 Error caddy "error","ts":"2025-03-15T03:03:05Z","logger":"admin.api","msg":"request error","error":"loading config: loading new config: starting caddy administration endpoint: unable to set permissions (--w--w----) on /var/run/caddy/caddy.sock: chmod /var/run/caddy/caddy.sock: operation not permitted","status_code":400}
I've stopped/started the Caddy service and also rebooted OPNsense but it didn't fix it. Should I need to change some permissions manually on the filesystem?
Its weird to see that error it should have been fixed here:
https://github.com/opnsense/plugins/pull/4403
Which version of the plugin do you use?
I have plugin version 1.8.3. I did a fresh install of OPN 25.1 and upgraded along the way to 25.1.3, and then installed os-caddy.
Well you should be able to delete the socket file, it will get recreated automatically when caddy starts.
I gonna see if there are issues next week.
Thanks- I deleted the file and don't see the error now after bouncing the service.
Here's the 'before' and 'after' state:
root@firewall:~ # cd /var/run/caddy/
root@firewall:/var/run/caddy # ls -l
total 2
-rw-rw---- 1 root www 6 Mar 14 23:00 caddy.pid
s-w--w---- 1 root www 0 Mar 14 23:00 caddy.sock
srw-rw-rw- 1 root www 0 Mar 14 22:57 log.sock
root@firewall:/var/run/caddy #
root@firewall:/var/run/caddy # rm caddy.sock
(bounced the service here)
root@firewall:/var/run/caddy # ls -l
total 2
-rw------- 1 www www 6 Mar 15 07:32 caddy.pid
s-w--w---- 1 www www 0 Mar 15 07:32 caddy.sock
srw-rw-rw- 1 root www 0 Mar 14 22:57 log.sock
I'm getting an error in the Caddy log when trying to use the reverse proxy on a Plex instance. The log shows:
"error","ts":"2025-03-19T23:36:47Z","logger":"http.log.error","msg":"EOF","request":{"remote_ip":"192.168.x.xxx","remote_port":"50589","client_ip":"192.168.x.xxx","proto":"HTTP/2.0","method":"GET","host":"plexsub.mydomain.com","uri":"/media/providers?X-Plex-Product=Plex%20Web&X-Plex-Version=4.145.1&X-Plex-Client-Identifier=y1574g5pgysu0b7435g9qsqd&X-Plex-Platform=Firefox&X-Plex-Platform-Version=136.0&X-Plex-Features=external-media%2Cindirect-media%2Chub-style-list&X-Plex-Model=bundled&X-Plex-Device=Windows&X-Plex-Device-Name=Firefox&X-Plex-Device-Screen-Resolution=1536x731%2C1536x864&X-Plex-Token=TWeNgtGispep-E4RBR1m&X-Plex-Language=en&X-Plex-Session-Id=72ff17fc-21db-4b3b-8437-9194ca66bd7d","headers":{"Referer":["http://192.168.x.xxx:32400/"],"Accept-Encoding":["gzip, deflate, br, zstd"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:136.0) Gecko/20100101 Firefox/136.0"],"Accept-Language":["en"],"Dnt":["1"],"Sec-Fetch-Site":["cross-site"],"Accept":["application/json"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["cors"],"Sec-Gpc":["1"],"Te":["trailers"],"Origin":["http://192.168.x.xxx:32400"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"plexsub.mydomain.com"}},"duration":0.000950754,"status":502,"err_id":"kr1iycyqd","err_trace":"reverseproxy.statusError (reverseproxy.go:1373)"}
Plex is stating that remote access through the reverse proxy doesn't work. Is this something easily fixed?
TWIMC Just warning here that DNS Providers might get a bit more inconvenient soon.
https://github.com/opnsense/plugins/issues/4643
Only cloudflare will remain default compiled in as it is maintained directly by the caddy organization. All other providers will be optionally installable via CLI with e.g.
caddy add-package github.com/caddy-dns/duckdns
https://caddyserver.com/docs/command-line#caddy-add-package
If they won't compile after the caddy binary is updated to caddy-v2.10.0 please reach out to their maintainers via https://github.com/caddy-dns
This had happened once already and I don't want to run after 40 repos with for something I don't even use personally (I dont use dns-challenge or dynamic-dns, I maintain this in my free time)
JFYI- the earlier issue about caddy.sock file being owned by root and causing errors when using the 'www' user in Caddy can still be seen. Yesterday I re-installed OPNsense and imported my config, then installed the Caddy plugin. Because of my configs the user was preset to 'www' but the sock file was owned by root and needed to be deleted manually.
Im a little surprised, can you find out the exact spot where it happens so it can be reproduced?
Please take this as reference what was implemented to mitigate it and the discussion:
https://github.com/opnsense/plugins/pull/4403
If there is still an issue please open one.
I tried to reproduce on a test install of OPNsense in Proxmox, but could not. After changing the user to 'www' and default ports to 8080/8443 the service started fine.
Looking at the GitHub ticket, the fix was merged in December. I started using OPNsense around 24.7 so I'm wondering if there's something in my main router config file from before the fix that is getting carried over and causing an issue? I know it sounds crazy but I can't imagine what else it could be. I blew away my previous root filesystem when I did the fresh install yesterday using the 25.1 installer, so literally the only difference between it and the test instance in Proxmox would be the config import.
The socket file does have the correct ownership & permissions (root:www, 0220), and I confirmed that the 'www' user is in the 'www' group as well, so there shouldn't be an issue with that.
I'll post back if I notice anything in my config.xml file, but since this is not reproducible on a clean install then I don't think it warrants a ticket.
Hello,
I recently upgraded from Opnsense (FreeBSD 14.2-RELEASE-p2) Business Edition 24.10 --> 25.4, and caddy server stopped working.
caddy --version
v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=
I have crowdsec plugin on caddy which I use in Caddyfile for integration w/ crowdsec. So, usually, after major upgrades on Opnsense I end up doing the following:
caddy add-package github.com/hslatman/caddy-crowdsec-bouncer github.com/caddyserver/transform-encoder
and off to the races I go. However, this time around, I'm getting a weird 400 error.
caddy add-package github.com/hslatman/caddy-crowdsec-bouncer github.com/caddyserver/transform-encoder
2025/05/01 01:17:17.322 INFO this executable will be replaced {"path": "/usr/local/bin/caddy"}
2025/05/01 01:17:17.322 INFO requesting build {"os": "freebsd", "arch": "amd64", "packages": ["github.com/caddy-dns/desec@v0.0.0-20240526070323-822a6a2014b2", "github.com/caddy-dns/scaleway@v0.0.0-20231227190624-561fd7f77b1b", "github.com/caddyserver/transform-encoder", "github.com/caddy-dns/namedotcom@v0.1.3-0.20231028060845-b9fae156cd97", "github.com/caddy-dns/ovh@v0.0.3", "github.com/caddy-dns/porkbun@v0.2.1", "github.com/caddy-dns/rfc2136@v0.1.1", "github.com/hslatman/caddy-crowdsec-bouncer", "github.com/caddy-dns/netcup@v0.1.1", "github.com/caddy-dns/vultr@v0.0.0-20230331143537-35618104157e", "github.com/caddyserver/ntlm-transport@v0.1.3-0.20230224201505-e0c1e46a3009", "github.com/caddy-dns/cloudflare@v0.0.0-20240703190432-89f16b99c18e", "github.com/caddy-dns/directadmin@v0.3.1", "github.com/caddy-dns/hetzner@v0.0.2-0.20240820184004-23343c04385f", "github.com/caddy-dns/linode@v0.7.2", "github.com/caddy-dns/namecheap@v0.0.0-20240114194457-7095083a3538", "github.com/caddy-dns/acmedns@v0.3.0", "github.com/caddy-dns/bunny@v0.1.1-0.20240209091254-71ced26b4224", "github.com/caddy-dns/acmeproxy@v1.0.6", "github.com/caddy-dns/infomaniak@v1.0.1", "github.com/caddy-dns/inwx@v0.3.1", "github.com/caddy-dns/mailinabox@v0.0.2-0.20240829173454-39d0e3ce8e25", "github.com/caddy-dns/powerdns@v1.0.1", "github.com/caddy-dns/azure@v0.5.0", "github.com/caddy-dns/gandi@v1.0.4-0.20240531160843-d814cce86812", "github.com/caddy-dns/hexonet@v0.1.0", "github.com/mholt/caddy-ratelimit@v0.1.0", "github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead", "github.com/caddy-dns/dnsmadeeasy@v1.1.3", "github.com/mholt/caddy-dynamicdns@v0.0.0-20241025234131-7c818ab3fc34", "github.com/caddy-dns/duckdns@v0.4.0", "github.com/caddy-dns/ionos@v1.1.0"]}
Error: download failed: download failed: HTTP 400: unable to fulfill download request (id=43358b0e-5041-4319-adac-d96d6a1e570e)
caddy upgrade
2025/05/01 01:24:33.309 INFO this executable will be replaced {"path": "/usr/local/bin/caddy"}
2025/05/01 01:24:33.309 INFO requesting build {"os": "freebsd", "arch": "amd64", "packages": ["github.com/caddy-dns/cloudflare@v0.0.0-20240703190432-89f16b99c18e", "github.com/caddy-dns/gandi@v1.0.4-0.20240531160843-d814cce86812", "github.com/caddy-dns/inwx@v0.3.1", "github.com/caddy-dns/acmeproxy@v1.0.6", "github.com/caddy-dns/dnsmadeeasy@v1.1.3", "github.com/caddy-dns/duckdns@v0.4.0", "github.com/caddy-dns/hetzner@v0.0.2-0.20240820184004-23343c04385f", "github.com/caddy-dns/mailinabox@v0.0.2-0.20240829173454-39d0e3ce8e25", "github.com/caddy-dns/namecheap@v0.0.0-20240114194457-7095083a3538", "github.com/mholt/caddy-ratelimit@v0.1.0", "github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead", "github.com/caddy-dns/bunny@v0.1.1-0.20240209091254-71ced26b4224", "github.com/caddy-dns/directadmin@v0.3.1", "github.com/caddy-dns/linode@v0.7.2", "github.com/caddy-dns/infomaniak@v1.0.1", "github.com/caddy-dns/netcup@v0.1.1", "github.com/caddy-dns/vultr@v0.0.0-20230331143537-35618104157e", "github.com/caddy-dns/acmedns@v0.3.0", "github.com/caddy-dns/azure@v0.5.0", "github.com/caddy-dns/desec@v0.0.0-20240526070323-822a6a2014b2", "github.com/caddy-dns/ovh@v0.0.3", "github.com/caddy-dns/porkbun@v0.2.1", "github.com/caddy-dns/scaleway@v0.0.0-20231227190624-561fd7f77b1b", "github.com/caddyserver/ntlm-transport@v0.1.3-0.20230224201505-e0c1e46a3009", "github.com/caddy-dns/rfc2136@v0.1.1", "github.com/caddy-dns/hexonet@v0.1.0", "github.com/caddy-dns/ionos@v1.1.0", "github.com/caddy-dns/namedotcom@v0.1.3-0.20231028060845-b9fae156cd97", "github.com/caddy-dns/powerdns@v1.0.1", "github.com/mholt/caddy-dynamicdns@v0.0.0-20241025234131-7c818ab3fc34"]}
Error: download failed: download failed: HTTP 400: unable to fulfill download request (id=704dc2db-afa9-4ee4-953a-6ba7ffec9803)
Firewall is not blocking either DNS translation of GitHub or ip connectivity to it. I am wondering if anyone else is having this issue?
caddy add-package uses the build servers of the caddy project to supply you with a binary.
They have no SLA and always serve the latest version of caddy.
Right now the latest version probably has build incompatabilities with the build you are requesting.
Try using xcaddy instead for your personal build.
Thank you, Cedrik.
Quote from: Monviech (Cedrik) on May 01, 2025, 06:55:57 AMTry using xcaddy instead for your personal build.
I presume this'd mean I'd issue xcaddy with all the package names indicated in the previously failed command, like:
xcaddy build \
--with github.com/caddy-dns/scaleway \
--with github.com/caddy-dns/desec \
...
(+ packages of my interest e.g. crowdsec)
/r
morik
PS: Your efforts in maintaining os-caddy port for OPNsense and other software are very much appreciated!
You only need to specify the packets you want to use. The base build uses the top 4 packages in this list and then just the single DNS provider you need + your custom packages.
https://github.com/opnsense/tools/blob/31002815e15e1f50cc7ab0af5c3f1cd155878926/config/25.1/make.conf#L100
If you cannot build caddy 2.10.0 try to pass version 2.9.1 and use the same commit hashes as I do in the link.
Thank you, Cedrik.
OPNSense: You want to do what?
Me: pkg install xcaddy
OPNSense: Did you take me for a fool?
OPNSense:
pkg install xcaddy
Updating OPNsense repository catalogue...
Fetching meta.conf: 100% 163 B 0.2kB/s 00:01
Fetching packagesite.pkg: 100% 249 KiB 255.0kB/s 00:01
Processing entries: 100%
OPNsense repository update completed. 870 packages processed.
All repositories are up to date.
pkg: No packages available to install matching 'xcaddy' have been found in the repositories
Me:
sed -in 's/no/yes/' /usr/local/etc/pkg/repos/FreeBSD.conf
FreeBSD: { enabled: yes }
pkg install xcaddy
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating OPNsense repository catalogue...
Fetching meta.conf: 100% 163 B 0.2kB/s 00:01
Fetching packagesite.pkg: 100% 249 KiB 255.0kB/s 00:01
Processing entries: 100%
OPNsense repository update completed. 870 packages processed.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):
Installed packages to be UPGRADED:
pkg: 1.19.2_5 -> 2.1.2 [FreeBSD]
Number of packages to be upgraded: 1
The process will require 31 MiB more space.
12 MiB to be downloaded.
Proceed with this action? [y/N]: N
OPNSense: Told you, I'll win.
Me: I give up
Also, few minutes later,
Me: having never done ports, why don't we hose our system...
pkg install git
...
git clone --depth=1 https://git.FreeBSD.org/ports.git /usr/ports
cd /usr/ports/www/xcaddy
make install clean
....
mkdir -p ~/caddy_build && cd ~/caddy_build
xcaddy build \
--with github.com/caddyserver/ntlm-transport \
--with github.com/mholt/caddy-dynamicdns \
--with github.com/mholt/caddy-l4 \
--with github.com/mholt/caddy-ratelimit \
--with github.com/hslatman/caddy-crowdsec-bouncer \
--with github.com/caddyserver/transform-encoder
...
...
././caddy version
v2.10.0 h1:fonubSaQKF1YANl8TXqGcn4IbIRUDdfAkpcsfI/vX5U=
<< make my changes on crowdsec >>
configctl caddy restart
OK
Phew.. No idea what else did I break, but the feature which I wanted works now. Of course, I do not know how future os-caddy updates will behave. Life is indeed an adventure :-)
You probably did not break anything, I plan to evaluate to make it more straight forward in the future, maybe.
https://github.com/opnsense/plugins/issues/4668
I evaluated it and its possible but very brittle. So its not going to be included in the plugin.
In your case you built caddy with go121 which might be an issue since go124 is required for all features to work correctly.
The build will be thinned out soon to only include cloudflare, which will make caddy add-package less prone to fail.
It failed because there was a breaking libdns change again and it includes a couple of dns providers that are dependent on it, thus it failed.
For xcaddy, using a separate freebsd vm where you can tightly control the build environment might be the best choice.
All in all, go limits the flexibility of caddy here, comparing it to dynamic modules in nginx or apache.
So in my config I have as follows:
1) OPNsense firewall sits behind ISP router.
2) ISP router is configured for DDNS, going to NO-IP
3) Cloudflare is handling external DNS, and CNAMEs are setup in Cloudflare to point back to the NO-IP DNS name
This way everything looks like a static IP/DNS setup. With that said I would assume I do NOT need to configure Caddy for DDNS correct? I would just use the CNAMEs that are stup for the services I want Caddy to proxy correct?
If you do a dig or nslookup and your FQDNs always point to the correct IP address with your current setup, you do not need to configure dynamic dns in the caddy plugin, it's that simple.
Im having a heck of a time getting opnsense caddy plugin working with immich. The logs are stating that CORS is failing. Immich states that all i should need to do is set the reverse_proxy (https://immich.app/docs/administration/reverse-proxy#caddy-example-config) which i have done but it fails on CORS requests. I was able to add 4 or 5 additional headers (more here: https://forum.opnsense.org/index.php?topic=46177.msg239034#msg239034) but its stating that the preflight response was a non 2xx response. Any help would be greatly appreciated.
Just spent a couple hours debugging a reload(not restart) of the caddy service from the acme client and gave up.
I'll try to briefly summarize the problem/observed behaviour.
System is on 25.1.8_1
Caddy package is 2.0.1
Caddy is setup to run as www and runs nicely.
- Checking /var/run/caddy after restarting the service via GUI shows caddy.sock owned by www:www with 0220
- Applying caddy config from GUI changes that to root:www with 0220
- Restarting the service via GUI changes that back to www:www with 0220
So far so good besides the strange ownership discrepancies between applying config and restarting via GUI.
Setting up an automation in ACME to reload(!) caddy is where things get strange and I can't figure out why.
- Reload fails with log:
"loading config: loading new config: starting caddy administration endpoint: unable to set permissions (--w--w----) on /var/run/caddy/caddy.sock: chmod /var/run/caddy/caddy.sock: operation not permitted"
- Ownership/permission on caddy.sock after running that automation are root:www with 0220 regardless what they where before.
If I use a caddy restart automation in ACME, everything is fine, ownership/permissions after run are www:www with 0220.
As soon as the ownership/permission on caddy.sock is root:www with 0220 running "service caddy reloadssl" fails with the same error, "service caddy restart" reverts ist back to www:www with 0200 after which "service caddy reloadssl" also works fine from the CLI.
So something goes wrong when "service caddy reloadssl" is run via an ACME automation, at least I think that is also run via the system action from within ACME.
I failed digging through all the scripts and includes and system actions etc to identify the cause and now had to give up due to time constraints, am now using caddy restart action from ACME which is fine for now.
Hopefully this helps someone way more knowledgeable then me in all the internals of the system to narrow down the issue.
Thanks for the report, if I have time I will look into it.
Best would be an issue in https://github.com/opnsense/plugins
Hi all, I'm setting up Caddy behind OPNsense and running into an issue that I suspect is either a misconfiguration on my part or a misunderstanding of how NAT and Caddy work together.
Setup details:
- OPNsense as the router
- VLAN 20 for LAN
- VLAN 50 for servers
Caddy is configured and successfully issues Let's Encrypt certificates
The upstream service (Minio) is running over HTTP on port 9001
I created a NAT rule to forward HTTPS traffic to port 9001 on the server, but I'm getting a ERR_SSL_PROTOCOL_ERROR. If I create a NAT rule for plain HTTP instead, it works fine.
A few questions:
- Has anyone experienced this behavior with Minio over HTTP? Could Minio be rejecting the SSL termination from Caddy?
- I assumed that once a handler is set up in the Caddyfile, Caddy would take care of routing automatically—but it seems like it doesn't touch NAT rules. Is that correct?
Any guidance or clarification on how Caddy and OPNsense should interact in this case would be greatly appreciated!
Thanks in advance.
Hi,
I have setup Caddy exactly as described in OPNsense documentation.
I changed the OPNsense https port to 8443, checked the Disable web GUI redirect rule.
I have created the WAN and LAN rules for This firewall destination.
Enabled Caddy, supplied e-mail, Auto HTTPS is ON.
Domain is https://test.duckdns.org. Certificate is Auto HTTPS.
Handler: Domain is https://test.duckdns.org, Directive: reverse_proxy
Upstream:
Protocol: http://
Upstream domain: 192.168.250.162
Upstream port: 2283
Access list private_ipv4 created as per documentation and added to the domain access list
OPNsense runs on a dedicated hardware box, WAN and LAN interfaces only. Dynamic DNS is configured in OPNsense to use DuckDNS. I can do nslookup test.duckdns.org and it resolves to my WAN IP address, no problem. If I access https://test.duckdns.org from within my local network, it can access the server on 192.168.250.162:2283 without any issues. But when I access from internet, I get the error:
net::ERR_HTTP2_PROTOCOL_ERROR
The Caddy file is:
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file
# caddy_user=root
# Global Options
{
log {
output net unixgram//var/run/caddy/log.sock {
}
format json {
time_format rfc3339
}
}
servers {
protocols h1 h2
}
email xyz@gmail.com
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}
# Reverse Proxy Configuration
test.duckdns.org {
@c1ac6e21-c93d-461e-8546-246789993f33_testduckdnsorg {
not client_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
}
handle @c1ac6e21-c93d-461e-8546-246789993f33_testduckdnsorg {
abort
}
handle {
reverse_proxy 192.168.250.162:2283 {
}
}
}
import /usr/local/etc/caddy/caddy.d/*.conf
I have done Google search but nothing. I have read all 21 pages of this thread but no matching issue. Caddy is getting certificate from Lets Encrypt. Any help would be appreciated.
OPNsense 25.1.9_2-amd647
os-caddy 2.0.1
Thanks,
Arun
Update: Issue resolved. The documentation is not good. It did not say when to stop creating reverse proxy. It went to configure access list for local access only. I did that and it blocked access from internet. I had to remove the access list from the domain and then reverse proxy started to work from internet. The documentation of Caddy needs to be revised as some options do not exist in OPNsense. It also needs to be made more descriptive to describe so many other options available.
I was able to fix it, it was a bad config on the DNS, it was pointing to the server IP instead of OPNsense IP to handle the reverse proxy correctly.
Quote from: lucasconde on June 28, 2025, 06:07:54 AMHi all, I'm setting up Caddy behind OPNsense and running into an issue that I suspect is either a misconfiguration on my part or a misunderstanding of how NAT and Caddy work together.
Setup details:
- OPNsense as the router
- VLAN 20 for LAN
- VLAN 50 for servers
Caddy is configured and successfully issues Let's Encrypt certificates
The upstream service (Minio) is running over HTTP on port 9001
I created a NAT rule to forward HTTPS traffic to port 9001 on the server, but I'm getting a ERR_SSL_PROTOCOL_ERROR. If I create a NAT rule for plain HTTP instead, it works fine.
A few questions:
- Has anyone experienced this behavior with Minio over HTTP? Could Minio be rejecting the SSL termination from Caddy?
- I assumed that once a handler is set up in the Caddyfile, Caddy would take care of routing automatically—but it seems like it doesn't touch NAT rules. Is that correct?
Any guidance or clarification on how Caddy and OPNsense should interact in this case would be greatly appreciated!
Thanks in advance.
Quote from: Monviech (Cedrik) on May 04, 2025, 04:47:53 PMI evaluated it and its possible but very brittle. So it's not going to be included in the plugin.
Thank you for taking the time to look into this matter!
QuoteThe build will be thinned out soon to only include cloudflare, which will make caddy add-package less prone to fail.
Good to hear! I finally got around to using DNS-01 challenge with Porkbun DNS, so had to rebuild caddy 2.10.1 with porkbun module using xcaddy. go124 upgrade was done implicitly. Slightly painful but works. Thank you for writing such a helpful plugin!
How to use environment variables with caddy?
I had a need to reuse certain parameters in different reverse proxy handler blocks. The manual way of duplicating the same value in multiple places works ofcourse. It'd be easier to use environment variables. So, I tried add caddy.conf in /usr/local/opnsense/service/conf/configd.conf.d with content like so
CADDY_RESOLVERS_DNS_TLS=149.112.112.112 1.1.1.1 8.8.8.8 8.8.4.4
followed by
service configd restart
configctl caddy restart
caddy environ
startup fails w/ the environment variable value not found. Perhaps a wiser being could tell me whether there is a trick to this getting the env variables working?