Using Caddy for local subdomains

[RobLatour]

Yesterday, with help from this forum, I got Caddy to provide ssl access to my OPNSense router - which was for me a great first step.

Today, for the second step, I had hoped to get it working for some local machines - for example one running Home Assistant and another PiKVM.

At first I tried the subdomain route - but couldn't get it to work.  Also, I found it didn't let me use specific ports - so I attempted to follow the advice of the documentation here: https://docs.opnsense.org/manual/how-tos/caddy.html#wildcard-domain-with-subdomains

If in doubt, do not use subdomains. If there should be foo.example.com, bar.example.com and example.com, just create them as three base domains. This way, there is the most flexibility, and the most features are supported.

but did not succeed.

For this this I removed the wildcard entries I had previously used, and replaced them with three domains:

router.example.com
ha.example.com
pikvm.example.com

router.example.com is working fine.

ha.example.com and pikvm.example.com are not.

When I try to browse to either ha.example.com or pikvm.example.com I get a CloudFlare page served up that says Connection timed out - 502.

There are no error, warning or even informational entries in my Caddy log to show what might be going on.

My registry domain dns records are set up as follows:

Name                     Type        TTL          RDATA 
@                           A            14400      xxx.xxx.xxx.xxx
@                           NS          86400      xxxxx.ns.cloudflare.com
@                           NS          86400      xxxx.ns.cloudflare.com
ha.example.com      A            14440      xxx.xxx.xxx.xxx
pikvm.example.com A            14440      xxx.xxx.xxx.xxx
router.example.com A            14440      xxx.xxx.xxx.xxx
www                        CNAME      14440      example.com

My Cloudflare records are as follows:
Type     Name            Content               Proxy status    TTL      
A          ha                 xxx.xxx.xxx.xxx   Proxied          Auto
A          pikvm            xxx.xxx.xxx.xxx   Proxied          Auto
A          example.com  xxx.xxx.xxx.xxx   Proxied          Auto
A          router            xxx.xxx.xxx.xxx   Proxied          Auto
CNAME  www              example.com       Proxied          Auto

For the IP addresses of the two machine, even though I think the right think to do is use the external addresses, I have also tired using my internal IP address (non proxied) such as 192.168.1.123 ; but when I used the internal address they didn't work either.  Also, CloudFlare throws an error if I try to use the internal ones and Proxy them.  My gut feel I should be using external proxied addresses so Caddy can set them to what it needs.

In any case, here too is my Caddy file:

Code Select Expand



# DO NOT EDIT THIS FILE -- OPNsense auto-generated file


# caddy_user=root

# Global Options
{
log {
output net unixgram//var/run/caddy/log.sock {
}
format json {
time_format rfc3339
}
}

dynamic_dns {
provider cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
domains {
ha.example.com @
pikvm.example.com @
router.example.com @
}
versions ipv4
}

email me@example.com
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file


# caddy_user=root

# Global Options
{
log {
output net unixgram//var/run/caddy/log.sock {
}
format json {
time_format rfc3339
}
}

dynamic_dns {
provider cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
domains {
ha.example.com @
pikvm.example.com @
router.example.com @
}
versions ipv4
}

email me@example.com
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}

# Reverse Proxy Configuration


# Reverse Proxy Domain: "bf960db4-a3f6-432c-8be4-be6ed247d7b2"
ha.example.com {
tls {
issuer acme {
dns cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
resolvers 1.1.1.1
}
}

@b91182d4-6b27-4ef1-a8a3-8d45e1578a76 {
client_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
}
handle @b91182d4-6b27-4ef1-a8a3-8d45e1578a76 {
handle {
reverse_proxy 192.168.1.173:8123 {
transport http {
tls_insecure_skip_verify
}
}
}
}
}
# Reverse Proxy Domain: "9a3a66c5-aeff-4401-a697-b34de6525a10"
pikvm.example.com {
tls {
issuer acme {
dns cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
resolvers 1.1.1.1
}
}

@b91182d4-6b27-4ef1-a8a3-8d45e1578a76 {
client_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
}
handle @b91182d4-6b27-4ef1-a8a3-8d45e1578a76 {
handle {
reverse_proxy 192.168.1.166:443 {
transport http {
tls_insecure_skip_verify
}
}
}
}
}
# Reverse Proxy Domain: "17af584e-7fcf-4ca0-b5f9-fbd9712f95e4"
router.example.com {
tls {
issuer acme {
dns cloudflare xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
resolvers 1.1.1.1
}
}

@b91182d4-6b27-4ef1-a8a3-8d45e1578a76 {
client_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
}
handle @b91182d4-6b27-4ef1-a8a3-8d45e1578a76 {
}
}

import /usr/local/etc/caddy/caddy.d/*.conf





I have also tried both handlers without tls_insecure_skip_verify - but it still didn't work.


Also, and as and aside although I don't think it matters much, when I deleted the wild card entry from before, and when I created and then deleted some other Services: Caddy Web Server: Reverse Proxy - Domains it appears their certificates are still hanging around (as I see them in the Dashboard under the Caddy Certificates widget) rather being cleaned up when their associated domains were deleted.  I suppose it is possible these unused certificates may be getting in the way some how - but I don't know how to otherwise get rid of them.

Any help would be much appreciated.

[Monviech (Cedrik)]

Hey,

your Caddyfile looks fine.

Read this please: https://docs.opnsense.org/manual/how-tos/caddy.html#help-nothing-works

Youll have to do the troubleshooting yourself here.

[cloudz]

For this to work you need to point your ha / pikvm DNS records to the IP address of the Caddy proxy. This can either be your internal FW ip or if you have turned on hairpinning you can use your external IP address.

I'm using it in the exact same fashion as what you are trying to achieve.

I'm using a DDNS record for my router at Bunny and CNAME all the internal domains I want accessible to that DDNS record. This way I can use hairpinning to the services on my DMZ to which I proxy.

[RobLatour]

@Monviech , @cloudz

Thank you for your suggestions:

I went through the https://docs.opnsense.org/manual/how-tos/caddy.html#help-nothing-works documentation but couldn't find the solution

However, I notice the command nslookup pikvm.example.com responds with:

Server:  router.example.com
Address:  192.168.1.1

Non-authoritative answer:
Name:    pikvm.example.com
Addresses:  2606:4700:3036::6815:a7b
          2606:4700:3030::ac43:be24
          104.21.10.123
          172.67.190.36

with those ipv4 and ipv6 addresses mapping back to CloudFlare not to my external IP address - so I'm not sure why?

Also, I am now getting these errors in my log (where xxx = each of router, ha, and pikvm)

Error   caddy   "error","ts":"2024-09-20T16:11:40Z","logger":"dynamic_dns","msg":"failed setting DNS record(s) with new IP address(es)","zone":"xxx.example.com","error":"expected 1 zone, got 0 for router.rlatour.ca"}

I had gotten them yesterday, but got rid of them by unchecking Dynamic DDS in Services: Caddy Web Server: Reverse Proxy but now that I have them checked again I see that the errors again.  So I assume the dynamic dns isn't working either.

@cloudz

I tried changing to cnames, but that didn't work either.   I ran into this yesterday, and had more success with A records.

Also, I read thru the documentation and tried truning on hairpinning but with no success - perhaps I did that wrong too as the doc assumes a dmz which I don't have.

I've been working on this literally all week, not sure what the issues are but have appreciated your help.

[Monviech (Cedrik)]

You should have stuck with the wildcard subdomain setup I told you yesterday. Checking dyndns for subdomains adds them to the zone like:

example.com router

right now they are added like:

router.example.com @

but you dont have a zone for router.example.com, only for example.com

Thats why it fails now in the logs.

[RobLatour]

Well I needed to use specific entries as opposed to one wildcard entry and subdomains as my individual machines on my internal network need to be accessed via different ports, for example HomeAsssistant uses 8123, while the PiKVM uses 443.  With subdomains I can't specify a port - which is what I suspect the documentation was alluding to where it reads:

If in doubt, do not use subdomains. If there should be foo.example.com, bar.example.com and example.com, just create them as three base domains. This way, there is the most flexibility, and the most features are supported.

regardless, what I've just done was add a *.example.com entry in my Services: Caddy Web Server: Reverse Proxy - Domains, and in CloudFlare, and in CanSpace (my domain registry service), updated my Cloudflare api token, and in the Caddy plugin removed dynamic dns from my pikvm.example.com, ha.example.com and router.example.com domains, and added it in the wildcard domain.

Now when I disable and enable Caddy I the error messages for the pikvm, ha, and pikvm are gone, but there is an information message for the wild card domain that reads:

   "info","ts":"2024-09-20T17:09:52Z","logger":"dynamic_dns","msg":"domain not found in DNS","domain":"*.example.com"}

Which probably means it is not working - any may have not been working yesterday as well and I didn't notice it as it comes up as info rather than a warning or an error.

[Monviech (Cedrik)]

You can add the same wildcard domain multiple times with different ports.

https://forum.opnsense.org/index.php?topic=38714.msg207433#msg207433

Its a bit complicated but it works.

The setup you try to do as your starting setup doesn't do you any favors in terms of complexity.

You should try to get one simple domain working, then it might be a bit easier, especially if you just start out with reverse proxies.

Due to the things I learned here, I have rewritten some parts of the docs:

https://github.com/opnsense/docs/blob/8c83692c4465ef07a33199b9239d5494a99c57e0/source/manual/how-tos/caddy.rst

Check out especially this section:

https://github.com/opnsense/docs/blob/8c83692c4465ef07a33199b9239d5494a99c57e0/source/manual/how-tos/caddy.rst#dynamic-dns

And this section:

https://github.com/opnsense/docs/blob/8c83692c4465ef07a33199b9239d5494a99c57e0/source/manual/how-tos/caddy.rst#wildcard-domain-with-subdomains

But I can not save every setup (sadly). I don't have Cloudflare. I use the plugin myself but with totally different configurations. I don't use the DNS Providers and API stuff at all. There are so many factors. Somebody with Cloudflare should really help out here or do a youtube tutorial sometime.

[Patrick M. Hausen]

If the DNS is pointing to the Cloudflare CDN, HTTP-01 won't work unless you configure CF to pass through all requests to /.well-known/acme-challenge. How exactly that is done, I don't know - CF should have documentation for that.

Or have the DNS point to your public IP. Again, how that is achieved is a CF question.

[RobLatour]

Thanks again.

I just spent the better part of another two hours at this and just can't get it to work.

While I can get ddns to work with cloudflare for the generic wild card *.example.com I can't get it to work with any of the subdomains.  HTTP-01 Challenge Redirection is not going to work, as I don't have a host behind the registered domain, just the registered domain - which means only DNS-01 Challenge is going to work but as I can only use  DNS-01 Challenge at the domain level, all my subdomains can't be setup for ddsn.

Beyond that, I never was able to get a subdomain to resolve correctly.

I tried the idea of various *.example.com subdomains, each with there own unique port - but even though that is noted as 'supported' they are not resolving correctly for me.   Also, even if I could get it to work the theory in behind using them assumes each unique port is only used once across all machines in my network.  So for example if pikvm uses port 443, then no other machine in my network would be able to use it.

I would like to thank @Monviech, @cloudz, and @Patrick M. Hausen for helping me out - but ultimately, I'm just going to have to put this down now as I've already dumped way too much time into it without getting the outcomes I was looking for.   However, my system does now resolve the opnsense box via ssl with is the silver lining in all of this for me, it just those pesky other machines on my internal network that I couldn't get working.

Again, with my sincere thanks!

[cloudz]

You just need to turn off 'proxying' in Cloudflare.

[RobLatour]

ok - since there was a new update to the plugin I decided to come back to this.

As suggested, I turned off proxy in Cloudflare but it did not work.  When I tried to browse to ha.example.com I got a BitDefender pop-up on my host machine saying 'Suspicious page blocked for your protection', and when I tried it in the W11 Sandbox of my host machine (that doesn't run Bitdefender) I got a message saying 'Your connection isn't private' and when I advance beyond that 'This page isn't working right now'.

So I decided to take Cloudflare out of the picture, and try it with Duckdns instead.

I setup duckdns entries for:  example.duckdns.org and ha-example.duckdns.org

In Caddy I setup up a domain record for example.duckdns.org and an associated http handler.  In the domain I used Certificate ACME (HTTP-1, TLS-ALPN-01), plus checked DNS-01 Challenge and Dynamic DNS.  In the Handler I checked TLS Insecure Skip Verify.  With this setup I can now browse to example.duckdns.org and get a valid ssl connection to myOPNSense box! Yeah!

I then tried to setup ha-example.duckdns.org in the same way to point to my HomeAssistant box.  The only difference was the IP address and the Port in the Handler.   However, when I browse to https://ha-example.duckdsn.org:8123  I get ERR_CONNECTION_TIMED_OUT. 

Now the scarry part.

Beyond that, and yikes, when I browse to https://ha-example.duckdns.org  (no port)  or https://ha-example.duckdsn.org:443 I get redirected to a malware site.  The same sort of thing happened last week in my testing - but I got directed to survey-smiles (or something like that) (after a few days it was clunen.com/xr.php?e=iqXRc and a bunch of other stuff which I assume amounted to some sort of injection attack ) (both of which Bitdefender was blocking) but now its redirecting to https : //crrds70hubcc73ba0m30.securedjointnetwork.co.in/01/?cid=ac910b88f486f12d .... (asking me to click on some box to prove I'm not a robot which I suspect Bitdefender should be blocking but is not)  Of note: in the link above I manually added a space between the s, colon, and slash so this post would not have that show up as a hyperlink.

Originally, I thought perhaps my PC had a virus, but I scanned it and it came up clean.  Also, at the time, as I remember it I was getting the same redirect in the W11 sandbox and on a Raspberry pi running Raspberry OS.   I am now not getting it in Windows sandbox, nor on the Raspberry Pi OS - but neither of those are connecting to my home assistant box either.

Of note - I do not have a certificate on my home assistant box (a dedicated Raspberry Pi) as I understood Caddy didn't need one to allow the connection to be secure.

A lot to digest for sure.

[Monviech (Cedrik)]

well you wrote duckDSN instead of duckDNS

[RobLatour]

well you wrote duckDSN instead of duckDNS

Well @Monviech when you are right you are right, and that does take care of the scarry part - thank you!

However, sadly the important although somewhat more mundane part remains:
https://ha-example.duckdns.org
https://ha-example.duckdns.org:8123
https://ha-example.org
all result with

This page isn't working
ha-example.duckdns.org is currently unable to handle this request.
HTTP ERROR 502

Here is the log (debug on):

Code Select Expand




2024-09-27T16:36:36-04:00 Error caddy "error","ts":"2024-09-27T20:36:36Z","logger":"http.log.error","msg":"tls: first record does not look like a TLS handshake","request":{"remote_ip":"192.168.1.10","remote_port":"11483","client_ip":"192.168.1.10","proto":"HTTP/2.0","method":"GET","host":"ha-example.duckdns.org","uri":"/","headers":{"Sec-Ch-Ua":["\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\""],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Priority":["u=0, i"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br, zstd"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"ha-example.duckdns.org"}},"duration":0.002232754,"status":502,"err_id":"ycczhrkwr","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}

2024-09-27T16:36:36-04:00 Error caddy "debug","ts":"2024-09-27T20:36:36Z","logger":"http.handlers.reverse_proxy","msg":"upstream
roundtrip","upstream":"192.168.1.173:8123","duration":0.002111749,"request":{"remote_ip":"192.168.1.10","remote_port":"11483","client_ip":"192.168.1.10","proto":"HTTP/2.0","method":"GET","host":"ha-example.duckdns.org","uri":"/","headers":{"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\""],"Sec-Fetch-Dest":["document"],"X-Forwarded-For":["192.168.1.10"],"X-Forwarded-Host":["ha-example.duckdns.org"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Priority":["u=0, i"],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"ha-example.duckdns.org"}},"error":"tls: first record does not look like a TLS handshake"}

2024-09-27T16:36:36-04:00 Debug caddy "debug","ts":"2024-09-27T20:36:36Z","logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.1.173:8123","total_upstreams":1}

2024-09-27T16:36:34-04:00 Error caddy "error","ts":"2024-09-27T20:36:34Z","logger":"http.log.error","msg":"tls: first record does not look like a TLS handshake","request":{"remote_ip":"192.168.1.10","remote_port":"11483","client_ip":"192.168.1.10","proto":"HTTP/2.0","method":"GET","host":"ha-example.duckdns.org","uri":"/","headers":{"Sec-Ch-Ua-Platform":["\"Windows\""],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"],"Sec-Fetch-User":["?1"],"Priority":["u=0, i"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Purpose":["prefetch;prerender"],"Purpose":["prefetch"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\""]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"ha-example.duckdns.org"}},"duration":0.002067182,"status":502,"err_id":"rq8r5ftj6","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}

2024-09-27T16:36:34-04:00 Error caddy "debug","ts":"2024-09-27T20:36:34Z","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.1.173:8123","duration":0.001940032,"request":{"remote_ip":"192.168.1.10","remote_port":"11483","client_ip":"192.168.1.10","proto":"HTTP/2.0","method":"GET","host":"ha-example.duckdns.org","uri":"/","headers":{"Sec-Purpose":["prefetch;prerender"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"],"X-Forwarded-Host":["ha-example.duckdns.org"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"X-Forwarded-For":["192.168.1.10"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\""],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Upgrade-Insecure-Requests":["1"],"Purpose":["prefetch"],"Priority":["u=0, i"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Sec-Fetch-Mode":["navigate"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"ha-example.duckdns.org"}},"error":"tls: first record does not look like a TLS handshake"}

2024-09-27T16:36:34-04:00 Debug caddy "debug","ts":"2024-09-27T20:36:34Z","logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.1.173:8123","total_upstreams":1}

192.168.1.1 correctly shows as the OPNSense box
192.168.1.10 correctly shows as the machine on which the request is made
192.168.1.173 correctly shows as the Home Assistant box

regarding "tls: first record does not look like a TLS handshake" I've tried it with and without the option 'TLS Insecure Skip Verify' checked.

on 192.168.1.10 if I ping ha-example.duckdns.org the ping works, and identifies my network's external IPV4 address correctly.

[Monviech (Cedrik)]

I sadly cant do much here. You should google the error you have. E.g.,

https://caddy.community/t/tls-handshake-error-first-record-does-not-look-like-a-tls-handshake/18350

[RobLatour]

Thank you.

Well, I finally got it working using a domain and cloudflare for machines running opnsense itself, open media vault, pikvm, and bitwarden.  I don't yet have it working for home assistant, but will keep working on that.

Also, of note, at one point I was digging quite deep trying to figure out why it wasn't working and noticed one of my certificates was issue by Bitdefender which didn't look right as I thought it should be LetsEncrypt or ZeroSSL.  I worked for about an hour trying to figure out how that was happening (and how I might stop it) but do have Bitdefender running on my PC.  In any case, I then switched over to testing on a Raspberry Pi using Firefox where it said I was using a LetsEncrypt certificate.  However, it also said the Issuer country was "RU"?? 

On OPNSense I have geo-blocking for certain countries, including Russia.  I turned that off for a short time to see what would happen, and shortly after that I finally got my first caddy redirect to work. 

I did some more config changes and somehow I think I ended up regenerating new LetsEncrypt certificates, as when I checked them later they had a Issuer country of "US".   Now given my prior experience with duckdsn/duckdns it is possible I key fumbled something, however at this time I wasn't testing with duckdns domains any more.  Also, I found this - which may tie into the problems I was having?

ref: https://community.sophos.com/utm-firewall/f/general-discussion/146144/let-s-encrypt-renewal-no-longer-works-with-country-blocking

In any case, at this point, I have geoblocking back on and as mentioned above caddy is working as expected for most of my machines (just not Home Assistant).  Also it is working fine when I am browsing on my PC with Bitdefender.  My PC is still showing that the certificates are issued by Bitdefender while my Pi is showing them as issued by LetsEncrypt.  So I imagine BitDefender is doing some sort of MITM work that I'm just not going to worry about for now.

If I figure out the Home Assistant thing, I'll post back here.

Again, as always, thanks for your help.