ACME cron renewal fails but manual renewal succeeds

[arrowd]

I set up the ACME service on my OPNsense (v26.7.4_1 on a Protectli V1410) in the spring. The problem I describe happened at the first cert renewal, but I decided to wait for the next run to see if it repeated. The certificate was due for replacement yesterday but the cron job failed. I then ran it manually and it succeeded without my changing any setting. I have my domain DNS at Cloudflare. My TrueNAS is set up similarly and the ACME runs correctly there. This is apparently a commmon problem on OPNsense as I found at least two separate forum posts saying essentially the same thing. Neither had a solution with people just saying they would renew manually when needed. Do you think this is a permission problem? A Cloudflare problem? Thanks for reading.

Below is the log in debug format. I replaced my domain name with 'mydomainname' and my unique numbers with equal length strings of 'a','b','c','x', or 'y'. The two sets of long log lines between cron/manual runs are identical.

2026-09-24T00:00:00-04:00opnsenseAcmeClient: issue/renewal not required for certificate: opnsense.mydomainname.net

2026-09-23T13:44:08-04:00configAcmeClient: running automation (configd): Restart Web GUI
2026-09-23T13:44:08-04:00configAcmeClient: running automations for certificate: opnsense.mydomainname.net
2026-09-23T13:44:08-04:00configAcmeClient: updated ACME X.509 certificate: opnsense.mydomainname.net (ccccccccccccccc)
2026-09-23T13:44:08-04:00opnsenseAcmeClient: successfully issued/renewed certificate: opnsense.mydomainname.net
2026-09-23T13:44:08-04:00opnsenseAcmeClient: AcmeClient: The shell command returned exit code '0': '/usr/local/sbin/acme.sh --renew --syslog 7 --debug --server 'letsencrypt' --dns 'dns_cf' --home '/var/etc/acme-client/home' --cert-home '/var/etc/acme-client/cert-home/xxxxxxxxxxxxxx.yyyyyyyy' --certpath '/var/etc/acme-client/certs/xxxxxxxxxxxxxx.yyyyyyyy/cert.pem' --keypath '/var/etc/acme-client/keys/xxxxxxxxxxxxxx.yyyyyyyy/private.key' --capath '/var/etc/acme-client/certs/xxxxxxxxxxxxxx.yyyyyyyy/chain.pem' --fullchainpath '/var/etc/acme-client/certs/xxxxxxxxxxxxxx.yyyyyyyy/fullchain.pem' --domain 'opnsense.mydomainname.net' --days '60' --keylength '4096' --accountconf '/var/etc/acme-client/accounts/aaaaaaaaaaaaaa.bbbbbbbb_prod/account.conf''
2026-09-23T13:43:38-04:00opnsenseAcmeClient: running acme.sh command: /usr/local/sbin/acme.sh --renew --syslog 7 --debug --server 'letsencrypt' --dns 'dns_cf' --home '/var/etc/acme-client/home' --cert-home '/var/etc/acme-client/cert-home/xxxxxxxxxxxxxx.yyyyyyyy' --certpath '/var/etc/acme-client/certs/xxxxxxxxxxxxxx.yyyyyyyy/cert.pem' --keypath '/var/etc/acme-client/keys/xxxxxxxxxxxxxx.yyyyyyyy/private.key' --capath '/var/etc/acme-client/certs/xxxxxxxxxxxxxx.yyyyyyyy/chain.pem' --fullchainpath '/var/etc/acme-client/certs/xxxxxxxxxxxxxx.yyyyyyyy/fullchain.pem' --domain 'opnsense.mydomainname.net' --days '60' --keylength '4096' --accountconf '/var/etc/acme-client/accounts/aaaaaaaaaaaaaa.bbbbbbbb_prod/account.conf'
2026-09-23T13:43:38-04:00opnsenseAcmeClient: using challenge type: OPNsense GUI Certificate
2026-09-23T13:43:38-04:00opnsenseAcmeClient: account config is valid (CERT_HOME): OPNsense GUI Certificate
2026-09-23T13:43:38-04:00opnsenseAcmeClient: account is registered: OPNsense GUI Certificate
2026-09-23T13:43:38-04:00opnsenseAcmeClient: using CA: letsencrypt
2026-09-23T13:43:38-04:00opnsenseAcmeClient: renew certificate: opnsense.mydomainname.net
2026-09-23T13:43:38-04:00opnsenseAcmeClient: certificate must be issued/renewed: opnsense.mydomainname.net

2026-09-23T00:00:01-04:00opnsenseAcmeClient: validation for certificate failed: opnsense.mydomainname.net
2026-09-23T00:00:01-04:00opnsenseAcmeClient: domain validation failed (dns01)
2026-09-23T00:00:01-04:00opnsenseAcmeClient: AcmeClient: The shell command returned exit code '2': '/usr/local/sbin/acme.sh --renew --syslog 7 --debug --server 'letsencrypt' --dns 'dns_cf' --home '/var/etc/acme-client/home' --cert-home '/var/etc/acme-client/cert-home/xxxxxxxxxxxxxx.yyyyyyyy' --certpath '/var/etc/acme-client/certs/xxxxxxxxxxxxxx.yyyyyyyy/cert.pem' --keypath '/var/etc/acme-client/keys/xxxxxxxxxxxxxx.yyyyyyyy/private.key' --capath '/var/etc/acme-client/certs/xxxxxxxxxxxxxx.yyyyyyyy/chain.pem' --fullchainpath '/var/etc/acme-client/certs/xxxxxxxxxxxxxx.yyyyyyyy/fullchain.pem' --domain 'opnsense.mydomainname.net' --days '60' --keylength '4096' --accountconf '/var/etc/acme-client/accounts/aaaaaaaaaaaaaa.bbbbbbbb_prod/account.conf''
2026-09-23T00:00:00-04:00opnsenseAcmeClient: running acme.sh command: /usr/local/sbin/acme.sh --renew --syslog 7 --debug --server 'letsencrypt' --dns 'dns_cf' --home '/var/etc/acme-client/home' --cert-home '/var/etc/acme-client/cert-home/xxxxxxxxxxxxxx.yyyyyyyy' --certpath '/var/etc/acme-client/certs/xxxxxxxxxxxxxx.yyyyyyyy/cert.pem' --keypath '/var/etc/acme-client/keys/xxxxxxxxxxxxxx.yyyyyyyy/private.key' --capath '/var/etc/acme-client/certs/xxxxxxxxxxxxxx.yyyyyyyy/chain.pem' --fullchainpath '/var/etc/acme-client/certs/xxxxxxxxxxxxxx.yyyyyyyy/fullchain.pem' --domain 'opnsense.mydomainname.net' --days '60' --keylength '4096' --accountconf '/var/etc/acme-client/accounts/aaaaaaaaaaaaaa.bbbbbbbb_prod/account.conf'
2026-09-23T00:00:00-04:00opnsenseAcmeClient: using challenge type: OPNsense GUI Certificate
2026-09-23T00:00:00-04:00opnsenseAcmeClient: account config is valid (CERT_HOME): OPNsense GUI Certificate
2026-09-23T00:00:00-04:00opnsenseAcmeClient: account is registered: OPNsense GUI Certificate
2026-09-23T00:00:00-04:00opnsenseAcmeClient: using CA: letsencrypt
2026-09-23T00:00:00-04:00opnsenseAcmeClient: renew certificate: opnsense.mydomainname.net
2026-09-23T00:00:00-04:00opnsenseAcmeClient: certificate must be issued/renewed: opnsense.mydomainname.net

2026-09-22T00:00:00-04:00opnsenseAcmeClient: issue/renewal not required for certificate: opnsense.mydomainname.net

[JeGr]

Seconded here. Was hitting a monitoring alert because 2 certs that are in use with Caddy were over there set up renewal timeframe. As I'm already using the new "tlsserver" profile that only has ~45days certs manually refreshing multiple certificates is tedious. It's only the cron that fails though, manually refreshing is fine, but somehow the --cron stuff isn't running correctly.

[JeGr]

Checked again today, still "broken":

Certificate is picked up by cron to be renewed but is listed as "validation failed" in "Last ACME status". If you renew it manually it works, just cron is borked. Anything we can do to debug that further? With reduced certificate runtime that now gets tedious to check and confirm every few days!

Cheers
Jens

[caoyadong888]

Cron and manual run the same acme.sh line in your paste, so the plugin path is fine; the difference is almost always environment or timing at midnight, not a different challenge type.

Useful checks, in order:

1. Capture one failed cron and one successful manual run with debug still on, and compare the first Cloudflare / dns_cf / TXT error above the exit code 2. That line usually says whether the API token failed, the TXT record was missing, or propagation timed out.

2. Confirm the Cloudflare token in the ACME account still has Zone:DNS:Edit on that zone, and that /var/etc/acme-client/accounts/.../account.conf is readable by the process that cron starts (manual GUI renew sometimes inherits a richer environment).

3. Around 00:00, look in System → Log Files (and any leftover _acme-challenge TXT records in Cloudflare). Leftover TXT from a half-failed run, or CF rate limiting at the top of the hour, both show up as "validation failed" even though a later manual renew works.

4. Shift the ACME auto-renewal window away from 00:00:00 if you can, and verify NTP is synced. Also check you do not have a second cron or external acme.sh --cron fighting the GUI job.

5. For day-to-day debugging: after a failed cron, do not change settings—renew once from the GUI and keep both log blocks. If the only difference is DNS propagation / API errors at night, the workaround of a later schedule is more reliable than chasing permissions that only fail once a day.

If you can paste just the dns_cf / Cloudflare error lines from a failed 00:00 run (redact tokens), it is usually enough to tell API vs propagation apart.