Hai all!
functional ha setup ... recently noticed that the 'accounts' and 'certificates' panes in the acme client portion of the UI have no content.
everything still "works" but
- nothing's shown in the UI....
- nothing is logged as to WHY nothing is shown in the UI.
This has become an increasingly common problem with various features within opnsense with the migration of various UI components.
While I applaud the innovation.... it's obscenely infuriating to run into these sorts of 'it just doesnt work' problems that present no errors when stuff's wonky.
querying the firewall's API:
apikey=keyhere; apisecret=secrethere; OPNapiCred="${apikey}:${apisecret}"; OPNSENSE="my.firewall.fqdn"; for jsonkey in account.id account.name account.ca.letsencrypt.selected; do
echo -ne "${jsonkey}: ";curl -sk -u "${OPNapiCred}" https://${OPNSENSE}/api/acmeclient/accounts/get|jq ".${jsonkey}";done
account.id: "6840c9029b1e24.89177252"
account.name: ""
account.ca.letsencrypt.selected: 1
which ... hmm ....
however...
[root@evey /tmp]# grep -c 6840c7f446c448.10990235 /conf/config.xml
0
Mostly sanitized xml snippit:
<AcmeClient version="4.2.0" persisted_at="1749070403.42">
<settings>
<enabled>1</enabled>
<autoRenewal>1</autoRenewal>
<UpdateCron>5050b8d5-285f-4e54-b405-2d9b0dbe0d86</UpdateCron>
<environment/>
<challengePort>43580</challengePort>
<TLSchallengePort>43581</TLSchallengePort>
<restartTimeout>600</restartTimeout>
<haproxyIntegration>0</haproxyIntegration>
<haproxyAclRef>xxx</haproxyAclRef>
<haproxyActionRef>yyy</haproxyActionRef>
<haproxyServerRef>uuu</haproxyServerRef>
<haproxyBackendRef>zzz</haproxyBackendRef>
<logLevel>normal</logLevel>
<showIntro>0</showIntro>
</settings>
<accounts>
<account uuid="65922bbc-a9fd-4f88-9ed3-4a4444bcf91e">
<id>5e5355ce0a8040.21993484</id>
<enabled>1</enabled>
<name>wpl LEstaging</name>
<description>base account</description>
<email>letsencrypt@medomain.com</email>
<ca>letsencrypt_test</ca>
<custom_ca/>
<eab_kid/>
<eab_hmac/>
<key>KEYHERE</key>
<statusCode>200</statusCode>
<statusLastUpdate>1611208803</statusLastUpdate>
</account>
<account uuid="82392a9d-c87c-4ddb-bfcb-9f2f1b3452f1">
<id>629e77ba2de515.54429234</id>
<enabled>1</enabled>
<name>mahdomain_io_prd</name>
<description>letsencrpyt for mahdomain.io</description>
<email>letsencrypt@mahdomain.io</email>
<ca>letsencrypt</ca>
<custom_ca/>
<eab_kid/>
<eab_hmac/>
<key>KEYHERE</key>
<statusCode>200</statusCode>
<statusLastUpdate>1654552524</statusLastUpdate>
</account>
<account uuid="ac5896ba-820f-4998-bfa8-c469b08f84e6">
<id>634d994e5e7622.19236562</id>
<enabled>1</enabled>
<name>LetsEncryptProd-letsencrypt@mahdomain.com</name>
<description>LetsEncryptProd-letsencrypt@mahdomain.com</description>
<email>letsencrypt@mahdomain.com</email>
<ca>letsencrypt</ca>
<custom_ca/>
<eab_kid/>
<eab_hmac/>
<key>KEYHERE</key>
<statusCode>200</statusCode>
<statusLastUpdate>1667152965</statusLastUpdate>
</account>
<account uuid="bfd665b5-d413-40c3-b9d7-54c02b521bfc">
<id>635ec8ac0ca223.65201712</id>
<enabled>1</enabled>
<name>letsencryptprod-letsencrypt@mahdomain.com</name>
<description>mahdomain.com certs</description>
<email>letsencrypt@mahdomain.com</email>
<ca>letsencrypt</ca>
<custom_ca/>
<eab_kid/>
<eab_hmac/>
<key>KEYHERE</key>
<statusCode>200</statusCode>
<statusLastUpdate>1667156154</statusLastUpdate>
</account>
<account uuid="3f9bc481-bb8a-49d1-9787-903d713d272b">
<id>65d7f338966714.88960649</id>
<enabled>1</enabled>
<name>2024_letsEncryptprod-skwirreltrap@mahdomain.com</name>
<description>LEProd to skwirreltrap</description>
<email>skwirreltrap@mahdomain.com</email>
<ca>letsencrypt</ca>
<custom_ca/>
<eab_kid/>
<eab_hmac/>
<key>KEYHERE</key>
<statusCode>200</statusCode>
<statusLastUpdate>1708651360</statusLastUpdate>
</account>
<account uuid="b7ca8960-5b0b-40ed-bfa8-27142c9be633">
<id>65d7ff9eaf5918.28302336</id>
<enabled>1</enabled>
<name>2024_mahdomain_LE_staging</name>
<description>mahdomain LE Staging</description>
<email>letsencrypt@mahdomain.com</email>
<ca>letsencrypt_test</ca>
<custom_ca/>
<eab_kid/>
<eab_hmac/>
<key/>
<statusCode>100</statusCode>
<statusLastUpdate/>
</account>
<account uuid="dbdb0671-6e7f-42f1-a332-fcb41cb4f04f">
<id>65d805e17c63b8.16159266</id>
<enabled>1</enabled>
<name>2024_mahdomain_io_prod</name>
<description>mahdomain.io - LE prod</description>
<email>domains@mahdomain.io</email>
<ca>letsencrypt</ca>
<custom_ca/>
<eab_kid/>
<eab_hmac/>
<key>KEYHERE</key>
<statusCode>200</statusCode>
<statusLastUpdate>1708656489</statusLastUpdate>
</account>
<account uuid="8524a51e-332b-4191-90f9-9503923b5abe">
<id>65d80672a9e4d0.44930860</id>
<enabled>1</enabled>
<name>2024_mahdomain_com_leprod</name>
<description>mahdomain.com - letsencrypt prod</description>
<email>domains@mahdomain.com</email>
<ca>letsencrypt</ca>
<custom_ca/>
<eab_kid/>
<eab_hmac/>
<key>KEYHERE</key>
<statusCode>200</statusCode>
<statusLastUpdate>1708656496</statusLastUpdate>
</account>
<account uuid="04f9bf67-9238-4db3-9bf5-0440514875a9">
<id>65d8074d76f383.85932244</id>
<enabled>1</enabled>
<name>2024_mahdomain_com_leprod</name>
<description>letsencrypt prod - mahdomain.com</description>
<email>domains@mahdomain.com</email>
<ca>letsencrypt</ca>
<custom_ca/>
<eab_kid/>
<eab_hmac/>
<key>KEYHERE</key>
<statusCode>200</statusCode>
<statusLastUpdate>1708656501</statusLastUpdate>
</account>
</accounts>
I'm a bit confused as to why the account value (singular) from the api isn't found within the config.xml
but okay...
[root@evey /tmp]# awk -F\" '/account uuid=/ {print $2}' /tmp/config.xml
65922bbc-a9fd-4f88-9ed3-4a4444bcf91e
82392a9d-c87c-4ddb-bfcb-9f2f1b3452f1
ac5896ba-820f-4998-bfa8-c469b08f84e6
bfd665b5-d413-40c3-b9d7-54c02b521bfc
3f9bc481-bb8a-49d1-9787-903d713d272b
b7ca8960-5b0b-40ed-bfa8-27142c9be633
dbdb0671-6e7f-42f1-a332-fcb41cb4f04f
8524a51e-332b-4191-90f9-9503923b5abe
04f9bf67-9238-4db3-9bf5-0440514875a9
for ACCOUNTUUID in $(awk -F\" '/account uuid=/ {print $2}' /tmp/config.xml); do curl -sk -u "${skwapi}" -d '{"uuid": "${ACCOUNTUID}" }' https://${skw}/api/acmeclient/accounts/get|jq .account.id ;done
"6840d4d3117fa6.81540267"
"6840d4d37687e2.58905660"
"6840d4d3dec033.66572673"
"6840d4d44eeac0.50883672"
"6840d4d4b649d7.69385764"
"6840d4d5270df9.27257680"
"6840d4d58e34e1.42312099"
"6840d4d6004852.43396427"
"6840d4d6696899.56775438"
as well as:
for CERTUUID in $(awk -F\" '/certificate uuid=/ {print $2}' /tmp/config.xml); do curl -sk -u "${skwapi}" -d '{"uuid": "${CERTUID}" }' https://${skw}/api/acmeclient/certificates/get|jq .certificate.id ;done
"6840d546937f81.27987089"
"6840d5470da2b1.66662430"
"6840d547778d40.03297044"
"6840d547dda4d3.95097759"
"6840d5484f4182.64843254"
"6840d548b56b69.37322399"
"6840d549280ad2.25541055"
"6840d5498dfcd0.06062109"
"6840d54a02f6e6.60224178"
"6840d54a6a3b97.86416756"
"6840d54acef892.64027770"
"6840d54b44aae4.47806167"
"6840d54baa8453.95779217"
"6840d54c1c04a1.40384875"
"6840d54c848c09.62165656"
"6840d54ce878b8.34705034"
"6840d54d5788e1.14449539"
"6840d54dbc5f44.63081339"
"6840d54e30abd7.22681242"
"6840d54e94c875.26891397"
"6840d54f0a26f8.09383901"
"6840d54f6e3960.63442021"
"6840d54fd0e409.40241557"
"6840d55041e5e3.38334821"
"6840d550ac6f68.85042886"
"6840d5511d5f68.53976736"
"6840d551840e13.41678246"
"6840d5520f8003.15666135"
"6840d55285f7f2.86828150"
"6840d552ed3517.53143605"
"6840d5535ffd23.63776110"
"6840d553c49313.46478168"
"6840d55467da21.13099485"
"6840d554cffe18.68834881"
"6840d555446310.48562770"
so.... things "are there" .... but the UI doesn't seem to agree .... there are simply no errors anywhere...
1) How should one go about diagnosing this?
The more concerning (to me) question tho: WHY is the software failing silently?
Failure is ..... to be expected occasionally
Doing so without any sort of explanation as to why feels .... not so awesome.
Sure, entirely possible I have some sort of wonk in my config somehow ....
why is there no noise about it?
:)
I'm having the exact same kind of thing. To make matters worse, I only discovered that there was a issue when it was reported to me that a domain had an expired cert (for HA proxy), after an hour of looking around and being unable to troubleshoot anything since like you found everything was blank, the only real solution from the GUI was a reboot. I'm not looking forward to repeating that when the next cert comes due
But what does the browser development tools say, any network errors? Any console log errors?
nadda on the error front, see the attached image. This instance has 2 ACME managed certs.
And the network tab? What does the API respond to the search?
Whats in the response?
API responds with a 200, and the relevant ACME sections from the json response is attached
Quote from: Monviech (Cedrik) on June 05, 2025, 09:37:04 PMBut what does the browser development tools say, any network errors? Any console log errors?
For me, no. nothing...
Quote from: DarcyB on June 05, 2025, 04:21:34 PMI'm having the exact same kind of thing. To make matters worse, I only discovered that there was a issue when it was reported to me that a domain had an expired cert (for HA proxy), after an hour of looking around and being unable to troubleshoot anything since like you found everything was blank, the only real solution from the GUI was a reboot. I'm not looking forward to repeating that when the next cert comes due
UGH!!!
I'm SORRY.
that super sucks.
What happens if you run (from shell on the fw)
[root@evey /home/wolfspyre]# configctl acmeclient cron-auto-renew
OK
While I see nothing at all added to the acmeclient log (/var/log/acmeclient/*)
Looking at the systemlog I see activity:
SESSION ONE:
[root@atticus /home/wolfspyre]# tail -f /var/log/system/latest.log |grep -i acmeclient
SESSION TWO:
[root@atticus /home/wolfspyre]# configctl acmeclient cron-auto-renew
OK
SESSION ONE:
<13>1 2025-06-06T02:36:34-05:00 atticus.wolfspyre.com opnsense-devel 65333 - [meta sequenceId="131"] AcmeClient: certificate must be issued/renewed: d....
MAYBE you'll have some luck with that? ((fingers crossed it at least gets you back to limping?))
plot thickens,
I cloned the instance into another VM, and attempted to restore a backup from the running instance.
I get a crash report, with the following.
[07-Jun-2025 17:42:22 Etc/UTC] Error: Call to undefined function system_trust_configure() in /usr/local/etc/inc/plugins.inc.d/core.inc:481
Stack trace:
#0 /usr/local/etc/inc/plugins.inc(323): core_trust_crl(false)
#1 /usr/local/opnsense/scripts/stunnel/generate_certs.php(90): plugins_configure('crl')
#2 {main}
[07-Jun-2025 17:42:28 Etc/UTC] Error: Call to undefined function system_trust_configure() in /usr/local/etc/inc/plugins.inc.d/core.inc:481
Stack trace:
#0 /usr/local/etc/inc/plugins.inc(323): core_trust_crl(false)
#1 /usr/local/opnsense/scripts/stunnel/generate_certs.php(90): plugins_configure('crl')
#2 {main}