Messages

But what does the browser development tools say, any network errors? Any console log errors?

For me, no. nothing...

I'm having the exact same kind of thing. To make matters worse, I only discovered that there was a issue when it was reported to me that a domain had an expired cert (for HA proxy), after an hour of looking around and being unable to troubleshoot anything since like you found everything was blank, the only real solution from the GUI was a reboot.    I'm not looking forward to repeating that when the next cert comes due

UGH!!!

I'm SORRY.

that super sucks.

What happens if you run (from shell on the fw)

Code Select Expand

[root@evey /home/wolfspyre]# configctl acmeclient cron-auto-renew
OK




While I see nothing at all added to the acmeclient log  (/var/log/acmeclient/*)


Looking at the systemlog I see activity:


SESSION ONE:

Code Select Expand

[root@atticus /home/wolfspyre]# tail -f /var/log/system/latest.log |grep -i acmeclient


SESSION TWO:

Code Select Expand

[root@atticus /home/wolfspyre]# configctl acmeclient cron-auto-renew
OK

SESSION ONE:

Code Select Expand

<13>1 2025-06-06T02:36:34-05:00 atticus.wolfspyre.com opnsense-devel 65333 - [meta sequenceId="131"] AcmeClient: certificate must be issued/renewed: d....


MAYBE you'll have some luck with that? ((fingers crossed it at least gets you back to limping?))

Hai all!

functional ha setup ... recently noticed that the 'accounts' and 'certificates' panes in the acme client portion of the UI have no content.

everything still "works" but
- nothing's shown in the UI....
- nothing is logged as to WHY nothing is shown in the UI.

This has become an increasingly common problem with various features within opnsense with the migration of various UI components.

While I applaud the innovation.... it's obscenely infuriating to run into these sorts of 'it just doesnt work' problems that present no errors when stuff's wonky.
 


querying the firewall's API:

Code Select Expand

apikey=keyhere; apisecret=secrethere; OPNapiCred="${apikey}:${apisecret}"; OPNSENSE="my.firewall.fqdn"; for jsonkey in account.id account.name account.ca.letsencrypt.selected; do
  echo -ne "${jsonkey}: ";curl -sk -u "${OPNapiCred}" https://${OPNSENSE}/api/acmeclient/accounts/get|jq ".${jsonkey}";done
account.id: "6840c9029b1e24.89177252"
account.name: ""
account.ca.letsencrypt.selected: 1


which ... hmm ....

however...

Code Select Expand

[root@evey /tmp]# grep -c 6840c7f446c448.10990235 /conf/config.xml
0

Mostly sanitized xml snippit:

Code Select Expand

    <AcmeClient version="4.2.0" persisted_at="1749070403.42">
      <settings>
        <enabled>1</enabled>
        <autoRenewal>1</autoRenewal>
        <UpdateCron>5050b8d5-285f-4e54-b405-2d9b0dbe0d86</UpdateCron>
        <environment/>
        <challengePort>43580</challengePort>
        <TLSchallengePort>43581</TLSchallengePort>
        <restartTimeout>600</restartTimeout>
        <haproxyIntegration>0</haproxyIntegration>
        <haproxyAclRef>xxx</haproxyAclRef>
        <haproxyActionRef>yyy</haproxyActionRef>
        <haproxyServerRef>uuu</haproxyServerRef>
        <haproxyBackendRef>zzz</haproxyBackendRef>
        <logLevel>normal</logLevel>
        <showIntro>0</showIntro>
      </settings>
      <accounts>
        <account uuid="65922bbc-a9fd-4f88-9ed3-4a4444bcf91e">
          <id>5e5355ce0a8040.21993484</id>
          <enabled>1</enabled>
          <name>wpl LEstaging</name>
          <description>base account</description>
          <email>letsencrypt@medomain.com</email>
          <ca>letsencrypt_test</ca>
          <custom_ca/>
          <eab_kid/>
          <eab_hmac/>
          <key>KEYHERE</key>
          <statusCode>200</statusCode>
          <statusLastUpdate>1611208803</statusLastUpdate>
        </account>
        <account uuid="82392a9d-c87c-4ddb-bfcb-9f2f1b3452f1">
          <id>629e77ba2de515.54429234</id>
          <enabled>1</enabled>
          <name>mahdomain_io_prd</name>
          <description>letsencrpyt  for mahdomain.io</description>
          <email>letsencrypt@mahdomain.io</email>
          <ca>letsencrypt</ca>
          <custom_ca/>
          <eab_kid/>
          <eab_hmac/>
          <key>KEYHERE</key>
          <statusCode>200</statusCode>
          <statusLastUpdate>1654552524</statusLastUpdate>
        </account>
        <account uuid="ac5896ba-820f-4998-bfa8-c469b08f84e6">
          <id>634d994e5e7622.19236562</id>
          <enabled>1</enabled>
          <name>LetsEncryptProd-letsencrypt@mahdomain.com</name>
          <description>LetsEncryptProd-letsencrypt@mahdomain.com</description>
          <email>letsencrypt@mahdomain.com</email>
          <ca>letsencrypt</ca>
          <custom_ca/>
          <eab_kid/>
          <eab_hmac/>
          <key>KEYHERE</key>
          <statusCode>200</statusCode>
          <statusLastUpdate>1667152965</statusLastUpdate>
        </account>
        <account uuid="bfd665b5-d413-40c3-b9d7-54c02b521bfc">
          <id>635ec8ac0ca223.65201712</id>
          <enabled>1</enabled>
          <name>letsencryptprod-letsencrypt@mahdomain.com</name>
          <description>mahdomain.com certs</description>
          <email>letsencrypt@mahdomain.com</email>
          <ca>letsencrypt</ca>
          <custom_ca/>
          <eab_kid/>
          <eab_hmac/>
          <key>KEYHERE</key>
          <statusCode>200</statusCode>
          <statusLastUpdate>1667156154</statusLastUpdate>
        </account>
        <account uuid="3f9bc481-bb8a-49d1-9787-903d713d272b">
          <id>65d7f338966714.88960649</id>
          <enabled>1</enabled>
          <name>2024_letsEncryptprod-skwirreltrap@mahdomain.com</name>
          <description>LEProd to skwirreltrap</description>
          <email>skwirreltrap@mahdomain.com</email>
          <ca>letsencrypt</ca>
          <custom_ca/>
          <eab_kid/>
          <eab_hmac/>
          <key>KEYHERE</key>
          <statusCode>200</statusCode>
          <statusLastUpdate>1708651360</statusLastUpdate>
        </account>
        <account uuid="b7ca8960-5b0b-40ed-bfa8-27142c9be633">
          <id>65d7ff9eaf5918.28302336</id>
          <enabled>1</enabled>
          <name>2024_mahdomain_LE_staging</name>
          <description>mahdomain LE Staging</description>
          <email>letsencrypt@mahdomain.com</email>
          <ca>letsencrypt_test</ca>
          <custom_ca/>
          <eab_kid/>
          <eab_hmac/>
          <key/>
          <statusCode>100</statusCode>
          <statusLastUpdate/>
        </account>
        <account uuid="dbdb0671-6e7f-42f1-a332-fcb41cb4f04f">
          <id>65d805e17c63b8.16159266</id>
          <enabled>1</enabled>
          <name>2024_mahdomain_io_prod</name>
          <description>mahdomain.io - LE prod</description>
          <email>domains@mahdomain.io</email>
          <ca>letsencrypt</ca>
          <custom_ca/>
          <eab_kid/>
          <eab_hmac/>
          <key>KEYHERE</key>
          <statusCode>200</statusCode>
          <statusLastUpdate>1708656489</statusLastUpdate>
        </account>
        <account uuid="8524a51e-332b-4191-90f9-9503923b5abe">
          <id>65d80672a9e4d0.44930860</id>
          <enabled>1</enabled>
          <name>2024_mahdomain_com_leprod</name>
          <description>mahdomain.com - letsencrypt prod</description>
          <email>domains@mahdomain.com</email>
          <ca>letsencrypt</ca>
          <custom_ca/>
          <eab_kid/>
          <eab_hmac/>
          <key>KEYHERE</key>
          <statusCode>200</statusCode>
          <statusLastUpdate>1708656496</statusLastUpdate>
        </account>
        <account uuid="04f9bf67-9238-4db3-9bf5-0440514875a9">
          <id>65d8074d76f383.85932244</id>
          <enabled>1</enabled>
          <name>2024_mahdomain_com_leprod</name>
          <description>letsencrypt prod - mahdomain.com</description>
          <email>domains@mahdomain.com</email>
          <ca>letsencrypt</ca>
          <custom_ca/>
          <eab_kid/>
          <eab_hmac/>
          <key>KEYHERE</key>
          <statusCode>200</statusCode>
          <statusLastUpdate>1708656501</statusLastUpdate>
        </account>
      </accounts>

I'm a bit confused as to why the account value (singular) from the api isn't found within the config.xml

but okay...

Code Select Expand

[root@evey /tmp]# awk -F\" '/account uuid=/ {print $2}' /tmp/config.xml
65922bbc-a9fd-4f88-9ed3-4a4444bcf91e
82392a9d-c87c-4ddb-bfcb-9f2f1b3452f1
ac5896ba-820f-4998-bfa8-c469b08f84e6
bfd665b5-d413-40c3-b9d7-54c02b521bfc
3f9bc481-bb8a-49d1-9787-903d713d272b
b7ca8960-5b0b-40ed-bfa8-27142c9be633
dbdb0671-6e7f-42f1-a332-fcb41cb4f04f
8524a51e-332b-4191-90f9-9503923b5abe
04f9bf67-9238-4db3-9bf5-0440514875a9


Code Select Expand

for ACCOUNTUUID in $(awk -F\" '/account uuid=/ {print $2}' /tmp/config.xml); do curl -sk -u "${skwapi}" -d '{"uuid": "${ACCOUNTUID}" }' https://${skw}/api/acmeclient/accounts/get|jq .account.id ;done
"6840d4d3117fa6.81540267"
"6840d4d37687e2.58905660"
"6840d4d3dec033.66572673"
"6840d4d44eeac0.50883672"
"6840d4d4b649d7.69385764"
"6840d4d5270df9.27257680"
"6840d4d58e34e1.42312099"
"6840d4d6004852.43396427"
"6840d4d6696899.56775438"


as well as:

Code Select Expand

for CERTUUID in $(awk -F\" '/certificate uuid=/ {print $2}' /tmp/config.xml); do curl -sk -u "${skwapi}" -d '{"uuid": "${CERTUID}" }' https://${skw}/api/acmeclient/certificates/get|jq .certificate.id ;done
"6840d546937f81.27987089"
"6840d5470da2b1.66662430"
"6840d547778d40.03297044"
"6840d547dda4d3.95097759"
"6840d5484f4182.64843254"
"6840d548b56b69.37322399"
"6840d549280ad2.25541055"
"6840d5498dfcd0.06062109"
"6840d54a02f6e6.60224178"
"6840d54a6a3b97.86416756"
"6840d54acef892.64027770"
"6840d54b44aae4.47806167"
"6840d54baa8453.95779217"
"6840d54c1c04a1.40384875"
"6840d54c848c09.62165656"
"6840d54ce878b8.34705034"
"6840d54d5788e1.14449539"
"6840d54dbc5f44.63081339"
"6840d54e30abd7.22681242"
"6840d54e94c875.26891397"
"6840d54f0a26f8.09383901"
"6840d54f6e3960.63442021"
"6840d54fd0e409.40241557"
"6840d55041e5e3.38334821"
"6840d550ac6f68.85042886"
"6840d5511d5f68.53976736"
"6840d551840e13.41678246"
"6840d5520f8003.15666135"
"6840d55285f7f2.86828150"
"6840d552ed3517.53143605"
"6840d5535ffd23.63776110"
"6840d553c49313.46478168"
"6840d55467da21.13099485"
"6840d554cffe18.68834881"
"6840d555446310.48562770"


so.... things "are there" .... but the UI doesn't seem to agree .... there are simply no errors anywhere... 

1) How should one go about diagnosing this?


The more concerning (to me) question tho:  WHY is the software failing silently?
Failure is ..... to be expected occasionally
Doing so without any sort of explanation as to why feels .... not so awesome.

Sure, entirely possible I have some sort of wonk in my config somehow ....

why is there no noise about it?

:)

on the one hand, that **KINDA** makes sense... (commit / stage -> apply ) 
because there's likely cases wherein one would want to batch several related-but-isolated alterations.... and apply them once....

and I agree that it's .... surprising if ya don't know about it...

is there any mention of this in the docs?
 

Okay...
so...
it's **SOMETHING** to do with my sysctls, but I've not quite narrowed down wot yet. more digging to come. but.....

as a note to others ...

if ya run into something wobbly like this... try backing up your config and resetting all yer sysctls custom tunables .... if it solves yer problem, start adding them back and rebooting til you find the cause of the borkedness :)

allright... went back to 25.1.2.... and removed all sysctls / loader changes.... and I have logs again.

will walk back to current, then start reintroducing sysctls

Dunno if it is just me...but I see all screenshots blurred.

What happens if you leave everything in place and only revert the opn package ? I would reboot after to make sure there's a clean slate before retesting.

Code Select Expand

opnsense-revert -r 25.1.3 opnsense

Good question. will try on the secondary node here in a bit....

yeah, I made the images small as I didn't figure they needed to be huge to be legible, but praps I went a lil too optimize-crazy ;)

(the forum wouldn't let me post an image larger than 250k)

yes, I run non-rfc1918 unroutable addresses, but it **REALLY** shouldn't matter.

yes, lagg -> vlan bridges ... multiple isolated segments....  not **TYPICAL** sure, but not really an antipattern... just occasionally finicky

yes, I have a /28. each fw gets a /32 for themselves;
(.17 / .18) .19 is the catchall nat, many other services behind the fw pair are natted to distinct addresses....
that's not **that** abnormal ;)

the tcpdump showing traffic on the fw interface wasn't locked down to proto / src ... it was just picking up all the traffic on that vlan ... which ... sure... there's some noise...

but all of that is unlikely to cause NOTHING to be shown in any of the inspection panes ... :)

lastly
the overview pane, showing 'No Data Available' for anything of significance

Something's borked.... but nothing (obvious) is logged to point me in the direction of the borkedness ;)
(yes, that's a technical term :P )

I appreciate your input, and your request strategy...

 (I **DID** do all this (altho admittedly not as pedantically) before starting this thread, but a second lobescratcher is appreciated ;) )

furthering:
- the reporting pane of the webui showing that traffic is indeed transiting the vlan/lagg and (at least some of) the opnsense componentry exposing that
- the live view filtered by 'dst 8.8.8.8' showing nothing
- the live view filtered by interface showing nothing
- the live view filtered by src with the ip of the canary/smurf host ... showing nothing

(with no filter whatsoever there's still nothing at all visible, but being explicit in the imagery for shiggles)

following
- view of the interface, showing the icmp rule ordered first.
- view of the interface rules with all autogenerated/group rules expanded
- the canary host (smurf) pinging and the tcpdump on the firewall's vlan interface receiving the traffic

Apparently I'm restricted to 4 imgs/post. so I'll make a few.

I happened to be in the middle of setting up a new vlan when I noticed this problem, so I had a blank canvas.. here's:
- the interface config
- the egress link config
- the rule explicitly permitting and logging icmp (198.18.14.0/24) -> (8.8.8.8/32)
- the egress nat rule.

I'd really appreciate some insight, as I'm kinda at a loss as to how to solve my own problem here.

what feeds the log plumbing here?

How can I walk back the cat, as it were?

What information would be helpful in further teasing apart the problem I've made for myself here? :)

Okay,

at this point, I'm really pretty stumped.

- I have rolled my standby firewall back to 25.1.5_5
- I have replaced all -devel packages with their "stable" counterparts

I am still unable to see any firewalling events in live view.
now, if my config broke something, that's .... okay, whatever... that's on me.

Nevertheless, The problem encountered **STILL** aught be logged, but I'm not seeing anything obvious.
Neither the dashboard's live view of firewall actions, nor the  live view display anything.

This **IS** somewhat troublesome, as I've been attempting to sort out some bad traffic, and the lack of ability to inspect this in the webui is, mildly problematic.


Could really use some pointers as to what to inspect or what might be contributing to the problem...


I'm sure it's my fault.... but I'm stumped as to how. ;)

 

Curious.... So, I still have no live-view logs.... Seeing this post gave me hope...

however:

Code Select Expand

root@evey /home/wolfspyre]#  /usr/local/opnsense/scripts/filter/read_log.py --limit '1000'  --stream
event: keepalive
data:
event: keepalive
data:
event: keepalive
data:
event: keepalive
data:
event: keepalive
data:
event: keepalive
data:
[root@evey /home/wolfspyre]# opnsense-version
OPNsense 25.7.a_329 (amd64)


got any suggestions on what to look into?

well.... at least it's not **JUST** me :)