HAproxy + ACME client something went wrong in renewing certificates

[alefello]

Hello
I've a OPNsense installation with HAproxy managing a couple of subdomains providing SSL offloading and certificates for every subdomain managed by ACME client and issued from Letsencrypt.
The configuration is running great from years and survived several software updates without a problem.
Now it stopped renewing the certificates, without any modification or software update. It did an auto update of the certificates in june without problems, but failed the update for a couple of other subdomains yesterday.
The error in ACME logs is:

Code Select Expand

2025-07-30T10:15:10 opnsense AcmeClient: validation for certificate failed: filetransfer.XXXXXXXXXX.XX
2025-07-30T10:15:10 opnsense AcmeClient: domain validation failed (http01)
2025-07-30T10:15:08 opnsense AcmeClient: AcmeClient: The shell command returned exit code '1': '/usr/local/sbin/acme.sh --issue --syslog 9 --debug 3 --server 'letsencrypt' --webroot /var/etc/acme-client/challenges --home '/var/etc/acme-client/home' --cert-home '/var/etc/acme-client/cert-home/64dde6d44d35b5.13257403' --certpath '/var/etc/acme-client/certs/64dde6d44d35b5.13257403/cert.pem' --keypath '/var/etc/acme-client/keys/64dde6d44d35b5.13257403/private.key' --capath '/var/etc/acme-client/certs/64dde6d44d35b5.13257403/chain.pem' --fullchainpath '/var/etc/acme-client/certs/64dde6d44d35b5.13257403/fullchain.pem' --domain 'filetransfer.XXXXXXXXXX.XX' --days '1' --force --keylength '4096' --accountconf '/var/etc/acme-client/accounts/64ddda572ce593.09763895_prod/account.conf''
Could you please help me to figure out what is going wrong?
Thank you

[viragomann]

Now it stopped renewing the certificates, without any modification

Are you sure?

Code Select Expand

2025-07-30T10:15:10    opnsense    AcmeClient: validation for certificate failed: filetransfer.XXXXXXXXXX.XX
2025-07-30T10:15:10    opnsense    AcmeClient: domain validation failed (http01)


So the client said, that domain validation failed using http challenge.
You can go to the ACME settings and set the log level to debug to get more details about what's failing.

Ensure that the domain is resolved properly to your public IP in the public DNS.

Also ensure that the redirect_acme_challenges rule is in place in the port 443 frontend for the concerned IP.
And if the certificate has expired already, also ensure that there is as well a frontend with the redirect_acme_challenges rule in place which is listening on port 80.

[alefello]

Thank you for your reply

Are you sure?

Yes I am. The only change was a power outage some days ago, but everything started just fine after the power went back. I'm not working on these tasks (firewall, networking, etc) since months.

The versions of today I think were the forced renewals I tried to do.

So the client said, that domain validation failed using http challenge.
You can go to the ACME settings and set the log level to debug to get more details about what's failing.

I did that before posting.

Ensure that the domain is resolved properly to your public IP in the public DNS.

Already done that too.

Also ensure that the redirect_acme_challenges rule is in place in the port 443 frontend for the concerned IP.
And if the certificate has expired already, also ensure that there is as well a frontend with the redirect_acme_challenges rule in place which is listening on port 80.

I think everything is in the right place, as it worked at the end of june but not now.
    

What else could I try or could I provide to continue debugging?

Thank you

[alefello]

I'm sorry for having opened the thread and maybe had the wrong debug method.
I solved the problem.
I forgot I set up a HTTP challenge in the begginning that pointed to the internal OpnSense webserver, because using the HAproxy integration things never worked right. Bu I was able to figure out what was wrong with the HAproxy configuration for the challenge and to move the challenge to HAproxy and have it working.
Tha problem was that I followed the most famous tutorial to set up the HAproxy SSL offloading and ACME Let'sEncrypt certificates and that didn't work in validating certificates, so I had the workaround to use the internal webserver for the challenge.
I had to add a rule in HAproxy, opposite to the rule that checks the challenge and redirects that traffic to the internal backend of ACME plugin to validate the certificate, and apply this opposite rule to the traffic that has to be redirected to the services backend. I think otherwise the traffic for ACME challenge was redirected to the service backends before evaluating the rule that made it redirect to the ACME webserver.

Anyway it remains a mistery that until june the challenge with internal webserver worked fine, but it doesn't matter.