acme not working anymore (since 21 Dec 2023)

[rudiratlos63]

my last automatic cert renewal was executed last December. After upgrading opensense, (couldnt remeber when), cert renewals are failing. I looks like that the lighthtpd process running on port 43580 respond with Forbidden.

1. test on opensense root cli:
# fetch http://localhost:43580
fetch: http://localhost:43580: Forbidden

2. test on desktop firefox, calling
http://<internal IP of my openses>/.well-known/acme-challenge/XXXXidXXXXX....
results in Forbidden


sockstat -4 -l
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS     
root     lighttpd   94028 4  tcp4   127.0.0.1:43580       *:*

[tuxlemmi]

same here.

OPNsense running on port 8443/tcp. Some hosts behind with Port-Forwarding to 443/tcp.
In acme.sh log it shows one of the hosts behind - accessible with Port-forwarding to 443/tcp - that it uses the OPNsense https-Port 8443 to validate with the http-01-challenge.

"only ports 80 and 443 are supported, not 8443"

[muchacha_grande]

I had the same issue.
In my case, the last renewal was on January, 1st.
I reported this on the General Discussion Forum https://forum.opnsense.org/index.php?topic=38484.0
I solved the issue changing the challenge to DNS-01 using cPanel API.
I was using HTTP-01 challenge and after the 23.7.11 update it stopped working.

[tuxlemmi]

there is no dns-api support at STRATO. >:(

[muchacha_grande]

From what I could see, the server responds with a 302 redirect when the HTTP-01 challenge is trying to download the test token at /.well-known/acme-challenge/<TOKEN>.
I saw this at the Nginx logs.

[apiros]

I have the same problem. I suppose it will be solved with a future patch. In the meantime I'm back to version 13.1.11_2. By the way, when the certificate request is run, the address http://<internal IP of my openses>/.well-known/acme-challenge/XXXXidXXXXX is available and even the cert is fetched by the browser.

[rudiratlos63]

my response for http://<internal IP of my openses>/.well-known/acme-challenge/XXXXidXXXXX
is still forbidden.  Opensense Version OPNsense 24.1_1-amd64

[apiros]

I can only access it when the certificate request is running, otherwise not.

[Fright]

something like "unable to setup a port forward (empty ruleset)" in Services: ACME Client: Log Files ?

[muchacha_grande]

This is the ACME log of the first failure on January, 22nd:

2024-01-22T05:30:29-03:00   acme.sh   [Mon Jan 22 05:30:29 -03 2024] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh
2024-01-22T05:30:29-03:00   acme.sh   [Mon Jan 22 05:30:29 -03 2024] Please add '--debug' or '--log' to check more details.
2024-01-22T05:30:29-03:00   acme.sh   [Mon Jan 22 05:30:29 -03 2024] Invalid status, example.com:Verify error detail:w.x.y.z: Fetching https://example.com/.well-known/acme-challenge/EREIaZNm_HFsxaz64fDfizrzUVKeGQ_0CPtkZYHmEmE: Timeout during connect (likely firewall problem)
2024-01-22T05:30:26-03:00   acme.sh   [Mon Jan 22 05:30:26 -03 2024] Pending, The CA is processing your order, please just wait. (9/30)
2024-01-22T05:30:23-03:00   acme.sh   [Mon Jan 22 05:30:23 -03 2024] Pending, The CA is processing your order, please just wait. (8/30)
2024-01-22T05:30:21-03:00   acme.sh   [Mon Jan 22 05:30:21 -03 2024] Pending, The CA is processing your order, please just wait. (7/30)
2024-01-22T05:30:18-03:00   acme.sh   [Mon Jan 22 05:30:18 -03 2024] Pending, The CA is processing your order, please just wait. (6/30)
2024-01-22T05:30:15-03:00   acme.sh   [Mon Jan 22 05:30:15 -03 2024] Pending, The CA is processing your order, please just wait. (5/30)
2024-01-22T05:30:12-03:00   acme.sh   [Mon Jan 22 05:30:12 -03 2024] Pending, The CA is processing your order, please just wait. (4/30)
2024-01-22T05:30:10-03:00   acme.sh   [Mon Jan 22 05:30:10 -03 2024] Pending, The CA is processing your order, please just wait. (3/30)
2024-01-22T05:30:07-03:00   acme.sh   [Mon Jan 22 05:30:07 -03 2024] Pending, The CA is processing your order, please just wait. (2/30)
2024-01-22T05:30:04-03:00   acme.sh   [Mon Jan 22 05:30:04 -03 2024] Pending, The CA is processing your order, please just wait. (1/30)
2024-01-22T05:30:04-03:00   acme.sh   [Mon Jan 22 05:30:04 -03 2024] Verifying: example.com
2024-01-22T05:30:04-03:00   acme.sh   [Mon Jan 22 05:30:04 -03 2024] Getting webroot for domain='example.com'
2024-01-22T05:30:01-03:00   acme.sh   [Mon Jan 22 05:30:01 -03 2024] Getting domain auth token for each domain
2024-01-22T05:30:01-03:00   acme.sh   [Mon Jan 22 05:30:01 -03 2024] Single domain='example.com'
2024-01-22T05:30:01-03:00   acme.sh   [Mon Jan 22 05:30:01 -03 2024] Using CA: https://acme-v02.api.letsencrypt.org/directory
2024-01-22T05:30:00-03:00   acme.sh   [Mon Jan 22 05:30:00 -03 2024] Renew to Le_API=https://acme-v02.api.letsencrypt.org/directory
2024-01-22T05:30:00-03:00   acme.sh   [Mon Jan 22 05:30:00 -03 2024] Renew: 'example.com'

And this is the System Log:

2024-01-22T05:30:30-03:00   opnsense   AcmeClient: validation for certificate failed: example.com
2024-01-22T05:30:30-03:00   opnsense   AcmeClient: domain validation failed (http01)
2024-01-22T05:30:00-03:00   opnsense   AcmeClient: using challenge type: HTTP-01
2024-01-22T05:30:00-03:00   opnsense   AcmeClient: using IPv4 address: w.x.y.z
2024-01-22T05:30:00-03:00   opnsense   AcmeClient: account is registered: Admin
2024-01-22T05:30:00-03:00   opnsense   AcmeClient: using CA: letsencrypt
2024-01-22T05:30:00-03:00   opnsense   AcmeClient: renew certificate: example.com
2024-01-22T05:30:00-03:00   opnsense   AcmeClient: certificate must be issued/renewed: example.com

[Fright]

I suspect possible issues (imho the backend response needs to be json_decode-ed) in https://github.com/opnsense/plugins/commit/834a0dfa55fb608e6126c1536db8a9070227154a for cases when only the interface is specified in http-01 challenge properties, but your logs show that the address has been received.
however, it still looks like there are problems with creating the firewall\translation rules (in the validation properties, what HTTP service is specified? OPNsense?).
It makes sense to look for errors in the general/backend/firewall log at the time of certificate renewal. maybe there will be a hint there

[muchacha_grande]

Filtered those logs for the time at which the renewal process happened and found nothing.
Firewall had not blocked anything between 05:30:00 and 05:30:29.

This is the backend log, I see nothing either:

2024-01-22T05:30:29-03:00   Notice   configd.py   [9e5c85a1-74b3-471b-9e9f-7d8c7263d326] request pf current overall table record count and table-entries limit   
2024-01-22T05:30:29-03:00   Notice   configd.py   [24b90037-00d9-47cb-be25-df1665c8a008] Reloading filter   
2024-01-22T05:30:00-03:00   Notice   configd.py   [10ab735c-cb6f-4e84-98bb-b5c227534100] Reading primary IPv4 of wan   
2024-01-22T05:30:00-03:00   Notice   configd.py   [696cfc90-e22e-4d31-90dd-b37cbfbb1a22] request pf current overall table record count and table-entries limit   
2024-01-22T05:30:00-03:00   Informational   configd.py   message d86e94ef-a777-4271-986c-c00934c2a21e [] returned OK   
2024-01-22T05:30:00-03:00   Notice   configd.py   [d86e94ef-a777-4271-986c-c00934c2a21e] cronjob running to sign or renew certificates

[rudiratlos63]

according to this message:
https://forum.opnsense.org/index.php?topic=38694.0
I reinstalled acme. But same result. certs can not be renewed

[Fright]

@muchacha_grande

Fetching https://example.com/.well-known/..

what if you "Disable web GUI redirect rule" at System: Settings: Administration ?

[muchacha_grande]

Hi Fright,

what if you "Disable web GUI redirect rule" at System: Settings: Administration ?

I have it disabled already. Since a long time ago.