OPNsense Forum

English Forums => 24.1 Production Series => Topic started by: rudiratlos63 on February 02, 2024, 11:50:18 am

Title: acme not working anymore (since 21 Dec 2023)
Post by: rudiratlos63 on February 02, 2024, 11:50:18 am: my last automatic cert renewal was executed last December. After upgrading opensense, (couldnt remeber when), cert renewals are failing. I looks like that the lighthtpd process running on port 43580 respond with Forbidden.

1. test on opensense root cli:
# fetch http://localhost:43580
fetch: http://localhost:43580: Forbidden

2. test on desktop firefox, calling
http://<internal IP of my openses>/.well-known/acme-challenge/XXXXidXXXXX....
results in Forbidden

sockstat -4 -l
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root lighttpd 94028 4 tcp4 127.0.0.1:43580 *:*
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: tuxlemmi on February 02, 2024, 01:12:49 pm: same here.

OPNsense running on port 8443/tcp. Some hosts behind with Port-Forwarding to 443/tcp.
In acme.sh log it shows one of the hosts behind - accessible with Port-forwarding to 443/tcp - that it uses the OPNsense https-Port 8443 to validate with the http-01-challenge.

"only ports 80 and 443 are supported, not 8443"
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: muchacha_grande on February 02, 2024, 06:48:04 pm: I had the same issue.
In my case, the last renewal was on January, 1st.
I reported this on the General Discussion Forum https://forum.opnsense.org/index.php?topic=38484.0
I solved the issue changing the challenge to DNS-01 using cPanel API.
I was using HTTP-01 challenge and after the 23.7.11 update it stopped working.
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: tuxlemmi on February 03, 2024, 07:29:11 pm: there is no dns-api support at STRATO. >:(
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: muchacha_grande on February 05, 2024, 12:52:40 pm: From what I could see, the server responds with a 302 redirect when the HTTP-01 challenge is trying to download the test token at /.well-known/acme-challenge/<TOKEN>.
I saw this at the Nginx logs.
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: apiros on February 05, 2024, 01:55:03 pm: I have the same problem. I suppose it will be solved with a future patch. In the meantime I'm back to version 13.1.11_2. By the way, when the certificate request is run, the address http://<internal IP of my openses>/.well-known/acme-challenge/XXXXidXXXXX is available and even the cert is fetched by the browser.
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: rudiratlos63 on February 05, 2024, 05:05:38 pm: my response for http://<internal IP of my openses>/.well-known/acme-challenge/XXXXidXXXXX
is still forbidden. Opensense Version OPNsense 24.1_1-amd64
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: apiros on February 05, 2024, 08:16:31 pm: I can only access it when the certificate request is running, otherwise not.
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: Fright on February 08, 2024, 07:01:09 pm: something like "unable to setup a port forward (empty ruleset)" in Services: ACME Client: Log Files ?
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: muchacha_grande on February 08, 2024, 08:33:16 pm: This is the ACME log of the first failure on January, 22nd:

Quote
2024-01-22T05:30:29-03:00   acme.sh   [Mon Jan 22 05:30:29 -03 2024] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh
2024-01-22T05:30:29-03:00   acme.sh   [Mon Jan 22 05:30:29 -03 2024] Please add '--debug' or '--log' to check more details.
2024-01-22T05:30:29-03:00   acme.sh   [Mon Jan 22 05:30:29 -03 2024] Invalid status, example.com:Verify error detail:w.x.y.z: Fetching https://example.com/.well-known/acme-challenge/EREIaZNm_HFsxaz64fDfizrzUVKeGQ_0CPtkZYHmEmE: Timeout during connect (likely firewall problem)
2024-01-22T05:30:26-03:00   acme.sh   [Mon Jan 22 05:30:26 -03 2024] Pending, The CA is processing your order, please just wait. (9/30)
2024-01-22T05:30:23-03:00   acme.sh   [Mon Jan 22 05:30:23 -03 2024] Pending, The CA is processing your order, please just wait. (8/30)
2024-01-22T05:30:21-03:00   acme.sh   [Mon Jan 22 05:30:21 -03 2024] Pending, The CA is processing your order, please just wait. (7/30)
2024-01-22T05:30:18-03:00   acme.sh   [Mon Jan 22 05:30:18 -03 2024] Pending, The CA is processing your order, please just wait. (6/30)
2024-01-22T05:30:15-03:00   acme.sh   [Mon Jan 22 05:30:15 -03 2024] Pending, The CA is processing your order, please just wait. (5/30)
2024-01-22T05:30:12-03:00   acme.sh   [Mon Jan 22 05:30:12 -03 2024] Pending, The CA is processing your order, please just wait. (4/30)
2024-01-22T05:30:10-03:00   acme.sh   [Mon Jan 22 05:30:10 -03 2024] Pending, The CA is processing your order, please just wait. (3/30)
2024-01-22T05:30:07-03:00   acme.sh   [Mon Jan 22 05:30:07 -03 2024] Pending, The CA is processing your order, please just wait. (2/30)
2024-01-22T05:30:04-03:00   acme.sh   [Mon Jan 22 05:30:04 -03 2024] Pending, The CA is processing your order, please just wait. (1/30)
2024-01-22T05:30:04-03:00   acme.sh   [Mon Jan 22 05:30:04 -03 2024] Verifying: example.com
2024-01-22T05:30:04-03:00   acme.sh   [Mon Jan 22 05:30:04 -03 2024] Getting webroot for domain='example.com'
2024-01-22T05:30:01-03:00   acme.sh   [Mon Jan 22 05:30:01 -03 2024] Getting domain auth token for each domain
2024-01-22T05:30:01-03:00   acme.sh   [Mon Jan 22 05:30:01 -03 2024] Single domain='example.com'
2024-01-22T05:30:01-03:00   acme.sh   [Mon Jan 22 05:30:01 -03 2024] Using CA: https://acme-v02.api.letsencrypt.org/directory
2024-01-22T05:30:00-03:00   acme.sh   [Mon Jan 22 05:30:00 -03 2024] Renew to Le_API=https://acme-v02.api.letsencrypt.org/directory
2024-01-22T05:30:00-03:00   acme.sh   [Mon Jan 22 05:30:00 -03 2024] Renew: 'example.com'

And this is the System Log:

Quote
2024-01-22T05:30:30-03:00   opnsense   AcmeClient: validation for certificate failed: example.com
2024-01-22T05:30:30-03:00   opnsense   AcmeClient: domain validation failed (http01)
2024-01-22T05:30:00-03:00   opnsense   AcmeClient: using challenge type: HTTP-01
2024-01-22T05:30:00-03:00   opnsense   AcmeClient: using IPv4 address: w.x.y.z
2024-01-22T05:30:00-03:00   opnsense   AcmeClient: account is registered: Admin
2024-01-22T05:30:00-03:00   opnsense   AcmeClient: using CA: letsencrypt
2024-01-22T05:30:00-03:00   opnsense   AcmeClient: renew certificate: example.com
2024-01-22T05:30:00-03:00   opnsense   AcmeClient: certificate must be issued/renewed: example.com
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: Fright on February 09, 2024, 05:23:16 am: I suspect possible issues (imho the backend response needs to be json_decode-ed) in https://github.com/opnsense/plugins/commit/834a0dfa55fb608e6126c1536db8a9070227154a for cases when only the interface is specified in http-01 challenge properties, but your logs show that the address has been received.
however, it still looks like there are problems with creating the firewall\translation rules (in the validation properties, what HTTP service is specified? OPNsense?).
It makes sense to look for errors in the general/backend/firewall log at the time of certificate renewal. maybe there will be a hint there
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: muchacha_grande on February 09, 2024, 01:35:52 pm: Filtered those logs for the time at which the renewal process happened and found nothing.
Firewall had not blocked anything between 05:30:00 and 05:30:29.

This is the backend log, I see nothing either:

Quote
2024-01-22T05:30:29-03:00   Notice   configd.py   [9e5c85a1-74b3-471b-9e9f-7d8c7263d326] request pf current overall table record count and table-entries limit
2024-01-22T05:30:29-03:00   Notice   configd.py   [24b90037-00d9-47cb-be25-df1665c8a008] Reloading filter
2024-01-22T05:30:00-03:00   Notice   configd.py   [10ab735c-cb6f-4e84-98bb-b5c227534100] Reading primary IPv4 of wan
2024-01-22T05:30:00-03:00   Notice   configd.py   [696cfc90-e22e-4d31-90dd-b37cbfbb1a22] request pf current overall table record count and table-entries limit
2024-01-22T05:30:00-03:00   Informational   configd.py   message d86e94ef-a777-4271-986c-c00934c2a21e [] returned OK
2024-01-22T05:30:00-03:00   Notice   configd.py   [d86e94ef-a777-4271-986c-c00934c2a21e] cronjob running to sign or renew certificates
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: rudiratlos63 on February 09, 2024, 06:29:06 pm: according to this message:
https://forum.opnsense.org/index.php?topic=38694.0
I reinstalled acme. But same result. certs can not be renewed
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: Fright on February 09, 2024, 08:06:12 pm: @muchacha_grande
Quote
Fetching https://example.com/.well-known/..
what if you "Disable web GUI redirect rule" at System: Settings: Administration ?
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: muchacha_grande on February 09, 2024, 10:05:06 pm: Hi Fright,

Quote from: Fright on February 09, 2024, 08:06:12 pm
what if you "Disable web GUI redirect rule" at System: Settings: Administration ?

I have it disabled already. Since a long time ago.
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: Fright on February 10, 2024, 01:44:09 pm: @ Hi
Hm
I tried it on 24.1 (with a small HttpOpnsense.php-patch (pr#3813) to get the interface address, but this is not your case if the address is already displayed in the log) - everything works as expected ..
Quote
I have it disabled already. Since a long time ago.
but something still redirects token request to https?
Code: [Select]
Fetching https://example.com/.well-known/acme-challenge/EREIaZNm_HFsxaz64fDfizrzUVKeGQ_0CPtkZYHmEmE: Timeout during connect
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: muchacha_grande on February 12, 2024, 06:30:07 pm: The interface address has been got correctly, I just obfuscated that.

Before switching to DNS-01 challenge I tried to renew the certificates a couple of times and I could see the requests on Nginx log. Then Nginx responded with a 302 code. I could not see the details of the response but the log showed a 302 code (temporary redirection).
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: Fright on February 12, 2024, 07:05:59 pm: ah. so the initial request is processed by nginx..
Probably "HTTPS Only" is set in the server settings in nginx.
the request is redirected to HTTPS (with 302 code).
then, if the "Enable Let's Encrypt Plugin Support" server settings are enabled, the request for /.well-known/acme-challenge/* should be sent to the acme lighttpd instance (https://github.com/opnsense/plugins/blob/cab29219d7fb43bc77bbffd8224a8a2cddb59b22/www/nginx/src/opnsense/service/templates/OPNsense/Nginx/http.conf#L253-L260).
there must be something more in the logs (access or errors) on nginx
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: muchacha_grande on February 13, 2024, 11:34:25 pm: @Fright, I tried to renew a certificate using HTTP-01 to catch the logs and look for clues based on your advice, but ACME skipped the HTTP-01 verification because the domain is already verified, so I couldn't see the complete process.
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: MarekWojtaszek on February 17, 2024, 04:34:07 pm: Same issue here :(
I can't even register any new certificate with the same challenge - I am getting the same errors.

Anybody?
Marek
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: MarekWojtaszek on February 17, 2024, 05:23:56 pm: Here is what I see in my syslog and what started in Dec 2023:

(https://i.postimg.cc/ncf7rgXw/le-error.png)

Any suggestion on how to troubleshoot would be greatly appreciated.
Marek
Title: Re: acme not working anymore (since 21 Dec 2023)
Post by: ilfalti on February 20, 2024, 03:37:54 pm: Hi, problem solved with 24.1.2
(os-acme-client 4.1)