[SOLVED] Problem with SSL Certificate / ACME / HAproxy

[andrew]

============================================
Final Update: scroll all the way down! It has been solved.
I still post all these notes unedited, to hopefully help others.
And to maybe get some answers to my stupid noob questions.
============================================

Hello guys,

I have a problem with ACME+HAproxy. Unfortunately, I'm not sure if it started right after the SSL cert was autorenewed,
or 1 day later when I updated OPNsense from an older 21.1.x to the latest 21.1.x.
The problem is, since either the renew or the update, the ACME/Letsencrypt SSL cert doesn't show up under Services -> HAProxy -> Maintenance -> SSL Certificates and HTTPS connections from the internet to HAproxy are not established anymore (smartphones who use MS Exchange ActiveSync (= HTTPS) through this reverse proxy).
Today I upgraded to 21.7. but it didn't fix the problem.


On the 28th the Cert was autorenewed.

ACME Log:

2021-09-28T00:00:33   acme.sh[1870]   ] Installing full chain to:/var/etc/acme-client/certs/604f8275329168.48298406/fullchain.pem
2021-09-28T00:00:33   acme.sh[53229]   ] Installing key to:/var/etc/acme-client/keys/604f8275329168.48298406/private.key
2021-09-28T00:00:33   acme.sh[23824]   ] Installing CA to:/var/etc/acme-client/certs/604f8275329168.48298406/chain.pem
2021-09-28T00:00:33   acme.sh[17629]   ] Installing cert to:/var/etc/acme-client/certs/604f8275329168.48298406/cert.pem


On the shell I actually found these 4 files, and they have the correct timestamp. Seems OK.


2021-09-28T00:00:32   acme.sh[31219]   ] And the full chain certs is there: /var/etc/acme-client/home/mail1.EXAMPLE.de/fullchain.cer
2021-09-28T00:00:32   acme.sh[7043]   ] The intermediate CA cert is in /var/etc/acme-client/home/mail1.EXAMPLE.de/ca.cer
2021-09-28T00:00:32   acme.sh[89542]   ] Your cert key is in /var/etc/acme-client/home/mail1.EXAMPLE.de/mail1.EXAMPLE.de.key
2021-09-28T00:00:32   acme.sh[34063]   ] Your cert is in /var/etc/acme-client/home/mail1.EXAMPLE.de/mail1.EXAMPLE.de.cer


On the shell I actually found these 4 files, but mail1.EXAMPLE.de.key has a timestamp from March! Is that OK?
Shouldn't it be a new one?



After forcing a reissue of the cert in the ACME Client:

ACME Log:

2021-09-30T13:55:40   acme.sh[55535]   ] Installing full chain to:/var/etc/acme-client/certs/604f8275329168.48298406/fullchain.pem
2021-09-30T13:55:40   acme.sh[94777]   ] Installing key to:/var/etc/acme-client/keys/604f8275329168.48298406/private.key
2021-09-30T13:55:40   acme.sh[32315]   ] Installing CA to:/var/etc/acme-client/certs/604f8275329168.48298406/chain.pem
2021-09-30T13:55:40   acme.sh[41172]   ] Installing cert to:/var/etc/acme-client/certs/604f8275329168.48298406/cert.pem
2021-09-30T13:55:40   acme.sh[10644]   ] And the full chain certs is there: /var/etc/acme-client/home/mail1.EXAMPLE.de/fullchain.cer
2021-09-30T13:55:40   acme.sh[49799]   ] The intermediate CA cert is in /var/etc/acme-client/home/mail1.EXAMPLE.de/ca.cer
2021-09-30T13:55:40   acme.sh[72121]   ] Your cert key is in /var/etc/acme-client/home/mail1.EXAMPLE.de/mail1.EXAMPLE.de.key
2021-09-30T13:55:39   acme.sh[4512]   ] Your cert is in /var/etc/acme-client/home/mail1.EXAMPLE.de/mail1.EXAMPLE.de.cer
2021-09-30T13:55:39   acme.sh[46883]   ] Cert success.
2021-09-30T13:55:38   acme.sh[25725]   ] Le_LinkCert='https://acme-v02.api.letsencrypt.org/acme/cert/031cf6ff4b15af3f48b8afad7df065c101a3'
2021-09-30T13:55:38   acme.sh[57964]   ] Downloading cert.
2021-09-30T13:55:36   acme.sh[93557]   ] Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/115804411/28318826760'
2021-09-30T13:55:36   acme.sh[35627]   ] Lets finalize the order.
2021-09-30T13:55:36   acme.sh[49165]   ] Verify finished, start to sign.
2021-09-30T13:55:36   acme.sh[30920]   ] autodiscover.EXAMPLE.de is already verified, skip http-01.
2021-09-30T13:55:36   acme.sh[51622]   ] mail1.EXAMPLE.de is already verified, skip http-01.
2021-09-30T13:55:36   acme.sh[99352]   ] mail1.EXAMPLE.de is already verified, skip http-01.
2021-09-30T13:55:36   acme.sh[51062]   ] Getting webroot for domain='autodiscover.EXAMPLE.de'
2021-09-30T13:55:35   acme.sh[65227]   ] Getting webroot for domain='mail1.EXAMPLE.de'
2021-09-30T13:55:35   acme.sh[49398]   ] Getting webroot for domain='mail1.EXAMPLE.de'
2021-09-30T13:55:28   acme.sh[96516]   ] Getting domain auth token for each domain
2021-09-30T13:55:28   acme.sh[90247]   ] Multi domain='DNS:mail1.EXAMPLE.de,DNS:mail1.EXAMPLE.de,DNS:autodiscover.EXAMPLE.de'
2021-09-30T13:55:28   acme.sh[69829]   ] Using CA: https://acme-v02.api.letsencrypt.org/directory


System Log:

2021-09-30T13:55:43   php[65196]   AcmeClient: running automation: Postfix
2021-09-30T13:55:41   php[65196]   AcmeClient: running automation: HAProxy
2021-09-30T13:55:41   php[65196]   AcmeClient: running automations for certificate: mail1.EXAMPLE.de
2021-09-30T13:55:40   opnsense[65196]   AcmeClient: updated ACME X.509 certificate: mail1.EXAMPLE.de
2021-09-30T13:55:40   opnsense[65196]   AcmeClient: successfully issued/renewed certificate: mail1.EXAMPLE.de
2021-09-30T13:55:26   opnsense[65196]   AcmeClient: using challenge type: http_validate
2021-09-30T13:55:25   opnsense[65196]   AcmeClient: account is registered: EXAMPLE
2021-09-30T13:55:25   opnsense[65196]   AcmeClient: using CA: letsencrypt
2021-09-30T13:55:25   opnsense[65196]   AcmeClient: issue certificate: mail1.EXAMPLE.de
2021-09-30T13:55:23   api[70849]   [2021-09-30T13:55:23+02:00][error] AcmeClient: HAProxy integration is complete
2021-09-30T13:54:28   opnsense[72181]   AcmeClient: issue/renewal not required for certificate: mail1.EXAMPLE.de
2021-09-30T13:54:26   api[61659]   [2021-09-30T13:54:26+02:00][error] AcmeClient: HAProxy integration is complete


Services -> ACME Client -> Certificates now shows the new cert with the new date/time

System -> Trust -> Certificates now shows the new cert with the new date/time

The HAproxy public service still has SSL Offloading selected, and the ACME cert is also still selected under "Certificates" and "Default Certificate".

/var/etc/acme-client/home/mail1.EXAMPLE.de/mail1.EXAMPLE.de.key STILL has a timestamp from March! Is that OK?

Services -> HAProxy -> Maintenance -> SSL Certificates STILL doesn't show any cert!



From this thread https://forum.opnsense.org/index.php?topic=9936.0 it seems HAproxy looks for the SSL cert in /var/etc/haproxy/ssl.
But /var/etc/haproxy doesn't exist:

root@OPNsense:/var/etc # ls -la
total 64
drwxr-xr-x   5 root  wheel   512 Sep 30 12:19 .
drwxr-xr-x  30 root  wheel   512 Sep  2 10:20 ..
drwxr-x---   8 root  wheel   512 Mar 15  2021 acme-client
-rw-------   1 root  wheel  1944 Sep 30 12:19 cert.pem
-rw-r--r--   1 root  wheel   187 Jul 11  2019 dhclient_wan.conf
-rw-------   1 root  wheel  3272 Sep 30 12:19 key.pem
-rw-r--r--   1 root  wheel  2522 Sep 30 14:06 lighttpd-acme-challenge.conf
-rw-r--r--   1 root  wheel  1968 Sep 30 12:18 lighttpd-api-dispatcher.conf
-rw-r--r--   1 root  wheel  5897 Sep 30 12:19 lighty-webConfigurator.conf
lrwxr-xr-x   1 root  wheel    26 Jul 11  2019 mpd.script -> /usr/local/sbin/mpd.script
-rw-r-----   1 root  wheel   968 Sep 30 12:19 mpd_wan.conf
-rw-r--r--   1 root  wheel    24 Sep 30 12:19 nameserver_pppoe0
-rw-r--r--   1 root  wheel     0 Sep 30 12:19 nameserver_v6pppoe0
-rw-r--r--   1 root  wheel   360 Sep 30 12:20 ntpd.conf
drwxr-x---   2 root  wheel   512 Mar 18  2020 openvpn
drwxr-x---   3 root  wheel   512 Jul 10  2019 openvpn-csc
-rw-r--r--   1 root  wheel  1789 Sep 30 12:19 syslog.conf

Is that OK in the current version of OPNsense and the HAproxy Plugin?
From which path and which filename does the current HAproxy plugin want to load the SSL cert?



UPDATE:
Now I'm reading this https://forum.opnsense.org/index.php?topic=24950.0


System -> Trust -> Authorities looks like this:

Name          Internal    Issuer    Certificates    Distinguished Name    
   
R3 (Let's Encrypt)    NO        external     1     O=Let's Encrypt, CN=R3, C=US
     Valid From:    Wed, 07 Oct 2020 21:21:40 +0200
     Valid Until:    Wed, 29 Sep 2021 21:21:40 +0200
   
R3 (ACME Client)    NO        external     0     O=Let's Encrypt, CN=R3, C=US
     Valid From:    Fri, 04 Sep 2020 02:00:00 +0200
     Valid Until:    Mon, 15 Sep 2025 18:00:00 +0200

Note that the old "Let's Encrypt" CA has 1 cert, while the new "ACME Client" CA has 0.
This matches System -> Trust -> Certificates:

Name                                Issuer          Distinguished Name
 mail1.EXAMPLE.de (ACME Client)      R3 (Let's Encrypt)     CN=mail1.EXAMPLE.de
CA: No, Server: Yes    
     Valid From:    Thu, 30 Sep 2021 12:55:38 +0200
     Valid Until:    Wed, 29 Dec 2021 11:55:37 +0100


So I followed this advice from mimugmail:
"You can also go to System : Trust : Authorities, remove the old CA which expires today, then go to LE plugin and renew all,
then go to your sevices and look if they are correctly linked and restart.
No patch necessary."

It worked fine. The newest renewed cert is now issued by the new "ACME Client" instead of "Let's Encrypt".
BUT: Services -> HAProxy -> Maintenance -> SSL Certificates STILL doesn't show any cert!
So I checked the HAproxy public service, and noticed that "Certificate" and "Default certificate" were empty now (as others have already warned).
So in both fields I selected the new "mail1.EXAMPLE.de (ACME Client)" cert, and clicked 'save' and then 'apply'.
BUT: Services -> HAProxy -> Maintenance -> SSL Certificates STILL doesn't show any cert!
So I stopped and started the HAproxy service from the Dashboard.
BUT: Services -> HAProxy -> Maintenance -> SSL Certificates STILL doesn't show any cert!
However, I noticed that external clients (such as my iPhones E-Mail app) can connect again without SSL error! Problem solved! Except for the Web GUI...

[5k7m4n]

Didn't work form me. I got ERR_CERT_DATE_INVALID after following your instructions.

[stesoell]

Didn't work form me. I got ERR_CERT_DATE_INVALID after following your instructions.

Check HAProxy settings - Public Service - HTTPS in (or similiar). Do you have selected a SSL offloading certificate (ACME)?