Exporting LetsEncrypt Certificates in Automated way

[gazd25]

Hi Everybody,

Firstly thanks very much to everybody involved in making OPNSense the amazing product it is, I have had great luck with it so far, but have hit a wall I seem unable to get over and am hoping somebody can help.

I'm in the process of moving away from my old StartSSL certificates which have been painful for a while and have effectively worked out how to use LetsEncrypt and HAProxy to give me all the HTTP/HTTPS functionality I require.

The only problem I have leftover is to do with importing the certificates from the OPNSense firewall in an automated way to two mail servers in my environment, one of which is a mail scanning gateway and the other is an Exchange 2016 server. the reason for the certificate requirement is because I need to enable starttls receive functionality.

It ultimately would be far easier to use the LetsEncrypt instance on OPNSense to renew/maintain the certificates for my domain and automatically export and import them in to the servers as required every 60-90 days, but trying to automate this process is proving difficult.

Unfortunately everything I've tried so far doesn't seem to be working, which includes trying to automate a forms login for a user that only has access to the cert page and trying to find the certificates in a location I can script an SCP session to download from the firewall.

I realise of course I can just login to the system manually, click the button to export and then run the process to import them to my mail servers, but that sort of breaks the whole point of automating the certs in the first place

The API doesn't currently appear to allow access to the System > Trust > Certificate menu's so after days of trying and reading I'm, at the point of asking.

I'd be happy enough to create a cron job on the firewall itself which copies the certs out to another location automatically, which works around the issue of incoming security that the firewall is clearly very well hardened to protect against and for good reason, but my skills on this type of scripting are fairly limited.

Does anybody have any ideas?

Thanks in advance for any help

Gareth

[ChrisH]

+1
I'm having the exact same problem.
HTTPS, SMTPS (TCP/465) and IMAPS (TCP/993) can be handled by HAProxy, but for explicit TLS with STARTTLS the cert needs to be on the Exchange box(es).
At the moment I export the certs manually every 60 days, which is manageable but annoying. I have a PRTG sensor to check the certificate and reminds me if I forget.

Ideal would be some kind of trigger that exports the certs via SMB or SSH to another server after every LetsEncrypt refresh.

Alternatively HAProxy could learn to handle STARTTLS, but I guess that's far more effort.

[rungekutta]

I had a similar problem. LetsEncrypt cert bots running on public web server, every time certs being refreshed they need to be distributed around a few more servers (another web server, a smtp server, etc). I solved it by writing a small script which runs weekly, checks the last-modified date on the cert files, and if less than a week old redistributes them to the other servers and remotely restarts the relevant services to pick up the new certs. Was pretty easy to do with a combination of ssh and scp, at least in an all-unix environment. I can paste the script here later if of interest.

[gazd25]

Thanks Rungekutta,

My environment is typically all Windows currently, but it might be interesting to review your script anyway to see if I can apply a similar logic.

[rungekutta]

Ok here they are.

fetch-push.sh (runs weekly on a command & control Linux VM on the lan via cron):

Code: [Select]

#!/bin/bash
function age() {
   local filename=$1
   local changed=`stat -c %Y "$filename"`
   local now=`date +%s`
   local elapsed
   let elapsed=now-changed
   echo $elapsed
}

/root/pki/fetch.sh
# push new certs if less that 7.5d old
if [ $(age "/root/pki/cert.pem") -lt 648000 ]; then
        /root/pki/push.sh
fi

fetch.sh

Code: [Select]

#!/bin/sh
rsync --checksum -Ltve 'ssh' root@192.168.xx.yy:/etc/letsencrypt/live/[mydomain]/* /root/pki/

push.se

Code: [Select]

#!/bin/sh
echo Pushing config
scp /root/pki/*.pem root@192.168.aaa.bbb:/etc/pki/tls/letsencrypt/
scp /root/pki/*.pem root@192.168.ccc.ddd:/etc/pki/tls/letsencrypt/
[...]
echo Restarting remote services
ssh root@192.168.aaa.bbb "systemctl restart postfix"
ssh root@192.168.aaa.bbb "systemctl restart dovecot"
ssh root@192.168.ccc.ddd "systemctl restart apache2"
[...]

192.168.xx.yy is the web server that also runs the LetsEncrypt certbots hence generates certificates
192.168.aaa.bbb is an email server
192.168.ccc.ddd is another web server

[gazd25]

ok,

So I've been playing around with this quite a bit and have come up with the beginnings of a working solution, steps to implement as below:

* Create a user in the firewall web gui System > Access > Users. Give the user shell access only by editing the effective privileges, search for the word "user" at the top and tick only the box for "System: Shell account access" This grants the account minimal rights to login to the firewall shell.

* The wheel group is the owner of the folder which stores the certificates generated by the LetsEncrypt acme
client at the following location "/var/etc/acme-client/home/your cert common name". I have tested and this folder appears to store the certificates after each refresh/renew so is the location we will use to download our certs and keys from.

* Login as root and edit this file using vi "/usr/local/etc/rc.subr.d/recover" at line 34 from this
"wheel:*:0:root" to this "wheel:*:0:root,%your user name%", substituting %your user name% with the user you created earlier. This file is used to regenerate the group memberships at reboot and is the only way I could find to make the membership of the wheel group persist across reboots.

* I would suggest test rebooting the firewall if possible to ensure your edit of the file at the preceding step was successful and the user you created can now login to the shell and view/read the files at /var/etc/acme-client/home/your cert common name.

* If you are unable to reboot, you can immediately add the user to the wheel group at the command line by logging in using putty and running this command "pw groupmod wheel -m %your user name%", you can then confirm the user has been added by running this command "pw groupshow wheel" which should show this = wheel:*:0:root,%your user name%.

* Next we can build the simple bat script that will download the cert files, I chose to use pscp.exe for this
since it's simple and easy to work with, essentially my script for downloading the files looks like below:

"C:\Program Files (x86)\PuTTY\pscp.exe" -pw yourpasssord youruser@yourfirewall:/var/etc/acme-client/home/yourcert CN/fullchain.cer "C:\youroutputpath\yourcert CN.cer"
"C:\Program Files (x86)\PuTTY\pscp.exe" -pw yourpassword youruser@yourfirewall:/var/etc/acme-client/home/yourcert CN/yourcert CN.key "C:\youroutputpath\yourcert CN.key."

The above commands assume you have Putty and pscp.exe installed at the default location and you can specify the output path by changing the "C:\youroutputpath\yourcert CN.cer" element on the end of the commands.

* You could of course just enter the root username and password here and basically have avoided all of the above steps to create and lockdown the user, but I prefer the idea of using a new user that has privileges limited
to those of the wheel group, which honestly isn't much.

* Once you have completed the above steps you should now have the full cert chain and the key file at the location you specified when creating the script. Once you have them there, you can manipulate them using openssl to create a P12 file which can then be imported in to Exchange.


I still have to work out automating that next step, but this is a good start at least

[gazd25]

Also,

Forgot to add that you will need to run the bat script for pscp for the first time interactively to allow you to accept the host key for the firewall, and if the host key is regenerated at any point in the future you will also need to do the same again or the script will fail.

[fabian]

you get the wheel group by the administrators group and it makes you more or less root.

[gazd25]

Ah,

Well in that case all that other stuff was a waste of time

I should have just used root.

[fabian]

It seems so but at least you have learnt something about the system (paths, users, groups etc.)

[rungekutta]

Gazd25, you may want to avoid storing passwords in scripts as well, not the least kept on hosts that are also public web servers. You could avoid passwords by using certificates, although that still doesn’t change the fact that an attacker who gains control over the web server can still access the router. How about doing it the other way around, and pushing the certs from opnsense onto the windows server as opposed to pulling them? If only you can figure out a mechanism for that in Windows, I guess ssh may be tricky, maybe an ftp server? Scripting and automation is easy on the unix side.

[gazd25]

Hi Rungekutta,

You are of course correct about storing passwords, but I am not storing them in anything that's publicly accessible. I'm using a non-published server to pull my certs from the firewall, run whatever tasks are necessary and then push them out to whatever location they need to be in, so I've minimised any risk as much as I possibly can.

Then lastly I have created scripts on the published servers which run a day later than the certificate update/renewal runs to import the new certificates in to which ever applications once the files have been placed there.

Actually I have just completed the task and have managed to get both my mail gateways and my Exchange server to recognise the certificates in a fully automated way, though I will admit it's been a pretty tough thing to do, requiring me to learn various scripting techniques.

I agree that pushing them from the firewall to the Windows servers would be preferable and will also look at this as a next step, but that's probably going to take me a while longer.

Thanks again for the help

[gazd25]

For those that might be interested, the rest of the solution after the last post looks like this:

My mail gateways are able to use the certificates in the standard .pem cert and key format that is downloaded by the bat script I mentioned earlier so that was a very easy job to script, which just involved a service restart after placing the certs in the correct location.

The Exchange import was a significantly more complicated, requring firstly a conversion of the .pem formart to PFX, then a Powershell script to import.

For the conversion to pem, I downloaded an openssl binary for windows from the openssl website directly and followed the instructions here:

https://www.tbs-certificates.co.uk/FAQ/en/openssl-windows.html

To create an example openssl.cnf file and then add the PATH variable for the location I had added the .cnf file too.

Once that was done I just pass the cert and key files in to openssl to output a .pfx with password using the below command:

"C:\pathtopenssl\openssl" pkcs12 -export -inkey "C:\pathtoyourcert\yourcert.key" -in "C:\pathtoyourcert\yourcert.cer" -out "C:\outputpathforyourpfx\yourpfx.pfx" -password pass:yourpassword

Once completed I now have a .pfx certificate file which contains the private key protected by the password from the command.

Lastly to perform the import of the certificate on Exchange I created a powershell script which I have santised and attached, I have made comments in the script about what each step is doing so you should be able to work out how to use if you wish to.

I have added the script using a task scheduler job for Powershell as per the instructions here:

https://community.spiceworks.com/how_to/17736-run-powershell-scripts-from-task-scheduler


The script performs the below actions in order:

* Add Exchange management tools snappin
* Create new self signed fake certificate to be used only during script
* Create variable for self-signed fake cert  and enable all Exchange services IIS,SMTP,POP,IMAP for interim
* Create variable for old LetsEncrypt certificate and remove
* Import new LetsEncrypt certificate
* Create variable for new LetsEncrypt cert and enable all Exchange services IIS,SMTP,POP,IMAP
* Remove fake cert now it's done it's job of taking Exchange roles for interim


A few very important things to keep in mind about the script:

The order of the script is very particular so be careful if you change it at all.

The script assumes either it is being ran from a server that has the Exchange remote management tools installed, or the Exchange server itself.

There are various locations in the script that generate Y/N requests for input, so to solve this I have used the .NET sendkeys method to insert the answer, and yes I know it looks daft having the sendkeys in before the actual command, but when running the script non-interactively, the timing of the key presses works nicely.

The sendkeys are used to send N in answer to all of the prompts, the reason for this is that the prompt is asking whether you would like to change the default SMTP certificate. Unless you enjoy pain, please DO NOT change this to a Y because it changes the certificate used for internal communication between Exchange servers leaving the server unable to relay internally correctly and causing potentially all kinds of other problems! Dont ask me how I know this .

The script assumes there is already an existing letsEncrypt certificate installed which will be removed in favour of the new LetsEncrypt cert, but you could edit it to remove whatever cert you want as long as you match one of the SAN/FQDN's from the command.

The script is provided as-is without any warranties for you to edit to suit your needs. I have tried my best to make it clear from the comments what you would need to change to make it functional for your environment but please make sure you understand what the script is doing and why before you try to deploy it, otherwise problems may result since it's editing things that are highly critical to email movement in your organisation.



Lastly, I recognise of course some of this is quite clunky, but it's the only way I've been able to make it work so far. I'm very much open to suggestions/improvements if anybody has any so please say.

Thanks

[fabian]

I would recommend you not to attach the file to the post and upload it to GitHub instead (as a Gist because it is a single file) because Post attachment is only visible to logged in users.

[gazd25]

Thanks Fabian,

I didn't realise users couldn't download unless logged in.

I've now created a public Gist of the file which anybody can download here:

https://gist.github.com/gazd25/2b7418f27562c80ed3a12d0f379a05d8