OPNWAF + HA - TLS certificates not synced

[ivosir]

Hello,

We have two OPNsense nodes in High Availability setup. We use the OPNWAF plugin to proxy back-end hosted web sites. The plugin automatically manages TLS certificates from Let's Encrypt using the ACME protocol.

I've realized that the certificates are not synced from Master to Backup. The folder structure /usr/local/md/domains, where the certificates are stored on Master, is created on Backup but each subfolder contains a fallback cert+key only. In the result, when the Backup node takes over, all web sites throw a certificate error. The certs do get gradually renewed from LE but it takes a while during which the web sites are unreachable.

Is this on purpose? Wouldn't it be better to synchronize the certs+keys from Master to Backup as well?

Thank you,
Ivo

[Monviech (Cedrik)]

The issue is that these certificates exist only on the filesystem and not inside the actual configuration (conf/config.xml) which gets synced by XMLRPC sync.

A HA setup with ACME automation is always a challenge.

In Caddy for example (a different Reverse Proxy that I maintain), the issue could be solved with some reverse proxy trickery, since it can do all ACME challenge types at the same time.

https://docs.opnsense.org/manual/how-tos/caddy.html#high-availability-setups

Another method would be using the ACME plugin, since it writes custom certificates into the config.xml. Custom certificates would also work seamlessly in OPNWAF since they're also stored in the config.xml.

I could look into OPNWAF to see if a reverse proxy trick like with caddy is possible to have both servers issue certificates at the same time with different challenge types, but that could take some time and probably not mature.

[ivosir]

Ah right, the OPNWAF-made certificates are not included in /conf/config.xml that is synced to the Backup. The Caddy way seems to be somewhat bumpy, but the ACME plugin + OPNWAF could be a viable solution, I will give it a try. Thanks for the hint!

Another possibility, somewhat harsh though, would be to reboot the Master from time to time to make the Backup take over and force it to renew the certs. :-)