HAProxy 4.3 Broken with Cloudflare Origin Cert and OCSP Automatic Update

[frunkaf]

I updated my OPNsense firmware yesterday to 24.1.1 and according to the changelog, that included an update to HAProxy, version 4.3 (2.8.5-aaba8d0). I didn't reboot until this morning and I noticed my HAProxy service was stopped. I tried restarting it to no avail so I went to settings and checked the syntax where I got the following error...

Code Select Expand

[NOTICE] (62651) : haproxy version is 2.8.5-aaba8d0
[NOTICE] (62651) : path to executable is /usr/local/sbin/haproxy
[ALERT] (62651) : config : parsing [/usr/local/etc/haproxy.conf.staging:73] : 'bind 127.4.4.3:443' in section 'frontend' : 'crt-list' : error processing line 1 in file '/tmp/haproxy/ssl/659dc76096fe83.76462299.certlist' : '/tmp/haproxy/ssl/65c3b55f42923.pem' has an OCSP URI and OCSP auto-update is set to 'on' but an error occurred (maybe the issuer could not be found)'.
[ALERT] (62651) : config : Error(s) found in configuration file : /usr/local/etc/haproxy.conf.staging
[ALERT] (62651) : config : Fatal errors found in configuration.

I googled and found the following issue on the HAProxy Github...
https://github.com/haproxy/haproxy/issues/2432

The issue being described in here is with regards to HAProxy producing this error message when evaluating a self-signed certificate with no URI declared in the pem file. This isn't my exact case as the Cloudflare origin cert I'm using does have a URI defined...

Code Select Expand

root@OPNsense:~ # openssl x509 -noout -text -in /tmp/haproxy/ssl/65c3b55f42923.pem | grep URI
OCSP - URI:http://ocsp.cloudflare.com/origin_ecc_ca
URI:http://crl.cloudflare.com/origin_ecc_ca.crl

The workaround is to disable 'Automatic OCSP updates' under Settings > Global Parameters > SSL settings. This allows me to use my Cloudflare Origin cert and keep the SSL/TLS encryption mode in Cloudflare to Full(Strict).

I can also keep 'Automatic OCSP updates' turned on, use any self-signed certificate for the HTTPS frontend public service, and dial back my SSL/TLS encryption mode in Cloudflare to Full(Not Strict).

Is anyone else experiencing this issue?

Also, side note...

I followed the amazing guide by TheHellSite...
https://forum.opnsense.org/index.php?topic=23339.0

Looks like it was recently updated to state, "OCSP updates are now built into HAProxy. No external Cron job is necessary anymore." AND it instructs you to add 'strict-sni' to the SSL Offloading section on your https frontend. This breaks my websites. Why? :(

[frunkaf]

Confirmed the issue was introduced with yesterday's update.

I reverted the haproxy plugin to 24.1's version...

Code Select Expand

opnsense-revert -r 24.1 os-haproxy

And now haproxy is working with the Cloudflare Origin cert and Automatic OCSP Updates enabled.

The only delta I see so far is the fact that the contents of the /tmp/haproxy/ssl/659dc76096fe83.76462299.certlist file doesn't have [] anymore...

/tmp/haproxy/ssl/65c3e946c3ef1.pem ocsp-update on

vs

/tmp/haproxy/ssl/65c3e946c3ef1.pem [ocsp-update on]

Not sure if that makes a difference...

[frunkaf]

I fixed it. Documenting this if anyone comes across this same issue in the future.

TLDR: Import the Cloudflare CA to your Authorities

The haproxy plugin in 4.1 was the bugged version. The updated plugin in 4.1.1 corrected the bug.

The valid syntax for the configuration is to have the [ocsp-update on] in brackets. So the reason my config worked on 4.1 is because the ocsp-update on parameter was invalid and not interpreted by the haproxy engine. 4.1.1 corrected the syntax and highlighted my actual issue which is that I needed to install the Certificate Authority for the Cloudflare Origin Certificate. This was made evident because the error states that, maybe the issuer could not be found. This refers to the .issuer file under the directory, /tmp/haproxy/ssl/, there was no .issuer file here for me.

I went here and downloaded the CA for my cert (ECC in my case)...
https://developers.cloudflare.com/ssl/origin-configuration/origin-ca/#cloudflare-origin-ca-root-certificate

Then I imported into System > Trust > Authorities, applied my haproxy configuration, and I got no error message in the GUI. The .issuer file appeared in the /tmp/haproxy/ssl/ directory.

Now everything works.

Thanks to these two folks, rlebreton and fraenki, for pointing me in the right direction on GH.
https://github.com/haproxy/haproxy/issues/2432#issuecomment-1935042531