Importing Intermediate (Issuing) CA Certificates

[benyamin]

A regression in handling the importation of intermediate CA certificates led to a general discussion regarding the importation of such certificates into the OPNsense trusted store.

The topic can be found here.

Given the discussion had thoroughly hijacked that topic, I thought it best to continue here...

Credit to Fright for continuing the conversation over there (and hopefully here too).

I think it remains unresolved as to if, when and how third-party or external intermediate CA certificates should be installed (at System: Trust).

For completeness, I note that it appears the internal intermediate CA (when established) cannot issue internal certificates.

To that end:

It is my understanding that it remains best practice to install intermediate CA certificates that form part of a certificate chain to internal services. Just to be clear, I'm not suggesting that intermediate CA certs should be imported for say external websites.

These certificates should normally form part of the Certificates message but in my experience that isn't always implemented. Courtesy of Fright's previous postings, the pertinent part of the relevant RFC is here for your reading pleasure.

I had suggested that any Intermediate CA should be imported into System: Trust: Certificates (marked as a CA) and the corresponding root CA to System: Trust: Authorities. I only said this because I presume a CA listed under Authorities cannot be revoked... I am not convinced this is the case. I have not been able to test whether a Root CA CRL could revoke an intermediate CA certificate imported into System: Trust: Authorities. I would normally assume this to be the expected behaviour.

Furthermore, after attempting to do this I realised it was not an appropriate solution as the private key also must be imported.

If I am not mistaken, it is noteworthy, that the change in default behaviour was prompted by the recent Let's Encrypt saga. It is worth looking at the discussions on GitHub for issues 5257 and 2554. It gives good insight into the mitigation employed and kulikov-a (which I presume is you Fright) explains the issues well.

I would like to continue a discussion about what should constitute best practice in the use of intermediate CA certificates. I expect the discussion will also include the importation of both Root and Intermediate CA CRLs. Perhaps this discussion will identify and / or resolve any necessary improvements in this space.

Please join in.

[benyamin]

It might be best to move this to General Discussion or another parent forum. I leave it to a mod to decide...

[Fright]

For completeness, I note that it appears the internal intermediate CA (when established) cannot issue internal certificates

sorry, dont understand this part..

and for the record: I am not claiming that there is only one right choice. I'm only wondering why admins trust intermediate CAs even when, in my opinion, this is not required. Everything else is just argumentation of my point of view

It is my understanding that it remains best practice to install intermediate CA certificates that form part of a certificate chain to internal services. Just to be clear, I'm not suggesting that intermediate CA certs should be imported for say external websites.

If security is a matter of trust, then from a best practice perspective, why should the behavior be different for internal services?

These certificates should normally form part of the Certificates message but in my experience that isn't always implemented.

Maybe. but in this case, imho the first attempt should be to eliminate the cause on the service side and only then workaround the problem using the trusted store (if it works). (in windows for such cases there is a separate store of intermediate certificates, but the certificates placed there do not become trusted. only helps to build the chain afaik)
and this is not counting the unpredictability of the result of adding an intermediate certificate to the store: openssl does not trust them, but uses them to select the chain. curl uses them as a trusted anchor and this can have unexpected consequences also (if you add an expired R3 intermediate certificate https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.pem to the trusted store (21.7.4 will not allow you to do this, you will have to add it by hand), then the 'curl https://letsencrypt.org' will give an error about an expired certificate, although this is not a root certificate and the server certificate is not even signed by this certificate (!)  LE used the same private key when requesting a new intermediate certificate. so curl defines it as issuer cert)

I had suggested that any Intermediate CA should be imported into System: Trust: Certificates

maybe I'm missing something. how should the system use intermediate certificates placed there (assuming that this would be possible)?

[Fright]

I also think that General Discussions are more suitable for such topics

[benyamin]

I've sent Ad a PM requesting he move the topic to General Discussions.

Regarding the internal intermediate CA: I was just saying that it cannot issue certificates via the GUI. Just for completeness, it really has no bearing on the discussion.

Regarding best practice and limiting the scope to internal services, I guess it would be better to include those services where OPNsense is acting as a client. I would say it best to exclude external websites because it should be presumed that those sites issue full and correct Certificate messages.

I agree that intermediate CA certs should only be used to build chains and verify trust as part of that chain. All implementations should require a root CA cert be present to establish trust. It is also my experience that M$, OpenSSL and Java treat intermediates CA certs this way.

@Fright, it looks like you have identified an issue with curl trusting intermediate certs. Would it be fair to say that a bug report would likely take some time to resolve (if ever)...?

I would like to propose a hypothetical, and see what everyone thinks: Say I have several web services hosted on servers in one or more DMZs behind OPNsense which is acting as a reverse proxy to clients in the LAN and WAN. Some of these DMZ servers use TLS channels to OPNsense and some are unencrypted. However, OPNsense always presents HTTPS sites to clients (effectively SSL offloading).

How would OPNsense build the chain? Would each server certificate have concatenated certificates containing the intermediate CA certs? Or would OPNsense build each chain as necessary from it's trusted store...? Can anyone see the advantage(s) of the latter?

If I was to perform a RASDUM analysis (Reliability, Availability, Serviceability, Dependability, Usability, Manageability - perhaps a little old school for most) along with a security analysis, I believe it would become evident that there are several advantages to having OPNsense build the chain and serve the Certificates message. I also don't see how it is possible for unencrypted (HTTP) server channels to be presented as HTTPS without OPNsense performing this anyway.

Is OPNsense capable of performing such reverse proxy features? IIRC, there are a couple of capable / required plugins.

[benyamin]

I had suggested that any Intermediate CA should be imported into System: Trust: Certificates

maybe I'm missing something. how should the system use intermediate certificates placed there (assuming that this would be possible)?

I noticed that certs placed in this store specify whether or not they are a CA. If this exposed some capacity to verify certificate chains, then I was thinking it might have been of use. I'm not sure why this info is displayed. I would have thought all CA certs should be in the Authorities store.

I note my previous comments regarding CRLs and the ability to revoke intermediates CA certs was really the impetus for my prior suggestion. I note it appears that OPNsense does not download published CRLs for imported CAs but provides an import data field for X509 CRL data. Is that correct? I think this could lead to potentially serious security issues if that field is not properly and timely maintained.

[benyamin]

Regarding the internal intermediate CA: I was just saying that it cannot issue certificates via the GUI. Just for completeness, it really has no bearing on the discussion.

I was just able to create a cert using the internal intermediate CA.

Not sure what was going on there before. Maybe I needed a GUI refresh... 8^d

[franco]

There was a bug with PHP/LibreSSL caused by a bad update of PHP itself where CSR wasn't working, which is used internally for issuing a certificate/CA.

https://forum.opnsense.org/index.php?topic=24304.msg116519#msg116519


Cheers,
Franco

[Fright]

I was just saying that it cannot issue certificates via the GUI.

it can. and as far as I understand you have already figured it out)

it looks like you have identified an issue with curl trusting intermediate certs. Would it be fair to say that a bug report would likely take some time to resolve (if ever)...?

in my understanding, this is not so much a curl bug as a flaw in the LE certs: curl uses the partial_chain openssl flag (it would be great if they documented it well, but who loves to mess around with the docs)). And imho if LE leaf (end-objects) certs contained all the fields in the authorityKeyIdentifier extension, then this would not happen, because openssl checks all extension fields to make sure the certificates match.

I mainly cited this as an example of possible unpredictability as a result of adding intermediate certificates to the store

How would OPNsense build the chain? Would each server certificate have concatenated certificates containing the intermediate CA certs? Or would OPNsense build each chain as necessary from it's trusted store...? Can anyone see the advantage(s) of the latter?

afaik chains for services on OPNsense are based on config (not on trust storage). when a certificate is added to the System: Trust: Certificates, a relationship is built between the certificate in System: Trust: Certificates and CA certs in System: Trust: Authorities. which allows (when specifying a certificate from System: Trust: Certificates as a service cert) to build a complete chain, appending the required certificates from the System: Trust: Authorities

Is OPNsense capable of performing such reverse proxy features?

HAProxy, nginx is there

[Fright]

I noticed that certs placed in this store specify whether or not they are a CA. If this exposed some capacity to verify certificate chains, then I was thinking it might have been of use. I'm not sure why this info is displayed. I would have thought all CA certs should be in the Authorities store.

I thought that this page only displays information from the cert itself (say "CA: No, Server: Yes")

I note it appears that OPNsense does not download published CRLs for imported CAs but provides an import data field for X509 CRL data

I did not digg into this part, but I do not think that this is somehow used to change the normal operation of crl check, but is probably used to work internal services with internal CAs (for example, for the OpenVPN operations)

[benyamin]

There was a bug with PHP/LibreSSL caused by a bad update of PHP itself where CSR wasn't working, which is used internally for issuing a certificate/CA.

Not in this case... Fat thumbs maybe... 8^d

[benyamin]

in my understanding, this is not so much a curl bug as a flaw in the LE certs: curl uses the partial_chain openssl flag (it would be great if they documented it well, but who loves to mess around with the docs)). And imho if LE leaf (end-objects) certs contained all the fields in the authorityKeyIdentifier extension, then this would not happen, because openssl checks all extension fields to make sure the certificates match.

I mainly cited this as an example of possible unpredictability as a result of adding intermediate certificates to the store

I note your concerns re the LE certs. I think they are somewhat valid. I also note @franco's comments on GitHub around the ordering of certs in the store which I found interesting too.

Can anyone recall any other examples where importing intermediate CA certs has become an issue?

On curl's use of the partial_chain option with openssl verify, could the version of curl used in OPNsense be compiled without that support? Would that solve this problem? It might break something else I guess, but isn't use of this option providing an exploit vector?

afaik chains for services on OPNsense are based on config (not on trust storage)...

It would be good to get a definitive answer on this. I presume it would also be plugin specific too. This is what I meant when I asked if OPNsense was capable of such reverse proxy features, i.e. does OPNsense (whether core or plugin) build the chain based upon the specified server certificate, or does each feature require the chain to be configured? I presume each chain is verified by each server process upon startup, so perhaps it would be process specific...

I thought that this page only displays information from the cert itself (say "CA: No, Server: Yes")

Yes, but I wondered why, what is the benefit? And if a CA, why is it not in Authorities.

I note it appears that OPNsense does not download published CRLs for imported CAs but provides an import data field for X509 CRL data

I did not digg into this part, but I do not think that this is somehow used to change the normal operation of crl check, but is probably used to work internal services with internal CAs (for example, for the OpenVPN operations

I think this is potentially a significant problem. OPNsense should really be checking CRLs at the distribution points specified in each CA certificate. As they are imported CAs and not internal, the CRL distribution point is very likely to be external too. As the certificate is imported in PEM format and not DER, doesn't that mean that CRL distribution points are not imported (or acted on). Manually adding the CRL in x509 format to OPNsense means that a manual and non-authoritative CRL is used.

[Fright]

I also note @franco's comments on GitHub around the ordering of certs in the store which I found interesting too.

I think it was @fraenki ) And yes, yet another example of unpredictability: order matters.
"The chain is built up iteratively, looking up in turn a certificate with suitable key usage that matches as an issuer of the current "subject" certificate as described below. If there is such a certificate, the first one found that is currently valid is taken, otherwise the one that expired most recently of all such certificates. For efficiency, no backtracking is performed, thus any further candidate issuer certificates that would match equally are ignored."
https://www.openssl.org/docs/manmaster/man1/openssl-verification-options.html#Certification-Path-Building

could the version of curl used in OPNsense be compiled without that support? Would that solve this problem?

afaik latest versions of curl have an option to disable this flag. but this does not change the fact: applications can use intermediate certificates in the store differently

It would be good to get a definitive answer on this. I presume it would also be plugin specific too.

technically yes. but it is very conveniently implemented in core and imho the devs will insist on forming chains in this way. the plugins mentioned do it in the manner indicated.

but I wondered why, what is the benefit?

to quickly understand the purpose of the imported certificate? (an administrator might accidentally import a non-server or CA certificate)

I think this is potentially a significant problem. OPNsense should really be checking CRLs at the distribution points specified in each CA certificate.

as far as i could find in the code, this is only used in openvpn with internal CA. and certainly does not affect the behavior of client apps in any way

[benyamin]

I think it was @fraenki ) And yes, yet another example of unpredictability: order matters.

It was Frank..! Franco just summed it up in one line. 
I think this particular "unpredictability" has probably graduated to a "known issue".
I'm a little surprised it hasn't been addressed upstream, but not really at the same time... 
I guess it comes down to maintenance of the store, whether that is by the upstream sources, by OPNsense devs or by the user.

I did like the idea of your blacklist - at least in principle. I should note that almost all competitor products have a CA certificate management capability. This could be accommodated in the existing view by adding a model populated by the upstream source(s). UI controls would probably best be dis/enable (or edit, but disable/enable would be better) and export but without delete. It would likely be helpful to display the fingerprint too (including for internal and imported certs). As for the controller, I am admittedly not at all familiar with what's specifically implemented, but it would require pointing to a new "edited" store, whether in config or mounting over the top of the upstream. Food for thought I guess.

could the version of curl used in OPNsense be compiled without that support? Would that solve this problem?

afaik latest versions of curl have an option to disable this flag. but this does not change the fact: applications can use intermediate certificates in the store differently

I should note I remain somewhat unconvinced that there is an inherent risk (due to unpredictability) with the importation of intermediate CA certificates. Rather, I think it is a matter of needing good store management and minimum standards for application selection (I'm thinking of curl and the partial_chain option).

Regarding building chains from config rather than the store (after supplying the leaf certificate):

technically yes. but it is very conveniently implemented in core and imho the devs will insist on forming chains in this way. the plugins mentioned do it in the manner indicated.

When doing config, would there be any value to selecting certs from the store in a drop down list, e.g. how it works now when setting the Web GUI SSL Certificate...? Just the same for issuing CA certs when needed. Or is that how it is already implemented..? 

Regarding the display of CAs in the Certificates store:

to quickly understand the purpose of the imported certificate? (an administrator might accidentally import a non-server or CA certificate)

It's certainly informative. 

I think this is potentially a significant problem. OPNsense should really be checking CRLs at the distribution points specified in each CA certificate.

as far as i could find in the code, this is only used in openvpn with internal CA. and certainly does not affect the behavior of client apps in any way

Firstly, I should say my comment quoted here is wrong:

As the certificate is imported in PEM format and not DER, doesn't that mean that CRL distribution points are not imported (or acted on).

The required extensions are still exported in PEM data: CDP (CRL Distribution Points) and AIA (Authority Information Access: used to publish issuing CA certs and OSCP Responders).
 
My concern was that OPNsense, when validating chains or behaving as a client (a relying party), might ignore CDP and AIA extensions (when present). Many clients aren't bothering anymore. Does OPNsense core or HAProxy or nginix even do CRL checking or OSCP verification or stapling? Are any of these verification techniques default behaviour for OpenSSL (presuming that this or LibreSSL are what all of core and the plugins rely on), or would such checks need to be invoked deliberately...?

I think this is more an issue for internal services, especially in enterprise use cases, where immediate revocation might be required. Many browser implementations don't even bother anymore. Yet, accessing my internal LDAP, ERP, CRM, SSH, email systems (whether webmail or fat-client) and intranet pages - well that could be a different issue...

It's noteworthy that AIA could potentially be used to prefill config fields for OPNsense core and plugins. For that matter it could supply clients with issuing CA certs too. I just don't think it's being used anymore...

[benyamin]

Also...

How does one make the internal Root CA offline...?

Perhaps:

1. Export cert and private key to secure storage
2. Delete the CA
3. Import the cert but not the private key
4. Optionally, spin the Root CA up on alternate air-gap hardware

Will this impact certificates issued by an internal intermediate CA or the intermediate's operation?

Can CRLs for the internal Root CA still be published against the root CA certificate?

This topic for context.