A regression in handling the importation of intermediate CA certificates led to a general discussion regarding the importation of such certificates into the OPNsense trusted store.
The topic can be found here (https://forum.opnsense.org/index.php?topic=25361.0).
Given the discussion had thoroughly hijacked that topic, I thought it best to continue here...
Credit to Fright for continuing the conversation over there (and hopefully here too).
I think it remains unresolved as to if, when and how third-party or external intermediate CA certificates should be installed (at System: Trust).
For completeness, I note that it appears the internal intermediate CA (when established) cannot issue internal certificates.
To that end:
It is my understanding that it remains best practice to install intermediate CA certificates that form part of a certificate chain to internal services. Just to be clear, I'm not suggesting that intermediate CA certs should be imported for say external websites.
These certificates should normally form part of the Certificates message but in my experience that isn't always implemented. Courtesy of Fright's previous postings, the pertinent part of the relevant RFC is here (https://datatracker.ietf.org/doc/html/rfc5246#section-7.4.2) for your reading pleasure.
I had suggested that any Intermediate CA should be imported into System: Trust: Certificates (marked as a CA) and the corresponding root CA to System: Trust: Authorities. I only said this because I presume a CA listed under Authorities cannot be revoked... I am not convinced this is the case. I have not been able to test whether a Root CA CRL could revoke an intermediate CA certificate imported into System: Trust: Authorities. I would normally assume this to be the expected behaviour.
Furthermore, after attempting to do this I realised it was not an appropriate solution as the private key also must be imported.
If I am not mistaken, it is noteworthy, that the change in default behaviour was prompted by the recent Let's Encrypt saga. It is worth looking at the discussions on GitHub for issues 5257 (https://github.com/opnsense/core/issues/5257) and 2554 (https://github.com/opnsense/plugins/issues/2554). It gives good insight into the mitigation employed and kulikov-a (which I presume is you Fright) explains the issues well.
I would like to continue a discussion about what should constitute best practice in the use of intermediate CA certificates. I expect the discussion will also include the importation of both Root and Intermediate CA CRLs. Perhaps this discussion will identify and / or resolve any necessary improvements in this space.
Please join in.
It might be best to move this to General Discussion or another parent forum. I leave it to a mod to decide...
QuoteFor completeness, I note that it appears the internal intermediate CA (when established) cannot issue internal certificates
sorry, dont understand this part..
and for the record: I am not claiming that there is only one right choice. I'm only wondering why admins trust intermediate CAs even when, in my opinion, this is not required. Everything else is just argumentation of my point of view
QuoteIt is my understanding that it remains best practice to install intermediate CA certificates that form part of a certificate chain to internal services. Just to be clear, I'm not suggesting that intermediate CA certs should be imported for say external websites.
If security is a matter of trust, then from a best practice perspective, why should the behavior be different for internal services?
QuoteThese certificates should normally form part of the Certificates message but in my experience that isn't always implemented.
Maybe. but in this case, imho the first attempt should be to eliminate the cause on the service side and only then workaround the problem using the trusted store (if it works). (in windows for such cases there is a separate store of intermediate certificates, but the certificates placed there do not become trusted. only helps to build the chain afaik)
and this is not counting the unpredictability of the result of adding an intermediate certificate to the store: openssl does not trust them, but uses them to select the chain. curl uses them as a trusted anchor and this can have unexpected consequences also (if you add an expired R3 intermediate certificate https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.pem to the trusted store (21.7.4 will not allow you to do this, you will have to add it by hand), then the 'curl https://letsencrypt.org' will give an error about an expired certificate, although this is not a root certificate and the server certificate is not even signed by this certificate (!) ;D LE used the same private key when requesting a new intermediate certificate. so curl defines it as issuer cert)
QuoteI had suggested that any Intermediate CA should be imported into System: Trust: Certificates
maybe I'm missing something. how should the system use intermediate certificates placed there (assuming that this would be possible)?
I also think that General Discussions are more suitable for such topics
I've sent Ad a PM requesting he move the topic to General Discussions.
Regarding the internal intermediate CA: I was just saying that it cannot issue certificates via the GUI. Just for completeness, it really has no bearing on the discussion.
Regarding best practice and limiting the scope to internal services, I guess it would be better to include those services where OPNsense is acting as a client. I would say it best to exclude external websites because it should be presumed that those sites issue full and correct Certificate messages.
I agree that intermediate CA certs should only be used to build chains and verify trust as part of that chain. All implementations should require a root CA cert be present to establish trust. It is also my experience that M$, OpenSSL and Java treat intermediates CA certs this way.
@Fright, it looks like you have identified an issue with curl trusting intermediate certs. Would it be fair to say that a bug report would likely take some time to resolve (if ever)...?
I would like to propose a hypothetical, and see what everyone thinks: Say I have several web services hosted on servers in one or more DMZs behind OPNsense which is acting as a reverse proxy to clients in the LAN and WAN. Some of these DMZ servers use TLS channels to OPNsense and some are unencrypted. However, OPNsense always presents HTTPS sites to clients (effectively SSL offloading).
How would OPNsense build the chain? Would each server certificate have concatenated certificates containing the intermediate CA certs? Or would OPNsense build each chain as necessary from it's trusted store...? Can anyone see the advantage(s) of the latter?
If I was to perform a RASDUM analysis (Reliability, Availability, Serviceability, Dependability, Usability, Manageability - perhaps a little old school for most) along with a security analysis, I believe it would become evident that there are several advantages to having OPNsense build the chain and serve the Certificates message. I also don't see how it is possible for unencrypted (HTTP) server channels to be presented as HTTPS without OPNsense performing this anyway.
Is OPNsense capable of performing such reverse proxy features? IIRC, there are a couple of capable / required plugins.
Quote from: Fright on October 31, 2021, 04:34:03 PM
QuoteI had suggested that any Intermediate CA should be imported into System: Trust: Certificates
maybe I'm missing something. how should the system use intermediate certificates placed there (assuming that this would be possible)?
I noticed that certs placed in this store specify whether or not they are a CA. If this exposed some capacity to verify certificate chains, then I was thinking it might have been of use. I'm not sure why this info is displayed. I would have thought all CA certs should be in the
Authorities store.
I note my previous comments regarding CRLs and the ability to revoke intermediates CA certs was really the impetus for my prior suggestion. I note it appears that OPNsense does not download published CRLs for imported CAs but provides an import data field for X509 CRL data. Is that correct? I think this could lead to potentially serious security issues if that field is not properly and timely maintained.
Quote from: benyamin on November 01, 2021, 02:40:38 AM
Regarding the internal intermediate CA: I was just saying that it cannot issue certificates via the GUI. Just for completeness, it really has no bearing on the discussion.
I was just able to create a cert using the internal intermediate CA.
Not sure what was going on there before. Maybe I needed a GUI refresh... 8^d
There was a bug with PHP/LibreSSL caused by a bad update of PHP itself where CSR wasn't working, which is used internally for issuing a certificate/CA.
https://forum.opnsense.org/index.php?topic=24304.msg116519#msg116519
Cheers,
Franco
QuoteI was just saying that it cannot issue certificates via the GUI.
it can. and as far as I understand you have already figured it out)
Quoteit looks like you have identified an issue with curl trusting intermediate certs. Would it be fair to say that a bug report would likely take some time to resolve (if ever)...?
in my understanding, this is not so much a
curl bug as a flaw in the
LE certs:
curl uses the
partial_chain openssl flag (it would be great if they documented it well, but who loves to mess around with the docs)). And imho if
LE leaf (end-objects) certs contained all the fields in the
authorityKeyIdentifier extension, then this would not happen, because openssl checks all extension fields to make sure the certificates match.
I mainly cited this as an example of possible unpredictability as a result of adding intermediate certificates to the store
QuoteHow would OPNsense build the chain? Would each server certificate have concatenated certificates containing the intermediate CA certs? Or would OPNsense build each chain as necessary from it's trusted store...? Can anyone see the advantage(s) of the latter?
afaik chains for services on OPNsense are based on config (not on trust storage). when a certificate is added to the System: Trust: Certificates, a relationship is built between the certificate in System: Trust: Certificates and CA certs in System: Trust: Authorities. which allows (when specifying a certificate from System: Trust: Certificates as a service cert) to build a complete chain, appending the required certificates from the System: Trust: Authorities
QuoteIs OPNsense capable of performing such reverse proxy features?
HAProxy, nginx is there
QuoteI noticed that certs placed in this store specify whether or not they are a CA. If this exposed some capacity to verify certificate chains, then I was thinking it might have been of use. I'm not sure why this info is displayed. I would have thought all CA certs should be in the Authorities store.
I thought that this page only displays information from the cert itself (say "CA: No, Server: Yes")
QuoteI note it appears that OPNsense does not download published CRLs for imported CAs but provides an import data field for X509 CRL data
I did not digg into this part, but I do not think that this is somehow used to change the normal operation of crl check, but is probably used to work internal services with internal CAs (for example, for the OpenVPN operations)
Quote from: franco on November 01, 2021, 03:27:37 PM
There was a bug with PHP/LibreSSL caused by a bad update of PHP itself where CSR wasn't working, which is used internally for issuing a certificate/CA.
Not in this case... Fat thumbs maybe... 8^d
Quote from: Fright on November 01, 2021, 04:35:01 PM
in my understanding, this is not so much a curl bug as a flaw in the LE certs: curl uses the partial_chain openssl flag (it would be great if they documented it well, but who loves to mess around with the docs)). And imho if LE leaf (end-objects) certs contained all the fields in the authorityKeyIdentifier extension, then this would not happen, because openssl checks all extension fields to make sure the certificates match.
I mainly cited this as an example of possible unpredictability as a result of adding intermediate certificates to the store
I note your concerns re the LE certs. I think they are somewhat valid. I also note @franco's comments on GitHub around the ordering of certs in the store which I found interesting too.
Can anyone recall any other examples where importing intermediate CA certs has become an issue?
On curl's use of the
partial_chain option with
openssl verify, could the version of curl used in OPNsense be compiled without that support? Would that solve this problem? It might break something else I guess, but isn't use of this option providing an exploit vector?
Quote from: Fright on November 01, 2021, 04:35:01 PM
afaik chains for services on OPNsense are based on config (not on trust storage)...
It would be good to get a definitive answer on this. I presume it would also be plugin specific too. This is what I meant when I asked if OPNsense was capable of such reverse proxy features, i.e. does OPNsense (whether core or plugin) build the chain based upon the specified server certificate, or does each feature require the chain to be configured? I presume each chain is verified by each server process upon startup, so perhaps it would be process specific...
Quote from: Fright on November 01, 2021, 04:38:30 PM
I thought that this page only displays information from the cert itself (say "CA: No, Server: Yes")
Yes, but I wondered why, what is the benefit? And if a CA, why is it not in
Authorities.
Quote from: Fright on November 01, 2021, 04:38:30 PM
QuoteI note it appears that OPNsense does not download published CRLs for imported CAs but provides an import data field for X509 CRL data
I did not digg into this part, but I do not think that this is somehow used to change the normal operation of crl check, but is probably used to work internal services with internal CAs (for example, for the OpenVPN operations
I think this is potentially a significant problem. OPNsense should really be checking CRLs at the distribution points specified in each CA certificate. As they are imported CAs and not internal, the CRL distribution point is very likely to be external too. As the certificate is imported in PEM format and not DER, doesn't that mean that CRL distribution points are not imported (or acted on). Manually adding the CRL in x509 format to OPNsense means that a manual and non-authoritative CRL is used.
QuoteI also note @franco's comments on GitHub around the ordering of certs in the store which I found interesting too.
I think it was @fraenki ) And yes, yet another example of unpredictability: order matters.
"The chain is built up iteratively, looking up in turn a certificate with suitable key usage that matches as an issuer of the current "subject" certificate as described below. If there is such a certificate, the first one found that is currently valid is taken, otherwise the one that expired most recently of all such certificates. For efficiency, no backtracking is performed, thus any further candidate issuer certificates that would match equally are ignored."
https://www.openssl.org/docs/manmaster/man1/openssl-verification-options.html#Certification-Path-Building
Quotecould the version of curl used in OPNsense be compiled without that support? Would that solve this problem?
afaik latest versions of curl have an option to disable this flag. but this does not change the fact: applications can use intermediate certificates in the store differently
QuoteIt would be good to get a definitive answer on this. I presume it would also be plugin specific too.
technically yes. but it is very conveniently implemented in core and imho the devs will insist on forming chains in this way. the plugins mentioned do it in the manner indicated.
Quotebut I wondered why, what is the benefit?
to quickly understand the purpose of the imported certificate? (an administrator might accidentally import a non-server or CA certificate)
QuoteI think this is potentially a significant problem. OPNsense should really be checking CRLs at the distribution points specified in each CA certificate.
as far as i could find in the code, this is only used in openvpn with internal CA. and certainly does not affect the behavior of client apps in any way
Quote from: Fright on November 02, 2021, 01:22:50 PM
I think it was @fraenki ) And yes, yet another example of unpredictability: order matters.
It was Frank..! Franco just summed it up in one line. :-X
I think this particular "unpredictability" has probably graduated to a "known issue".
I'm a little surprised it hasn't been addressed upstream, but not really at the same time... :-\
I guess it comes down to maintenance of the store, whether that is by the upstream sources, by OPNsense devs or by the user.
I did like the idea of your blacklist - at least in principle. I should note that almost all competitor products have a CA certificate management capability. This could be accommodated in the existing view by adding a model populated by the upstream source(s). UI controls would probably best be
dis/enable (or
edit, but disable/enable would be better) and
export but without
delete. It would likely be helpful to display the fingerprint too (including for internal and imported certs). As for the controller, I am admittedly not at all familiar with what's specifically implemented, but it would require pointing to a new "edited" store, whether in config or mounting over the top of the upstream. Food for thought I guess.
Quote from: Fright on November 02, 2021, 01:22:50 PM
Quotecould the version of curl used in OPNsense be compiled without that support? Would that solve this problem?
afaik latest versions of curl have an option to disable this flag. but this does not change the fact: applications can use intermediate certificates in the store differently
I should note I remain somewhat unconvinced that there is an inherent risk (due to unpredictability) with the importation of intermediate CA certificates. Rather, I think it is a matter of needing good store management and minimum standards for application selection (I'm thinking of curl and the
partial_chain option).
Regarding building chains from config rather than the store (after supplying the leaf certificate):
Quote from: Fright on November 02, 2021, 01:22:50 PM
technically yes. but it is very conveniently implemented in core and imho the devs will insist on forming chains in this way. the plugins mentioned do it in the manner indicated.
When doing config, would there be any value to selecting certs from the store in a drop down list, e.g. how it works now when setting the Web GUI SSL Certificate...? Just the same for issuing CA certs when needed. Or is that how it is already implemented..?
Regarding the display of CAs in the Certificates store:
Quote from: Fright on November 02, 2021, 01:22:50 PM
to quickly understand the purpose of the imported certificate? (an administrator might accidentally import a non-server or CA certificate)
It's certainly informative. ;)
Quote from: Fright on November 02, 2021, 01:22:50 PM
QuoteI think this is potentially a significant problem. OPNsense should really be checking CRLs at the distribution points specified in each CA certificate.
as far as i could find in the code, this is only used in openvpn with internal CA. and certainly does not affect the behavior of client apps in any way
Firstly, I should say my comment quoted here is wrong:
QuoteAs the certificate is imported in PEM format and not DER, doesn't that mean that CRL distribution points are not imported (or acted on).
The required extensions are still exported in PEM data: CDP (CRL Distribution Points) and AIA (Authority Information Access: used to publish issuing CA certs and OSCP Responders).
My concern was that OPNsense, when validating chains or behaving as a client (a relying party), might ignore CDP and AIA extensions (when present). Many clients aren't bothering anymore. Does OPNsense core or HAProxy or nginix even do CRL checking or OSCP verification or stapling? Are any of these verification techniques default behaviour for OpenSSL (presuming that this or LibreSSL are what all of core and the plugins rely on), or would such checks need to be invoked deliberately...?
I think this is more an issue for internal services, especially in enterprise use cases, where immediate revocation might be required. Many browser implementations don't even bother anymore. Yet, accessing my internal LDAP, ERP, CRM, SSH, email systems (whether webmail or fat-client) and intranet pages - well that could be a different issue...
It's noteworthy that AIA could potentially be used to prefill config fields for OPNsense core and plugins. For that matter it could supply clients with issuing CA certs too. I just don't think it's being used anymore...
Also...
How does one make the internal Root CA offline...?
Perhaps:
1. Export cert and private key to secure storage
2. Delete the CA
3. Import the cert but not the private key
4. Optionally, spin the Root CA up on alternate air-gap hardware
Will this impact certificates issued by an internal intermediate CA or the intermediate's operation?
Can CRLs for the internal Root CA still be published against the root CA certificate?
This topic (https://forum.opnsense.org/index.php?topic=25387.0) for context.
QuoteRather, I think it is a matter of needing good store management and minimum standards for application selection (I'm thinking of curl and the partial_chain option).
I have no idea how you can predict at what point developers of third-party applications will decide to change their behavior
QuoteWhen doing config, would there be any value to selecting certs from the store in a drop down list, e.g. how it works now when setting the Web GUI SSL Certificate...? Just the same for issuing CA certs when needed. Or is that how it is already implemented..?
sorry, did not quite understand the question. user selects the server certificate for application in GUI. if there are referenced CA's certificates in System: Trust: Authorities, the chain is built automatically and placed in the appropriate application directories
QuoteDoes OPNsense core or HAProxy or nginix even do CRL checking or OSCP verification or stapling? Are any of these verification techniques default behaviour for OpenSSL (presuming that this or LibreSSL are what all of core and the plugins rely on), or would such checks need to be invoked deliberately...?
it's all on the app (nginx, for example, does not work with CRL distibuton points or some. CRLs are checked only if the file is explicitly specified). I do not think that the OPNsense should and in the right to do its work for the app.
QuoteHow does one make the internal Root CA offline...?
QuotePerhaps:
I think it's technically possible
QuoteCan CRLs for the internal Root CA still be published against the root CA certificate?
you can import CRL
but. if we are talking about best practices, then IMHO it is better not to use an internal opnsense CA in a production environment at all (for debugging and emergency use only). OPNsense is not a CA and does not try to seem like ..
Quote from: Fright on November 03, 2021, 08:26:11 PM
I have no idea how you can predict at what point developers of third-party applications will decide to change their behavior
Well obviously you can't predict such changes, but you can respond to them, even if the response is to recognise the problem and do nothing. For example, in the case of curl, an issue could be raised to
assess curl's use of the
partial_chain option. It might be determined that it does present a security issue that needs to be resolved, if say, a critical core or plugin function relies on curl and fully valid trust chains are expected. Or it might be determined that nothing should be done, but at least that decision is then documented.
Quote from: Fright on November 03, 2021, 08:26:11 PM
QuoteWhen doing config, would there be any value to selecting certs from the store in a drop down list, e.g. how it works now when setting the Web GUI SSL Certificate...? Just the same for issuing CA certs when needed. Or is that how it is already implemented..?
sorry, did not quite understand the question. user selects the server certificate for application in GUI. if there are referenced CA's certificates in System: Trust: Authorities, the chain is built automatically and placed in the appropriate application directories
Yes, that was essentially my question. I did assume an interim step: that when required for any particular configuration, the issuing CA would be pre-populated in the relevant GUI field.
I've seen over the last several days that there is quite the mixture of different implementations. Are you aware of any OPNsense standard for plugin development is this space, or is it left up to the plugin dev to make these calls...?
I guess I see some value in
trusting what is in the store to populate GUI fields and subsequent configurations. One feature I find notably missing in OPNsense is an indication of all places in the system where the certificates are in use. I think if something like that was implemented, then a system-wide framework for managing certificates would be more feasible. This would allow for the replacement of certificates in plugins before deleting expired or revoked certs from the store. Automated replacement of plugin certificates via, e.g. a wizard at
System: Trust, would also be feasible.
Quote from: Fright on November 03, 2021, 08:26:11 PM
QuoteDoes OPNsense core or HAProxy or nginix even do CRL checking or OSCP verification or stapling? Are any of these verification techniques default behaviour for OpenSSL (presuming that this or LibreSSL are what all of core and the plugins rely on), or would such checks need to be invoked deliberately...?
it's all on the app (nginx, for example, does not work with CRL distibuton points or some. CRLs are checked only if the file is explicitly specified). I do not think that the OPNsense should and in the right to do its work for the app.
I agree in principle.
Provided the GUI exposes the functionality of the plugin to use such features (otherwise the novice may not be aware of the feature or be able to configure it). I note nginx has true OSCP validation and stapling capabilities built in. It is my understanding HAProxy relies on running a cron job to update files in a watch directory. I couldn't tell if OPNsense does
external CRL or OSCP validation at all (with a preference for the latter). I would hope that it does.
If a cron job or daemon ran to check CRLs / OSCP validation of certificates in the OPNsense store at some reasonable (and configurable) interval, I see some benefit to a system-wide notification of invalid certificates being in use. Such a notification could be an alert to support personnel or even a watchdog process that can optionally shutdown plugins affected by revoked or invalidated certificates.
I think, if OPNsense should play any role in this space, the above would be it. If all plugins only make use of certificates in the OPNsense store, then this would mean any issues with certificate validity could be captured and acted upon.
An extension of this would be timely notifications for expiring certificates. :)
Quote from: Fright on November 03, 2021, 08:26:11 PM
QuoteHow does one make the internal Root CA offline...?
... but. if we are talking about best practices, then IMHO it is better not to use an internal opnsense CA in a production environment at all (for debugging and emergency use only). OPNsense is not a CA and does not try to seem like ..
Yes, clearly this would be the better practice. It is also somewhat dubious operating multiple CAs on the one system. ;)
Quote from: benyamin on November 06, 2021, 03:19:13 PM
One feature I find notably missing in OPNsense is an indication of all places in the system where the certificates are in use.
Just noticed a basic indication of where certificates are in use on the far RHS of the
System: Trust: Certificates view.
If it could provide more info that would be awesome. Is the data available in the model?
Quotebut you can respond to them, even if the response is to recognise the problem and do nothing.
it will be even more confusing as the behavior of even familiar utilities will change. not to mention that it will be a nightmare for the maintainers imho (in addition to the lots of openssl options itself, each application has the right to introduce its own behavior)
QuoteOne feature I find notably missing in OPNsense is an indication of all places in the system where the certificates are in use. I think if something like that was implemented, then a system-wide framework for managing certificates would be more feasible.
I think it will be too cumbersome (at least in terms of certs update automation from core) to ensure simplicity and stability. and if I understand the architecture correctly, devs quite clearly separates the core and plugins (plugins should not change the core behavior and the core should not spend too much on maintaining plugins. The main thing is to provide the infrastructure for work)
QuoteJust noticed a basic indication of where certificates are in use on the far RHS of the System: Trust: Certificates view.
yes. but only for core services. not plugins