OPNsense Forum
Archive => 18.1 Legacy Series => Topic started by: NOYB on December 20, 2017, 11:48:02 pm
-
Python is using the certifi CA bundle (.../site-packages/certifi/cacert.pem).
Is there also an equivalent to the OpenSSL CAPATH? A hash dir of CA's. That Python/certifi can be configured to check if the CA is not found in cacert.pem? Like OpenSSL does?
e.g. /etc/ssl/certs or /usr/local/openssl/certs
Could add own CA's to the cacert.pem, but that is ungainly approach.
Is OpenSSL going by the wayside in favor of Certifi?
18.1.b
-
Full path: /usr/local/lib/python2.7/site-packages/certifi/cacert.pem
The common way on linux is to run a tool called update-ca-trust, which should update certificate stores. On OPNsense, this tool does not exist. but it is probably possible to write it.
The important store is the store of OpenSSL/LibreSSL because most tools use it for TLS verification. In my opinion this file should not exist. It may be a symlink to the OpenSSL/LibreSSL store as everything else does not really make sense in my opinion.
-
Not trying to update the certificate stores. Rather trying to have my own CA's also checked for TLS verification.
OpenSSL does this by looking for a matching cert in the hash dir (CAPATH) if a match wasn't already found in the CA bundle.
Typical OpenSSL hash dir CA path: /etc/ssl/certs or /usr/local/openssl/certs
Does python certifi have a similar capability to look elsewhere if a match wasn't found in their cacert.pem bundle?
-
I'm not exactly sure what you are looking for, but the default ssl documentation for python is here https://docs.python.org/2/library/ssl.html (https://docs.python.org/2/library/ssl.html)
Which has a handle to tell which paths it will consider:
ssl.get_default_verify_paths()
Is there any specific piece of code you're having issues with?
The requests library seem to be using it https://github.com/requests/requests/blob/master/requests/certs.py , although I'm not 100% sure it will keep the ssl defaults in tact here.
-
The issue is that url table alias updates fail for servers that have a certificate signed by a "personal" CA. i.e. the CA is not in the distributed CA bundle.
The previous url table alias updates were done with PHP CURL. Pointing CURLOPT_CAPATH at the OpenSSL CAPATH directory allowed the CA's there to be used.
Email notifications are also able to use the CA's there. Because it is using OpenSSL.
Suspect that ssl is not being imported and used by the Python url alias tables update. It seems to be using the certifi bundle exclusively. Manually putting the CA in the certifi bundle, as a test, works. But that is not a good solution. Would like for the OpenSSL capath to be included and checked if a matching CA isn't found in the bundle.
python2.7 -c "import ssl; print(ssl.get_default_verify_paths())"
DefaultVerifyPaths(
cafile='/usr/local/openssl/cert.pem',
capath='/usr/local/openssl/certs',
openssl_cafile_env='SSL_CERT_FILE',
openssl_cafile='/usr/local/openssl/cert.pem',
openssl_capath_env='SSL_CERT_DIR',
openssl_capath='/usr/local/openssl/certs'
)
-
The culprit should be the python requests library, from the library source it looks like you can set an environment variable to merge your own ca pack.
https://github.com/requests/requests/blob/v2.18.1/requests/sessions.py#L673
If you have the chance to test the env variable on your situation, that would be helpful, if that's the (only?) solution the next step could be to open an issue (on our GitHub repo) to handle this automatically. (export the path and make sure the registered CA's in OPNsense are included as well).
-
Added a couple get environment variables to sessions.py, and exported them from both /etc/rc and /usr/local/etc/rc. But no magic occurred. Is that the correct place to export those environment variables for them to be available to python?
rc
#Create environment variables pointing to trusted certificates hash directory.
export SSL_CA_CERT_PATH=/etc/ssl/certs # See man fetch(3)
export SSL_CERT_DIR=/etc/ssl/certs # See man SSL_CTX_load_verify_locations(3)
export SSL_CERT_FILEx=/etc/ssl/certs/be4b640d.0
export SSL_CERT_DIRx=/etc/ssl/certs
sessions.py
# Look for requests environment configuration and be compatible
# with cURL.
if verify is True or verify is None:
verify = (os.environ.get('REQUESTS_CA_BUNDLE') or
os.environ.get('CURL_CA_BUNDLE') or
os.environ.get('SSL_CERT_FILEx') or
os.environ.get('SSL_CERT_DIRx'))
Clarification re: the desired end state; Don't want to merge own ca "pack". Want to use OpenSSL's hash dir. That is a dir containing individual CA files named by the hash of their subject, etc. (ex: /etc/ssl/be4b640d.0) The cert is then able to be selected directly by the hash name, rather than having to look through all the files for the correct one.
-
I think the best place to export the variables is in configd as it will call most python scripts and it can read the config.xml as well.
-
I don't think the requests library supports a hash dir, but concatenating all certificates registered in OPNsense would probably be good enough.
But to know if that would work, you should try to export the path in your shell and start the python script manually, if it works then (with a cert pack) we could probably work something out to properly fix this.
If you really need a cert hash directory, you should investigate python-requests further.
Changes to python-requests have to flow in via them, so if you need changes there, their github issue tracker is likely the way to go.
-
I think this sort of ties into this discussion if we could add a trusted store from a file we manage and write to disk?
https://github.com/opnsense/core/issues/1460
Cheers,
Franco
-
Yes, it does, but we first need to know the correct settings to supply our own store, then we can proceed with adding functionality.
-
I don't think the requests library supports a hash dir, but concatenating all certificates registered in OPNsense would probably be good enough.
But to know if that would work, you should try to export the path in your shell and start the python script manually, if it works then (with a cert pack) we could probably work something out to properly fix this.
If you really need a cert hash directory, you should investigate python-requests further.
Changes to python-requests have to flow in via them, so if you need changes there, their github issue tracker is likely the way to go.
Right. The goal is to use the OpenSSL hash dir. Don't think that requires a Python change. Just that it be configured to utilize OpenSSL. OpenSSL should take care of the rest.
-
The problem with that verify variable in sessions.py is that it is an "OR" of which "ONE" to use, rather than a list of what to use. Must have been half a sleep when tried that before. Setting verify to the hash dir (/etc/ssl/certs) it finds the cert. But we need it to also try the bundle too. Either before or after the hash dir.
-
MAGIC TIME!!!
Set verify to false and import ssl
Updates from both servers using "personal" certs with CA in the hash dir and servers using "public" certs in one of the distributed bundles.
Forget that. That was dumb.
-
Like stated before, I don't expect the python-requests library (which we use for most https requests) to work with a hash directory, but you never know...
My advise however stays the same, first try to set the environment option and run your script, see if that works, if it does, we can proceed to the next step....
-
It's an ugly duckling hack, but it's functional with the hash dir. See attached patch.
If first connection attempt fails it then tries again with verify set to the hash dir. A tacky hack.
To get hash filename for certs with PHP:
$crt_inf = openssl_x509_parse(base64_decode($crt));
$cert_hash_filename = $crt_inf['hash'] . ".0";
-
What can be done to get the requests module to try additional certificate store(s) instead of exclusively the Certifi store?
It seems like the best solution would be for Certifi to try additional certificate store(s) when a match is not found in its own bundle. But don't see an obvious way to do that to the Certifi code.
There must be a better way than another connection attempt with a different certificate store specified. Functional but unseemly. At least the second connection attempt is only done to my "own" servers.
-
What can be done to get the requests module to try additional certificate store(s) instead of exclusively the Certifi store?
It seems like the best solution would be for Certifi to try additional certificate store(s) when a match is not found in its own bundle. But don't see an obvious way to do that to the Certifi code.
Certifi seems to be only some code returning the path to its bundled cacert.pem.
There must be a better way than another connection attempt with a different certificate store specified. Functional but unseemly. At least the second connection attempt is only done to my "own" servers.
In theory the certificates could be added from config.xml
-
In theory the certificates could be added from config.xml
Adding custom system certs to the distributed Certifi bundle has it own baggage issues. Amongst them not being a system wide solution. Objective is for system config CA certs to be trusted by not only Python requests, but everything that supports secure connections (notifications, custom DynDNS, URL table updates, etc.).
With exception of URL table updates (Python requests) they all have native support for using a hash dir (capath) fallback when a cert match is not found in the bundle.
Been using this for quite awhile to trust own CA's from the system config. But Python requests exclusively using Certifi bundle is a barrier.