Using own CA's w/Python (update_tables.py, etc.)

[NOYB]

Python is using the certifi CA bundle (.../site-packages/certifi/cacert.pem).

Is there also an equivalent to the OpenSSL CAPATH?  A hash dir of CA's.  That Python/certifi can be configured to check if the CA is not found in cacert.pem?  Like OpenSSL does?
e.g. /etc/ssl/certs or /usr/local/openssl/certs

Could add own CA's to the cacert.pem, but that is ungainly approach.

Is OpenSSL going by the wayside in favor of Certifi?

18.1.b

[fabian]

Full path: /usr/local/lib/python2.7/site-packages/certifi/cacert.pem

The common way on linux is to run a tool called update-ca-trust, which should update certificate stores. On OPNsense, this tool does not exist. but it is probably possible to write it.

The important store is the store of OpenSSL/LibreSSL because most tools use it for TLS verification. In my opinion this file should not exist. It may be a symlink to the OpenSSL/LibreSSL store as everything else does not really make sense in my opinion.

[NOYB]

Not trying to update the certificate stores.  Rather trying to have my own CA's also checked for TLS verification.

OpenSSL does this by looking for a matching cert in the hash dir (CAPATH) if a match wasn't already found in the CA bundle.
Typical OpenSSL hash dir CA path: /etc/ssl/certs or /usr/local/openssl/certs

Does python certifi have a similar capability to look elsewhere if a match wasn't found in their cacert.pem bundle?

[AdSchellevis]

I'm not exactly sure what you are looking for, but the default ssl documentation for python is here https://docs.python.org/2/library/ssl.html

Which has a handle to tell which paths it will consider:

Code: [Select]

ssl.get_default_verify_paths()
Is there any specific piece of code you're having issues with?
The requests library seem to be using it https://github.com/requests/requests/blob/master/requests/certs.py , although I'm not 100% sure it will keep the ssl defaults in tact here.

[NOYB]

The issue is that url table alias updates fail for servers that have a certificate signed by a "personal" CA.  i.e. the CA is not in the distributed CA bundle.

The previous url table alias updates were done with PHP CURL.  Pointing CURLOPT_CAPATH at the OpenSSL CAPATH directory allowed the CA's there to be used.

Email notifications are also able to use the CA's there.  Because it is using OpenSSL.


Suspect that ssl is not being imported and used by the Python url alias tables update.  It seems to be using the certifi bundle exclusively.  Manually putting the CA in the certifi bundle, as a test, works.  But that is not a good solution.  Would like for the OpenSSL capath to be included and checked if a matching CA isn't found in the bundle.

Code: [Select]

python2.7 -c "import ssl; print(ssl.get_default_verify_paths())"

DefaultVerifyPaths(

cafile='/usr/local/openssl/cert.pem',
capath='/usr/local/openssl/certs',

openssl_cafile_env='SSL_CERT_FILE',
openssl_cafile='/usr/local/openssl/cert.pem',

openssl_capath_env='SSL_CERT_DIR',
openssl_capath='/usr/local/openssl/certs'
)

[AdSchellevis]

The culprit should be the python requests library, from the library source it looks like you can set an environment variable to merge your own ca pack.

https://github.com/requests/requests/blob/v2.18.1/requests/sessions.py#L673

If you have the chance to test the env variable on your situation, that would be helpful, if that's the (only?) solution the next step could be to open an issue (on our GitHub repo) to handle this automatically. (export the path and make sure the registered CA's in OPNsense are included as well).

[NOYB]

Added a couple get environment variables to sessions.py, and exported them from both /etc/rc and /usr/local/etc/rc.  But no magic occurred.  Is that the correct place to export those environment variables for them to be available to python?

rc

Code: [Select]

#Create environment variables pointing to trusted certificates hash directory.
export SSL_CA_CERT_PATH=/etc/ssl/certs # See man fetch(3)
export SSL_CERT_DIR=/etc/ssl/certs # See man SSL_CTX_load_verify_locations(3)

export SSL_CERT_FILEx=/etc/ssl/certs/be4b640d.0
export SSL_CERT_DIRx=/etc/ssl/certs

sessions.py

Code: [Select]

# Look for requests environment configuration and be compatible
# with cURL.
if verify is True or verify is None:
    verify = (os.environ.get('REQUESTS_CA_BUNDLE') or
    os.environ.get('CURL_CA_BUNDLE') or
    os.environ.get('SSL_CERT_FILEx') or
    os.environ.get('SSL_CERT_DIRx'))

Clarification re: the desired end state; Don't want to merge own ca "pack".  Want to use OpenSSL's hash dir.  That is a dir containing individual CA files named by the hash of their subject, etc.  (ex: /etc/ssl/be4b640d.0)  The cert is then able to be selected directly by the hash name, rather than having to look through all the files for the correct one.

[fabian]

I think the best place to export the variables is in configd as it will call most python scripts and it can read the config.xml as well.

[AdSchellevis]

I don't think the requests library supports a hash dir, but concatenating all certificates registered in OPNsense would probably be good enough.

But to know if that would work, you should try to export the path in your shell and start the python script manually,  if it works then (with a cert pack) we could probably work something out to properly fix this.

If you really need a cert hash directory, you should investigate python-requests further.

Changes to python-requests have to flow in via them, so if you need changes there, their github issue tracker is likely the way to go.

[franco]

I think this sort of ties into this discussion if we could add a trusted store from a file we manage and write to disk?

https://github.com/opnsense/core/issues/1460


Cheers,
Franco

[AdSchellevis]

Yes, it does, but we first need to know the correct settings to supply our own store, then we can proceed with adding functionality.

[NOYB]

I don't think the requests library supports a hash dir, but concatenating all certificates registered in OPNsense would probably be good enough.

But to know if that would work, you should try to export the path in your shell and start the python script manually,  if it works then (with a cert pack) we could probably work something out to properly fix this.

If you really need a cert hash directory, you should investigate python-requests further.

Changes to python-requests have to flow in via them, so if you need changes there, their github issue tracker is likely the way to go.

Right.  The goal is to use the OpenSSL hash dir.  Don't think that requires a Python change.  Just that it be configured to utilize OpenSSL.  OpenSSL should take care of the rest.

[NOYB]

The problem with that verify variable in sessions.py is that it is an "OR" of which "ONE" to use, rather than a list of what to use.  Must have been half a sleep when tried that before.  Setting verify to the hash dir (/etc/ssl/certs) it finds the cert.  But we need it to also try the bundle too.  Either before or after the hash dir.

[NOYB]

MAGIC TIME!!!

Set verify to false and import ssl

Updates from both servers using "personal" certs with CA in the hash dir and servers using "public" certs in one of the distributed bundles.


Forget that.  That was dumb.

[AdSchellevis]

Like stated before, I don't expect the python-requests library (which we use for most https requests) to work with a hash directory, but you never know...

My advise however stays the same, first try to set the environment option and run your script, see if that works, if it does, we can proceed to the next step....