OPNsense Forum
Archive => 20.7 Legacy Series => Topic started by: henningkessler on January 11, 2021, 12:13:38 pm
-
Hello,
I am trying to configure a TCP stream proxy for LDAP with the NGINX plugin using a cert from the lets encrypt plugin. Unfortunately starting Nginx always fails with the following error:
cannot load certificate "/usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
this is the nginx conf:
...
stream {
...
# UPSTREAM SERVERS
upstream upstream123dd4ddf4c74e86bce785b2817d5096 {
hash $remote_addr consistent;
server IP:636 weight=1 max_conns=5 max_fails=2 fail_timeout=20;
server IP:636 weight=1 max_conns=5 max_fails=2 fail_timeout=20;
}
# upstream maps
include opnsense_stream_vhost_plugins/*.conf;
# servers
server {
listen 63636 ssl;
listen [::]:63636 ssl;
access_log /var/log/nginx/stream_f5e949f2-0d6b-42a8-8c52-9706945f9454.access.log main;
error_log /var/log/nginx/stream_f5e949f2-0d6b-42a8-8c52-9706945f9454.error.log info;
ssl_client_certificate /usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454_ca.pem;
ssl_verify_client off;
ssl_certificate_key /usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454.key;
ssl_certificate /usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_dhparam /usr/local/etc/dh-parameters.4096;
ssl_ciphers 'ECDHE-ECDSA-CAMELLIA256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CAMELLIA256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-CAMELLIA128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CAMELLIA128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-CAMELLIA256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-CAMELLIA256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-CAMELLIA128-SHA256:ECDHE-RSA-AES128-SHA256';
ssl_session_timeout 1d;
ssl_session_cache shared:sslcachef5e949f20d6b42a88c529706945f9454:50m;
ssl_session_tickets off;
ssl_prefer_server_ciphers on;
proxy_ssl on;
proxy_pass upstream123dd4ddf4c74e86bce785b2817d5096;
proxy_protocol off;
}
}
...
Is this a bug or am I holding it wrong?
Henning
-
hi
is the file in /keys?
any clue in logs?
can run configctl nginx restart in shell?
-
Sorry for the late reply. I can see the certificate from the upstream server in /keys but nothing else:
drwxr-x--- 2 root wheel 512 Jan 11 10:39 .
drwxr-xr-x 6 root wheel 512 Jan 11 10:22 ..
-rw------- 1 root wheel 0 Jan 11 11:58 trust_upstream_123dd4dd-f4c7-4e86-bce7-85b2817d5096.pem
this is what shows up in the general log:
2021-01-11T22:47:22 configd.py[20102] [cf2d62f8-6b50-41c4-96ad-32fde3196537] returned exit status 1
2021-01-11T22:47:21 configd.py[20102] [cf2d62f8-6b50-41c4-96ad-32fde3196537] restarting nginx
2021-01-11T22:46:57 configd.py[20102] [4b91bdca-a882-417e-b71e-c578a7fb58d6] returned exit status 1
2021-01-11T22:46:57 configd.py[20102] [4b91bdca-a882-417e-b71e-c578a7fb58d6] starting nginx
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //etc/newsyslog.conf.d/nginx
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //usr/local/etc/php-fpm.d/webgui.conf
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //usr/local/etc/php-fpm.d/www.conf
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //etc/rc.conf.d/php_fpm
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //usr/local/etc/nginx/mime.types
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //usr/local/etc/nginx/nginx_web.conf
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //usr/local/etc/nginx/nginx.conf
2021-01-11T22:46:57 configd.py[20102] OPNsense/Nginx generated //etc/rc.conf.d/nginx
2021-01-11T22:46:55 configd.py[20102] [531046f7-c20b-48ea-9a1d-8c50c13ef04a] trigger config changed event
2021-01-11T22:46:55 configd.py[20102] generate template container OPNsense/Nginx
2021-01-11T22:46:55 configd.py[20102] [2ef28f04-4391-4987-a910-a2111951eb69] generate template OPNsense/Nginx
and the result from the shell is:
# configctl nginx restart
Error (1)
Nothing real helpful I am afraid..
Henning
-
hmm. are you sure you are referring to an existing certificate in the stream server settings?
any errors when
/usr/local/opnsense/scripts/nginx/setup.php?
-
No errors show up when I run that command...
The certificates I use for SSL termination are from letsencrypt and I selected them in the UI. those are the ones that are missing completely, the one that shows up but is empty is the one for the upstream server. All are in the trust store of the device.
Henning
-
I can reproduce this behavior (no errors when executing the setup.php, no .pem in folder, error loading nginx), if I assign a certificate for the stream server with empty or missing <crt> key in the config.
so please сheck cert content in System-trust-certificates-"i button" and config file for <crt> key for your cert.
something wrong with this cert
--
perhaps it is worth adding some kind of diagnostic message to the setup.php if the file_put_contents returns false
-
mmh none of the certs seam wrong for me. The cert for the stream server is an letsencrypt crt which is in use by HAProxy for a website at the same time. I changed the config for upstream to not use a cert at all but my problem persists.
here are some screenshots form the configuration:
(https://cloud.henningkessler.de/s/pgPH5FsmRAC3yZw/preview)
(https://cloud.henningkessler.de/s/ntRjKT5NLbKdy52/preview)
(https://cloud.henningkessler.de/s/cgjb2jCbeWgiX85/preview)
-
new idea: try to create any second stream server with any parameters (can do without TLS). there is a chance everything will work out. then I'll look deeper
it looks like the setup.php does not quite correctly checks the servers arrays on strings 79-83 and 124-128, because of which the conditions are not met if only one server is defined
-
adding a different stream server did the trick :-) nginx starts and the cert and key for the stream server is copied into /key only the cert for the upstream server is still 0 byte....
Henning
-
glad it works)
so yes, the setup.php needs to be fixed
-
Good to know but sorry for causing the work ;-)
-
Opened a issue for this https://github.com/opnsense/plugins/issues/2189 (https://github.com/opnsense/plugins/issues/2189)
-
/key only the cert for the upstream server is still 0 byte....
sorry, didn't notice that this part still doesn't work
will try to understand
-
upd. tested on test VM: setup.php just put empty .pem in /key for proxy_ssl_trusted_certificate
need some time. will update
-
if (!empty($upstream['tls_trusted_certificate'])) {
$cas = array();
if (is_array($http_server['ca'])) {
foreach ($http_server['ca'] as $caref) {
$ca = find_ca($caref);
if (isset($ca)) {
$cas[] = $ca;
}
}
}
export_pem_file(
'/usr/local/etc/nginx/key/trust_upstream_' . $upstream_uuid . '.pem',
implode("\n", $cas)
);
}
so at the moment it won't work
setup.php makes trust_upstream_ CA pem-file (if we talk about this cert) from HTTP server client-verify CA
for me this block seems to be inoperative at all (it uses $http_server in upstreams part of the code)
you need trust cert for proxy verify or client cert to connect to upstream?
what is the end goal?
-
Hi, the end goal is to make an LDAPS request from an MDM system which does not accept self-signed certifcates to 2 Samba domain controller which need to use private certs from a private CA. The idea is terminate TLS with Nginx using an lets encrypt certificate and the use TLS to connect to the DCs
-
got it. so there are no serious requirements for the DCs certificates verification on nginx?
there is a feeling that the setup.php needs serious rework.
as well as the stream template (I don't see many directives there, including upstream verifications).
so I don't think you can do much with streams (via GUI). unless you are ready to try it through the hooks and make some changes by hands
looking at the activity on github, a huge plugin updates is expected (though not in the part of streams). thanks to @8191!.
I would gladly contribute in this, but, unfortunately, I still don't really understand how to contact the maintainer correctly for this
-
Thats really unfortunate but thanks a lot for your help !!!
I will See how I can workaround this issue...
-
actually, it's not that bad.
nginx supports so many directives that imho there will never be a GUI for all of them.
for this, hooks are used.
unfortunately, these hooks are not already present in all templates (mainly in the part of the http server) and, of course, it is worth making a FR at least to add _pre and _post hooks to all templates so they don't get overwritten during plugin update.
in general, you can always try to add missing directives through hooks and add key, pem-files to /key dir by hands.
hook adding\usage example:
https://forum.opnsense.org/index.php?topic=19758.0