NGINX plugin does not copy cert to /keys for TCP stream proxy

[henningkessler]

Hello,

I am trying to configure a TCP stream proxy for LDAP with the NGINX plugin using a cert from the lets encrypt plugin. Unfortunately starting Nginx always fails with the following error:

Code Select Expand

cannot load certificate "/usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)

this is the nginx conf:

Code Select Expand


...
stream {
...

    # UPSTREAM SERVERS
    upstream upstream123dd4ddf4c74e86bce785b2817d5096 {
        hash $remote_addr consistent;
        server IP:636 weight=1 max_conns=5 max_fails=2 fail_timeout=20;
        server IP:636 weight=1 max_conns=5 max_fails=2 fail_timeout=20;
    }

    # upstream maps


    include opnsense_stream_vhost_plugins/*.conf;

    # servers
    server {
        listen  63636 ssl;
        listen  [::]:63636 ssl;

        access_log  /var/log/nginx/stream_f5e949f2-0d6b-42a8-8c52-9706945f9454.access.log main;
        error_log  /var/log/nginx/stream_f5e949f2-0d6b-42a8-8c52-9706945f9454.error.log info;

        ssl_client_certificate /usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454_ca.pem;
        ssl_verify_client off;
        ssl_certificate_key /usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454.key;
        ssl_certificate /usr/local/etc/nginx/key/f5e949f2-0d6b-42a8-8c52-9706945f9454.pem;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_dhparam /usr/local/etc/dh-parameters.4096;
        ssl_ciphers 'ECDHE-ECDSA-CAMELLIA256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CAMELLIA256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-CAMELLIA128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CAMELLIA128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-CAMELLIA256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-CAMELLIA256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-CAMELLIA128-SHA256:ECDHE-RSA-AES128-SHA256';
        ssl_session_timeout 1d;
        ssl_session_cache shared:sslcachef5e949f20d6b42a88c529706945f9454:50m;
        ssl_session_tickets off;
        ssl_prefer_server_ciphers on;

        proxy_ssl on;
        proxy_pass upstream123dd4ddf4c74e86bce785b2817d5096;
        proxy_protocol off;

    }
}
...


Is this a bug or am I holding it wrong?

Henning

[Fright]

hi
is the file in /keys?
any clue in logs?
can run

Code Select Expand

configctl nginx restart in shell?

[henningkessler]

Sorry for the late reply. I can see the certificate from the upstream server in /keys but nothing else:

drwxr-x---  2 root  wheel  512 Jan 11 10:39 .
drwxr-xr-x  6 root  wheel  512 Jan 11 10:22 ..
-rw-------  1 root  wheel    0 Jan 11 11:58 trust_upstream_123dd4dd-f4c7-4e86-bce7-85b2817d5096.pem

this is what shows up in the general log:

2021-01-11T22:47:22   configd.py[20102]   [cf2d62f8-6b50-41c4-96ad-32fde3196537] returned exit status 1
2021-01-11T22:47:21   configd.py[20102]   [cf2d62f8-6b50-41c4-96ad-32fde3196537] restarting nginx
2021-01-11T22:46:57   configd.py[20102]   [4b91bdca-a882-417e-b71e-c578a7fb58d6] returned exit status 1
2021-01-11T22:46:57   configd.py[20102]   [4b91bdca-a882-417e-b71e-c578a7fb58d6] starting nginx
2021-01-11T22:46:57   configd.py[20102]   OPNsense/Nginx generated //etc/newsyslog.conf.d/nginx
2021-01-11T22:46:57   configd.py[20102]   OPNsense/Nginx generated //usr/local/etc/php-fpm.d/webgui.conf
2021-01-11T22:46:57   configd.py[20102]   OPNsense/Nginx generated //usr/local/etc/php-fpm.d/www.conf
2021-01-11T22:46:57   configd.py[20102]   OPNsense/Nginx generated //etc/rc.conf.d/php_fpm
2021-01-11T22:46:57   configd.py[20102]   OPNsense/Nginx generated //usr/local/etc/nginx/mime.types
2021-01-11T22:46:57   configd.py[20102]   OPNsense/Nginx generated //usr/local/etc/nginx/nginx_web.conf
2021-01-11T22:46:57   configd.py[20102]   OPNsense/Nginx generated //usr/local/etc/nginx/nginx.conf
2021-01-11T22:46:57   configd.py[20102]   OPNsense/Nginx generated //etc/rc.conf.d/nginx
2021-01-11T22:46:55   configd.py[20102]   [531046f7-c20b-48ea-9a1d-8c50c13ef04a] trigger config changed event
2021-01-11T22:46:55   configd.py[20102]   generate template container OPNsense/Nginx
2021-01-11T22:46:55   configd.py[20102]   [2ef28f04-4391-4987-a910-a2111951eb69] generate template OPNsense/Nginx

and the result from the shell is:

# configctl nginx restart
Error (1)

Nothing real helpful I am afraid..

Henning

[Fright]

hmm. are you sure you are referring to an existing certificate in the stream server settings?
any errors when

Code Select Expand

/usr/local/opnsense/scripts/nginx/setup.php?

[henningkessler]

No errors show up when I run that command...

The certificates I use for SSL termination are from letsencrypt and I selected them in the UI. those are the ones that are missing completely, the one that shows up but is empty is the one for the upstream server. All are in the trust store of the device.

Henning

[Fright]

I can reproduce this behavior (no errors when executing the setup.php, no .pem in folder, error loading nginx), if I assign a certificate for the stream server with empty or missing  <crt> key in the config.
so please сheck cert content in System-trust-certificates-"i button" and config file for <crt>  key for your cert.
something wrong with this cert
--
perhaps it is worth adding some kind of diagnostic message to the setup.php if the file_put_contents returns false

[henningkessler]

mmh none of the certs seam wrong for me. The cert for the stream server is an letsencrypt crt which is in use by HAProxy for a website at the same time. I changed the config for upstream to not use a cert at all but my problem persists.
here are some screenshots form the configuration:

[Fright]

new idea: try to create any second stream server with any parameters (can do without TLS). there is a chance everything will work out. then I'll look deeper
it looks like the setup.php does not quite correctly checks the servers arrays on strings 79-83 and 124-128, because of which the conditions are not met if only one server is defined

[henningkessler]

adding a different stream server did the trick :-) nginx starts and the cert and key for the stream server is copied into /key only the cert for the upstream server is still  0 byte....

Henning

[Fright]

glad it works)
so yes, the setup.php needs to be fixed

[henningkessler]

Good to know but sorry for causing the work ;-)

[henningkessler]

Opened a issue for this https://github.com/opnsense/plugins/issues/2189

[Fright]

/key only the cert for the upstream server is still  0 byte....

sorry, didn't notice that this part still doesn't work
will try to understand

[Fright]

upd. tested on test VM: setup.php just put empty .pem in /key for proxy_ssl_trusted_certificate
need some time. will update

[Fright]

Code Select Expand


            if (!empty($upstream['tls_trusted_certificate'])) {
                $cas = array();
                if (is_array($http_server['ca'])) {
                    foreach ($http_server['ca'] as $caref) {
                        $ca = find_ca($caref);
                        if (isset($ca)) {
                            $cas[] = $ca;
                        }
                    }
                }
                export_pem_file(
                    '/usr/local/etc/nginx/key/trust_upstream_' . $upstream_uuid . '.pem',
                    implode("\n", $cas)
                );
            }

so at the moment it won't work
setup.php makes trust_upstream_  CA pem-file (if we talk about this cert) from HTTP server client-verify CA
for me this block seems to be inoperative at all (it uses $http_server in upstreams part of the code)

you need trust cert for proxy verify or client cert to connect to upstream?
what is the end goal?