Hi all,
NGINX fails with the following message after I upgraded to 20.7.8:
SSL_CTX_load_verify_locations("/usr/local/etc/nginx/key/trust_upstream_***GUID***.pem") failed (SSL: error:0B084088:x509 certificate routines:X509_load_cert_crl_file:no certificate or crl found)
Any suggestions for how I might rectify?
opnsense-revert -r 20.7.7 os-nginx
Dies this fix it?
certificate validation of upstreams is working now. so nginx starts to load trust_upstream_* certs.
is pem-file in place?
whats your nginx config? (is it http or stream?)
(maybe a problem with setup.php? try to uncheck "tls verify" in upstream config or dont select any in TLS:Trusted cert)
I have exactly the same issue. OpnSense updated to 20.1.8 and nginx is not starting anymore.
2021/01/20 22:03:14 [emerg] 95587#100595: SSL_CTX_load_verify_locations("/usr/local/etc/nginx/key/trust_upstream_228ce5a1-*****.pem") failed (SSL: error:0B084088:x509 certificate routines:X509_load_cert_crl_file:no certificate or crl found)The file mentioned in the log file is not existing. But I have no glue how to figure out where it should belong to. I have a couple of upstream and a couple of server defined. So far nothing suspicious detected. But I have no idea where to search.
Quote from: mimugmail on January 20, 2021, 06:22:53 AM
opnsense-revert -r 20.7.7 os-nginx
Dies this fix it?
Yes. In my case it helped. Nginx is running again.
made a PR for quck fix
https://github.com/opnsense/plugins/pull/2198
works well on test VM
@Tubs
have you tried not to select CAs in TLS:Trusted Certificate in Upstream config page?
should work if upstream cert issued CAs is in trusted store on OPN and nginx is happy on cert check
Quote from: mimugmail on January 20, 2021, 06:22:53 AM
opnsense-revert -r 20.7.7 os-nginx
Dies this fix it?
It did, yes. Thank you @mimugmail!!
Hopefully the pull mentioned above will address whatever the issue was during future version upgrades.
As a workaround I turned off "TLS: Verify Certificate" on the upstream. It is not recomended, but...
Quote from: Fright on January 20, 2021, 02:24:43 PM
made a PR for quck fix
https://github.com/opnsense/plugins/pull/2198
works well on test VM
@Tubs
have you tried not to select CAs in TLS:Trusted Certificate in Upstream config page?
should work if upstream cert issued CAs is in trusted store on OPN and nginx is happy on cert check
Thx, wasn't aware kulikov is you :)
QuoteThx, wasn't aware kulikov is you
;D thinking about changing name on forum (i don't even remember where it came from once) ;)
Dont ask me about "mimugmail" :P
Quote from: Fright on January 20, 2021, 02:24:43 PM
@Tubs
have you tried not to select CAs in TLS:Trusted Certificate in Upstream config page?
should work if upstream cert issued CAs is in trusted store on OPN and nginx is happy on cert check
I could not reproduce anymore.
One day after I went back to to nginx 1.19 opnsense automatically updated again to version 1.20. But it is running now. I guess the fix already was implemented.
@Tubshmm. a little weird. I think that for this it was necessary either to press "check for updates" or execute "# opnsense-revert os-nginx". and there was no fix yet.
but glad it works anyway)
@mimugmailQuoteDont ask me about "mimugmail"
;D ;)
Quote from: Fright on January 23, 2021, 06:10:34 AM
@Tubs
hmm. a little weird. I think that for this it was necessary either to press "check for updates" or execute "# opnsense-revert os-nginx".
It is possible that I have done this. If so, for a different reason. I cannot remember. But in every case I did not confirmed any upgrade process.
Hi,
We're experimenting another (probably related) issue since the update to 20.7.8. The log states:
*1 upstream SSL certificate does not match "upstream5959918e46f84fbb8bbf02dc24f2bbc5" while SSL handshaking to upstream, client: 10.6.97.138, server: server.domain, request: "GET / HTTP/2.0", upstream: "https://10.6.105.115:443/", host: "server.domain:444"
I disabled certificate validation but did not revert to a previous version. Certificate validation is failing even when no specific CA is selected in the UI. The whole TLS chain is already present in OPNsense certificate store. The only constraint we add is chain depth = 2. Removing it does not help either.
@marinbernard-pep06
hi.
Quoteupstream SSL certificate does not match "upstream5959918e46f84fbb8bbf02dc24f2bbc5"
since nginx plugin uses upstreams with uniform names, for verification to work you need specify the name in the "TLS: Servername override" field so that nginx compares the name in the certificate with this name, not with the name of the upstream with UID ("upstream5959918e46f84fbb8bbf02dc24f2bbc5" in your case). should work then.
this is not a bug, this is how it should work imho
---
but
if you use Name-based virtual hosts, then imho a situation is possible (depends on SNI implementation on backend) when this will not be enough and you will need to enable "TLS SNI Forwarding" on location but SNI does not work quite correctly for now (mentioned in the PR). waiting for review and permissions to add a SNI fix to PR to fix both issues at once
Quote from: Fright on January 25, 2021, 10:13:13 AM
@marinbernard-pep06
hi.
Quoteupstream SSL certificate does not match "upstream5959918e46f84fbb8bbf02dc24f2bbc5"
since nginx plugin uses upstreams with uniform names, for verification to work you need specify the name in the "TLS: Servername override" field so that nginx compares the name in the certificate with this name, not with the name of the upstream with UID ("upstream5959918e46f84fbb8bbf02dc24f2bbc5" in your case). should work then.
this is not a bug, this is how it should work imho
Hi,
I don't think so: our config used to work before the update, and stopped working after. Our
upstream references several
upstream servers, each of them using an individual certificate matching its real host name. Forcing the use of a single SNI would mean re-issuing certificates for all those hosts. This is a no-go for us.
QuoteI don't think so: our config used to work before the update
its because upstream verifying not worked at all before 1.20 )
https://github.com/opnsense/plugins/blob/master/www/nginx/pkg-descr
proxy_ssl_verify directive was missing
QuoteOur upstream references several upstream servers, each of them using an individual certificate matching its real host name. Forcing the use of a single SNI would mean re-issuing certificates for all those hosts. This is a no-go for us.
yep, and if you want to verify upstreams or your upstreams strictly checks SNI headers (like WAP\ADFS do) you will have to make separate upstreams for each.
Quote from: Fright on January 25, 2021, 03:22:09 PM
yep, and if you want to verify upstreams or your upstreams strictly checks SNI headers (like WAP\ADFS do) you will have to make separate upstreams for each.
It seems nginx works quite differently than HAProxy regarding TLS validation. With nginx, the validated SNI is the one set on the
upstream, not on the
upstream server. HAProxy does exactly the contrary. This comment (https://trac.nginx.org/nginx/ticket/1307#comment:5) from the nginx support team might help people dealing with the same problem.
Thank you for your help!
QuoteWith nginx, the validated SNI is the one set on the upstream, not on the upstream server.
technically nginx does it in the location block (or in server block for streams) (only the plugin GUI does it in upstream. imho there is logic in this, but is a little confusing)
QuoteHAProxy does exactly the contrary
it always takes time to switch brains when working with both)
Quote from: Tubs on January 23, 2021, 04:58:11 AM
Quote from: Fright on January 20, 2021, 02:24:43 PM
@Tubs
have you tried not to select CAs in TLS:Trusted Certificate in Upstream config page?
should work if upstream cert issued CAs is in trusted store on OPN and nginx is happy on cert check
I could not reproduce anymore.
One day after I went back to to nginx 1.19 opnsense automatically updated again to version 1.20. But it is running now. I guess the fix already was implemented.
The issue popped up again when I updated to 21.1.
Yes, when I untick "check trusted certificate" all is working again.
Quote from: Fright on January 20, 2021, 02:24:43 PM
made a PR for quck fix
https://github.com/opnsense/plugins/pull/2198
works well on test VM
Hi -- doesn't look like this fix has made it in yet, correct? After updating to 20.7.8_4, I had to revert nginx to 2.19 again.
hi
yes. not ready yet
no need to revert. you can uncheck all CAs in "TLS: Trusted Certificate" for Upstreams or uncheck "TLS: Verify Certificate" temporary
Fabian isn't responding at the moment. :(
I merged it now so it'll be in 21.1.1 ....
Sorry,
Franco
I hope everything is fine..forum profile shows activity today
thanks!!
After updating to 21.1.1 the problem is still there.
I've tested checking "TLS: Verify Certificate" with and without "TLS: Trusted Certificate".
For now I leave tls verification off.
Best regards...
@muchacha_grande
what is the exact error?
Quotewith and without "TLS: Trusted Certificate".
most likely a missconfig
Hi @Fright, I mean that I tested checking the appropriate CA and then unchecking all CAs as you pointed:
Quote from: Fright on January 31, 2021, 07:29:27 AM
hi
yes. not ready yet
no need to revert. you can uncheck all CAs in "TLS: Trusted Certificate" for Upstreams or uncheck "TLS: Verify Certificate" temporary
Unchecking "TLS: Verify Certificate" is still the only workaround for me.
Hi
it is difficult to assume something without knowing the error
is it "upstream SSL certificate does not match" or "pem file not found".
pem-file generation should be fixed in 21.1.1.
Is there some log or debug information? I can check.
Services: Nginx: Logs::Global Error Log
Services: Nginx: Logs::HTTP Error Logs
Global Error Log is full of "signal process started" messages. One per minute.
HTTP Error Log have many of this messages:
If I select the appropriate CA:
1 upstream SSL certificate verify error: (2:unable to get issuer certificate) while SSL handshaking to upstream, client: x.x.x.x, server: cloud.xxxx.com, request: "GET /login HTTP/2.0", upstream: "https://y.y.y.y:443/login", host: "cloud.xxxx.com"
And if I uncheck all CAs the error is:
1 upstream SSL certificate does not match "upstream0768447a263a475ab813a0e17926c340" while SSL handshaking to upstream, client: x.x.x.x, server: cloud.xxxx.com, request: "GET /login HTTP/2.0", upstream: "https://y.y.y.y:443/login", host: "cloud.xxxx.com"
so verification works.
Quote(2:unable to get issuer certificate
nginx cannot build a chain to a trusted CA certificate from upstream cert
is the certificate of the certification authority in the trusted list?
how long is the chain from the root certificate to the upstream certificate?
Quoteupstream SSL certificate does not match "upstream0768447a263a475ab813a0e17926c340" while SSL
add upstream name from cert to **TLS: Servername override**
https://github.com/opnsense/docs/pull/305/files
Quote from: Fright on February 10, 2021, 05:34:08 PM
add upstream name from cert to **TLS: Servername override**
https://github.com/opnsense/docs/pull/305/files
This worked. So now I have no CA checked and with the server override.
Still not working if I check a CA.
Quote from: Fright on February 10, 2021, 05:05:15 PM
is the certificate of the certification authority in the trusted list?
It is. Is a Lets Encrypt certificate. I imported the CA and the certificate in OPNSense.
Quote from: Fright on February 10, 2021, 05:05:15 PM
how long is the chain from the root certificate to the upstream certificate?
It's a Lets Encrypt certificate directly signed by the Lets Encrypt authority.
QuoteLets Encrypt authority
LE CA is not root CA afaik
need to add DST Root CA X3 (or what is root CA for LE certs) to System: Trust: Authorities and select this CA in upstream
Quote from: Fright on February 10, 2021, 06:45:01 PM
LE CA is not root CA afaik
need to add DST Root CA X3 (or what is root CA for LE certs) to System: Trust: Authorities and select this CA in upstream
You are right. I checked the certificate chain and it ends at DST Root CA X3. So I did what you pointed out and now it's working.
Thank you very much @Fright for your help.
Best regards.