OPNsense Forum

Nginx website protection opnsense

Hi all, I am recently running opnsense and I love all that it does. It works like a charm, but now I want to make my website protected behind nginx plugin with WAF.
I changed the default port of OPNsense to 444, so now port conflicts

My setup is now ISP – Opnsense – access point – Synology webstation.
I have on my opnsense – firewall – NAT - a port forward  for 80 and 443 to the IP of the Synology webstation 198.168.1.133
Synology runs the webstation with an Lets encrypt certificate for the webserver, and the site runs correctly with https. Port 80 is for lets encrypt to renew certificates and 443 for the ssl for instance I think.

Now I want to remove the port forward and have the website behind / through the nginx plugin with waf. This is what I have done, but it does not work. What have I done wrong? What steps do I have to add / change?
Your help is very much appreciated!

1.  I have configured the upstream server. Description "webserver", ip 198.168.1.133, port 80, priority 1, maximum connections 2500, max fails 10 and time out 60

2. I have configured the upstream. Description "backend", server  I selected the above "webserver", load balancing "weighted round robin", enable tls: selected/checked, tls supported tls v1.2 and v1.3, tls session reuse: selected/checked, tls trusted certificate: none selected

3. I have downloaded the NAXSI WAF rules

4. I added a new location HTTPS:  description "webserver_root", URL pattern / , match type " none", url rewriting " nothing selected", enable security rules: selected/checked, then 3 blank, custom security policy: I have selected all the options. Upstream servers "backend", patch prefix empty, cache directory " empty", Force HTTPS: selected/checked, enable HTTP/2 preloading: selected/checked.

5. I added HTTP server: http listen port: 80, https listen port: 443, domain "mywebserverdomain.com", locations "webserver root", url rewriting " nothing selected", file system root: empty, TLS sertificate "none", client CA certificate "none", access log "default", enable Lets Encrypt plugin support: selected/checked, Charset utf-8, HTTPS only: selected/checked, IP ACL "none", advanced ACL_authentication backend "local database", security header "none", limit request " nothing selected"

(Do I need to install the lets encrypt plugin in opnsense? And how to configure that? I have not done so because the certificates are on the sinology. )

6. I made a port forward rule: NAT – portfoward: interface WAN, tcp/ip version: IPv4, protocal TCP/UDP, destination " wan address", destination port range from HTTPS to HTTPS, redirect target ip " 192.168.1.1" (opnsense ip), redirect target port HTTPS, description "port forward to nginx proxy". (p.s. I have deleted the earlier made port forward rules 80 and 443 to the ip of the synology web station.)

7. Nginx global settings,  enable nginx selected/checked

The result is that my website does not load on LAN devices, and it does not load from external devices.
What have I done wrong / forgot / have to change?!

Thank you very much in advance for your help!!

Regards
Ramon

Port forward is definitly wrong -> Firewall rule on WAN and all the other interfaces where you would like to access nginx on the required tcp ports (I guess TCP/80 and TCP/443).

The next thing is you do not have configured a Certificate so the HTTP-Server cannot run in HTTPS mode. Without opening the TLS connection on OPNsense in the nginx plugin, you will not be able to access the data so the WAF will not work. If you want to pass through the connection to your NAS you can still do that using a stream server but I would recommend to have the LE-Certificate and Key on OPNsense so nginx can look into the connection and can handle it as it should (as an HTTP aware server).
It is possible to have LE outside of OPNsense but then you have to update it from time to time.

Thank you Fabian for your reply.
So in the firewall I should delete the portforward, go to firewall rule- wan  - and add a rule for ports HTTPS and HTTP, and to what destination do I point? the wan address?
And I have to export the lets encrypt certificate from synology and import it in opnsense. ok.
And every time the free certificated is being updated I have to change it in opnsens? Is there not an automation way like synology updates it with lets encrypt? Sorry if this sounds like easy questions. Opnsense is very new for me

thanks again,
Ramon

For the firewall I would recommend to use 'this firewall' as destination.

If you install the acme plugin, you can issue the certificates directly on OPNsense. The nginx plugin has an advanced option in the http server to forward the challenge to the acme plugin.

Thanks Fabian!
The firewall rule was the key. Got nginx running... somewhat.
Now I have to figure out how to get it working with proper ssl certificate and HTTPS.
i will do a web search/google into ACME plugin..

do you know of some sort of a guide for nginx https and acme?

Thanks again for pointing me into the right direction!

If you install the acme plugin, you can enable the checkbox for lets encrypt in nginx and reconfigure nginx. Then the HTTP-01 type challenge works immediately if you configure it. If you have it, you should get a certificate, which can be used by nginx.

Dear Fabian, First Merry Christmas!
I have nginx running with the " manual mode". The lets encrypt certificate imported at opnsense and used them in nginx. The website is running, so thank you for your help with that. With this manual mode I have to manually import the certificate every time there is a renewal. I understand that.

Now if I go for the automation option with ACME. Do I understand it correctly that ACME will make and renew the lets encrypt certificate automatically and I use then in nginx proxy. And does the synology webstation also maintain its lets encrypt certificate like currently in the manual version I am using now, or is that not correct? and has synology webstation in this ACME solution no longer a certificate on the synology itself?

Kind regards
Ramon

If you enable the automatic certificate creation on OPNsense, you cannot use the HTTP-01 challenge type on your NAS. That is all. If you use the DNS challenge on your NAS, both will work. The good thing is that you do not really need a public trusted certificate there because nginx will replace it by Let's Encrypt so you have a publicly trusted certificate there and you can use your internal CA on your home which needs to be trusted by OPNsense.

Thanks again for your help!
That sounds logic. I am going to experiment further with this. It feels great to have this working. Could not have done it without you! Loving OPNsense more and more everyday.

Best regards
Ramon

Hi Fabian,
I have one strange thing left. When I use my ISP ip on an external device, it shows up my webserver website, while using the mydomain.com show the correct website, I do not want to have my ISP ip to load a website.
do you know how to prevent this?

best regards
Ramon

This is how nginx works, It will choose the first server if there is no default if none of them is matching. I do not know a way to turn off this behaviour entirely.

Thanks for the explanation. Maybe I can change the behavior by pointing the isp ip in nginx to another (empty) webpage on the synology? should have a self signed certificate I presume?

thnx again for helping with my learning curve in OPNSense

If you do not point nginx to anything, it will serve the content fom the default directory which should be empty. No need to involve a 3rd party application.

thnx again for your help and fast reply.

regards
Ramon

Thank you for sharing such information, but my browser block that proxies.

@Fabian
Strange things here. All was working fine until I did the update to opnsense 20.7.8. I read in the update file that there is nginx 1.20 included.
What happens is: Before the update: All nginx proxy to https website synology was working thanks to your help. I did an update to opnsense 20.7.8 last night and no other changes in config. Than after the update the website is no longer accessible.
I did another reboot of Opnsense, but still no difference. This error appears when going to the website url:
------
Server Error

Sorry, but something went wrong on our side.

There is nothing you can do except waiting until we fix the issue.
Web Application Protection by OPNsense
-----

What is going wrong? error in the update? or is there something with nginx 1.20 that I have to add in the configuration due to this update?

Thank you in advance for your help!

is upstream cert verify enabled?
is trusted cert defined?

TLS: Verify Certificate -> enabled

TLS: Trusted Certificate -> nothing selected

TLS: Verify Certificate -> when disabled the site loads again?

QuoteTrusted Certificate -> nothing selected

QuoteVerify Certificate -> when disabled the site loads again

good
is upstream cert\issuedCAs certs added to trusts on OPN?

Under system - trust - authorities - domain chain pem listed
under system - trust - certificates - domain cert listed

before the update of opnsense site's it was working.....

Quotebefore the update of opnsense site's it was working.

upstream verification didn't work until 1.20.
it would be great to know what the errors are in logs
is it "upstream SSL certificate verify error" or some..

Quotedomain chain pem listed

sorry, is it certs used by nginx or certs used by upstream (backend)?

sorry,
cert for the upstream - configured under nginx - configuration - http server; TLS Certificate and Client CA Certificate selected with the cert of the domain running on synology webstation.

global error log is only showing : signal process started

" upstream verification didn't work until 1.20."
Maybe that is what I have to configure? How can I add this?

if I understood you correctly, since it starts to work with TLS:Verify unchecked, it should be something with upstream settings or plugin issue.
for now the plugin issue i know about is empty trust_upstream_*.pem files (and error on nginx loading so nginx not loads at all) when the trusted CA's is selected in TLS:Trusted certificate.
since you dont use it, I assume that your settings could not work with upstream verification. just before this function did not work in the plugin itself and it did not cause problems.
now it is enabled, but the settings do not allow the upstream certificate to be verified by nginx

Quoteglobal error log is only showing : signal process started

please look at Logs-HTTP Error logs-YourHTTPServer

QuoteMaybe that is what I have to configure? How can I add this?

its enabled by default. Configuration-Upstream-YourUpstream-AdvancedMode-TLS:Verify Certificate

that is correct " when the trusted CA's is selected in TLS:Trusted certificate"
As soon as I select te ca's for the sites, and refresh nginx it stops!

is this a bug? is there a solution? So I understand I have to select the ca's and than it should work with upstream verification what was empty at my config and did work with version before 1.20. Now with 1.20 this is available and I have to select the upstream ca's and nginx stops working when added.

error log: I see those: upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream, client:

Quoteis this a bug? is there a solution?

I hope @fabian is already aware of the problem and it will be fixed

QuoteNow with 1.20 this is available and I have to select the upstream ca's and nginx stops working when added

not really. the ability to select CA certificates in this field is provided to restrict trust to an upstream certificate to only selected CAs.
you can (and should imho. until bug will fixed) leave this field blank ("Nothing selected") and then the plugin will point nginx to the /etc/ssl/cert.pem file. i.e. all OS trusted roots plus CAs from System->Trust

Quoteunable to get local issuer certificate

it looks like the trust store is missing one or more CA certificates for your synology cert (may be Intermediate CA).

humm.. So I have to leave " TLS: Trusted Certificate" blank for now and disable TLS: Verify Certificate?
and wait for the fix?

sorry for the unclear wording )
you can use upstream verify right now.
but only with blank " TLS: Trusted Certificate" (ie verify upstream cert against all trusts store: OS trusts and System->trust of opnsense)
"unable to get local issuer certificate" error is not a plugin error, but misconfiguration imho

ah ok.. That sounds logic.
The fact that my site does give the error is than because the webserver has a cert that is not added in Opnsense?
I have all certs added (lets encrypt on my synology webstation) also to my opnsense. Only the internal synology cert is not on the opnsense. But can't see why that would cause the problem? and is internal / no authority /not lets encrypt.

QuoteOnly the internal synology cert

sorry, where is this certificate used?
when you access your synology directly, what cert you see in your browser?
LE?
so  Let's Encrypt Authority and  DST Root CA X3 (or what root was used when cert was renewed) should be in System->Trust

and one more thing: the next error you are likely to get (after cert chain will be correct) will be:
'upstream SSL certificate does not match 'upstreamblahblahblah'.
since plugin uses upstream uids in proxy_pass directives, you need specify the server name from the upstream certificate in the "TLS: Servername override" field. so nginx can verify name from upstream certificate against this field and not against upstream UID

when you access your synology directly, what cert you see in your browser? -> internal cert made at the synology. It is only used to locally access the synology, not for the websites. Those have an lets encrypt cert.

can the field " "TLS: Servername override" contain more than 1 domain name?

Quotenot for the websites. Those have an lets encrypt cert

got it, thanks
so LE CA's from this cert should be presented in System->trust

Quotecan the field " "TLS: Servername override" contain more than 1 domain name?

no. and there is no need. the field does not have to match all subject names at once )

i have added in system - trust - certificate the self signed synology cert. -> still the error when going to the websites
Since internal cert I do not have an system-trust-authority for the synology. (?)

TLS Servername Override -> so only 1 domain should be mentioned. Maybe silly question, but when I have multiple websites, it does not matter which domain I add here? only one is enough? I have one upstream server.

and really thank you for your help and explanation

Quotei have added in system - trust - certificate the self signed synology cert.

no-no. no need to add this cert to OPN if you not using it for websites on synology.
need to add CAs certs from LetsEncrypt chain on synology cert to OPN->System->Trust, restart ngnix and look in HTTP Error Logs for changes in behavior

Quote, it does not matter which domain I add here?

yes (in your case since you don't use SNI). better to use name from the cert's CN

Quoteno-no. no need to add this cert to OPN if you not using it for websites on synology.
need to add CAs certs from LetsEncrypt chain on synology cert to OPN->System->Trust, restart ngnix and look in HTTP Error Logs for changes in behavior

Ok, but I already had my lets encrypt cert used on synology added on OPNsense-system-trust. That was why it worked before Nginx 1.20. Or must I delete those, add them again an see if it works?

QuoteOk, but I already had my lets encrypt cert used on synology added on OPNsense-system-trus

again: not cert itself. certs of CA's that issued this cert should be in OPNsense-system-trust-Authorities.
and although it may sound strange, try to play with the value of "TLS: Verify Depth" field (make it 2 o bigger).
(although the error message should be different in this case, I have seen messages that it caused an "unable to get local issuer certificate" error.  for me it works well with depth 2 for certs with 2 intermediate CAs in chain. I have seen messages that it requires depth=4 for this)

I have added the content of cert.pem and privkey.pem in the system-trust-certificates
and the content of chain.pem in the system-trust-authorities.
Those where already added to opnsense. Or I must have missed something and it somehow did work on the previous nginx?

I tried just setting verify depth to 2 and 4 but no difference.
Don't know what I am missing

QuoteI have added the content of cert.pem and privkey.pem in the system-trust-certificates

no need to import private key for this

QuoteOr most have missed something and it somehow did work on the previous nginx?

in previous versions, verification was not allowed (the line was missing in the template). so the error just could not occur

QuoteI tried just setting verify depth to 2 and 4 but no difference.

hm. the same error in the log?

and can you share result of

# openssl s_client -connect 198.168.1.133:443

? (you can omit the content between BEGIN CERT--END CERT to save space. and sanitize private info)

the error is the same, only the first digit can be different per line.
it is *8 and *1, *5, thereafter it is the same
*11 upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream, client: <my device ip>, server: <my website url>, request: "GET /HTTP/2.0", upstream:

Quoteand can you share result of
Code: [Select]

This is the result:
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 C = NL, ST = xx, L = xxxxx, O = xxxxx, OU = xxxxx, CN = 192.168.1.133, emailAddress = info@xxxx
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = NL, ST = xx, L = xxxxx, O = xxxxx, OU = xxxxx, CN = 192.168.1.133, emailAddress = info@xxxx
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:C = NL, ST = xx, L = xxxxx, O = xxxxx, OU = xxxxx, CN = 192.168.1.133, emailAddress = info@xxxx
   i:C = NL, ST = xx, L = xxxxx, O = xxxxx, OU = xxxxx, CN = xxxxx, emailAddress = info@xxxx
---
Server certificate
-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----
subject=C = NL, ST = xx, L = xxxxx, O = xxxxx, OU = xxxxx, CN = 192.168.1.133, emailAddress = info@xxxx

issuer=C = NL, ST = xx, L = xxxxx, O = xxxxx, OU = xxxxx, CN = xxxx, emailAddress = info@xxxx

---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2372 bytes and written 419 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 533E94733C1CA4F3F9C852F5B57970F34DCF969F2B4EDE0CC1D03FE27964627A
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1611250116
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
---
closed

Quoteverify error:num=21:unable to verify the first certificate

so as you can see there is cert issue on upstream

QuoteCertificate chain
0 s:C = NL, ST = xx, L = xxxxx, O = xxxxx, OU = xxxxx, CN = 192.168.1.133, emailAddress = info@xxxx
   i:C = NL, ST = xx, L = xxxxx, O = xxxxx, OU = xxxxx, CN = xxxxx, emailAddress = info@xxxx

upstream server sends only leaf cert (should send chain with issued intermediate (or root) CAs cert) and there is a strong feeling that this is a self-signed certificate. Not letsencrypt

or the synology web-intraface bind to 443 also?

can you repeat this command but with internal site name?
openssl s_client -connect OneOfYourInternalSiteOnSynology:443

yeah the error is there.
The only thing I do not understand is that it was working before the update and now with 1.20 it does not.

And now the error seems to be something like verifying the self signed cert in stead of the lets encrypt... and all after the update, or is that a coincidence (I have not changed anything in the config on opnsense or synology)

can you repeat this command but with internal site name?
openssl s_client -connect OneOfYourInternalSiteOnSynology:443

QuoteThe only thing I do not understand is that it was working before the update and now with 1.20 it does no

as I said, 1.20 enabled upstream verification. before that it did not work and therefore these errors could not exist at all

no errors there to the site url:

CONNECTED(00000005)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = <domainname>
verify return:1
---
Certificate chain
0 s:CN = <domainname>
   i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----
subject=CN = <domainname>

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3055 bytes and written 403 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 40ECCDD0F5259C9C5A4DA19303D796865BEE345A6CD04B7BD958F12CD76C8B9C
    Session-ID-ctx:
    Resumption PSK: C9D1C9FC9CEEB39B86191369CD7CEA30F9AC48FE57DD788A3834CE25F04842201914FD5E5772E16C9929BC086451EA55
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - b8 36 42 74 c7 75 cc f1-8b 17 ae 92 be 6c 34 86   .6Bt.u.......l4.
    0010 - b2 15 ff f1 1d 64 0f 0d-b0 77 f2 2e 42 c4 07 16   .....d...w..B...

    Start Time: 1611255660
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 80921EF250DE4870CA312FF6382871B6F979853F03A425FB8ADA222D01D56038
    Session-ID-ctx:
    Resumption PSK: 4C20487328E90DFC5CFF1D4B6A14A649F86ABEBEFE886F58027E60022FDA33C2225AAAE570BAB6B78D098866C47C33CF
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - 5c 5f d4 84 d6 65 e8 16-34 07 16 62 e5 b9 5c d9   \_...e..4..b..\.
    0010 - 75 28 b8 10 8d 76 df 39-fb 28 25 f2 f1 f6 33 39   u(...v.9.(%...39

    Start Time: 1611255660
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0

thanks.
hmm..so if on Synology you used "Name-based virtual hosts" (or as it is called in Synology) this may be SNI issue.
so when nginx connects to upstream it sends wrong SNI and synology starts with self-signed cert.
in this case you can try:
1) change default cert (if synology allows that) to LE and test again
2) fix SNI directives on nginx plugin. but in this case you will need to edit location.conf template by hand (my request is still not reviewed on github). if you are ready to try, I will write what and where to change and you can try to verify upstream again

Your the best!

I changed the default synology cert from self signed to lets encrypt of a website domain for synology as default cert
I changed in nginx - upstream - TLS: Servername override -> added that domain used as synology default cert

and website loads again!
Strange... but it works

So now it is to wait for the fix for nginx 1.20 to being able to add the < TLS: Trusted Certificate> domains. ?

* one thing I think of now is: with the default domain for synology not being the internal cert and using the lets encrypt domain for a website, I will get an error on e.g. iPhone while going to the synology domain cert not matching I think.
How to fix that? or is this a temporary work around and will I have to set the default cert synology back to self singed cert after the nginx update?

glad it worked

QuoteStrange... but it works

why its strange? finally a configuration is created that meets the requirements and it works  ;)

QuoteSo now it is to wait for the fix for nginx 1.20 to being able to add the < TLS: Trusted Certificate> domains. ?

yes. but I am starting to think that fixing of SNI directives in locations templates also becomes critical for some configurations (upstream verification with SNI virtual hosting)

QuoteI will get an error on e.g. iPhone while going to the synology domain cert not matching I think

maybe the LE certificate does not contain synology domain name?

Quotewill I have to set the default cert synology back to self singed cert after the nginx update?

yes, hopefully

thnx again for your help and fast replies!
Glad it works now and will wait for the nginx update to configure it further how it should be with < TLS: Trusted Certificate>

and getting synology default back to self signed.
you are correct with the LE nog containing the synology domain while I use only the local ip to connect to synology itself or from outside with mobile using vpn to local ip synology, so no domain is being used therefor.

Is it to soon to change back to the self signed cert for Synology?
Im running on opnsense 21.1.5
with the latest Nginx in it.
I am now able to add my domain cert to TLS: Trusted Certificate
And I removed TLS: Servername override

Sites keep working, so there has been made improvements. But when I change my default cert on the synology to the self signed cert, the site gives the error page from nginx/opnsense. So I think I was to early trying for that, or I am missing something in my settings for nginx in doing so (?)

hi
21.1.5 includes fixes for sni. so it should work imho

QuoteBut when I change my default cert on the synology to the self signed cert, the site gives the error page

any clue in Services: Nginx: Logs-HTTP Error logs at this moment?

QuoteAnd I removed TLS: Servername override

TLS SNI Forwarding should be enabled in Location
TLS: Servername override should be enabled in Upstream and set to name in cert (nginx will compare names in cert with name in this field)

Thanks for your help.
I tried again with synology to the self singed cert and
I have TLS SNI Forwarding enabled now in Location, but still getting:
Server Error

Sorry, but something went wrong on our side.

There is nothing you can do except waiting until we fix the issue.

and in Nginx Logs-http error:
*1 upstream SSL certificate verify error: (2:unable to get issuer certificate) while SSL handshaking to upstream, client:

switching off TLS SNI Forwarding and Synology back to the previous default cert, kept the system not running anymore, while it did work previously.
I restored a opnsense backup config and system is still not working. Don't know why. Only 2 things I did was enabling TLS SNI forwarding on this time, and off while I got the above error....

rebooted the Synology, and still not working... Strange.
So I went back to the backup settings that was working yesterday, but not working now anymore.....

in addition:
I have now removed  TLS: Trusted Certificate in upstream 
And website is running again. Strange while I had it yesterday working with TLS Trusted certificates listed in upstream.

What can this be?

Testing further:

Now I have TLS SNI Forwarding enabled in Location
TLS: Servername override enabled and filled with domain

and as long as I keep TLS: Trusted Certificate empty my site works. As soon as I enable the certs I get the error again.....

hi.sorry, I can react slowly soon

Quoteand as long as I keep TLS: Trusted Certificate empty my site works. As soon as I enable the certs I get the error again....

so what's in the Services: Nginx: Logs: HTTP Error logs  at this moment? the name must be mismatched or the root certificate cannot be found

I just tested it:
without TLS: Trusted Certificate I get: no HTTP error

with TLS: Trusted Certificate I get this on 2 of the 3 domains:
*1 upstream SSL certificate verify error: (2:unable to get issuer certificate) while SSL handshaking to upstream, client:

But with TLS: Trusted Certificate filled the default Lets encrypt synology NAS domain cert still does work, the other 2 domains do not.(?). That default cert change from self signed to one of the 3 domains lets encrypt cert was your earlier solution replacing the self singed cert on synology. That one does work. So has this to do with the virtual hosts also?

Hi

Quote:unable to get issuer certificate

so there is not enough depth or the CA's certificate is not in the trusted list

quick tested on test VM with 21.1.5 without any customs\tweaks:
-change gui cert to self-signed with 'configctl webgui restart renew'
-copy gui cert to trusted CA's
-add upstream to 127.0.0.1 with  TLS: Verify Certificate enabled and gui cert as trusted CA
-add server with location pointed to 127.0.0.1 upstream
works well accessing gui via nginx
if i set wrong trusted CA (say R3 LE) nginx throws "upstream SSL certificate verify error: (18:self signed certificate) while SSL handshaking to upstream"

can you check what certificates synology returns when accessing each site?

Hi Fright,

I tried to set with TLS: Trusted Certificate  and with TLS: Verify Depth from 1 (what it already was), 2, 3 all to 8 but no difference.

Quotecan you check what certificates synology returns when accessing each site?

When I go to all of the 3 domains, the 3 domain/sites show all the right specific SSL cert (lets encrypt) for that specific domain on the site.
Or do I have to check this some other way?

Quotehe 3 domain/sites show all the right specific SSL cert (lets encrypt) for that specific domain on the site.

more like sites do not pass the intermediate CA certificate (R3)
can you share the result of
'openssl s_client -connect OneOfYourInternalSiteOnSynology:443'
connecting to one of the virtual hosts?

Dear Fright,

I just sent you a PM with the info of openssl s_client -connect website.com:443
without TLS: Verify Certificate
and with TLS: Verify Certificate enabled.

Hope you can find what the problem is...

hi
please clarify: is this a client-nginx or client-synology connection? (in the example you sent, the server correctly transmitted its certificate and the certificate of an intermediate CA).
would like to see client-synology connection

My system is Opnsense -> nginx -> synology (virtual hosts)

I disabled nginx, did a port forward to the synology and run openssl s_client -connect website.com:443

The output I have just sent in a PM to you.

hi
synology behavior looks correct.
looks like some kind of obvious error in nginx config that we are missing
sorry if I repeat myself, I understand correctly that now everything is working fine if upstream verification is enabled but no CA is specified in "TLS: Trusted Certificate"? and stops working if you specify a CA in the settings?
then it looks like just a wrong choice of CA in "TLS: Trusted Certificate". although then the error would have to be "20:unable to get local issuer certificate" imho

Yes that is correct.
When I enable TLS Trusted certificate with the 3 cert than than only 1 site keeps working. The working one is the cert for the domain that is also the default cert on synology (by your earlier advice before the updated nginx, instead of using self signed cert for default cert on synology). The other 2 stop working with the mentioned error.
You mentioned earlier SNI that could cause problems if I remember correctly. Is that something?

QuoteYou mentioned earlier SNI that could cause problems if I remember correctly. Is that something?

21.1.5 with 1.22 nginx plugin contains fixes for the mentioned problem
besides, if I understand correctly, the problems arise only when trying to narrow the number of trusted root CA's. if the problems were in SNI, then it would not work even with an empty "TLS Trusted certificate" field imho.
What trusted CAs do you specify in the "TLS Trusted certificate" field? maybe this is the problem?

ok,
I specify the 3 domains on synology (and only 3 cert's used on synology). Those 3 have their own lets encrypt cert.
When I only TLS Trusted certificate the domain that is also used for the default synology cert, than there is no difference. that domain (lets say site1) keeps working, the other 2 don't.

When I add all 3 domains to TLS Trusted certificate, I get the same result. That one site (site 1) keeps working, and the other 2 don't.

I tried something further. When I TLS Trusted certificate 1 other domain, cert site 2, site 1 keeps working, 2 and 3 error

When I TLS Trusted certificate site 3, than again, only site 1 works.

this site 1 is also the domain mentioned in TLS: Servername override

So it looks like no matter wat I select in TLS Trusted certificate, only site 1 keeps working, suspiciously also the default synology cert

Hi
sorry, I still have the feeling that I did not fully understand your actions. for sites using LE certificates, the TLS Trusted certificate field should contain the certificate of the root certification authority that issued the certificate to the LE intermediate authority (that is, DST Root CA X3). what certificate do you specify for sites with LE certificates in TLS Trusted certificate?
maybe you can send screenshots?

Maybe I did not understand it correctly than?

I have in opnsense under Trust - Authorities -> I have added the details of 3 LE chain.pem from the 3 domains.

Those site1chain.pem, site2chain.pem and site3chain.pem are the only 3 I can select at nginx-configuration-upstream-TLS: Trusted Certificate

But your text

QuoteDST Root CA X3

: do you mean that I only select 1 chain.pem? and that that is the info from LE? So in opnsense I ad one more Trust Authoritie Like from this url: https://letsencrypt.org/certs/trustid-x3-root.pem.txt (https://letsencrypt.org/certs/trustid-x3-root.pem.txt)
and I select that one in TLS: Trusted Certificate?

Or ?

---- addon
I added the info from https://letsencrypt.org/certs/trustid-x3-root.pem.txt to opnsense trust-authorities and than selected in nginx - configuration - upstream - TLS: Trusted Certificate -<selected this trustid-x3 I just added to opnsense>

And all the 3 sites work like it should. I hope I did follow you correctly and this was what I was doing wrong. (?)

hi
glad it works

QuoteI hope I did follow you correctly and this was what I was doing wrong. (?)

may be  those  "site1chain.pem, site2chain.pem and site3chain.pem" did not contain the  root cert?
then this could be the reason

Well it works, or it looks like it works :-) (is there a verification method?)
What I do not understand is that with the site1chain.pem, site2chain.pem and site3chain.pem and no matter what chain pem I selected in TLS: Trusted Certificate, only the default synology cert, say site1chain.pem, kept working and not the other 2.... Was it than not more "logic" that also site1 should stop working when selecting one of those in TLS: Trusted Certificate than also??

Quotemay be  those  "site1chain.pem, site2chain.pem and site3chain.pem" did not contain the  root cert?

How can I verify this? I have those Letsencrypt certs created in synology for that specific domain/site. Than I have exported the cert and put the info in opnsense trust authorities and certificate. 

QuoteI have those Letsencrypt certs created in synology for that specific domain/site

then it probably is. the chain on the server usually does not need to contain the root certificate. only endpoint certificate and intermediate authority certificate

QuoteHow can I verify this?

can try to look in pem-file. does it contain certificate from https://letsencrypt.org/certs/trustid-x3-root.pem.txt ?

QuoteWell it works, or it looks like it works :-) (is there a verification method?)

can try to change certificate on web-server to some non-LE or cert with CN\SAN that does not match the TLS: Servername override field and check the result )

thnx for explaining.

Quotecan try to look in pem-file. does it contain certificate from https://letsencrypt.org/certs/trustid-x3-root.pem.txt ?

-> no the info from the pem-file is different than the info in https://letsencrypt.org/certs/trustid-x3-root.pem.txt

Quotecan try to change certificate on web-server

I changed the cert on the synology part to a self signed cert for a domain, but no difference. The site loaded with the correct LE cert for that domain... Than I changed the cert in opnsense-nginx-HTTPserver to a cert for another domain, so like I gave site2 the cert for site1. And loaded the site. Then I noticed that the site was given the cert1. So that change worked... Can it be that nginx on the opnsense part overrides / makes synology cert obsolete?

QuoteCan it be that nginx on the opnsense part overrides / makes synology cert obsolete?

when a client accesses the site through a nginx, client sees the certificate that nginx provides. only the nginx itself "sees" the certificate that upstream provides.

QuoteI changed the cert on the synology part to a self signed cert for a domain, but no difference

if you change the certificate on synology web to a certificate with mismatched (with TLS: Servername override field) name (in CN\SAN) or not signed by a root from the list in TLS Trusted certificate, access to the site through nginx should stop working

Quoteif you change the certificate on synology web to a certificate with mismatched (with TLS: Servername override field)

that was it! as long as the cert for the server name override was correct, the site keeps working. when I set that (with TLS: Servername override field) to the changed site with wrong cert, the site stopped working as you described and as should.
Thnx! I understand now how it works.

I have now also changed the default cert for synology back to the self signed cert and there sites keep working as they should. Great Feeling!

And thank you for making those updates to opnsense/nginx.