[solved] SSL cert validation broken after upgrade

[irgendwr]

Hi,

this is my first post and it might be a bit long but I tried to include all of the details and things I tried.
I would be very glad if someone is able to help 

After upgrading OPNsense from 19.1.6 to >= 19.1.7 the upgrade process got stuck at fetching a file and would not finish and just fill the log with dots. When clicking the "Check for updates" button again it would tell me:

Could not authenticate the selected mirror.

I then decided to roll back to a snapshot before the upgrade and export my config. I reinstalled the current version (19.7) and restored my config. After restoring I got the same behavior as described before; Clicking "Check for updates" would tell me:

Could not authenticate the selected mirror.

I then tried to update via the terminal and it showed the following:

Code: [Select]

Fetching change log information, please wait... Certificate verification failed for /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
4033068160632:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/s3_clnt.c:1269:
fetch: https://pkg.opnsense.org/FreeBSD:11:amd64/19.7/sets/changelog.txz.sig: Authentication error

This seemed to indicate that something is wrong with the SSL certificates, which a quick test confirmed:

Code: [Select]

root@OPNsense:~ # fetch https://example.com
Certificate verification failed for /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
3276227729016:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/s3_clnt.c:1269:
fetch: https://example.com: Authentication error

root@OPNsense:~ # curl https://example.com
curl: (77) error setting certificate verify locations:
  CAfile: /usr/local/etc/ssl/cert.pem
  CApath: none

I tried looking for similar issues in the forum but only found two threads:
1) https://forum.opnsense.org/index.php?topic=12824.msg59372#msg59372
=> not helpful, different issue
2) https://forum.opnsense.org/index.php?topic=12550.0 (german)
=> same issue! But...

It mentions that the cause might be "cert.pem" being empty so I checked it:

Code: [Select]

root@OPNsense:~ # ls -lah /usr/local/etc/ssl/
total 852
drwxr-xr-x   2 root  wheel   512B Aug  2 03:14 .
drwxr-xr-x  32 root  wheel   4.0K Aug  2 01:49 ..
-rw-r--r--   1 root  wheel   798K Aug  2 04:18 cert.pem
lrwxr-xr-x   1 root  wheel    38B Jul 24 00:51 cert.pem.sample -> /usr/local/share/certs/ca-root-nss.crt
-rw-r--r--   1 root  wheel   9.3K Jul 24 09:52 opnsense.cnf
root@OPNsense:~ # ls -lah /usr/local/share/certs/
total 840
drwxr-xr-x   2 root  wheel   512B Aug  2 03:14 .
drwxr-xr-x  29 root  wheel   512B Aug  2 01:49 ..
-rw-r--r--   1 root  wheel   793K Jul 24 00:51 ca-root-nss.crt
Not sure if the files are alright, but at least they are not empty.
The Administrator @franco mentioned that 19.1.7 would fix that but for me anything >= 19.1.7 seems to break SSL instead of fixing anything 

Nevertheless I tried his suggestion:

Code: [Select]

cp /usr/local/etc/ssl/cert.pem.sample /usr/local/etc/ssl/cert.pem
and... it actually WORKED! \o/
So why am I posting this then?

After "reloading all services" or rebooting the "cert.pem" file is 798K again, instead of the 793K from /usr/local/share/certs/ca-root-nss.crt and everything is broken again 

I'm stuck and don't know what else to do anymore. How can I fix this? 

[bartjsmit]

What are the differences between the certificates?

openssl x509 -in cert.pem -noout -text | less

Are the valid dates and the subjects different?

Bart...

[irgendwr]

Are the valid dates and the subjects different?

Both files produce the exact same output with the same date and subject.

What are the differences between the certificates?

I checked the diff and found something interesting:
OPNsense appends the CA's from "System: Trust: Authorities" to the end of the file...
The Let's Encrypt client plugin (os-acme-client) seems to have added three CA's for Let's Encrypt and one seems to be invalid  . Besides that I only added one for self-signing the Webinterface and OpenVPN certs.

I removed the invalid looking LE CA and now it works! Thank you! 
Weird that this only caused issues for me after upgrading.