24.10 BE upgrade CRL errors

[franco]

24.10_1 is available now so it doesn't have to be picked up manually via opnsense-patch anymore.


Cheers,
Franco

[DEC670airp414user]

I did attempt but from the gui I could not get past the error messages so it could not find anything new
In the past downgrading and upgrading fixed it👍

[really_lost]

I'm also at 24.10 BE and receive the could not load CRL file error:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.10 at Thu Oct 24 20:15:38 UTC 2024
Fetching subscription information, please wait... Could not load CRL file /tmp/libfetch_crl.24102420
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/subscription: Authentication error
Fetching changelog information, please wait... Could not load CRL file /tmp/libfetch_crl.24102420
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/sets/changelog.txz: Authentication error
Updating OPNsense repository catalogue...
Waiting for another process to update repository OPNsense
Updating SunnyValley repository catalogue...
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
pkg: https://updates.zenarmor.com/opnsense/FreeBSD:14:amd64/24.7/${SUBSCRIPTION}/meta.txz: Authentication error
repository SunnyValley has no meta file, using default settings
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
pkg: https://updates.zenarmor.com/opnsense/FreeBSD:14:amd64/24.7/${SUBSCRIPTION}/packagesite.pkg: Authentication error
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
pkg: https://updates.zenarmor.com/opnsense/FreeBSD:14:amd64/24.7/${SUBSCRIPTION}/packagesite.txz: Authentication error
Unable to update repository SunnyValley
Error updating repositories!
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***

I did attempt

# opnsense-patch 372c9c98 70df0a15f7e
# /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com

and the update-crl-fetch.py worked.  However, a check for udpates in the GUI still fails.

[franco]

Just make sure to

# rm /tmp/libfetch_crl.*

And check from the GUI again. If it complains about a file post the contents of it here, e.g.

# cat /tmp/libfetch_crl.24102420

(I'm assuming the file is empty which is a risk with OpenSSL and file-based CRL provider). We're switching to the trust store integration in 24.10.1, but it needs a bit more tweaking at the moment.

Worst case you cannot fetch the CRL because a strict block rule was implemented.


Cheers,
Franco

[really_lost]

Thanks!  Removing all the various certificates downloaded since the upgrade to 24.10 and re-running an update worked.

Code Select Expand

# rm /tmp/li
libfetch_crl.24101713  libfetch_crl.24102022  libfetch_crl.24102420
libfetch_crl.24101722  libfetch_crl.24102122  libfetch_crl.24102422
libfetch_crl.24101822  libfetch_crl.24102222  lighttpdcompress/
libfetch_crl.24101922  libfetch_crl.24102322
root@OPNsense:~ # rm /tmp/libfetch_crl.2410*

The tab complete shows roughly one a day since the upgrade on 17 Oct.

I am now at 24.10_7.

[franco]

Ok, good. I'm hopeful the issue will not be back with the hotfixes in place.


Cheers,
Franco

[Mbl]

I'm still having the same problem with 24.10_7:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.10_7 at Thu Oct 31 11:42:39 CET 2024
Fetching subscription information, please wait... done
Fetching changelog information, please wait... done
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 856 packages processed.
Updating SunnyValley repository catalogue...
No CRL was provided for /CN=zenarmor.com
No CRL was provided for /C=US/O=Google Trust Services/CN=WE1
No CRL was provided for /C=US/O=Google Trust Services LLC/CN=GTS Root R4
No CRL was provided for /CN=zenarmor.com
No CRL was provided for /C=US/O=Google Trust Services/CN=WE1
No CRL was provided for /C=US/O=Google Trust Services LLC/CN=GTS Root R4
Fetching meta.conf: . done
No CRL was provided for /CN=zenarmor.com
No CRL was provided for /C=US/O=Google Trust Services/CN=WE1
No CRL was provided for /C=US/O=Google Trust Services LLC/CN=GTS Root R4
No CRL was provided for /CN=zenarmor.com
No CRL was provided for /C=US/O=Google Trust Services/CN=WE1
No CRL was provided for /C=US/O=Google Trust Services LLC/CN=GTS Root R4
Fetching packagesite.pkg: ... done
Processing entries: ....... done
SunnyValley repository update completed. 66 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (13 candidates): .......... done
Processing candidates (13 candidates): .. done
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***

I have deleted all /tmp/libfetch_crl.* files and retried but still get the same error. Current file libfetch_crl.24103111 has valid crl information as it looks like:

Code Select Expand

# [i] fetch certificate for https://opnsense-update.deciso.com
# [i] fetch CRL from http://cdp.rapidssl.com/RapidSSLTLSECCCAG1.crl
# [i] fetch CRL from http://crl3.digicert.com/DigiCertGlobalRootG3.crl

Is the mentioned hotfix already included in 24.10_7 or will it be available in a later version?

[franco]

Neither the "same":

These CRL errors are about Zenarmor, not OPNsense repo.

Nor a "problem":

Third party repo CRL verification is not included in 24.10.x yet and you can still connect to Zenarmor just fine.


Cheers,
Franco

[TotalGriffLock]

Just upgraded some BE to 24.10_7 and am having this same issue, however I suspect there is a different root cause: my 2x BE devices are behind a proxy. They both have proxy environment variables set as per documentation here:

https://docs.opnsense.org/development/backend/configd.html#extending-the-environment

However this does not appear to be passing down the python requests library as it is clearly trying to connect directly.

Code Select Expand

root@x-y-z:~ # /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com
# [i] fetch certificate for https://opnsense-update.deciso.com
[!!] Chain fetch failed for https://opnsense-update.deciso.com (HTTPSConnectionPool(host='opnsense-update.deciso.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1ee0c0bfa110>: Failed to establish a new connection: [Errno 65] No route to host')))


And from the gui:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.10_7 at Thu Nov  7 20:41:56 GMT 2024
Fetching subscription information, please wait... Could not load CRL file /tmp/libfetch_crl.24110720
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/subscription: Authentication error
Fetching changelog information, please wait... Could not load CRL file /tmp/libfetch_crl.24110720
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/sets/changelog.txz: Authentication error
Updating OPNsense repository catalogue...
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/meta.txz: Authentication error
repository OPNsense has no meta file, using default settings
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/packagesite.pkg: Authentication error
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/packagesite.txz: Authentication error
Unable to update repository OPNsense
Error updating repositories!
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***

Edit: Seems python-requests wants the environment variables in lower case. Added those and now the cmdline script runs without errors. The updates web gui still doesn't, though.

[franco]

Thanks, looking into it now.


Cheers,
Franco

[franco]

After editing the script to run fine try to delete the old crl files:

# rm /tmp/libfetch_crl.*

And try the GUI again.


Cheers,
Franco

[TotalGriffLock]

No, trying again from the GUI just makes it recreate that file again. Each time I rm, the check from the gui brings it back.

However as suggested by another user in this thread, ticking System -> Trust -> Settings -> Autofetch CRLs (which shouldn't have an apostrophe in it :D ) and leaving it for a few hours has done the job. I suspect then from this that the fetch command that is run when you click 'Check for updates' is somehow running in some different environment/context which isn't inheriting the environment variables from configd, where the scheduled check (and pkg, when running 'Check for updates') is.

[franco]

CRL auto fetch is a workaround for an empty CRL file, but basically you are left without a valid CRL download which we need to debug (and I'm having a deja-vu here writing this).

What's the content of the libfetch_crl file in the error case?

HTTP(S)_PROXY should work in upper and lower case according to the Python code itself.


Cheers,
Franco

[TotalGriffLock]

Unfortunately in fixing it, I don't have that evidence from when it was broken anymore. I can't roll the environment back as it is semi-live... I'll see if I can recreate it in test.

[franco]

Going back to the old situation is relatively easy without interfering with operation:

# rm /usr/local/share/certs/ca-crl-upd-* /tmp/libfetch_crl.*

And check for updates again. The new libfetch_crl file should have enough diagnostics to hint at the issue.

Just making sure this is 24.10_7 from updates and not a fresh install 24.10?

Have you set both HTTP_PROXY and HTTPS_PROXY in the configd template? And if you add the lower case ones it starts working or still doesn't?


Cheers,
Franco