24.10 BE upgrade CRL errors

[Steve]

Just upgraded to 24.10 BE on my test device.  Upgrade itself went fine, but when I go to check for updates, I'm getting a CRL error:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.10 at Thu Oct 17 10:50:46 EDT 2024
Fetching subscription information, please wait... Could not load CRL file /tmp/libfetch_crl.24101710
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/subscription: Authentication error
Fetching changelog information, please wait... Could not load CRL file /tmp/libfetch_crl.24101710
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/sets/changelog.txz: Authentication error
Updating OPNsense repository catalogue...
Could not load CRL file /tmp/libfetch_crl.24101710
Could not load CRL file /tmp/libfetch_crl.24101710
Could not load CRL file /tmp/libfetch_crl.24101710
Could not load CRL file /tmp/libfetch_crl.24101710
Could not load CRL file /tmp/libfetch_crl.24101710
Could not load CRL file /tmp/libfetch_crl.24101710
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/meta.txz: Authentication error
repository OPNsense has no meta file, using default settings
Could not load CRL file /tmp/libfetch_crl.24101710
Could not load CRL file /tmp/libfetch_crl.24101710
Could not load CRL file /tmp/libfetch_crl.24101710
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/packagesite.pkg: Authentication error
Could not load CRL file /tmp/libfetch_crl.24101710
Could not load CRL file /tmp/libfetch_crl.24101710
Could not load CRL file /tmp/libfetch_crl.24101710
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/packagesite.txz: Authentication error
Unable to update repository OPNsense
Error updating repositories!
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***
This is what's in /tmp/libfetch_crl.24101710:

Code Select Expand

# [i] fetch certificate for https://opnsense-update.deciso.com

Figured a new file would be updated on the hour and it was.  libfetch_crl.24101711 also only contains that single line.

Thanks.    -Steve

[DEC670airp414user]

I received an error before it rebooted    It rebooted rather quicker than expected
But says it's updated and shows the posted above

[franco]

Can you fetch http://x1.c.lencr.org/ from the box? It looks like you cannot.


Cheers,
Franco

[Steve]

It looks like I can, but strangely it doesn't contain any revoked certs:

Code Select Expand

root@testrange:~ # curl http://x1.c.lencr.org/ --output /tmp/test.crl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   717  100   717    0     0   9425      0 --:--:-- --:--:-- --:--:--  9434
root@testrange:~ # openssl crl -in /tmp/test.crl -inform DER -text -noout
Certificate Revocation List (CRL):
        Version 2 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Last Update: Feb  5 00:00:00 2024 GMT
        Next Update: Jan  4 23:59:59 2025 GMT
        CRL extensions:
            X509v3 Authority Key Identifier:
                79:B4:59:E6:7B:B6:E5:E4:01:73:80:08:88:C8:1A:58:F6:E9:9B:6E
            X509v3 CRL Number:
                104
No Revoked Certificates.
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        59:26:d6:a5:01:52:f5:e1:20:f8:e7:5d:6d:28:5c:a1:6f:39:
        1e:ee:92:a8:4d:07:f9:a4:65:af:37:db:f8:a9:4f:df:a1:b4:
        96:e5:61:1a:84:3c:03:66:0d:4f:6c:33:1c:97:b1:e5:33:9e:
        4a:d9:1e:88:c7:42:8e:fd:36:21:24:e6:a0:87:b0:d2:c4:34:
        41:7c:d3:68:9c:50:f2:a5:a6:09:8c:8b:c2:62:63:dc:26:a4:
        12:ae:c3:81:65:c0:44:2a:35:01:49:b2:cd:59:6a:e7:5d:f1:
        1f:63:84:aa:a1:53:3a:5f:7f:f3:9a:ed:42:4b:64:21:52:fa:
        9d:e9:b9:af:bb:c7:5c:e4:78:3c:47:f3:be:16:78:c4:23:63:
        c1:6a:e9:8e:65:31:9b:00:24:0f:91:20:98:1f:47:55:ca:ab:
        6a:72:ad:ac:b9:c0:f9:3c:4f:1b:46:58:d8:50:8f:e7:13:7b:
        ff:fb:5f:8b:c1:ba:01:97:37:77:34:20:a8:d5:4d:b0:9c:f2:
        8f:6d:22:b2:dd:5f:05:b6:2c:de:99:a2:b6:ea:ef:59:64:d5:
        c1:b0:7f:80:45:cc:68:87:7c:63:eb:63:07:f1:49:1a:8f:38:
        e4:05:2d:7c:e0:42:98:ae:07:07:8b:f7:9c:3b:a9:09:70:bf:
        8f:52:d3:30:ea:df:42:67:88:6b:d2:de:ab:3d:28:a4:7a:d7:
        7d:bb:82:6f:6a:10:96:01:4a:3f:81:d7:e1:e3:5a:91:58:9e:
        2d:f2:f9:5e:58:cf:ce:63:a3:bd:46:8f:0c:97:6c:4f:97:d5:
        48:de:9c:cb:57:c3:9a:c6:a2:92:78:e6:05:3d:d5:4e:14:d9:
        f8:f4:09:9e:d2:fe:13:38:5b:e9:af:0a:ec:92:e7:bf:ee:5a:
        33:48:ee:31:82:d7:6f:0b:cd:ec:aa:db:66:9f:d8:a1:63:20:
        57:7b:76:aa:d0:d6:a5:1e:c9:44:45:dd:3c:18:bd:6f:05:b8:
        19:58:a0:e9:c5:8a:58:70:3b:e4:22:bf:0d:c8:a3:e0:53:a6:
        7f:2b:a6:39:14:ad:2b:0d:b7:4a:46:d2:78:21:67:6b:25:33:
        23:d6:ab:17:80:bb:66:22:ec:ee:6d:b1:e5:01:ae:4e:5b:5c:
        3b:35:54:3e:5a:94:51:f5:81:eb:cb:10:ca:d6:39:7e:17:ae:
        f0:4d:25:81:64:cd:b6:06:09:ea:75:eb:0e:06:e5:a4:c0:1e:
        0e:24:9f:33:bf:fd:1f:12:48:57:60:e1:a4:e8:aa:b2:30:e9:
        ec:e0:52:76:44:4e:bd:42:69:69:b5:de:51:ef:84:a4:16:19:
        49:a2:2b:d2:3d:62:b4:6e
root@testrange:~ #

[franco]

Empty CRL is ok but it has to be there. Can you run this manually?

# /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com

Just tested again from here and I get it loaded into the CRL file so that it proceeds to talk to the update server.

We will likely switch the chain away from LE, but that won't address the issue of not being able to download CRLs. This is a certification requirement so we can't just disable it when it fails. But the OpenSSL situation itself is sticky. My favourite mail exchange on the subject:

https://openssl-users.openssl.narkive.com/qKrxQx5U/certificate-crls-x509-v-err-unable-to-get-crl


Thanks,
Franco

[Steve]

Code Select Expand

root@testrange:~ # /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com
# [i] fetch certificate for https://opnsense-update.deciso.com
[!!] Chain fetch failed for https://opnsense-update.deciso.com


[Steve]

I had checked the box for Auto fetch CRLs in System:Trust:Settings, and the 1pm update of /tmp/libfetch_crl.24101713 has CRLs in the file now:

Code Select Expand

root@testrange:/tmp # cat libfetch_crl.24101713
-----BEGIN X509 CRL-----
MIIBVDCB3AIBATAKBggqhkjOPQQDAzBIMQswCQYDVQQGEwJERTEVMBMGA1UEChMM
RC1UcnVzdCBHbWJIMSIwIAYDVQQDExlELVRSVVNUIEVWIFJvb3QgQ0EgMSAyMDIw
Fw0yNDAxMTUxNDAwNDlaFw0yNTAxMTYxMzU5NDlaMDEwLwIQaOXtSmaheM68bssu
...
sptVq1l7Np3JGm7LV6UkiPfEHWbtz3BXrGWe8TYJacs1ekmk06yHivnEFZHUVNAf
CzjrFOBpb2V0giVc4FT1pFyyAsHYJ8CEACufMddpJcnmqsm+u5v5s5ECvwcIVcrV
iyfpHYL9lVi4VI7b2k++Adl0fJXojykktFeycNFYCYP/fw==
-----END X509 CRL-----
# [i] fetch certificate for https://opnsense-update.deciso.com


Check for Updates appears to be working now:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.10 at Thu Oct 17 13:16:14 EDT 2024
Fetching subscription information, please wait... No CRL was provided for /CN=opnsense-update.deciso.com
No CRL was provided for /C=US/O=Let's Encrypt/CN=R11
No CRL was provided for /C=US/O=Internet Security Research Group/CN=ISRG Root X1
done
Fetching changelog information, please wait... No CRL was provided for /CN=opnsense-update.deciso.com
No CRL was provided for /C=US/O=Let's Encrypt/CN=R11
No CRL was provided for /C=US/O=Internet Security Research Group/CN=ISRG Root X1
No CRL was provided for /CN=opnsense-update.deciso.com
No CRL was provided for /C=US/O=Let's Encrypt/CN=R11
No CRL was provided for /C=US/O=Internet Security Research Group/CN=ISRG Root X1
done
Updating OPNsense repository catalogue...
No CRL was provided for /CN=opnsense-update.deciso.com
No CRL was provided for /C=US/O=Let's Encrypt/CN=R11
No CRL was provided for /C=US/O=Internet Security Research Group/CN=ISRG Root X1
Fetching meta.conf: . done
No CRL was provided for /CN=opnsense-update.deciso.com
No CRL was provided for /C=US/O=Let's Encrypt/CN=R11
No CRL was provided for /C=US/O=Internet Security Research Group/CN=ISRG Root X1
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 856 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (0 candidates): . done
Processing candidates (0 candidates): . done
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***
Still seeing Chain fetch failed from the python script, but it's now printing the CRLs as well:

Code Select Expand

root@testrange:/tmp # /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com
# [i] fetch certificate for https://opnsense-update.deciso.com
[!!] Chain fetch failed for https://opnsense-update.deciso.com
-----BEGIN X509 CRL-----
MIIBVDCB3AIBATAKBggqhkjOPQQDAzBIMQswCQYDVQQGEwJERTEVMBMGA1UEChMM
RC1UcnVzdCBHbWJIMSIwIAYDVQQDExlELVRSVVNUIEVWIFJvb3QgQ0EgMSAyMDIw
Fw0yNDAxMTUxNDAwNDlaFw0yNTAxMTYxMzU5NDlaMDEwLwIQaOXtSmaheM68bssu
...


[franco]

This is a "workaround" of sorts. The problem is OpenSSL fails hard if the CRL file is empty, but it gladly accepts multiple ones even if unrelated. It does insist on one, even though that's technically wrong as it would later complain about the missing CRL anyway (the latter is ok, the former fail is more dangerous when you don't know where the user is going to go). I don't think any of this OpenSSL-specific behaviour can be made much better than it is (or nobody will be willing to accept it for "security" reasons).

For now we need to know what the issue during your fetch is:

# opnsense-patch https://github.com/opnsense/core/commit/372c9c98
# /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com


Thanks,
Franco

[Steve]

Code Select Expand

root@testrange:~ # /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com
# [i] fetch certificate for https://opnsense-update.deciso.com
[!!] Chain fetch failed for https://opnsense-update.deciso.com (HTTPSConnectionPool(host='opnsense-update.deciso.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x2cedeac42d10>, 'Connection to opnsense-update.deciso.com timed out. (connect timeout=0.1)')))
-----BEGIN X509 CRL-----
MIIBVDCB3AIBATAKBggqhkjOPQQDAzBIMQswCQYDVQQGEwJERTEVMBMGA1UEChMM
RC1UcnVzdCBHbWJIMSIwIAYDVQQDExlELVRSVVNUIEVWIFJvb3QgQ0EgMSAyMDIw
...


[franco]

Ok, timeout=0.1 looks like an obvious problem. Another patch coming up in a minute. Thanks!

[franco]

# opnsense-patch https://github.com/opnsense/core/commit/70df0a15f7e
# /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com


Cheers,
Franco

[Steve]

Code Select Expand

root@testrange:/tmp # /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com
# [i] fetch certificate for https://opnsense-update.deciso.com
# [i] fetch CRL from http://x1.c.lencr.org/
-----BEGIN X509 CRL-----
MIIBVDCB3AIBATAKBggqhkjOPQQDAzBIMQswCQYDVQQGEwJERTEVMBMGA1UEChMM
RC1UcnVzdCBHbWJIMSIwIAYDVQQDExlELVRSVVNUIEVWIFJvb3QgQ0EgMSAyMDIw
...

I see this additional CRL added to libfetch_crl.24101714:

Code Select Expand

Certificate Revocation List (CRL):
        Version 2 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Last Update: Feb  5 00:00:00 2024 GMT
        Next Update: Jan  4 23:59:59 2025 GMT
        CRL extensions:
            X509v3 Authority Key Identifier:
                79:B4:59:E6:7B:B6:E5:E4:01:73:80:08:88:C8:1A:58:F6:E9:9B:6E
            X509v3 CRL Number:
                104
No Revoked Certificates.
Check for updates is now only showing one No CRL:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.10 at Thu Oct 17 14:15:45 EDT 2024
Fetching subscription information, please wait... No CRL was provided for /CN=opnsense-update.deciso.com
done
Fetching changelog information, please wait... No CRL was provided for /CN=opnsense-update.deciso.com
No CRL was provided for /CN=opnsense-update.deciso.com
done
Updating OPNsense repository catalogue...
No CRL was provided for /CN=opnsense-update.deciso.com
Fetching meta.conf: . done
No CRL was provided for /CN=opnsense-update.deciso.com
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 856 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (0 candidates): . done
Processing candidates (0 candidates): . done
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***

Thanks!    -Steve

[yourfriendarmando]

Hi all

Where are we with this?

I just upgraded both, and this only happens on BE, the most up to date CE appears to be doing just fine.

It's just in my VMs. I'm glad I tried the upgrade, which both succeeded fine. I'll hang back and re-upgrade my 24.4 VM at a later time, try again and make sure it's working fine before I upgrade my main system, as well as of my subscribing clients.

Thank you

[franco]

@Steve

Thanks for your help. Looks good now on your end. The other CRL related message will go away when we switch to a better chain. LE allegedly thinks their CRL handling is complete but OpenSSL certainly doesn't agree.

@yourfriendarmando

Where we are right now is here:

# opnsense-patch 372c9c98 70df0a15f7e
# /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com

Will hotfix tomorrow morning.


Cheers,
Franco

[DEC670airp414user]

I moved back to community version yesterday
When I woke up this early morning I moved back to BE

all is working now