timeouts on updates (commercial version)

Started by mgambacorta, August 28, 2025, 10:39:01 AM

Previous topic - Next topic
Print
Go Down Pages1 2
User actions
August 28, 2025, 10:39:01 AM
Hi all,

I am trying to use a OPNsense 25.4.2 VM (host is Proxmox VE 9.0). This is hosted on a dedicated server on Hetzner.

The subscription seems ok:
Type   opnsense-business   
Version   25.4.2   
Architecture   amd64   
Commit   c9f8b1676   
Mirror   https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/25.4   
Repositories   OPNsense (Priority: 11)   
Updated on   Wed Aug 27 11:54:33 CEST 2025   
Checked on   Thu Aug 28 09:37:19 CEST 2025   
Licensed until   2026-08-30


on the backend logs I see these error messages:
2025-08-28T09:41:17   Error   configd.py    Timeout (120) executing : firmware remote
2025-08-28T09:39:15   Error   configd.py    Timeout (120) executing : firmware tiers
2025-08-28T09:21:59   Error   configd.py    Timeout (120) executing : firmware tiers
2025-08-28T09:05:24   Error   configd.py    [319efc73-7aaa-423a-af4d-6cccbf3b34a9] Script action stderr returned "b'No CRL was provided for /CN=opnsense-update.deciso.com\nNo CRL was provided for /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL TLS ECC CA G1\nNo CRL was provided for /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root G3\nNo CRL was provid'"
2025-08-28T09:03:39   Error   configd.py    Timeout (120) executing : firmware tiers
2025-08-28T08:53:04   Error   configd.py    [8ce505bd-a2f2-4c08-bfd5-675412ba3f82] Script action stderr returned "b'No CRL was provided for /CN=opnsense-update.deciso.com\nNo CRL was provided for /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL TLS ECC CA G1\nNo CRL was provided for /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root G3\nNo CRL was provid'"
2025-08-28T08:53:04   Error   configd.py    [16061049-c950-461d-b7d3-f5b18ac33083] Script action stderr returned "b'No CRL was provided for /CN=opnsense-update.deciso.com\nNo CRL was provided for /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL TLS ECC CA G1\nNo CRL was provided for /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root G3\npkg: https://opns'"
2025-08-28T08:51:07   Error   configd.py    Timeout (120) executing : firmware remote
2025-08-28T08:49:05   Error   configd.py    Timeout (120) executing : firmware tiers
2025-08-28T08:46:02   Error   configd.py    Timeout (120) executing : firmware remote
2025-08-28T08:44:00   Error   configd.py    Timeout (120) executing : firmware tiers
2025-08-28T08:40:48   Error   configd.py    Configd disconnected while executing : firmware tiers
2025-08-28T08:37:33   Error   configd.py    [35e7a19b-bbfe-462d-9135-83bfc304ee6f] Script action stderr returned "b'No CRL was provided for /CN=opnsense-update.deciso.com\nNo CRL was provided for /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL TLS ECC CA G1\nNo CRL was provided for /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root G3\nNo CRL was provid'"


I thought of an issue with connectivity, but the firewall if allowing a test linux machine to browse easily , it is fast and no pings are ever lost.

I have tried to log on to ssh on the firewall and tried :
fetch https://opnsense-update.deciso.com/<...>/FreeBSD:14:amd64/25.4/latest/packagesite.pkg

sometimes it responds fine others it does not.

I have asked the provider to test connectivity.

One of my options would be to reinstall the firewall but my hopes are not high this will solve the issue.

Has anyone run into a situation like this ?

Anyone knows what could be happening?
August 28, 2025, 11:25:20 AM #1
The timeouts happen because the process can't finish, likely because it eventually fails to connect or is otherwise stuck while trying to connect.

The CRL error shows it can't even download a CRL from the Internet which points in the same direction.

The firmware connectivity audit is the best tool to further diagnose this.


Cheers,
Franco
August 28, 2025, 12:37:50 PM #2
Franco, thanks for your answer.
I run the connectivity audit and this is the result (see below).

I saw a line that seems to be related to ipv6:
Checking connectivity for host: opnsense-update.deciso.com -> 2001:1af8:4f00:a005:5::

After that there are errors. I tried to disable ipv6 on all interfaces on firewall and on the host (Proxmox VE).

I checked the "Prefer to use IPv4 even if IPv6 is available".


Why is OPNsense still trying ipv6 ?





***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 25.4.2 (amd64) at Thu Aug 28 12:26:57 CEST 2025
Strict TLS 1.3 and CRL checking is enabled.
Checking connectivity for host: opnsense-update.deciso.com -> 89.149.211.205
PING 89.149.211.205 (89.149.211.205): 1500 data bytes
1508 bytes from 89.149.211.205: icmp_seq=0 ttl=54 time=27.798 ms
1508 bytes from 89.149.211.205: icmp_seq=1 ttl=54 time=27.732 ms
1508 bytes from 89.149.211.205: icmp_seq=2 ttl=54 time=27.843 ms
1508 bytes from 89.149.211.205: icmp_seq=3 ttl=54 time=27.835 ms

--- 89.149.211.205 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 27.732/27.802/27.843/0.044 ms
Checking connectivity for repository (IPv4): https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/25.4
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 908 packages processed.
All repositories are up to date.
Checking connectivity for host: opnsense-update.deciso.com -> 2001:1af8:4f00:a005:5::
ping: UDP connect: No route to host
Checking connectivity for repository (IPv6): https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/25.4
Updating OPNsense repository catalogue...
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/25.4/latest/meta.txz: No route to host
repository OPNsense has no meta file, using default settings
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/25.4/latest/packagesite.pkg: No route to host
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/25.4/latest/packagesite.txz: No route to host
Unable to update repository OPNsense
Error updating repositories!
Checking server certificate for host: opnsense-update.deciso.com
002001D6FF470000:error:8000003C:system library:BIO_connect:Operation timed out:/usr/src/crypto/openssl/crypto/bio/bio_sock2.c:125:calling connect()
002001D6FF470000:error:10000067:BIO routines:BIO_connect:connect error:/usr/src/crypto/openssl/crypto/bio/bio_sock2.c:127:
002001D6FF470000:error:80000041:system library:BIO_connect:No route to host:/usr/src/crypto/openssl/crypto/bio/bio_sock2.c:125:calling connect()
002001D6FF470000:error:10000067:BIO routines:BIO_connect:connect error:/usr/src/crypto/openssl/crypto/bio/bio_sock2.c:127:
connect:errno=65
***DONE***
August 28, 2025, 12:43:43 PM #3
I'm in doubt if IPv6 is the culprit but since it's not functional maybe it's still your issue:

Checking server certificate for host: opnsense-update.deciso.com
002001D6FF470000:error:8000003C:system library:BIO_connect:Operation timed out:/usr/src/crypto/openssl/crypto/bio/bio_sock2.c:125:calling connect()
002001D6FF470000:error:10000067:BIO routines:BIO_connect:connect error:/usr/src/crypto/openssl/crypto/bio/bio_sock2.c:127:
002001D6FF470000:error:80000041:system library:BIO_connect:No route to host:/usr/src/crypto/openssl/crypto/bio/bio_sock2.c:125:calling connect()
002001D6FF470000:error:10000067:BIO routines:BIO_connect:connect error:/usr/src/crypto/openssl/crypto/bio/bio_sock2.c:127:

Can you set Prefer IPv4 over IPv6 in system settings general? And/or disable IPv6 in the WAN interface as it makes more trouble than worth it.


Cheers,
Franco
August 28, 2025, 12:52:56 PM #4
Franco,

this behaviour is happening after I had already set Prefer IPv4 over IPv6 in system settings general

is there a way to unload the ipv6 stack on OPNsense?

Any other ideas ?
August 28, 2025, 01:04:58 PM #5
Disable IPv6 in WAN interface. You can also disallow IPv6 under Interfaces: Settings, but that has an effect on your LAN clients as well.

I'm still just assuming this is the issue, but I could be wrong.


Cheers,
Franco
August 28, 2025, 01:29:32 PM #6 Last Edit: August 28, 2025, 01:48:35 PM by mgambacorta
Franco,

thanks for keeping up with me.

ipv6 was already disabled on all interfaces.
I am trying now a reboot after disabling IPv6 in Interfaces -> Settings.
I cannot understand why I still see an ipv6 address on the WAN interface (after the reboot):

vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: WAN (wan)
   options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
   ether 00:50:56:00:45:37
   inet x.x.x.x netmask 0xfffffff0 broadcast x.x.x.x
   inet6 fe80::250:56ff:fe00:4537%vtnet0 prefixlen 64 scopeid 0x1
   media: Ethernet autoselect (10Gbase-T <full-duplex>)
   status: active
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

system is really slow when I ask to check for updates or install a plugin ... I tried to install the os acme client and it has been impossible.

I already downloaded 24.10 ISO and will try to start from scratch if I cannot solve this issue.

August 28, 2025, 01:40:57 PM #7
It all sounds like an issue with your DNS resolution in front of the OPNsense.

The audit shows you get the right IPv4 and IPv6 addresses and IPv4 works, but IPv6 doesn't. What the audit doesn't show is how long the working IPv4 takes, which could give hints you're running into a DNS resolution timeout.

The link local fe80:: IPv6 addresses are fine. They cannot connect to anything by itself.


Cheers,
Franco
August 28, 2025, 03:14:00 PM #8
I am using google 8.8.8.8

I can resolve names properly and fast when I try to ping using a FQDN from the OPNsense shell.

This is very strange ...
August 28, 2025, 03:22:06 PM #9
That's all fair but something is still off with regards to the final connection:

# echo | openssl s_client -no_ign_eof -brief opnsense-update.deciso.com:443
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: CN = opnsense-update.deciso.com
Hash used: SHA256
Signature type: ECDSA
Verification: OK
Server Temp Key: X25519, 253 bits
DONE

What does it return on your end?
August 28, 2025, 04:21:58 PM #10
Hi Franco,

I got this:
root@opn2:~ #  echo | openssl s_client -no_ign_eof -brief opnsense-update.deciso.com:443
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: CN = opnsense-update.deciso.com
Hash used: SHA256
Signature type: ECDSA
Verification: OK
Server Temp Key: X25519, 253 bits
DONE
August 28, 2025, 05:00:29 PM #11
Hmm, but the audit now? And check for updates?
August 28, 2025, 08:43:46 PM #12
Hi Franco,

thanks again for your help.

This is the result of Connectivity audit. After the ipv6 line things go wrong :-(

***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 25.4.2 (amd64) at Thu Aug 28 20:36:23 CEST 2025
Strict TLS 1.3 and CRL checking is enabled.
Checking connectivity for host: opnsense-update.deciso.com -> 89.149.211.205
PING 89.149.211.205 (89.149.211.205): 1500 data bytes
1508 bytes from 89.149.211.205: icmp_seq=0 ttl=54 time=27.747 ms
1508 bytes from 89.149.211.205: icmp_seq=1 ttl=54 time=27.752 ms
1508 bytes from 89.149.211.205: icmp_seq=2 ttl=54 time=27.737 ms
1508 bytes from 89.149.211.205: icmp_seq=3 ttl=54 time=27.745 ms

--- 89.149.211.205 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 27.737/27.745/27.752/0.005 ms
Checking connectivity for repository (IPv4): https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/25.4
Updating OPNsense repository catalogue...
No CRL was provided for /CN=opnsense-update.deciso.com
Fetching meta.conf: . done
No CRL was provided for /CN=opnsense-update.deciso.com
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 908 packages processed.
All repositories are up to date.
Checking connectivity for host: opnsense-update.deciso.com -> 2001:1af8:4f00:a005:5::
ping: UDP connect: No route to host
Checking connectivity for repository (IPv6): https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/25.4
Updating OPNsense repository catalogue...
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/25.4/latest/meta.txz: Non-recoverable resolver failure
repository OPNsense has no meta file, using default settings
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/25.4/latest/packagesite.pkg: Non-recoverable resolver failure
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/25.4/latest/packagesite.txz: Non-recoverable resolver failure
Unable to update repository OPNsense
Error updating repositories!
Checking server certificate for host: opnsense-update.deciso.com
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G3
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = RapidSSL TLS ECC CA G1
verify return:1
depth=0 CN = opnsense-update.deciso.com
verify return:1
DONE
***DONE***
August 28, 2025, 09:04:02 PM #13
Ok, but "Checking server certificate for host: opnsense-update.deciso.com" now works when before it didn't. Since we disabled IPv6 that's expected to not work.

Does checking for updates work now?


Cheers,
Franco
August 28, 2025, 11:31:25 PM #14
Quote from: franco on August 28, 2025, 09:04:02 PMOk, but "Checking server certificate for host: opnsense-update.deciso.com" now works when before it didn't. Since we disabled IPv6 that's expected to not work.

Does checking for updates work now?


Cheers,
Franco

Checking for available updates takes an unusual amount of time. Same thing happens for the firmware status tab refreshes.
Checking updates might take 30 minutes... you give up.
If I then go to System - Log files - Backend:
2025-08-28T21:40:17   Error   configd.py    [66445729-0920-4065-a177-0d09fba179c6] Script action stderr returned "b"[!!] Chain fetch failed for https://opnsense-update.deciso.com (HTTPSConnectionPool(host='opnsense-update.deciso.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2d192b61cc90>""
2025-08-28T21:40:17   Error   configd.py    [ede0413f-a7da-46d3-a760-4f9056df1595] Script action stderr returned "b"[!!] CRL fetch failed for http://crl3.digicert.com/DigiCertGlobalRootG3.crl (HTTPConnectionPool(host='crl3.digicert.com', port=80): Max retries exceeded with url: /DigiCertGlobalRootG3.crl (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection ""
2025-08-28T21:40:15   Error   configd.py    Timeout (120) executing : firmware tiers
2025-08-28T21:01:50   Error   configd.py    [2b5bf0fa-3ed9-4c01-9668-df5a8888d8fd] Script action stderr returned "b'No CRL was provided for /CN=opnsense-update.deciso.com\nNo CRL was provided for /CN=opnsense-update.deciso.com'"
2025-08-28T21:01:31   Error   configd.py    Timeout (120) executing : firmware tiers
2025-08-28T20:59:19   Error   configd.py    [8f1c7ddf-117d-49d2-9f83-55a5a301d727] Script action stderr returned "b'No CRL was provided for /CN=opnsense-update.deciso.com\nNo CRL was provided for /CN=opnsense-update.deciso.com'"
2025-08-28T20:58:18   Error   configd.py    Timeout (120) executing : firmware tiers
2025-08-28T20:43:49   Error   configd.py    [0d6e618d-db60-44ec-a93f-b71aac6eb06c] Script action stderr returned "b'No CRL was provided for /CN=opnsense-update.deciso.com\nNo CRL was provided for /CN=opnsense-update.deciso.com\nNo CRL was provided for /CN=opnsense-update.deciso.com\nNo CRL was provided for /CN=opnsense-update.deciso.com'"
2025-08-28T20:41:47   Error   configd.py    Timeout (120) executing : firmware remote
2025-08-28T20:39:44   Error   configd.py    Timeout (120) executing : firmware tiers
2025-08-28T20:37:03   Error   configd.py    [fef2691c-ba97-4c57-9f30-d5a6f64bb55a] Script action stderr returned "b"[!!] CRL fetch failed for http://cdp.rapidssl.com/RapidSSLTLSECCCAG1.crl (HTTPConnectionPool(host='cdp.rapidssl.com', port=80): Max retries exceeded with url: /RapidSSLTLSECCCAG1.crl (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object""
2025-08-28T20:32:18   Error   configd.py    Timeout (120) executing : firmware remote
2025-08-28T20:30:17   Error   configd.py    Timeout (120) executing : firmware tiers

I do not know what is happening.
Print
Go Up Pages1 2
User actions