23.1.4:1 Update to 23.1.5 not working - Repositories broken --- sort of fixed

[MajStealth]

i have a strange problem to update after having 23.1.4:1 for ~2 weeks

i were able to update fine and quick before but am blocked to go any further.

i tried to find and resolve the error but am at a wall now, dns is working, there is no other firewall that could block it...

i send the request to update, sometimes get the changelog "fast" ~20sec, sometimes after 15min, sometimes hours later, but thats it.

now i got a new error in the log:

Code Select Expand


2023-04-04T22:30:18 Notice configd.py unable to sendback response [Judy|||1.0.5_3|||General purpose dynamic array|||1.19MiB
|||0|||0|||LGPL21|||OPNsense|||devel/judy acme.sh|||3.0.5|||ACME protocol client written in shell|||1.12MiB
|||0|||0|||GPLv3+|||OPNsense|||security/acme.sh addrwatch|||1.0.2|||Supports IP/Ethernet pairing for IPv4 and IPv6|||81.5KiB
|||0|||0|||GPLv3|||OPNsense|||net/addrwatch ap24-mod_security|||2.9.6|||Intrusion detection and prevention engine|||1.91MiB
|||0|||0|||APACHE20|||OPNsense|||www/mod_security apache24|||2.4.56|||Version 2.4.x of Apache web server|||5.59MiB
|||0|||0|||APACHE20|||OPNsense|||www/apache24 apcupsd|||3.14.14_4|||Set of programs for controlling APC UPS|||588KiB
|||0|||0|||GPLv2|||OPNsense|||sysutils/apcupsd apr|||1.7.0.1.6.1_2|||Apache Portability Library|||2.37MiB
|||0|||0|||APACHE20|||OPNsense|||devel/apr1 arc|||5.21p|||Create & extract files from DOS .ARC files|||87.5KiB
|||0|||0|||GPLv2|||OPNsense|||archivers/arc argp-standalone|||1.5.0|||Standalone version of arguments parsing functions from GLIBC|||119KiB
|||0|||0|||LGPL21+|||OPNsense|||devel/argp-standalone arj|||3.10.22_9|||Open source implementation of the ARJ archiver|||460KiB
|||0|||0|||GPLv2|||OPNsense|||archivers/arj asterisk16|||16.30.0|||Open Source PBX and telephony toolkit|||40.0MiB
|||0|||0|||GPLv2|||OPNsense|||net/asterisk16 augeas|||1.12.0_3|||Configuration editing tool|||3.34MiB
|||0|||0|||LGPL21|||OPNsense|||textproc/augeas autoconf|||2.71|||Generate configure scripts and related files|||3.12MiB
|||0|||0|||GPLv3+|||OPNsense|||devel/autoconf autoconf|||2.71|||Generate configure scripts and related files|||3.12MiB
|||0|||0|||GPLv2+|||OPNsense|||devel/autoconf autoconf|||2.71|||Generate configure scripts and related files|||3.12MiB
|||0|||0|||GFDL|||OPNsense|||devel/autoconf autoconf|||2.71|||Generate configure scripts and related files|||3.12MiB
|||0|||0|||EXCEPTION|||OPNsense|||devel/autoconf autoconf-switch|||20220527|||Wrapper script to switch between autoconf versions|||524B
|||0|||0|||BSD2CLAUSE|||OPNsense|||devel/autoconf-switch automake|||1.16.5|||GNU Standards-compliant Makefile generator|||2.03MiB
|||0|||0|||GPLv2+|||OPNsense|||devel/automake automake|||1.16.5|||GNU Standards-compliant Makefile generator|||2.03MiB
|||0|||0|||GFDL|||OPNsense|||devel/automake autossh|||1.4g|||Automatically restart SSH sessions and tunnels|||32.5KiB
|||0|||0|||BSD3CLAUSE|||OPNsense|||security/autossh avahi-app|||0.8_1|||Service discovery on a local network|||1.60MiB
|||0|||0|||LGPL21+|||OPNsense|||net/avahi-app awscli|||1.20.61|||Universal Command Line Interface for Amazon Web Services|||9.47MiB
|||0|||0|||APACHE20|||OPNsense|||devel/awscli azure-agent|||2.8.0.11|||Microsoft Azure Linux Agent|||3.14MiB
|||0|||0|||APACHE20|||OPNsense|||sysutils/azure-agent bandwidthd|||2.0.1_12|||Tracks bandwidth usage by IP address|||62.1KiB
|||0|||0|||GPLv3+|||OPNsense|||net-mgmt/bandwidthd bash|||5.2.15|||GNU Project's Bourne Again SHell|||2.19MiB
|||0|||0|||GPLv3+|||OPNsense|||shells/bash beats7|||7.17.9_3|||Send logs, network, metrics and heartbeat to elasticsearch or logstash|||155MiB
|||0|||0|||APACHE20|||OPNsense|||sysutils/beats7 beep|||1.0_1|||Beeps a certain duration and pitch out of the PC Speaker|||9.76KiB
|||0|||0|||BSD4CLAUSE|||OPNsense|||audio/beep bind-tools|||9.18.13|||Command line tools from BIND: delv, dig, host, nslookup...|||9.58MiB
|||0|||0|||MPL20|||OPNsense|||dns/bind-tools bind918|||9.18.13|||BIND DNS suite with updated DNSSEC and DNS64|||10.9MiB
|||0|||0|||MPL20|||OPNsense|||dns/bind918 bird2|||2.0.12|||Dynamic IP routing daemon|||1.05MiB
|||0|||0|||GPLv2|||OPNsense|||net/bird2 bison|||3.8.2,1|||Parser generator from FSF, (mostly) compatible with Yacc|||2.03MiB
|||0|||0|||GPLv3+|||OPNsense|||devel/bison boehm-gc|||8.2.2|||Garbage collection and memory leak detection for C and C++|||776KiB
|||0|||0|||BDWGC|||OPNsense|||devel/boehm-gc boehm-gc-threaded|||8.2.2|||Garbage collection and memory leak detection for C and C++|||661KiB
|||0|||0|||BDWGC|||OPNsense|||devel/boehm-gc-threaded boost-
2023-04-04T22:23:28 Notice configd.py [dcd5f4df-fbe9-4500-bbf6-1a4bc53f682d] Retrieve upgrade progress status
2023-04-04T22:23:27 Notice configd.py [5f730416-4c30-4a82-99f1-e0ce12c94da8] Retrieve firmware product info
2023-04-04T22:23:27 Notice configd.py [a1986260-42c1-4de6-9463-bad18a107dfe] Retrieve changelog index
2023-04-04T22:23:27 Notice configd.py [36c691f2-f7c5-4b3e-a358-7ad082ca903f] view local packages
2023-04-04T22:23:27 Error configd.py Timeout (120) executing : firmware remote
2023-04-04T22:21:26 Notice configd.py [7940535d-6ce8-4665-ad72-38c5f62fc42a] view remote packages

the timeout (120) does come more than a dozend times

when it is "finished"

Code Select Expand


***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 23.1.4_1 at Tue Apr  4 20:18:33 CEST 2023
Fetching changelog information, please wait... done
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 817 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (73 candidates): .......... done
Processing candidates (73 candidates): .. done
The following 13 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
ca_root_nss: 3.88.1 -> 3.89
dnsmasq: 2.89,1 -> 2.89_1,1
glib: 2.74.6,2 -> 2.76.1,2
libcbor: 0.10.1 -> 0.10.2
libcjson: 1.7.15 -> 1.7.15_1
libfido2: 1.12.0 -> 1.13.0
libmspack: 0.10.1 -> 0.11alpha
libnghttp2: 1.51.0_1 -> 1.52.0
openssl: 1.1.1t,1 -> 1.1.1t_1,1
opnsense: 23.1.4_1 -> 23.1.5_4
opnsense-update: 23.1.2 -> 23.1.5
py39-ujson: 5.0.0 -> 5.7.0
radvd: 2.19_1 -> 2.19_2

Number of packages to be upgraded: 13

11 MiB to be downloaded.
***DONE***

but still on 23.1.4 and also health audit seems to be fine

I changed to another internal and then external DNS, DNS tests always worked, for usual stuff and the repository domain. other repositories were also tested.

this is ONE update-try - around 2.5k lines of the "same" infos

Code Select Expand


2023-04-05T08:44:16 Notice configd.py [0926b134-bcb3-40e1-a744-b61fb9e9e1b1] Retrieve upgrade progress status
2023-04-05T08:44:16 Notice configd.py [912e84b6-8069-4ff6-ad1c-e73f1b066cbf] Retrieve firmware product info
2023-04-05T08:44:16 Notice configd.py [bf2e343b-ee33-42d4-8333-297c20130481] Retrieve changelog index
2023-04-05T08:44:16 Notice configd.py [b7cf7006-8e0b-41ad-b5ae-c57c9fe7dd5a] view local packages
2023-04-05T08:44:16 Notice configd.py [d13d95e0-5368-43b1-a94d-bb3b6b669210] view remote packages
2023-04-05T08:44:16 Notice configd.py [388de048-86c5-4d72-ab8c-1011d785dbcf] Retrieve upgrade progress status
...
2023-04-05T08:21:01 Notice configd.py [e3a10092-058b-48fc-b84c-15b37012b97d] Retrieve upgrade progress status
2023-04-05T08:21:01 Notice configd.py [947cf331-2a45-4101-8fb6-ebc44c4bfa53] Retrieve firmware product info
2023-04-05T08:21:01 Notice configd.py [4b2c781c-243a-417c-b98a-9481147ac0a6] Retrieve changelog index
2023-04-05T08:21:01 Notice configd.py [358252d2-fb57-4655-9472-e2e7e319b4a5] view local packages
2023-04-05T08:21:01 Notice configd.py [d5b94475-f598-49f1-ae47-803e1737cd64] Retrieve upgrade progress status
...
2023-04-05T08:19:46 Notice configd.py [884b2ed6-79e8-4200-91fe-2208e9df36a4] Retrieve upgrade progress status
2023-04-05T08:19:46 Notice configd.py [4a6857da-afb9-4022-ba28-a11f005bace7] system status
2023-04-05T08:19:46 Informational configd.py message 2ee35bbf-d5ac-4665-85b5-0fc77d83c6dd [firmware.audit] returned OK
2023-04-05T08:19:46 Notice configd.py [2ee35bbf-d5ac-4665-85b5-0fc77d83c6dd] Retrieve vulnerability report
2023-04-05T08:19:46 Notice configd.py [0572ae8f-a6ad-4cf5-9411-2ec91e06f6d2] retrieve firmware execution status
2023-04-05T08:19:46 Notice configd.py [b33d5370-92eb-410e-8946-c92db88d0abd] view remote packages
...
2023-04-05T08:11:47 Notice configd.py [8a524bba-c9e8-441e-a109-9bbea480a20d] Retrieve upgrade progress status
2023-04-05T08:11:46 Notice configd.py [6f860a25-3e61-420b-8c78-748a255d8d50] Retrieve firmware product info
2023-04-05T08:11:46 Notice configd.py [dbe1933e-636d-45ae-855b-aaf78fb22830] Retrieve changelog index
2023-04-05T08:11:46 Notice configd.py [bbbb3055-5f53-4078-adac-80f0f03e3754] view local packages
2023-04-05T08:11:46 Error configd.py Timeout (120) executing : firmware remote
2023-04-05T08:11:46 Notice configd.py [b6ea7aa7-b661-498a-8f7f-38c825bbc5ca] Retrieve upgrade progress status
...
2023-04-05T08:09:45 Notice configd.py [0d49ecc6-1b57-4170-b48b-1693d9cf3860] Retrieve upgrade progress status
2023-04-05T08:09:45 Notice configd.py [0c88853e-e8d8-4e9f-b854-21fba4119904] system status
2023-04-05T08:09:45 Informational configd.py message 831f239c-37e3-44a3-9ad8-82ab45c83b4b [firmware.audit] returned OK
2023-04-05T08:09:45 Notice configd.py [831f239c-37e3-44a3-9ad8-82ab45c83b4b] Retrieve vulnerability report
2023-04-05T08:09:45 Notice configd.py [584344fd-d2b2-4d69-a706-41eb0031d69e] view remote packages
2023-04-05T08:09:45 Notice configd.py [0c71a470-356e-4932-8f58-4e778bbe87de] retrieve firmware execution status
2023-04-05T08:09:43 Notice configd.py [c2e21f5d-31ac-4a8b-bd8b-b7ebee16f1b0] Retrieve firmware product info
2023-04-05T08:09:43 Notice configd.py [39ceaacd-b574-4b37-abb0-a62bc845dfd3] request traffic stats
2023-04-05T08:09:42 Notice configd.py [d0b6c47f-2adc-484d-8410-daaf10a9b308] list gateway status
2023-04-05T08:09:38 Notice configd.py [37c37169-b1df-4b7c-918a-c46927486837] system status
2023-04-05T08:09:38 Notice configd.py [a7817e17-4a0d-44e3-a5ae-3878a9ff694c] request traffic stats
2023-04-05T08:09:37 Notice configd.py [3d9b3eac-028a-4c92-a71a-aa82e458bc41] request traffic stats
2023-04-05T08:09:37 Notice configd.py [15eb39c8-d6fd-48f7-933b-d4e1a27cebb9] Retrieve firmware product info
2023-04-05T08:09:37 Notice configd.py [3eabaf8f-d86d-4d0e-b9c4-1136a9d2aba1] list gateway status
2023-04-05T08:09:36 Notice configd.py [0e3b3368-ce4c-492f-9a2e-ae8ba4b725a4] Query OpenVPN status (client,server)
2023-04-05T08:01:00 Informational configd.py message 6b4d79c2-15b7-44ea-b1e8-e76135485a3b [syslog.archive] returned OK
2023-04-05T08:01:00 Notice configd.py [6b4d79c2-15b7-44ea-b1e8-e76135485a3b] Archive syslog files


this looks interesting:

Code Select Expand

***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 23.1.4_1 at Wed Apr  5 09:00:15 CEST 2023
Checking connectivity for host: pkg.opnsense.org -> 89.149.211.205
PING 89.149.211.205 (89.149.211.205): 1500 data bytes
1508 bytes from 89.149.211.205: icmp_seq=0 ttl=54 time=22.223 ms
1508 bytes from 89.149.211.205: icmp_seq=1 ttl=54 time=20.832 ms
1508 bytes from 89.149.211.205: icmp_seq=2 ttl=54 time=21.040 ms
1508 bytes from 89.149.211.205: icmp_seq=3 ttl=54 time=20.781 ms

--- 89.149.211.205 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 20.781/21.219/22.223/0.588 ms
Checking connectivity for repository (IPv4): https://pkg.opnsense.org/FreeBSD:13:amd64/23.1
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 817 packages processed.
All repositories are up to date.
Checking connectivity for host: pkg.opnsense.org -> 2001:1af8:4f00:a005:5::
ping: UDP connect: No route to host
Checking connectivity for repository (IPv6): https://pkg.opnsense.org/FreeBSD:13:amd64/23.1
Updating OPNsense repository catalogue...
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/23.1/latest/meta.txz: Non-recoverable resolver failure
repository OPNsense has no meta file, using default settings
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/23.1/latest/packagesite.pkg: Non-recoverable resolver failure
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/23.1/latest/packagesite.txz: Non-recoverable resolver failure
Unable to update repository OPNsense
Error updating repositories!
***DONE***


IPv6 is router-side disabled in our network, so there should be no way for opnsense to get anything IPv6 from and to the outside

----------------------------------------------------------------------------------------------------

"FIX"

Use the 1 DE leaseweb mirror, it ran 04/05.04.2023 - there might be more that work

"/FIX"

[dd2594opn]

Getting a like error from my system ... what has happened?

[j_s]

I'm having similar issues at both friends' Opnsense systems I manage.  I manage to get mine to update, but it took like an hour.

One friend updated and rebooted as though everything was normal; typical 10-15 minutes for installation and reboot.

The other friend I cannot update despite trying multiple times from the WebGUI and the CLI.  Here's the CLI output from tonight.

Code Select Expand


  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option: 12

Fetching change log information, please wait... done

This will automatically fetch all available updates and apply them.

load: 0.28  cmd: sh 70400 [wait] 358.96r 0.00u 0.00s 0% 2980k
mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 kern_wait6+0x527 sys_wait4+0x7d amd64_syscall+0x10c fast_syscall_common+0xf8
load: 0.26  cmd: sh 70400 [wait] 361.24r 0.00u 0.00s 0% 2980k
mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 kern_wait6+0x527 sys_wait4+0x7d amd64_syscall+0x10c fast_syscall_common+0xf8
load: 0.36  cmd: sh 70400 [wait] 1318.61r 0.00u 0.00s 0% 2980k
mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 kern_wait6+0x527 sys_wait4+0x7d amd64_syscall+0x10c fast_syscall_common+0xf8
This update requires a reboot.

Proceed with this action? [y/N]: y


Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Updating OPNsense repository catalogue...
pkg-static: https://pkg.opnsense.org/FreeBSD:13:amd64/23.1/latest/packagesite.txz: No route to host
Unable to update repository OPNsense
Error updating repositories!
Starting web GUI...done.
Generating RRD graphs...done.


Note that the part where you see CTRL+T output ultimately took over an hour before I got the "proceed with this action" query.

Clearly, *something* is going on, but I have no idea what.

At this particular location, ipv6 is disabled on the WAN side as there is no ipv6 support. I just tried to do the upgrade from the CLI again, but got a new message:

Code Select Expand


Enter an option: 12

Fetching change log information, please wait... fetch: transfer timed out

This will automatically fetch all available updates and apply them.


This has been frozen with no additional output for probably 30 minutes or so.  I'm hoping this is just a problem with a cdn or something and its an easy fix.

[j_s]

After posting this I noticed the last entry for the OP wasn't a signature but was what he did to fix it.  I tried both US repos, and no change.  I then changed to the DE repo the OP used, and its updating right now.

Sounds like some kind of problem with the repos or something.

[MajStealth]

i hopefully made it more clear now, but "nice" that i am not the only one....

[dd2594opn]

My test one finally updated -- after just letting it sit over night (took better than 8 hours). I will try an update from one of the other mirrors today on a second system.

Hopefully someone is watching and can fix the mirror issue.

[lulemos]

For me after enabling the Hardware CRC option, it worked:

Interfaces -- Settings

Disable hardware checksum offload