OPNsense Forum

Archive => 23.7 Legacy Series => Topic started by: BruceOS on October 04, 2023, 11:58:31 PM

Title: 23.7 Fetch stops for unknown reason - Update not working
Post by: BruceOS on October 04, 2023, 11:58:31 PM
Hi there - I had it running for 2 years with a major new isntallation last year (when introducing ZFS)

Basically since 23.7 i have massive problems.

ping works

pkg update -f

Updating update catalogue for eternal.

I cant run updates anymore stuck on 23.7.4

That's already one of the nastiest one so far this year. I do not even have a clue what is happing - Basicall my config on a fresh installed = same output.

That setup was running for years.

Recreating the failure makes updates stoping as well.

Fetching timed out -

I tried I guess all the stuff written somewhere in the internet, but nothing helps. What i do not get, why in 23.7.4 - and next thing, why cannot I not solve such an "easy" problem ? :) Network there, Nameserver there, Update on the Leaswebserver alvailable but no fetching.

an i do not have the debug skilly for freebsd nor opnsense to find why it is timeouting suddenly.

I tried to disable the hole firewall (pfctl) changed and checked DNS -

Fetch will not work at all. Even not after setting pk

Any help appriciated...
Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: newsense on October 05, 2023, 04:04:18 AM
Try changing the mirror, and post the output here please if still having errors.
Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: BruceOS on October 05, 2023, 07:06:41 AM
So i tried half night -

this is the most workable output I can get.

***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 23.7.4 at Thu Oct  5 07:02:50 CEST 2023
Checking connectivity for host: mirror.ams1.nl.leaseweb.net -> 5.79.108.33
PING 5.79.108.33 (5.79.108.33): 1500 data bytes

--- 5.79.108.33 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
Checking connectivity for repository (IPv4): http://mirror.ams1.nl.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 851 packages processed.
All repositories are up to date.
Checking connectivity for host: mirror.ams1.nl.leaseweb.net -> 2001:1af8:4700:b210::33
PING6(1548=40+8+1500 bytes) fe80::6a05:caff:fe20:c61c%em0 --> 2001:1af8:4700:b210::33

--- 2001:1af8:4700:b210::33 ping6 statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
Checking connectivity for repository (IPv6): http://mirror.ams1.nl.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7
Updating OPNsense repository catalogue...


Ping same mirror
root@OPNsense:~ # ping mirror.ams1.nl.leaseweb.net
PING mirror.ams1.nl.leaseweb.net (5.79.108.33): 56 data bytes
64 bytes from 5.79.108.33: icmp_seq=0 ttl=50 time=48.138 ms
64 bytes from 5.79.108.33: icmp_seq=1 ttl=50 time=60.030 ms
64 bytes from 5.79.108.33: icmp_seq=2 ttl=50 time=49.899 ms
--- mirror.ams1.nl.leaseweb.net ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 48.138/52.689/60.030/5.240 ms


Ping after setting to another mirror
ping mirror.fra10.de.leaseweb.net
PING mirror.fra10.de.leaseweb.net (37.58.58.140): 56 data bytes
64 bytes from 37.58.58.140: icmp_seq=0 ttl=47 time=49.068 ms
64 bytes from 37.58.58.140: icmp_seq=1 ttl=47 time=51.030 ms
64 bytes from 37.58.58.140: icmp_seq=2 ttl=47 time=44.430 ms
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 37.108/46.482/51.030/5.251 ms


New Mirror

***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 23.7.4 at Thu Oct  5 07:10:28 CEST 2023
Checking connectivity for host: mirror.fra10.de.leaseweb.net -> 37.58.58.140
PING 37.58.58.140 (37.58.58.140): 1500 data bytes

--- 37.58.58.140 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
Checking connectivity for repository (IPv4): http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 851 packages processed.
All repositories are up to date.
Checking connectivity for host: mirror.fra10.de.leaseweb.net -> 2a00:c98:2030:a034::21
ping: UDP connect: No route to host
Checking connectivity for repository (IPv6): http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7
Updating OPNsense repository catalogue...
pkg: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/meta.txz: No route to host
repository OPNsense has no meta file, using default settings
pkg: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg: No route to host
pkg: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.txz: No route to host
Unable to update repository OPNsense
Error updating repositories!
***DONE***



Third time mirror change

***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 23.7.4 at Thu Oct  5 07:14:19 CEST 2023
Checking connectivity for host: mirror.dns-root.de -> 172.67.206.93
PING 172.67.206.93 (172.67.206.93): 1500 data bytes

--- 172.67.206.93 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
Checking connectivity for repository (IPv4): https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.7
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 851 packages processed.
All repositories are up to date.
Checking connectivity for host: mirror.dns-root.de -> 2606:4700:3036::ac43:ce5d
ping: UDP connect: No route to host
Checking connectivity for repository (IPv6): https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.7
Updating OPNsense repository catalogue...
pkg: https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.7/latest/meta.txz: No route to host
repository OPNsense has no meta file, using default settings
pkg: https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg: No route to host
pkg: https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.txz: No route to host
Unable to update repository OPNsense
Error updating repositories!
***DONE***


I have MultiWAN (Deactivated Gateway 1 to make changing the mirror not take forever
I have IPv6 deactivated.. if this helps - i tried to activate it so only the firewall host can use it, but with close to same results. As if pkg update is using another ecosystem for DNS (?)

Setup was workign finde since Opnsense 16 -I renewded the whole image at Opnsense 20 and tried to make a fresh install -

Virtualized with Proxmox QUEMU for 5 years without any problems.

I 'd like to focus on the pkg update mechanism and the not possible resolving, while the firewall resolves everything fine.

UPDATE

After changing the 1t Gateway as off and added ipv6 compatibility to Gateway 2 it still put out negative connection logs, but updated after hiting the button. - I guess it is a problem on Gatweway 1 which is MAIN WAN.

Since the problem occured also with only one WAN after fresh install, I would consider this now a RULE or OUTBOUND PROBLEM -

I did not change the configuration . so something must be changed during the Updates.

Any Ideas on creating a Outbound rule for pkg to test this - I tried some stuff, but failed, since I still have no clue how to debug the system wenn internal program like ping or traceroute (in opnsense) is still working.

New output

***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 23.7.5 at Thu Oct  5 07:24:55 CEST 2023
Checking connectivity for host: mirror.dns-root.de -> 104.21.22.179
PING 104.21.22.179 (104.21.22.179): 1500 data bytes
1508 bytes from 104.21.22.179: icmp_seq=0 ttl=57 time=91.703 ms
1508 bytes from 104.21.22.179: icmp_seq=1 ttl=57 time=73.999 ms
1508 bytes from 104.21.22.179: icmp_seq=2 ttl=57 time=82.238 ms
1508 bytes from 104.21.22.179: icmp_seq=3 ttl=57 time=75.749 ms

--- 104.21.22.179 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 73.999/80.922/91.703/6.940 ms
Checking connectivity for repository (IPv4): https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.7
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 851 packages processed.
All repositories are up to date.
Checking connectivity for host: mirror.dns-root.de -> 2606:4700:3034::6815:16b3
ping: UDP connect: No route to host
Checking connectivity for repository (IPv6): https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.7
Updating OPNsense repository catalogue...
pkg: https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.7/latest/meta.txz: No route to host
repository OPNsense has no meta file, using default settings
pkg: https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg: No route to host
pkg: https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.txz: No route to host
Unable to update repository OPNsense

Error updating repositories!
***DONE***

Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: newsense on October 05, 2023, 07:21:42 AM
You're still failing in IPv6 there it seems.

System - Settings - General

Make sure Prefer to use IPv4 even if IPv6 is available is checked
Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: BruceOS on October 05, 2023, 05:29:30 PM
Hi newsense!

First of all thank you for looking into it.

I try to provide you with more information.

I dsiabled IPv6 a while ago: https://www.thomas-krenn.com/en/wiki/OPNsense_disable_IPv6

Never the less i did activate it in the past and then now activate it again.
Setting: "Prefer to use IPv4 even if IPv6 is available is checked"
was before like that - - was unchecked when i succeded via gateway two - i now checked as well as dhcpv6 is deactivcated for interface of Gateway No. 2

All set know is still behaving like before

Fetched timed out ..
Update circling forever and Status is cricling forever

Best regards


Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: BruceOS on October 05, 2023, 05:42:35 PM
+ Info - The Interface Diagnostics tool stops after one ping - The tool changed totally and is now producing ping jobs. but as mentioned they get stuck, while ping from console is working without problem.

+ Info I AM NOT ALONE
https://forum.opnsense.org/index.php?topic=33202.0

+ Info Investigating freebsd settings at the moment

/usr/local/etc/pkg.conf
https://forums.freebsd.org/threads/forcing-pkg-bootstrap-to-use-ip4-not-ipv6.78223/

+ Info finding more wired stuff
https://www.reddit.com/r/OPNsenseFirewall/comments/mwgl7r/update_issue/

my /etc/resolf.conf
root@OPNsense:~ # cat /etc/resolv.conf
domain orangetree
nameserver 127.0.0.1
nameserver 1.1.1.1
nameserver 9.9.9.9
search orangetree


Update forcing IPv4

root@OPNsense:~ # pkg -4 update
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.


Upgrade forcing IPv4
root@OPNsense:~ # pkg -4 upgrade
Updating OPNsense repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01   
pkg: https://mirror.dns-root.de/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg: Operation timed out
Fetching packagesite.txz: 100%  237 KiB 242.5kB/s    00:01   
Processing entries: 100%
OPNsense repository update completed. 851 packages processed.
All repositories are up to date.
Checking for upgrades (0 candidates): 100%
Processing candidates (0 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.



Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: BruceOS on October 13, 2023, 07:05:32 AM
This is the output now from the GUI


***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 23.7.5 at Fri Oct 13 06:49:16 CEST 2023
Fetching changelog information, please wait... fetch: transfer timed out
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 852 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (28 candidates): .......... done
Processing candidates (28 candidates): .......... done
The following 28 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
libedit: 3.1.20221030,1 -> 3.1.20230828,1
opnsense: 23.7.5 -> 23.7.6
php82: 8.2.10 -> 8.2.11
php82-ctype: 8.2.10 -> 8.2.11
php82-curl: 8.2.10 -> 8.2.11
php82-dom: 8.2.10 -> 8.2.11
php82-filter: 8.2.10 -> 8.2.11
php82-gettext: 8.2.10 -> 8.2.11
php82-ldap: 8.2.10 -> 8.2.11
php82-mbstring: 8.2.10 -> 8.2.11
php82-pcntl: 8.2.10 -> 8.2.11
php82-pdo: 8.2.10 -> 8.2.11
php82-session: 8.2.10 -> 8.2.11
php82-simplexml: 8.2.10 -> 8.2.11
php82-sockets: 8.2.10 -> 8.2.11
php82-sqlite3: 8.2.10 -> 8.2.11
php82-xml: 8.2.10 -> 8.2.11
php82-zlib: 8.2.10 -> 8.2.11
py39-Babel: 2.12.1 -> 2.13.0
py39-boto3: 1.28.52 -> 1.28.62
py39-botocore: 1.31.52 -> 1.31.62
py39-cffi: 1.15.1 -> 1.16.0
py39-charset-normalizer: 3.2.0 -> 3.3.0
py39-numexpr: 2.8.6 -> 2.8.7
py39-s3transfer: 0.6.2 -> 0.7.0
py39-urllib3: 1.26.16,1 -> 1.26.17,1
ruby31-gems: 3.4.19 -> 3.4.20
syslog-ng: 4.3.1_1 -> 4.4.0

Number of packages to be upgraded: 28

24 MiB to be downloaded.
self: No packages available to install matching 'opnsense'
***DONE***


self: No packages available to install matching 'opnsense' ?

AND if I start a PING job on dns-root.de it is stopping after 1 ping from they GUI

Firewall analyses pass to dns-root.de , pass to 104.21.22.179 (ip of dns-root.de)

The source is up, dns is working , fw is working - It's a pain in the ass -

Anyone here who is able to debug pkg-update ? DNS resolution , Download Gateway etc..



Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: franco on October 13, 2023, 09:41:18 AM
The sqlite database appears to be damaged.

All it tries to verify is:

# pkg rquery %n opnsense

but if that comes up empty it aborts for safety reasons.

If that is the case what does returning all packages say?

# pkg rquery %n | wc -l


Cheers,
Franco
Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: BruceOS on October 13, 2023, 11:13:46 PM
Hey franco,

thank you for reply  - i  had no time for more text when i sent the analyse. So thank you for havign a look.

Update: it is maybe connected to the WAN interface itself. - but i do not see the problem. I tried fire in the whole optimize settings from random forum users for the "em" interface. But it did not change anything.

I had overwrite MTU activated and deactivated it now, because in a freebsd forum i read something of mismatched mtu.

Since i switched that, traceroute is going much faster. So maybe it is a special network setting which I am missing out. But i don't get why everything (vm,s containers have prober connection via opnsense router firewall but itself has problems with the pkg.

So the WAN interface is the only pci-e  passthrough device for isolating it. I am going to change passthrough parameters tomorrow.

So what I can say for sure now:

Timeout on WAN (dedicated NIC)
No timeout on WAN2 (VLAN)

Was not before 23.7 in my opinion. Defnitly not before 23.x

Every new install of opnsense makes the same behavior. - to be fair i am also trying an install of opnsense to become more clear if this is a fact. 

Wtih best regards,
Bruce



oot@OPNsense:/usr/local/etc/pkg/repos # pkg rquery %n opnsense
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/23.7/latest/packagesite.pkg: Operation timed out
opnsense


oot@OPNsense:/usr/local/etc/pkg/repos # pkg -4 rquery %n opnsense
opnsense

root@OPNsense:/usr/local/etc/pkg/repos # pkg rquery %n | wc -l
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/23.7/latest/meta.txz: Operation timed out
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/23.7/latest/packagesite.txz: Operation timed out
       0


Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: BruceOS on October 15, 2023, 11:46:49 AM
Hey all,
Hey franco,

More analysis - looks like light at the end of the tunnel

It is a problem between IPv6 and DNS for MultiWAN kinda

How i found out



root@OPNsense:~ # fetch -v https://pkg.opnsense.org/FreeBSD:13:amd64/23.7/latest/Latest/nmap.pkg
resolving server address: pkg.opnsense.org:443

root@OPNsense:~ # fetch -v -4 https://pkg.opnsense.org/FreeBSD:13:amd64/23.7/latest/Latest/nmap.pkg
resolving server address: pkg.opnsense.org:443


root@OPNsense:~ # nano /etc/resolv.conf
domain orangetree
nameserver 127.0.0.1
nameserver 1.1.1.1
#nameserver 9.9.9.9 <- comment out WAN2 DNS
search orangetree

new result :

root@OPNsense:~ # fetch -v https://pkg.opnsense.org/FreeBSD:13:amd64/23.7/latest/Latest/nmap.pkg
resolving server address: pkg.opnsense.org:443

root@OPNsense:~ # fetch -v -4 https://pkg.opnsense.org/FreeBSD:13:amd64/23.7/latest/Latest/nmap.pkg
resolving server address: pkg.opnsense.org:443
SSL options: 82004854
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-CHACHA20-POLY1305
Certificate subject: /CN=pkg.opnsense.org
Certificate issuer: /C=BE/O=GlobalSign nv-sa/CN=GlobalSign GCC R3 DV TLS CA 2020
requesting https://pkg.opnsense.org/FreeBSD:13:amd64/23.7/latest/Latest/nmap.pkg
remote size / mtime: 5787392 / 1697092665
nmap.pkg                                              5651 kB 5506 kBps    01s

drill pkg.freebsd.org SRV
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 51814
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;; pkg.freebsd.org.     IN      SRV

;; ANSWER SECTION:
pkg.freebsd.org.        300     IN      CNAME   pkgmir.geo.freebsd.org.

;; AUTHORITY SECTION:
geo.freebsd.org.        900     IN      SOA     gns1.freebsd.org. hostmaster.freebsd.org. 1 7200 1800 259200 900

;; ADDITIONAL SECTION:

;; Query time: 60 msec
;; SERVER: 1.1.1.1
;; WHEN: Sun Oct 15 11:01:08 2023
;; MSG SIZE  rcvd: 110
root@OPNsense:~ # drill pkg.opnsense.org SRV
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 23183
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; pkg.opnsense.org.    IN      SRV

;; ANSWER SECTION:
pkg.opnsense.org.       0       IN      SRV     2570 513

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 840 msec
;; SERVER: 9.9.9.9    <<< THIS LINE HAD TO MATCH THEN WAN GATEWAY 1 - BUT IT ROUTED OVER DNS OF WAN2
;; WHEN: Sun Oct 15 11:01:15 2023
;; MSG SIZE  rcvd: 50






and the same for pkg update -and pkg -4 update

pkg update time out

pkg -4 update

In new installations my external "DNS" (pihole) was not connected so I rule that out

in the chain of updating and upgrading OPNSENSE is something wrong with:

IF IPv6 does not work go to IPv4 instead
AND
IF you DO NOT find an IP ON DNS1
GOTO DNS2
IF you DO NOT find an IP ON DNS2
GOTO DNS 3

(3 nameserver allowed in /etc/resolv.conf)
I try to make  it a bit like gibbish programm code so maybe the problem becomes clear for any freebsd / opnsense programmer . Was there a Version change in fetch or pkg or the script for update in opnsense ?

Problem Now - Update is nown (pkg works kinda ) - but fetching is not initialized when I hit the button


***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 23.7.5 at Sun Oct 15 11:32:06 CEST 2023
Fetching changelog information, please wait... fetch: transfer timed out
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 852 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (28 candidates): .......... done
Processing candidates (28 candidates): .......... done
The following 28 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
libedit: 3.1.20221030,1 -> 3.1.20230828,1
opnsense: 23.7.5 -> 23.7.6
php82: 8.2.10 -> 8.2.11
php82-ctype: 8.2.10 -> 8.2.11
php82-curl: 8.2.10 -> 8.2.11
php82-dom: 8.2.10 -> 8.2.11
php82-filter: 8.2.10 -> 8.2.11
php82-gettext: 8.2.10 -> 8.2.11
php82-ldap: 8.2.10 -> 8.2.11
php82-mbstring: 8.2.10 -> 8.2.11
php82-pcntl: 8.2.10 -> 8.2.11
php82-pdo: 8.2.10 -> 8.2.11
php82-session: 8.2.10 -> 8.2.11
php82-simplexml: 8.2.10 -> 8.2.11
php82-sockets: 8.2.10 -> 8.2.11
php82-sqlite3: 8.2.10 -> 8.2.11
php82-xml: 8.2.10 -> 8.2.11
php82-zlib: 8.2.10 -> 8.2.11
py39-Babel: 2.12.1 -> 2.13.0
py39-boto3: 1.28.52 -> 1.28.62
py39-botocore: 1.31.52 -> 1.31.62
py39-cffi: 1.15.1 -> 1.16.0
py39-charset-normalizer: 3.2.0 -> 3.3.0
py39-numexpr: 2.8.6 -> 2.8.7
py39-s3transfer: 0.6.2 -> 0.7.0
py39-urllib3: 1.26.16,1 -> 1.26.17,1
ruby31-gems: 3.4.19 -> 3.4.20
syslog-ng: 4.3.1_1 -> 4.4.0

Number of packages to be upgraded: 28

24 MiB to be downloaded.


UPDATE : when i remove CHECK from "Prefer IPv4 over IPv6    Prefer to use IPv4 even if IPv6 is available" then i DO NOT to delete any nameserver "fetch -v -4" is working always



Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: BruceOS on October 16, 2023, 07:17:19 AM
after another day and night session with my beloved firewall .

I am nearly going to be crazy -
[ x ] Prefer to use IPv4 even if IPv6 is available
[ x ] IPv6 disabled system wide  (https://www.thomas-krenn.com/en/wiki/OPNsense_disable_IPv6)
[ x ] setting mirror to http instead of https to rule out certification problems
[ x ] Setting the DNS manually (temporary) in /etc/hosts
[ x ] going crazy about name resolution and ipv6 and pkg



***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 23.7.5 at Mon Oct 16 06:54:41 CEST 2023
Checking connectivity for host: mirror.fra10.de.leaseweb.net -> 37.58.58.140
PING 37.58.58.140 (37.58.58.140): 1500 data bytes
1508 bytes from 37.58.58.140: icmp_seq=0 ttl=52 time=93.476 ms
1508 bytes from 37.58.58.140: icmp_seq=1 ttl=52 time=99.754 ms
1508 bytes from 37.58.58.140: icmp_seq=2 ttl=52 time=85.262 ms
1508 bytes from 37.58.58.140: icmp_seq=3 ttl=52 time=97.281 ms

--- 37.58.58.140 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 85.262/93.943/99.754/5.488 ms
Checking connectivity for repository (IPv4): http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 852 packages processed.
All repositories are up to date.
Checking connectivity for host: mirror.fra10.de.leaseweb.net -> 2a00:c98:2030:a034::21
ping: UDP connect: No route to host
Checking connectivity for repository (IPv6): http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7
Updating OPNsense repository catalogue...
pkg: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/meta.txz: Non-recoverable resolver failure
repository OPNsense has no meta file, using default settings
pkg: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg: Non-recoverable resolver failure
pkg: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.txz: Non-recoverable resolver failure
Unable to update repository OPNsense
Error updating repositories!
***DONE***

----

output cli
root@OPNsense:~ # pkg -4 -d update -f
DBG(1)[49882]> pkg initialized
Updating OPNsense repository catalogue...
DBG(1)[49882]> PkgRepo: verifying update for OPNsense
DBG(1)[49882]> Pkgrepo, begin update of '/var/db/pkg/repo-OPNsense.sqlite'
DBG(1)[49882]> Request to fetch pkg+http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/meta.conf
DBG(1)[49882]> opening libfetch fetcher
DBG(1)[49882]> Fetch > libfetch: connecting
DBG(1)[49882]> Fetch: fetching from: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/meta.conf with opts "i4"
DBG(1)[49882]> Fetch: fetching from: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/meta.conf with opts "i4"
DBG(1)[49882]> Fetch: fetching from: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/meta.conf with opts "i4"
DBG(1)[49882]> Request to fetch pkg+http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/meta.txz
DBG(1)[49882]> opening libfetch fetcher
DBG(1)[49882]> Fetch > libfetch: connecting
DBG(1)[49882]> Fetch: fetching from: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/meta.txz with opts "i4"
DBG(1)[49882]> Fetch: fetching from: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/meta.txz with opts "i4"
DBG(1)[49882]> Fetch: fetching from: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/meta.txz with opts "i4"
pkg: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/meta.txz: Operation timed out
repository OPNsense has no meta file, using default settings
DBG(1)[49882]> Request to fetch pkg+http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg
DBG(1)[49882]> opening libfetch fetcher
DBG(1)[49882]> Fetch > libfetch: connecting
DBG(1)[49882]> Fetch: fetching from: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg with opts "i4"
DBG(1)[49882]> Fetch: fetching from: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg with opts "i4"
DBG(1)[49882]> Fetch: fetcher chosen: http
Fetching packagesite.pkg: 100%  237 KiB 243.0kB/s    00:01   
DBG(1)[49882]> PkgRepo: extracting packagesite.yaml of repo OPNsense
DBG(1)[79533]> PkgRepo: extracting signature of repo in a sandbox
DBG(1)[49882]> Pkgrepo, reading new packagesite.yaml for '/var/db/pkg/repo-OPNsense.sqlite'
Processing entries: 100%
OPNsense repository update completed. 852 packages processed.
All repositories are up to date.

root@OPNsense:~ # opnsense-update
Nothing to do.  <- LIAR you have 27.5 installed and 27.6 is already available! ;)

Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: newsense on October 16, 2023, 07:23:45 AM
In Settings - General do you have a DNS or each WAN ?

Can you temporarily disable WAN2 and see if you can get to the updates ?


As for HTTPS, as long as the time is correct on the FW you have no reason to worry about.
Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: BruceOS on October 16, 2023, 04:46:37 PM
Ay ay --

two DNS for two Gateways - I marked gateway as down, disbaled the gateway removed 2nd DNS  and now I disabled the whole interface.

it's a really unicorn mistake

i love opjnsense too much, - so i will update via WAN2 for updates - I even swtiched the WANs WAN 1 x WAN 2 - I switched DNS .. nothing works :)

so I am happy for any help. Maybe I just want to know why this is hapening. - But WAN 2 is only temporaly activated.

WAN1 is going to a cable bridge
WAN2 is going to a LTE router bridge

ps: did no work

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 23.7.5 at Mon Oct 16 16:40:38 CEST 2023
Fetching changelog information, please wait... fetch: transfer timed out
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
pkg: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg: Operation timed out
Fetching packagesite.txz: .......... done
Processing entries: .......... done
OPNsense repository update completed. 852 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (28 candidates): .......... done
Processing candidates (28 candidates): .......... done
The following 28 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
libedit: 3.1.20221030,1 -> 3.1.20230828,1
opnsense: 23.7.5 -> 23.7.6
php82: 8.2.10 -> 8.2.11
php82-ctype: 8.2.10 -> 8.2.11
php82-curl: 8.2.10 -> 8.2.11
php82-dom: 8.2.10 -> 8.2.11
php82-filter: 8.2.10 -> 8.2.11
php82-gettext: 8.2.10 -> 8.2.11
php82-ldap: 8.2.10 -> 8.2.11
php82-mbstring: 8.2.10 -> 8.2.11
php82-pcntl: 8.2.10 -> 8.2.11
php82-pdo: 8.2.10 -> 8.2.11
php82-session: 8.2.10 -> 8.2.11
php82-simplexml: 8.2.10 -> 8.2.11
php82-sockets: 8.2.10 -> 8.2.11
php82-sqlite3: 8.2.10 -> 8.2.11
php82-xml: 8.2.10 -> 8.2.11
php82-zlib: 8.2.10 -> 8.2.11
py39-Babel: 2.12.1 -> 2.13.0
py39-boto3: 1.28.52 -> 1.28.62
py39-botocore: 1.31.52 -> 1.31.62
py39-cffi: 1.15.1 -> 1.16.0
py39-charset-normalizer: 3.2.0 -> 3.3.0
py39-numexpr: 2.8.6 -> 2.8.7
py39-s3transfer: 0.6.2 -> 0.7.0
py39-urllib3: 1.26.16,1 -> 1.26.17,1
ruby31-gems: 3.4.19 -> 3.4.20
syslog-ng: 4.3.1_1 -> 4.4.0

Number of packages to be upgraded: 28

24 MiB to be downloaded.


cheers Bruce
Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: newsense on October 16, 2023, 06:50:34 PM
That's the expected output, so it was working. Did you interrupt it ?
Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: BruceOS on October 16, 2023, 08:17:35 PM
Well that's the outpuz but

pkg: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg: Operation timed out <- is this right?

and then 28MB are not fetched ... or in other words

fetch -s is running on timeout (in the script, i don't know where it is - yet)
In other wors: The GUI does not pop ups the message and the button to upgrade. :/

Update is also from 5. October in the "Status" - not from today - 5 October was the last time WAN2 had a valid internet connection


Updated on Thu Oct 5 07:19:08 CEST 2023
Checked on N/A



[/code]
***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 23.7.5 at Mon Oct 16 20:17:59 CEST 2023
Fetching changelog information, please wait... fetch: transfer timed out
Updating OPNsense repository catalogue...
Fetching meta.txz: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 852 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (28 candidates): .......... done
Processing candidates (28 candidates): .......... done
The following 28 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
   libedit: 3.1.20221030,1 -> 3.1.20230828,1
   opnsense: 23.7.5 -> 23.7.6
   php82: 8.2.10 -> 8.2.11
   php82-ctype: 8.2.10 -> 8.2.11
   php82-curl: 8.2.10 -> 8.2.11
   php82-dom: 8.2.10 -> 8.2.11
   php82-filter: 8.2.10 -> 8.2.11
   php82-gettext: 8.2.10 -> 8.2.11
   php82-ldap: 8.2.10 -> 8.2.11
   php82-mbstring: 8.2.10 -> 8.2.11
   php82-pcntl: 8.2.10 -> 8.2.11
   php82-pdo: 8.2.10 -> 8.2.11
   php82-session: 8.2.10 -> 8.2.11
   php82-simplexml: 8.2.10 -> 8.2.11
   php82-sockets: 8.2.10 -> 8.2.11
   php82-sqlite3: 8.2.10 -> 8.2.11
   php82-xml: 8.2.10 -> 8.2.11
   php82-zlib: 8.2.10 -> 8.2.11
   py39-Babel: 2.12.1 -> 2.13.0
   py39-boto3: 1.28.52 -> 1.28.62
   py39-botocore: 1.31.52 -> 1.31.62
   py39-cffi: 1.15.1 -> 1.16.0
   py39-charset-normalizer: 3.2.0 -> 3.3.0
   py39-numexpr: 2.8.6 -> 2.8.7
   py39-s3transfer: 0.6.2 -> 0.7.0
   py39-urllib3: 1.26.16,1 -> 1.26.17,1
   ruby31-gems: 3.4.19 -> 3.4.20
   syslog-ng: 4.3.1_1 -> 4.4.0

Number of packages to be upgraded: 28

24 MiB to be downloaded.
self: No packages available to install matching 'opnsense'
***DONE***
[/code]

AND I did not interrupt it.

Scripts are here: /usr/local/opnsense/scripts/firmware
Title: [SOLVED] Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: BruceOS on October 16, 2023, 09:18:52 PM
haha news - i downloaded one package :D ...wtf :D


  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option: 12

Fetching change log information, please wait... fetch: transfer timed out

This will automatically fetch all available updates and apply them.

Proceed with this action? [y/N]: y

Updating OPNsense repository catalogue...
Fetching meta.conf: . done
pkg-static: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg: Operation timed out
Fetching packagesite.txz: .......... done
Processing entries: .......... done
OPNsense repository update completed. 852 packages processed.
All repositories are up to date.
Updating OPNsense repository catalogue...
pkg-static: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/meta.txz: Operation timed out
repository OPNsense has no meta file, using default settings
OPNsense repository is up to date.
All repositories are up to date.
Checking for upgrades (28 candidates): .......... done
Processing candidates (28 candidates): .......... done
The following 28 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        libedit: 3.1.20221030,1 -> 3.1.20230828,1
        opnsense: 23.7.5 -> 23.7.6
        php82: 8.2.10 -> 8.2.11
        php82-ctype: 8.2.10 -> 8.2.11
        php82-curl: 8.2.10 -> 8.2.11
        php82-dom: 8.2.10 -> 8.2.11
        php82-filter: 8.2.10 -> 8.2.11
        php82-gettext: 8.2.10 -> 8.2.11
        php82-ldap: 8.2.10 -> 8.2.11
        php82-mbstring: 8.2.10 -> 8.2.11
        php82-pcntl: 8.2.10 -> 8.2.11
        php82-pdo: 8.2.10 -> 8.2.11
        php82-session: 8.2.10 -> 8.2.11
        php82-simplexml: 8.2.10 -> 8.2.11
        php82-sockets: 8.2.10 -> 8.2.11
        php82-sqlite3: 8.2.10 -> 8.2.11
        php82-xml: 8.2.10 -> 8.2.11
        php82-zlib: 8.2.10 -> 8.2.11
        py39-Babel: 2.12.1 -> 2.13.0
        py39-boto3: 1.28.52 -> 1.28.62
        py39-botocore: 1.31.52 -> 1.31.62
        py39-cffi: 1.15.1 -> 1.16.0
        py39-charset-normalizer: 3.2.0 -> 3.3.0
        py39-numexpr: 2.8.6 -> 2.8.7
        py39-s3transfer: 0.6.2 -> 0.7.0
        py39-urllib3: 1.26.16,1 -> 1.26.17,1
        ruby31-gems: 3.4.19 -> 3.4.20
        syslog-ng: 4.3.1_1 -> 4.4.0

Number of packages to be upgraded: 28

24 MiB to be downloaded.
[1/28] Fetching php82-session-8.2.11.pkg: ..... done
pkg-static: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/All/php82-zlib-8.2.11.pkg: Operation timed out
Starting web GUI...done.
Generating RRD graphs...done.



i guess
pkg-static: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg
is the part which makes the update fail


later three fetches then - TImeout - any way to set the TImeout ?


***GOT REQUEST TO UPDATE***
Currently running OPNsense 23.7.5 at Tue Oct 17 22:01:45 CEST 2023
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 852 packages processed.
All repositories are up to date.
Checking for upgrades (28 candidates): .......... done
Processing candidates (28 candidates): .......... done
The following 28 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
libedit: 3.1.20221030,1 -> 3.1.20230828,1
opnsense: 23.7.5 -> 23.7.6
php82: 8.2.10 -> 8.2.11
php82-ctype: 8.2.10 -> 8.2.11
php82-curl: 8.2.10 -> 8.2.11
php82-dom: 8.2.10 -> 8.2.11
php82-filter: 8.2.10 -> 8.2.11
php82-gettext: 8.2.10 -> 8.2.11
php82-ldap: 8.2.10 -> 8.2.11
php82-mbstring: 8.2.10 -> 8.2.11
php82-pcntl: 8.2.10 -> 8.2.11
php82-pdo: 8.2.10 -> 8.2.11
php82-session: 8.2.10 -> 8.2.11
php82-simplexml: 8.2.10 -> 8.2.11
php82-sockets: 8.2.10 -> 8.2.11
php82-sqlite3: 8.2.10 -> 8.2.11
php82-xml: 8.2.10 -> 8.2.11
php82-zlib: 8.2.10 -> 8.2.11
py39-Babel: 2.12.1 -> 2.13.0
py39-boto3: 1.28.52 -> 1.28.62
py39-botocore: 1.31.52 -> 1.31.62
py39-cffi: 1.15.1 -> 1.16.0
py39-charset-normalizer: 3.2.0 -> 3.3.0
py39-numexpr: 2.8.6 -> 2.8.7
py39-s3transfer: 0.6.2 -> 0.7.0
py39-urllib3: 1.26.16,1 -> 1.26.17,1
ruby31-gems: 3.4.19 -> 3.4.20
syslog-ng: 4.3.1_1 -> 4.4.0

Number of packages to be upgraded: 28

24 MiB to be downloaded.
[1/27] Fetching php82-zlib-8.2.11.pkg: ... done
[2/27] Fetching php82-dom-8.2.11.pkg: ......... done
[3/27] Fetching php82-simplexml-8.2.11.pkg: ... done
pkg-static: http://mirror.fra10.de.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/All/php82-pdo-8.2.11.pkg: Operation timed out
Starting web GUI...done.
Generating RRD graphs...done.
***DONE***

Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: wired2network on October 18, 2023, 02:14:17 AM
BruceOS

I am receiving this same error with only 1 WAN interface on multiple boxes. I too am at wits end, but the only solution I found was to disable the firewall and go and run updates which works but reinstating firewall goes back to the same issues. I receive the error of:

Fetching changelog information, please wait... fetch: transfer timed out
fetch: /usr/local/opnsense/changelog/changelog.txz appears to be truncated: 0/115144 bytes

The configs didn't change, so I am inclined to think the error isn't with that. I've tried the disabling IPS from previous posts that had something similar. I can use OPNsense to ping from LAN, WAN and 127.0.0.1 to 89.149.222.99 as well as computer with all pings going through. I can even open the URL for the updates without issue.

Performing a Status verification takes minutes, much longer than normal. At one point it showed a Firmware: Reporter error but I can't get it to show again. If I remember correctly, it was a phalcon MVC error in pulling the status.

Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: newsense on October 18, 2023, 02:23:57 AM
Can you post a screenshot from Unbound Settings - General, Query Forwarding and DNSoverTLS please ?

This still loos like a misconfiguration, so reinstalling and importing the old configuration will bring you back to the same roadblock.

Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: wired2network on October 18, 2023, 03:27:26 AM
Here are my screenshots from unbound
Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: newsense on October 18, 2023, 07:07:40 AM
You need to forward the queries received by Unbound to an upstream resolver - preferably over TLS.

If using DoT then add 1.1.1.2 and 9.9.9.11 as IPs on port 853 - and it should suffice to get you going.


BruceOS - please check if you're in a similar situation as seen in the screenshots posted above.
Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: wired2network on October 18, 2023, 02:10:02 PM
I figured out what I had changed that caused the issue! I had started to play with RSS and enabled it as per OPNsense's guide setting net.inet.rss.enabled = 1. I set it back to '0' and everything started to pull correctly!

newsense, thank you for your suggestions! I did set the query forward settings but that didn't resolve the issue. made the RSS change, removed the query forward configuration, and did a reboot and the system is still back to working.

BruceOS, if you set up the RSS try disabling and running again.

Franco, I think something may be off with RSS which is denying the ability to fetch update status and reach the update server.
Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: BruceOS on October 18, 2023, 03:40:02 PM
net.inet.rss.enabled=0  (Fetching via IPv4 works again via WAN1)
8) THANK YOU -Progress - added that rss value to tunables when going to multicore CPU - setting it to 0 solved fetching problem. - But it still does not update correctly. but with the knowledege i have i post a working setting for MultiWAN in the evening.


pkg-static  (fetch)  :   Systems -> Settings -> Tunables -> net.inet.rss.enabled=0  (Fetching via IPv4 works again via WAN1

pkg update            :    Systems -> Settings -> General ->  [ ] Prefer to use IPv4 even if IPv6 is available (NOT CHECKED)

for me the Problem is solved

PS: added "net.inet.rss.enabled=1" in 2022 and it was running until "now" without problems.
Title: Re: 23.7 Fetch stops for unknown reason - Update not working
Post by: rrosson on October 18, 2023, 05:35:05 PM
@BruceOS You are not alone. I am also multi-wan and seeing the same issues as you are.