Update-Problems 21.7.6

[hendrikrhl]

Hi everyone,

we are using an DEC3840 and are experiencing issues with updates.
I already read other posts in this forum, but they all don't really match our problem.

First of all, we already trief a bunch of different DNS servers, local ones, cloudflare, pihole, etc.

Pinging pkg.opnsense.org is no problem, domain gets resolved instantly:

Code: [Select]

PING pkg.opnsense.org (89.149.211.205): 56 data bytes
64 bytes from 89.149.211.205: icmp_seq=0 ttl=53 time=11.470 ms
64 bytes from 89.149.211.205: icmp_seq=1 ttl=53 time=11.247 ms
64 bytes from 89.149.211.205: icmp_seq=2 ttl=53 time=23.779 ms
64 bytes from 89.149.211.205: icmp_seq=3 ttl=53 time=25.262 ms
64 bytes from 89.149.211.205: icmp_seq=4 ttl=53 time=18.210 ms
64 bytes from 89.149.211.205: icmp_seq=5 ttl=53 time=11.422 ms
64 bytes from 89.149.211.205: icmp_seq=6 ttl=53 time=11.382 ms
64 bytes from 89.149.211.205: icmp_seq=7 ttl=53 time=12.614 ms
64 bytes from 89.149.211.205: icmp_seq=8 ttl=53 time=17.008 ms
64 bytes from 89.149.211.205: icmp_seq=9 ttl=53 time=11.487 ms
64 bytes from 89.149.211.205: icmp_seq=10 ttl=53 time=11.575 ms
64 bytes from 89.149.211.205: icmp_seq=11 ttl=53 time=11.422 ms
64 bytes from 89.149.211.205: icmp_seq=12 ttl=53 time=11.285 ms
64 bytes from 89.149.211.205: icmp_seq=13 ttl=53 time=11.504 ms
64 bytes from 89.149.211.205: icmp_seq=14 ttl=53 time=11.364 ms
64 bytes from 89.149.211.205: icmp_seq=15 ttl=53 time=11.486 ms
64 bytes from 89.149.211.205: icmp_seq=16 ttl=53 time=11.620 ms
64 bytes from 89.149.211.205: icmp_seq=17 ttl=53 time=13.507 ms
64 bytes from 89.149.211.205: icmp_seq=18 ttl=53 time=36.484 ms
64 bytes from 89.149.211.205: icmp_seq=19 ttl=53 time=46.795 ms
64 bytes from 89.149.211.205: icmp_seq=20 ttl=53 time=11.375 ms
64 bytes from 89.149.211.205: icmp_seq=21 ttl=53 time=11.948 ms
64 bytes from 89.149.211.205: icmp_seq=22 ttl=53 time=11.457 ms
64 bytes from 89.149.211.205: icmp_seq=23 ttl=53 time=11.438 ms
64 bytes from 89.149.211.205: icmp_seq=24 ttl=53 time=11.190 ms
64 bytes from 89.149.211.205: icmp_seq=25 ttl=53 time=11.264 ms
^C
--- pkg.opnsense.org ping statistics ---
26 packets transmitted, 26 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 11.190/15.369/46.795/8.553 ms
When i try

Code: [Select]

pkg -d update -f, the meta.conf sometimes gets fetched after a few minutes, but the update gets stuck afterwards.
In the meantime, fetch does not complete due to a timeout.

In the livelog, the requests gets through the firewall with the "let out anything from firewall itself" rule.
I can also see the ip getting resolved in our dns servers log.
Traceroute to pkg.opnsense.org also working.

I already tried different mirrors and already cloned the mirror to a local server in our network, causing the exact same issue.

I also attachen one screenshot from a partially fetching

Code: [Select]

pkg -d update -f.

Usually,

Code: [Select]

pkg -d update -f returns this:

Code: [Select]

root@turing:~ # pkg -d update -f
DBG(1)[98888]> pkg initialized
Updating OPNsense repository catalogue...
DBG(1)[98888]> PkgRepo: verifying update for OPNsense
DBG(1)[98888]> PkgRepo: need forced update of OPNsense
DBG(1)[98888]> Pkgrepo, begin update of '/var/db/pkg/repo-OPNsense.sqlite'
DBG(1)[98888]> Request to fetch pkg+https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.conf
DBG(1)[98888]> opening libfetch fetcher
DBG(1)[98888]> Fetch > libfetch: connecting
DBG(1)[98888]> Fetch: fetching from: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.conf with opts "i"
DBG(1)[98888]> Fetch: fetching from: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.conf with opts "i"
DBG(1)[98888]> Fetch: fetching from: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.conf with opts "i"
DBG(1)[98888]> Request to fetch pkg+https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz
DBG(1)[98888]> opening libfetch fetcher
DBG(1)[98888]> Fetch > libfetch: connecting
DBG(1)[98888]> Fetch: fetching from: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz with opts "i"
DBG(1)[98888]> Fetch: fetching from: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz with opts "i"
DBG(1)[98888]> Fetch: fetching from: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz with opts "i"
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz: No route to host
repository OPNsense has no meta file, using default settings
DBG(1)[98888]> Request to fetch pkg+https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/packagesite.txz
DBG(1)[98888]> opening libfetch fetcher
DBG(1)[98888]> Fetch > libfetch: connecting
DBG(1)[98888]> Fetch: fetching from: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/packagesite.txz with opts "i"

DBG(1)[98888]> Fetch: fetching from: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/packagesite.txz with opts "i"
DBG(1)[98888]> Fetch: fetching from: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/packagesite.txz with opts "i"
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/packagesite.txz: No route to host
Unable to update repository OPNsense
Error updating repositories!
GUI update log:

Code: [Select]

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 21.7.6 (amd64/OpenSSL) at Tue Jan 25 13:55:31 CET 2022
Fetching changelog information, please wait... fetch: transfer timed out
Updating OPNsense repository catalogue...
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz: No route to host
repository OPNsense has no meta file, using default settings
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/packagesite.txz: No route to host
Unable to update repository OPNsense
Error updating repositories!
pkg: Repository OPNsense cannot be opened. 'pkg update' required
Checking integrity... done (0 conflicting)
Your packages are up to date.
Unbound is currently disabled.
Currently no plugin conflicts.


Thanks for your help!

[franco]

pkg/libfetch doesn't really tell you if you use IPv4 or IPv6 and using "ping" to verify is misleading. The system prefers IPv6 by default (disable under System: Settings: General) and we do have a connectivity audit to spot such issues more easily nowadays.


Cheers,
Franco

[hendrikrhl]

Hi Franco,

thanks for your reply.

Our Network is currently IPv4 only and that is also configured in the general tab.

Connectivity Audit returns the following:

Code: [Select]

***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 21.7.6 (amd64/OpenSSL) at Wed Jan 26 10:30:35 CET 2022
Checking connectivity for host: pkg.opnsense.org
PING 89.149.211.205 (89.149.211.205): 56 data bytes
64 bytes from 89.149.211.205: icmp_seq=0 ttl=53 time=11.529 ms
64 bytes from 89.149.211.205: icmp_seq=1 ttl=53 time=12.500 ms
64 bytes from 89.149.211.205: icmp_seq=2 ttl=53 time=11.493 ms
64 bytes from 89.149.211.205: icmp_seq=3 ttl=53 time=11.925 ms

--- 89.149.211.205 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 11.493/11.862/12.500/0.406 ms
ping6: UDP connect: No route to host
Checking connectivity for URL: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7
Updating OPNsense repository catalogue...
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz: No route to host
repository OPNsense has no meta file, using default settings
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/packagesite.txz: No route to host
Unable to update repository OPNsense
Error updating repositories!
***DONE***
Also executing fetch/curl with the -4 parameter fails due to a timeout.

I am completely confused why this is happening.

Regards

[hendrikrhl]

The following output shows two executions of

Code: [Select]

fetch -vvv -4 https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz executed directly after each other.
The first one passes without any problems within a few milliseconds, the second execution times out.
Is this a common issue based on a setting?

Code: [Select]

root@turing:~/HR-Temp # fetch -vvv -4 https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz
resolving server address: pkg.opnsense.org:443
SSL options: 82004854
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-CHACHA20-POLY1305
Certificate subject: /CN=pkg.opnsense.org
Certificate issuer: /C=BE/O=GlobalSign nv-sa/CN=GlobalSign GCC R3 DV TLS CA 2020
requesting https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz
remote size / mtime: 1460 / 1639495028
meta.txz                                              1460  B   23 MBps    00s
root@turing:~/HR-Temp # fetch -vvv -4 https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz
resolving server address: pkg.opnsense.org:443
failed to connect to pkg.opnsense.org:443
fetch: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/meta.txz: Operation timed out

[franco]

Well first of all make sure you have IPv6 disabled on WAN and LAN as it is set by default and can go undetected. As you can see the ping6 and pkg update make the same assumption (and you get an IPv6 from DNS as it is implied by the ping6 message).

Other than that maybe the issue has to do with multi-wan, load balancing, gateway group use. It's a bit difficult to tell but you can always packet capture from your WAN interface(s) to see where the traffic is flowing. Might also be a block policy in the firewall rules worst case.


Cheers,
Franco

[hendrikrhl]

Thanks for your reply!

Taking a look in the capture, I can see that the Handshake is most of the time incomplete,
shown by the TCP Retransmission after the server's SYN ACK.

It confuses me that the transmission is able to SYNC sometimes, so it looks like a firewall timeout or something like that to me.
Problem could also be related to these posts:
1. https://forum.opnsense.org/index.php?topic=23857.0
2. https://forum.opnsense.org/index.php?topic=4582.0

I'm gonna have a look into it and post it on here if I foud a solution.

Regards

[hendrikrhl]

Did a backup, firmware reset and restore, fixed it somehow