pkg, fetch & update check issues in 21.7.5

[survive]

Hi guys,

I'm running 21.7.5 in a vm on ESXi and I'm trying to understand why the update checks are taking so long when I check in both the gui & cli.

From what I can tell my GUI checks are slow because fetch is timing out connecting to pkg.opnsense.org, but I have confirmed I can ping\ping6 that host in an ssh session. The GUI eventually exits with this message:

pkg: Repository OPNsense cannot be opened. 'pkg update' required

There are some other topics here that come close to this problem, one suggested this command:

sh -x /usr/local/opnsense/scripts/firmware/changelog.sh fetch

Which results in fetch timing out.

Running an update from the console results in this error:

pkg-static: Repository OPNsense missing. 'pkg update' required
pkg-static: No package database installed.  Nothing to do!

Running "pkg update" from the CLI seems to start the same dialog as the GUI web update.

I saved my config & spun up a 21.7.1 vm, which I was able to update to 21.7.5 as expected. I can't say if I imported my config first or updated, but I was able to run the plug-in resolution tool & get my plug-ins all installed.

Any idea what's going on here?

-Will

[survive]

So I repeated my process with a fresh vm.

I installed 21.7.1 & gave it an IP, then I went in through the GUI & imported my config. Swapped in the new vm & was able to install all my plug-ins. Rebooted, checked for updates, & installed 21.7.5.

Same thing as before. Fetch times out.

I have just now seen that 21.7.6 is out. I did the same process starting with the fresh 21.7.1 vm & updating to 21.7.6. Fetch is timing out & my plug-ins are all listed as orphaned. I can ping & ping6 update site same as before.

-Will

[cookiemonster]

there's a good chance that either no DNS so it can't resolve the address of the repositories or no access to the internet. Check for that.

[survive]

That's what the other threads I've reviewed have said.

I can nslookup pkg.opnsense.org successfully from the shell while I'm trying. I can even run an un-interrupted ping going to the update site & it will fail when I check. And this seems to only happen once I've gone to 21.7.5 or .6.

I have no issues with 21.7.1 checking for updates or the updating process itself, but once I'm on 21.7.5 update checks don't complete.

-Will

[cookiemonster]

Right. Aside from checking other mirrors I'm not sure what else to suggest.

[franco]

Can you post the output of the (quite helpful) connectivity audit please.


Cheers,
Franco

[survive]

Sure thing!

Here are the (unsuccessful) results of an audit I just completed with a vm I just updated to 21.7.6:

Code Select Expand


***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 21.7.6 (amd64/OpenSSL) at Fri Nov 26 14:21:08 CST 2021
Checking connectivity for host: pkg.opnsense.org
PING 89.149.211.205 (89.149.211.205): 56 data bytes
64 bytes from 89.149.211.205: icmp_seq=0 ttl=49 time=103.515 ms
64 bytes from 89.149.211.205: icmp_seq=1 ttl=49 time=103.167 ms
64 bytes from 89.149.211.205: icmp_seq=2 ttl=49 time=105.620 ms
64 bytes from 89.149.211.205: icmp_seq=3 ttl=49 time=104.161 ms

--- 89.149.211.205 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 103.167/104.116/105.620/0.939 ms
PING6(56=40+8+8 bytes) 2600:1700:5db0:6050:207:e9ff:fe18:beef --> 2001:1af8:4f00:a005:5::
16 bytes from 2001:1af8:4f00:a005:5::, icmp_seq=0 hlim=45 time=106.239 ms
16 bytes from 2001:1af8:4f00:a005:5::, icmp_seq=1 hlim=45 time=106.698 ms
16 bytes from 2001:1af8:4f00:a005:5::, icmp_seq=2 hlim=45 time=105.724 ms
16 bytes from 2001:1af8:4f00:a005:5::, icmp_seq=3 hlim=45 time=105.261 ms

--- 2001:1af8:4f00:a005:5:: ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 105.261/105.981/106.698/0.540 ms
Checking connectivity for URL: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7
Updating OPNsense repository catalogue...
Fetching meta.txz: . done
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/packagesite.txz: Operation timed out
Unable to update repository OPNsense
Updating SunnyValley repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.txz: .. done
Processing entries: .... done
SunnyValley repository update completed. 31 packages processed.
Error updating repositories!
***DONE***


And here is an audit I did on 21.7.1 right before the update above:

Code Select Expand


***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 21.7.1 (amd64/OpenSSL) at Fri Nov 26 12:47:16 CST 2021
Checking connectivity for host: pkg.opnsense.org
PING 89.149.211.205 (89.149.211.205): 56 data bytes
64 bytes from 89.149.211.205: icmp_seq=0 ttl=49 time=103.411 ms
64 bytes from 89.149.211.205: icmp_seq=1 ttl=49 time=103.397 ms
64 bytes from 89.149.211.205: icmp_seq=2 ttl=49 time=103.582 ms
64 bytes from 89.149.211.205: icmp_seq=3 ttl=49 time=103.459 ms

--- 89.149.211.205 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 103.397/103.462/103.582/0.073 ms
PING6(56=40+8+8 bytes) 2600:1700:5db0:6050:207:e9ff:fe18:beef --> 2001:1af8:4f00:a005:5::
16 bytes from 2001:1af8:4f00:a005:5::, icmp_seq=0 hlim=45 time=105.174 ms
16 bytes from 2001:1af8:4f00:a005:5::, icmp_seq=1 hlim=45 time=105.476 ms
16 bytes from 2001:1af8:4f00:a005:5::, icmp_seq=2 hlim=45 time=105.212 ms
16 bytes from 2001:1af8:4f00:a005:5::, icmp_seq=3 hlim=45 time=105.955 ms

--- 2001:1af8:4f00:a005:5:: ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 105.174/105.454/105.955/0.312 ms
Checking connectivity for URL: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.txz: .......... done
Processing entries: .......... done
OPNsense repository update completed. 775 packages processed.
Updating SunnyValley repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.txz: .. done
Processing entries: .... done
SunnyValley repository update completed. 31 packages processed.
All repositories are up to date.
***DONE***


-Will

[Superduke]

+1 to this issue...I myself just updated via GUI to 21.7.6...although I would normally do it via CL but it said there was nothing to do.

After update, asking for another check causing a hang and time out.  And trying to install another plugin (in my case I wanted to install the unifi plugin) hangs as well.

Graphically the Update tab just continues to spin even after the update request has stopped.

Hi guys,

I'm running 21.7.5 in a vm on ESXi and I'm trying to understand why the update checks are taking so long when I check in both the gui & cli.

From what I can tell my GUI checks are slow because fetch is timing out connecting to pkg.opnsense.org, but I have confirmed I can ping\ping6 that host in an ssh session. The GUI eventually exits with this message:

pkg: Repository OPNsense cannot be opened. 'pkg update' required

There are some other topics here that come close to this problem, one suggested this command:

sh -x /usr/local/opnsense/scripts/firmware/changelog.sh fetch

Which results in fetch timing out.

Running an update from the console results in this error:

pkg-static: Repository OPNsense missing. 'pkg update' required
pkg-static: No package database installed.  Nothing to do!

Running "pkg update" from the CLI seems to start the same dialog as the GUI web update.

I saved my config & spun up a 21.7.1 vm, which I was able to update to 21.7.5 as expected. I can't say if I imported my config first or updated, but I was able to run the plug-in resolution tool & get my plug-ins all installed.

Any idea what's going on here?

-Will

[survive]

Hi Superduke,

Thanks for confirming that I'm not alone in seeing this.

Are your plug-ins listed as orphaned now as well?

-Will

[survive]

Hi guys,

Following up on this after only getting one response. I can't imagine I'm the only one seeing this.

I have a 21.7.1 image running on ESXi. Been working great for years. Since 21.7.5 I haven't been able to resolve the package manage site to check for updates. To work around this I just run on a snapshot, and when a new release comes out I revert back to 21.7.1.

I just did this with my vm, updating to 22.1.b and as expected I get this in the audit

Code Select Expand


***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 22.1.b_146 (amd64/OpenSSL) at Wed Jan 12 23:31:50 CST 2022
Checking connectivity for host: pkg.opnsense.org
PING 89.149.211.205 (89.149.211.205): 56 data bytes
64 bytes from 89.149.211.205: icmp_seq=0 ttl=49 time=105.870 ms
64 bytes from 89.149.211.205: icmp_seq=1 ttl=49 time=106.065 ms
64 bytes from 89.149.211.205: icmp_seq=2 ttl=49 time=105.280 ms
64 bytes from 89.149.211.205: icmp_seq=3 ttl=49 time=106.036 ms

--- 89.149.211.205 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 105.280/105.813/106.065/0.317 ms
PING6(56=40+8+8 bytes) 2600:1700:5db0:6050:207:e9ff:fe18:beef --> 2001:1af8:4f00:a005:5::
16 bytes from 2001:1af8:4f00:a005:5::, icmp_seq=0 hlim=45 time=107.744 ms
16 bytes from 2001:1af8:4f00:a005:5::, icmp_seq=1 hlim=45 time=107.511 ms
16 bytes from 2001:1af8:4f00:a005:5::, icmp_seq=2 hlim=45 time=107.520 ms
16 bytes from 2001:1af8:4f00:a005:5::, icmp_seq=3 hlim=45 time=107.336 ms

--- 2001:1af8:4f00:a005:5:: ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 107.336/107.527/107.744/0.145 ms
Checking connectivity for URL: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7
Updating OPNsense repository catalogue...
Fetching meta.txz: . done
pkg: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/latest/packagesite.txz: Operation timed out
Unable to update repository OPNsense
Updating SunnyValley repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.txz: .. done
Processing entries: .... done
SunnyValley repository update completed. 31 packages processed.
Error updating repositories!
***DONE***


It's clear that I can reach the test systems post update, but something sure isn't making it work!

Updating stopped working after 21.7.5. I'd like to figure out why. Any thoughts?

-Will

[franco]

Try a different mirror please.

It's likely still a local issue. I've seen VMs not handling fragmentation correctly and discarding packets because of it or VLAN tags getting in the way on the wire (4 extra bytes are bad under certain circumstances).

Keep in mind the pings are without payload. We might add a full payload of 1500 bytes to then just to make the test more realistic.


Cheers,
Franco

[iamperson347]

I experienced the same issue with updates in a VM but with physical nics passed through (no virtual nic). Does this rule out fragmenation due to running in a VM?

[survive]

So I think the latest update fixed it.

I have no explanation why and I didn't change anything, I swear! Mirror was set to default.

I was on the development branch on this vm so I got prompted for an update today which applied fine. Then saw that 22.1.r1 was out so I updated to that. It looked like it worked, all the packages were fetched but I wasn't on 22.1.r1.

I changed the type to "community" and updated without issue.

I hope it was some weird edge case & we never speak of it again.

-Will

[franco]

Problems that disappear for no reason usually come back for the same reason. Keep an eye out for this. :)


Cheers,
Franco

[hendrikrhl]

Hi,

we are currently experiencing the same issue on our DEC3840.

All Mirrors resolvable and pingable. Tried with OpenSSL and LibreSSL.
Also different DNS Servers bring no change.

Going to set our OPNSense to factory defaults in a few days to have a look if the issue can be resolved.

Also firewall live log does not show any blocked traffic from firewall itself.


No unbound installed.
Tested different DNS Servers (Cloudflare, Google, etc.)
Tested "do not use the local dns service as a nameserver for this system"

Our Network and uplink is currently only ipv4. May that be a problem? Even with ipv6 completely disabled, the OPNSense sometimes tries to resolve requests over ipv6