OPNsense Forum

Archive => 21.1 Legacy Series => Topic started by: waxhead on January 28, 2021, 10:38:46 pm

Title: Upgrade to 21.1 may take *very* long time (>1h)
Post by: waxhead on January 28, 2021, 10:38:46 pm
Hi,

I just wanted to warn that upgrading to 21.1 on my old box (core 2 duo era) took a very long time (over 1 hour) after all packages was downloaded. If this is indeed expected it might be a good idea to add a warning somewhere so that people don't assume that their box is hanging.

Except from that - everything seems to be running perfectly, so congratulations to the OPNsense team with yet another great release as usual! Thanks a bunch! :)
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: lxray on January 28, 2021, 10:57:40 pm
Thanks for your post. I had already given up hope, having tried for 3 times to install the update only to power off and on manually after nothing had happened 10 minutes later. I always came back to 20.8.4, assuming somehting got stuck during the update. Running on an Intel Nuc 7th Gen i5.

Edit: Tried a 4th time but gave up after 45 minutes. Via physical power button the machine came off and on in a minute and is now still sitting at 20.7.8_4. I guess something IS stuck. Should probably not take so long on i5 CPU with NVME SSD and 16GB RAM.
Advise?
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: mr.yx on January 28, 2021, 11:31:02 pm
Upgraded to 21.1 on an old APU2 took about 10mins.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: waxhead on January 29, 2021, 12:04:55 am
I was able to figure out for how long the update took.
I simply went to Reporting->Health->System->Processor and can see that the graph shows that there was no activity from 17:06 until about 19:26.

That is 2 hours and 20 minutes !! (which is a rather long time)

I also checked the mbuf,states,cputemp and memory graphs which show the same phenomenon.

Could it be that the installed downloads something in the background after all the packages are downloaded? Is there a install log available somewhere that maybe can help the OPNsense devs figure out what may have happened? My box have been running since the install and is backed by a UPS, so unless there is a major power outage I might be able to help out with additional information. (but that is tomorrow - now I need to get some sleep) :)
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: meazz1 on January 29, 2021, 01:00:58 am
Just update from 21.1r to 21.1, took about less than 10 min, smooth sailing all the way.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: gpb on January 29, 2021, 01:28:46 am
Wow...2 hours...?  About 6 minutes here...a bit took longer than a point release.  Verified the hdd light was flashing during upgrade and after the NIC went dark a few times, finally lit up and started flashing in a normal state.  No monitor hooked to it and thankfully didn't have to drag one out.  Zabbix reported "problem" resolved in 5 minutes.  This was an upgrade via GUI following the 2 package update (sudo, opnsense 20.7.8_4).
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: franco on January 29, 2021, 07:13:57 am
No CPU time means a lot of IO time. It heavily depends on the write performance of your disk. If it is an SD or CF card... slower than slow :)


Cheers,
Franco
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: mimugmail on January 29, 2021, 07:34:16 am
I had one system where the update window freezed and in the background it was already restarted.
Maybe open a separate tab and try to connect again.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: retatefw on January 29, 2021, 07:43:51 am
My upgrade to 21.1 was successful, but it took a long time. The delay seemed to be with the download process. The steps I went through via web interface are:

- Upgraded to 20.7.8_4 which took about 3 minutes.
- Attempted upgrade to 21.1. It failed after 30+ minutes on the first package download with a checksum error.
- Rebooted firewall
- Started upgrade again. The first package download took about 90 minutes. The next two downloads took a couple of minutes. The firewall then rebooted and was fully operational in less than 5 minutes.

Most OPNSense upgrades take me less than 5 minutes. Opnsense is running on an SSD drive and has sufficient  CPU (quad core I7-3820) to sustain 1.2 gigabit downloads (Xfinity Gigabit service provisioned at 1.2 gigabit).

Just wanted to confirm that for some patience may be needed for this upgrade.

Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: banym on January 29, 2021, 07:52:24 am
The first box I updated took a long time for reboot. 10-20 Minutes.
It has fast SSD but it was my old APU1 that I use for testing and is usually the first box I migrate to new releases.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: diablo on January 29, 2021, 10:02:29 am
FWIW my APU2c with AMD GX-412TC SOC (4 cores) took around 10 minutes. Was a bit surprised it took so long, but it worked 100%.

As always many thanks to the devs for making such a great product.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: waxhead on January 29, 2021, 10:13:44 am
No CPU time means a lot of IO time. It heavily depends on the write performance of your disk. If it is an SD or CF card... slower than slow :)


Cheers,
Franco

I rather interpret this as if OPNsense was not running e.g. while the box was rebooting/the system was loading, but then again I don't know the details of the logging. The box has two hard drives in Raid1 transparent to OPNsense and actually the BIOS as well. It is a separate external board that handles the magic all by itself. Never had any IO slowness with it before as far as I know.

Maybe FreeBSD / HardenedBSD have some watchdog functionality? In any case is there some log files available that may help shed light on the situation?! If so I'll happily provide them.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: flushell on January 29, 2021, 10:53:39 am
It took longer than a point upgrade, but not extermely long.
I clocked it: 7min 10sec.
This is in a Qemu box on Proxmox with 2 Xeon 1260L cores, 8Gb Mem on a NVME SSD.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: lxray on January 29, 2021, 12:19:43 pm
Well I can't make the upgrade work.
It takes forever and when I give up and reboot the machine, it's untouched and still on 20.7.8_4.

When I go to the update page, I see this: (please see attachment)

Ideas?
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: chemlud on January 29, 2021, 12:27:19 pm
I updated one box yesterday from GUI without problems. You could try from console...
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: lxray on January 29, 2021, 12:31:07 pm
I updated one box yesterday from GUI without problems. You could try from console...

Tried console too.
Won't work.

It's downloading packages, then rebooting and then nothing is happening (as far as I can tell).  :-\
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: franco on January 29, 2021, 12:58:40 pm
@lxray: please post your /var/cache/opnsense-update/.upgrade.log

For one reason or another the packages install fails... Mostly due to third party packages from non-primary source or manual port builds.


Cheers,
Franco
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: lxray on January 29, 2021, 01:19:36 pm
@franco: Thanks. See attached file.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: franco on January 29, 2021, 01:27:47 pm
The upgrade did not come back and you forcefully restarted? At least the upgrade log simply stops without an error to find during os-vnstat reinstall. Will try to reproduce.


Cheers,
Franco
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: lxray on January 29, 2021, 01:33:57 pm
The upgrade did not come back and you forcefully restarted? At least the upgrade log simply stops without an error to find during os-vnstat reinstall. Will try to reproduce.


Cheers,
Franco

Yes. It took so long, that I restarted (semi forcefully: I pressed the power button, waited until power was off and then powered on again).

Maybe it helps if I show the output of uname:
FreeBSD OPNsense.schlattau.net 12.1-RELEASE-p12-HBSD FreeBSD 12.1-RELEASE-p12-HBSD #0  3c6040c7243(stable/21.1)-dirty: Mon Jan 25 12:27:52 CET 2021     root@sensey:/usr/obj/usr/src/amd64.amd64/sys/SMP  amd64

Thanks Franco
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: waxhead on January 29, 2021, 01:36:08 pm
....At least the upgrade log simply stops....

So this means there is a an upgrade log available? Where can I find it?

FWIW I poked around in dmesg , but there is no timestamps so it's hard to tell. I am not well versed in the BSD world.
From what I could piece together it looked ok with possibly the exceptio of a couple of GIANT-LOCKED messages which after a few quick searches at duckduckgo did not seem significant - but what do I know.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: lucasplcorrea on January 29, 2021, 02:07:27 pm
Good Morning!
It didn't take more than 15 minutes to upgrade here, but after the upgrade, the firewall was very slow, did anyone else happen?

it works very well controlling the rules and ports that are blocked, the functions are operating correctly, but access to the GUI and CLI is very slow, the boot took more than 20 minutes to complete.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: franco on January 29, 2021, 02:11:46 pm
Yes there is a log at /var/cache/opnsense-update/.upgrade.log which records upgrade progress (or lack thereof). But it's the same as watching the console... the same verbosity, no timestamps (as is the case with FreeBSD package manager).

I can't reproduce this hang from here so far. Not sure what I am missing.


Cheers,
Franco
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: hescominsoon on January 29, 2021, 03:43:33 pm
Upgraded to 21.1 on an old APU2 took about 10mins.

instead of waiting for the upgrade.  make a backup..nuke the box..install the latest version..install config backup.  much easier.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: Spoonman2002 on January 29, 2021, 04:27:31 pm

instead of waiting for the upgrade.  make a backup..nuke the box..install the latest version..install config backup. 

- that's exactly what I did (upgrade from 20.7.8 to 21.1 did not work on APU2C4)
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: baqwas on January 29, 2021, 04:35:43 pm
Sorry for appending unexciting feedback. My upgrade to 21.1 (on ProtectLi FW4B - amd64) took around 7 mins + reboot. Forgot to alert Nagios about the downtime and it dutifully alerted me.  :D Other than that, no issue and my modest farm is chugging along nicely. Thank you to all those folks who continue to contribute towards OPNsense advancements and also to "First Responders" at the forum for newbie questions that come from folks like us.

Kind regards.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: gpb on January 29, 2021, 05:01:50 pm
Upgraded to 21.1 on an old APU2 took about 10mins.

instead of waiting for the upgrade.  make a backup..nuke the box..install the latest version..install config backup.  much easier.

Question about doing this...what data do you need to backup first?  For example, does the VNSTAT database get backed up via the configuration (I don't want to lose that history)?  What about data from other packages?  Thanks!
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: miruoy on January 29, 2021, 06:32:47 pm
Upgrade went super smooth as always.

Good job guys and thanks for all the hard work!
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: waxhead on January 29, 2021, 06:55:13 pm
Yes there is a log at /var/cache/opnsense-update/.upgrade.log which records upgrade progress (or lack thereof). But it's the same as watching the console... the same verbosity, no timestamps (as is the case with FreeBSD package manager).

I can't reproduce this hang from here so far. Not sure what I am missing.


Cheers,
Franco

Ok thanks , I looked through the upgrade.log, but could not find anything of interest. And since there is no timestamps it is hard to tell. I guess a potential solution would be to scan all files that have been updated for timestamps and figure out where there was a delay.... the best I could do was  find / -ls -newerct "2021-01-28 18:00:00" -not -newerct "2021-01-28 19:00:00" | grep "Jan 28 18:3" | less   and then tuning this command a bit. Not very useful , but perhaps there is a path that is more interesting than others - if so please let me know and I'll try to figure out what caused the delay.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: Meditux on January 29, 2021, 07:33:05 pm
Hi guys

The upgrade went through without a hitch within 4 minutes. Everything works great. My thanks to all who participated in this. Great job!

Greetings Meditux
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: thowe on January 29, 2021, 09:59:52 pm
On the APU2E4, the update ran completely automatically and smooth in less than 10 minutes. Many thanks to all who contributed to this!
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: 4r7ur on January 30, 2021, 05:04:57 pm
Upgraded two APU2C4, without issues. It took about 12 minutes each time.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: abraxxa on January 30, 2021, 08:52:27 pm
Upgraded my Gigabyte GA-J3455N-D3H based firewall successfully, took about 80 seconds until it was pingable again.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: sot3 on January 30, 2021, 09:02:38 pm
Well I can't make the upgrade work.
It takes forever and when I give up and reboot the machine, it's untouched and still on 20.7.8_4.

When I go to the update page, I see this: (please see attachment)

Ideas?

I'm getting the same result upgrading from 21.1.b_140-amd64, except that I have 2 updates pending instead of 3.  When I attempt either update it seems to download and install and reboot but then boots to the same state.

How does one access this console?  When I ssh to the system I get a bash prompt.
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: pablo on January 30, 2021, 11:09:54 pm
Hi everyone,

I upgraded a VM before upgrading one of my GWs.  I monitored the Console and noticed one of the longest steps was the extraction of the new packages.  This takes CPU + I/O.

My GW upgrade time was four minutes.

Below is my write-time on the SSD gumstick:

Code: [Select]
# dd if=/dev/zero of=x bs=64k count=10000;rm x
10000+0 records in
10000+0 records out
655360000 bytes transferred in 1.967271 secs (333131472 bytes/sec)
Title: Re: Upgrade to 21.1 may take *very* long time (>1h)
Post by: mullcom on April 18, 2021, 09:37:30 am
Hello!

It taket Long time for me also. Don't now but more then 1h after that I feel a sleep.