[SOLVED] opnsense-update fails

[shade73]

opnsense-update fails both from web gui and console.

Message in console:

opnsense-update
Updating OPNsense repository catalogue...
OPNsense repository is up-to-date.
All repositories are up-to-date.
Updating OPNsense repository catalogue...
OPNsense repository is up-to-date.
All repositories are up-to-date.
New version of pkg detected; it needs to be installed first.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
   pkg: 1.6.2_1 -> 1.6.4

The process will require 8 KiB more space.
[1/1] Upgrading pkg from 1.6.2_1 to 1.6.4...
pkg: sqlite error while executing INSERT OR REPLACE INTO packages( origin, name, version, comment, desc, message, arch, maintainer, www, prefix, flatsize, automatic, licenselogic, mtree_id, time, manifestdigest, dep_formula) VALUES( ?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, (SELECT id FROM mtree WHERE content = ?14), NOW(), ?15, ?16) in file pkgdb.c:1705: database disk image is malformed

[franco]

Your database has been trashed for some reason or another. Try to bootstrap the whole system again by issuing:

# opnsense-bootstrap

The side effect is that it'll automatically upgrade to the latest version and also preserve your previous settings.


Cheers,
Franco

[shade73]

Same results.

# opnsense-bootstrap
This utility will attempt to turn this installation into the latest
OPNsense 16.1 release.  All packages will be deleted, the base
system and kernel will be replaced, and if all went well the system
will automatically reboot.

Proceed with this action? [y/N]: y

pkg already bootstrapped at /usr/local/sbin/pkg
Updating OPNsense repository catalogue...
Fetching meta.txz: 100%    1 KiB   1.5kB/s    00:01   
Fetching packagesite.txz: 100%   67 KiB  68.9kB/s    00:01   
Processing entries: 100%
OPNsense repository update completed. 226 packages processed.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
   pkg: 1.6.2_1 -> 1.6.4

The process will require 8 KiB more space.
2 MiB to be downloaded.
Fetching pkg-1.6.4.txz: 100%    2 MiB   2.5MB/s    00:01   
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.6.2_1 to 1.6.4...
pkg: sqlite error while executing INSERT OR REPLACE INTO packages( origin, name, version, comment, desc, message, arch, maintainer, www, prefix, flatsize, automatic, licenselogic, mtree_id, time, manifestdigest, dep_formula) VALUES( ?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, (SELECT id FROM mtree WHERE content = ?14), NOW(), ?15, ?16) in file pkgdb.c:1705: database disk image is malformed

[franco]

Let's apply more force, not unsolvable. I don't know wether you use a nano image or not, but run all of these to be sure even if they throw an error...

# rm -rf /var/db/pkg /root/var/db/pkg
# rm /var/db/pkg
# mkdir -p /var/db/pkg

And then again opnsense-bootstrap.

[shade73]

It a nano image, runing on a opnsense appliance box (delivered with pfsense). I am testing opnsense on my small home box, before changing the production boxes.

But for now I will throw in the towel, just killed the box for the 10th time, 3 times to day.

Your above tip got it one step further, now it could start the opnsense-bootstrap but failed because of not enough space in /tmp (a tmpfs drive on 53mb). That was sovled by deleting the /tmp/opnsense-bootstrap folder, think there was leftover from the first attemt.
Next it failed because of not enough space in the /var folder (a tmpfs drive on 19mb) and the space needed was way above the total space, so I created a symlink in the subfolder used to new folder on the main drive.
Now the opnsense-bootstrap command could continue, and began updating, but died while unpacking the openssl package (have tried that many times when I updated from stock 16.1 to 16.1.2).
That kills the box, and the only solution is to remove the CFcard and flash it with stock 16.1 from a computer (dd i Linux).

When I updated to 16.1.2, it keept dying many times in the same way, but I learned if aborted the setup wizard (after a stock dd install) I could update, so long no config changes where made first.
So I tried that again, and after a long update it finished a 16.1.4 update, but died agian after a reboot. Have also experienced that in my 16.1.2 update struggles. Something in the last step fails, and the kernel arent written correct so it boot into no kernel.
In the 16.1.2 update I ended up fetching the kernel file from your repository before reboot from a shell. I forgot that step here the last time.

I dont know if my opnsense hardware box is to small?, or the opnsense software just arent ready for production use?

[franco]

Small /var and /tmp points to old embedded hardware, did you check with our requirements?

https://docs.opnsense.org/manual/hardware.html#hardware-requirements

The bug you describe is a sqlite database corruption that causes a FreeBSD 10.2 tool to refuse operation. It is very likely that one can run into it with all pkg-based firmware solutions.

[shade73]

It is a Decsio OPNsense hardware box, with a 500mhz AMD geode, with 512mb ram and a 4GB CFcard.

So it is just Right on the minium requirements

[franco]

Okay,thank you. So far so good. I'm not sure why the tmpfs is so low, are you using advanced features such as the proxy server or intrusion detection?

Can you provide output of the following command:

# df -h

Was the card preflashed or did you do this yourself? Was it from the 16.1 install or an older 15.7?

There are a multitude of issues, worst case the nano images have a integrity problem, but let's take it one step at a time.

[shade73]

I flashed the card myself (on new 4GB card), have pfsense on the old 2GB card. I used dd in linux to write the card, the Window command kept failed, it says the image is oversized.

I used the 16.1 install.

/tmp is a total of 53MB in size.
/var is a total of 19MB in size.

Don't know if it is related, but think about the oversize message in Windows. If I use the card reader/write to "look" on the written card it shows this:

[ 4949.993995] sd 4:0:0:1: [sdc] 7806960 512-byte logical blocks: (3.99 GB/3.72 GiB)
[ 4950.020563]  sdc: sdc1 sdc2 sdc3
                sdc1: <bsd: >
                sdc2: <bsd: >
[ 4950.020711] sdc: p3 size 102816 extends beyond EOD, enabling native capacity
[ 4950.026248]  sdc: sdc1 sdc2 sdc3
                sdc1: <bsd: >
                sdc2: <bsd: >
[ 4950.026342] sdc: p3 size 102816 extends beyond EOD, truncated

Can the extends beuond EOD mess with the update proces?

[franco]

/tmp is a total of 53MB in size.
/var is a total of 19MB in size.

This seems overly low for an installation that has indeed 512 MB of RAM and may be the cause of your problems: if files cannot be downloaded completely, they cannot be processed and pkg may get stuck trying to update files that it requires to be available. What helps in these instances is to disable /var and /tmp MFS mount temporarily for the update, reboot, do the actual update (it will take its time) and then flip /var and /tmp MFS it back on and reboot  again to get back to the original system state.

If the file system is active, the EOD is not an issue.

[shade73]

I did something like that in reply #4 I synlinked the dirs out of the tmpfs partitions, and then the upgrade finish. It is very slow but it does finish, but then it dies on reboot. The kernel is missing.

When you write "has indeed 512 MB of RAM" got me thinking, you guys got to know a Descio box you build them.. So I checked again (this time in software) and it have 233 MB total for system, so I guess the board only have 256 MB RAM total on chip.

Does opnSense use more RAM/ressources than the pfSense the box where deliverey with? Is it still the better/more stable choice for my 2 big Descio boxes?

I will try to get some bigger hardware to test opnSense on, have a fitlet X box I can replace my small home Descio box with.

[franco]

I'm an unaffiliated software engineer. I wouldn't know which hardware this is without an exact model name and cross-checking the specs.

opnsense-update tries hard not to remove the kernel, but running it subsequently with too little space will eventually shift out any remaining kernel. This can be improved, but won't fix the fact that it's impossible to properly upgrade the kernel on such a system.

pfSense is moving into the same direction, e.g. finally adopting pkg-based updates for 2.3, obsoleting read-only mounts for nano-style images and adding a larger ecosystem that may or may work on the hardware that you currently have.

[shade73]

Sorry, forgot to answer this.

I got a Fitlet XA10, and it run OPNsense very smooth, the update from the 16.1.0 installer to lastest took about 30 secounds

Thanks for your input, it keept me running until the Fitlet arrived.

[franco]

Hi shade73,

Thanks for checking back. 

A lot happened since March in this general area: we now have all runtime errors in the GUI, all firmware downloads move to persistent storage with 16.7, a locking mechanism prevents races during upgrades and the GUI firmware page can be closed and reopenend at any given time even when an update is running -- it'll just reattach and continue to show the upgrade progress.


Cheers,
Franco