[SOLVED] Can't upgrade freshly installed nano 15.7 to 15.7.7 on ALIX

[franco]

256 MB and IDS are really not meant to go together. The fact that it works is miracle. It also explains your low tmpfs capacity when suricata is running in the background. My advice: don't use Intrusion Detection or Proxy Server on the hardware, or replace it, or switch off those services when you upgrade (might be a tad quicker). We cannot afford to strip down OPNsense like other distributions, because it is one of ours strengths. If anything, others should step in and do a lightweight version of OPNsense if there is much demand.

Moving the pkg fetch location only shifts the problem, it doesn't solve it. Especially for nano images, space is always scarce and only RAM in newer devices offers enough space for future upgrades. The main issue is that FreeBSD and in turn OPNsense have outgrown certain older hardware.

The keys changed since OpenSSH 7.1 doesn't have support for SSHv1 anymore, so the keys were rotated by our scripts. This won't happen again any time soon though.

[gpac]

Thanks a lot for your feedback.

Based on your advice I just switched off suricata (who any way never achieved to start completely and was failing while loading). After rebooting, df -h brings :
Filesystem            Size    Used   Avail Capacity  Mounted on
/dev/ufs/OPNsense0    1.8G    783M    919M    46%    /
devfs                 1.0K    1.0K      0B   100%    /dev
tmpfs                  97M     48K     97M     0%    /tmp
tmpfs                 110M     13M     97M    12%    /var
devfs                 1.0K    1.0K      0B   100%    /var/dhcpd/dev


From you point of view what kind of minimum hardware would be required for a "nano" type of system able to run IDS ?

Thanks for your advice.

Best regards.

[franco]

That's a little better, I see, but might still give you trouble on larger updates (normally the updates are applied except the base/kernel).

Our recommendations for the nano image are here: https://opnsense.org/users/get-started/#hardware-requirements

[gpac]

Hello,

I'm back with my poor 256 Mb Alix...

To have automatic updates working, I'v plugged a 2 Gb USB key on the alix and mounted that USB key on /var/cache/pkg.

Now the point is when a kernel update is required opnsense-update uses /tmp to fetch its .txz and .obsolete files.
+ Why aren't they going in /var/cache/pkg like other .txz files ?
+ Is there a way to pass a parameter to opnsense-update to use another directory as working directory ?

Thanks for your kind advice.

Best regards.

[franco]

You can manually edit the working path of /usr/local/sbin/opnsense-update, it's a shell script. Note that the update of the utility currently overwrites these changes.

The reason was that opnsense-update is written by us and we did not want to write to a supposed disk for CF and SD cards while FreeBSD has more of a server focus and persistency of downloaded files on proper hard disks. There's some more things on my TODO list like a config file for opnsense-update, I will try to weave that in for 16.1.

[gpac]

OK Thanks a lot for the reply.

Basically changing the lines
rm -rf /tmp/opnsense-update.* (somewhere at the beginning of the file)
and
WORKDIR=/tmp/opnsense-update.${$}  (somewhere in the middle of the file after the KERNEL UPDATE warning)

by
rm -rf /var/cache/pkg/opnsense-update.*
WORKDIR=/var/cache/pkg/opnsense-update.${$} 

would be great help for me while staying consistent with what the updater is doing for other packages (fetching them in /var/cache/pkg).

Regards.

[franco]

/var/cache/pkg is not a directory we should reuse, the utilities serve different purposes and I don't know the side effects, you should use:

rm -rf /var/cache/opnsense-update.*
WORKDIR=/var/cache/opnsense-update.${$}

This behaviour has unintended side effects on non-Nano images as the updates will become non-volatile and will also make issues with the /var MFS and reversal of the setting, as we had very very early in 15.7 with pkg.

opnsense-update is pretty cool, but was designed because freebsd-update was too complex a tool for the system upgrades. But truth is opnsense-upgrade will become useless with FreeBSD 11 when we an have pkg take care of kernel and base system too.

So all in all I'm not sure how much work should be put in here in the long run, potentially adding regressions and seeing old problems reappear. Not very good style.

I'll have to rethink this some more.

[gpac]

OK I understand.

Then another idea could be to have a different directory name while staying in /tmp :

If you would use /tmp/opnsense-update/${$} instead of /tmp/opnsense-update.${$}, then one could create a mount to a permanent slice of a USB stick for /tmp/opnsense-update

[franco]

Hmm, we can do that actually and just push the files in there flat. Just have to make sure two processes don't race on the same files there (more hypothetical madness, but better safe than sorry when it comes to upgrades).

See: https://github.com/opnsense/update/issues/2

Thanks!

[franco]

I've added your suggestion to opnsense-update. It will be updated as soon as the next base/kernel update is needed. I don't think there's any need before this, right?

[gpac]

Now it works, thanks a lot (kernel updates are going to /tmp/opnsense-update/...)

So I re-formated by USB key with 2 partitions (one for /var/cache/pkg et the other one for /tmp/opnsense-update), and my alix is now updating the kernel very slooooooowly.

Finally it took 21 minutes for the kernel upgrade stage followed by a 6 minutes reboot !

[8191]

@gpac: which filesystem do you use on your USB pen drive? FAT32 or UFS?
Do you mount the drives manually before each upgrade or did you configure them in the fstab? Is the /etc/fstab persistent between updates?

I've just tried an update from 15.7.18 to 15.7.20 (the fifth time in a row) and failed again. I always end up with a corrupted file system. That time I had two external UFS partitions mounted to /var/cache and /tmp/opnsense-update and previously changed the opnsense-update script to adopt a predictable WORKDIR.

Is there any other way in upgrading to 15.7.20? I guess I'll build my own image and flash it directly to the CF...

[franco]

Mf, how much RAM does the board have? This sounds like an Out-Of-Memory kill mid-upgrade.

/etc/fstab entries are never touched. I guess FAT32 works, if you need the stick for other things. We'll improve the procedure further, but the CF card upgrades are tricky because the base system has grown considerably for OPNsense to allow easy access to build tools and a multitude of programming languages (PHP, Python, Perl) along with full manual pages.

[gpac]

Sorry for late feedback.

With recent upgrades I had to create 2 partitions on my USB key
/dev/da0p1 on /var/cache/pkg (ufs, local)
/dev/da0p2 on /tmp/opnsense-update (ufs, local)

Nota an unclean shutdown of my Alix recently corrupted the file system on /var/da0p1 => adding an fsck in the small batch who mounts those seems usefull.

I mounting them manuall through a simple batch file I'm launching via ssh. For sure it would be better to be able to add something in a rc file to have it launched at startup. But I don't know exactly where / how to do that (in regular linux adding a Sxx and a Kxx file in the init.d would to the trick but on freebsd I don't how to do that), nor how to do that in a location / file not touched by future upgrades.

Best regards.

[franco]

Gpac, the best way to launch a script is to use our rc.syshook facility.

Rename the (executable) shell script to /usr/local/etc/rc.syshook.d/mymount.early (mymount is a name you can freely choose)

The boot will then pick it up automatically.