OPNsense Forum
Archive => 19.1 Legacy Series => Topic started by: chemlud on May 05, 2019, 03:18:28 pm
-
Hi!
Tryingto update a 4GB Cf-card install of 386 nano (currently on 19.1.4), but after downloading all updates the install process gives "disk full" and update fails...
Any ideas?
-
Tried 4-5 more times, always stops with disk full when installing ruby. But in Dashboard disk is only 33% used. Very strange.
-
The packages are compressed. Did you try to download and extract the ruby package on other pc? Is the extracted size + the size of all downloaded update packages smaller than your free diskspace?
-
32% of 4 GB means roughly 2.6 GB free, do you really think there are packages of that size?
Additional info:
RAM = 2 GB
/var is on disk, /temp in RAM
I tried to reboot the system with /var in RAM, but it never came back (mixed system, as some updates had been done already). Have a dd-copy of the functional card (before updating) and now I'm ready for next try. Better to have /var and /temp in memory?
-
...total download for this update is 163 MiB, so not really sure there will be coming out something with more than 1 GiB size?
-
Next try with /var and /tmp in RAM, same result:
-
Uh, oh: https://github.com/opnsense/changelog/blob/9473107466ada781911c6a9a8c353f9c8ee9ee9c/doc/19.1/19.1.7#L11-L14
If you have ruby try to deinstall the plugins using it to salvage a bit of free space first. Then upgrade and try to reinstall the plugins:
$ git grep 'PLUGIN_DEPENDS.*ruby'
benchmarks/iperf/Makefile:PLUGIN_DEPENDS= iperf3 ruby
net/frr/Makefile:PLUGIN_DEPENDS= frr6 ruby
security/tor/Makefile:PLUGIN_DEPENDS= tor ruby
It might be that pkg does something weird in /var and /tmp so the best option is to *not* use RAM MFS in these scenarios, because 1G may be a bit small...
Cheers,
Franco
-
Plugins can be uninstalled how? :-)
-
FYI
My x64 Nano OPNsense (Intel Atom D525, 2 GB RAM, 8 GB CF-Card) updated without any problems to 19.1.7.
- /var and /tmp in RAM
- only ftpproxy-plugin installed
-
...so only apparent difference is size of CF-card? (and 64bit board)
But as the little machines are end-of-life for opnsense by the end of the year I don't want to buy larger cards now. Don't use CF-cards anymore in other devices.
-
@franco
dyndns, tucan and wol ar the only plugins installed by me on this system. No way to remove any preinstalled plugins, or?
PS: Tried with /var and /tmp on disk, not in RAM, same result....
-
You're probably out of inodes, like me.
df -ih
dmesg should show something to the effect of: pid 62215 (pkg-static), uid 0 inumber 16868 on : out of inodes
Nano install, 4GB disk. This is irritating to say the least...
Pretty sure we cannot expand this, even in single user mode. Best course of action here, devs?
-
Is it possible to specify inodes at some point in the ISO installer? Or does the ISO installer format with more UFS inodes by default than the NANO disk image was created with?
Please, please bump up the inode count in the NANO image :P ;) I have 1 plugin installed, DynDNS. Can't remember which OPNsense version I started with, but it has been at least 1 year since I setup this install, applied pretty much every update over the course. I would say that this is an undesired usage pattern for resulting in total system breakdown :'(
-
Are we really the only two users in the whole wide world with thisproblem? :-/
-
You can deinstall ruby easily, not sure how it ended up there:
# pkg remove ruby
Do you have /usr/ports on your install? I remember you building Unbound...
Yes, no inodes is an issue... a lot of small files will "fill" your system even though space is not an issue. Find the lots of files and delete them.
Cheers,
Franco
-
Thanks for reply!
Will check asap, hopefully before end of next week...
Unbound building was not on this machine (but on a 64bit full install, which updated 19.1.7 just fine, except for unbound 1.9.1 still not working with LibreSSL and DNS-over-TLS ;-) ). As you can see the disk full can appear even with php stuff installing (see pic appended above). IIRC the error with ruby came up when trying to install 19.1.7 ***again*** after first try failing with "disk full". In principle it's a plain vanilla nano setup about 2 years ago and updated regularly.
-
That does not exempt it from adding temporary files over the years which may linger depending on your usage of the system.
Cheers,
Franco
-
PS: there may be files hidden in /var or /tmp underneath your RAM MFS mount points ;)
-
Are we really the only two users in the whole wide world with thisproblem? :-/
Lol, maybe we are the only people left running from a Nano image :P
-
This is the output of
du -a | cut -d/ -f2 | sort | uniq -c | sort -nr
42408 usr
1800 boot
356 etc
345 var
161 root
89 sbin
77 lib
75 dev
73 tmp
72 conf
38 bin
12 home
9 libexec
1 sys
1 rescue
1 proc
1 net
1 mnt
1 media
1 entropy
1 COPYRIGHT
The largest folders in usr are 8397 python3.6
7841 python2.7
3373 perl5
So, my issue is completely unrelated to temporary files. NANO images simply don't have enough inodes. Also, var and tmp are RAM disks, as default in NANO images.
-
Could be, that's why I put the following into the notes:
https://github.com/opnsense/changelog/blob/9473107466ada781911c6a9a8c353f9c8ee9ee9c/doc/19.1/19.1.7#L11-L14
I'm not sure how to solve it just yet, but the problem is pretty clear. Maybe this only happens on 4GB cards. Long term goal is obviously to get rid of Python 2.7, but it'll be hard to kill a little while longer...
Cheers,
Franco
-
Yeah I definitely saw that. I checked my disk space usage first, didn't even think about inodes until the update failed half way through the packages. All well, I took a config backup first. Will reload it from ISO, and hopefully be able to specify a custom format command during the installer.
The issue is in the build system, wherever the format command lies for the (assuming memory?) disk that the NANO disk image file is eventually created from.
The UFS filesystem just needs to be created with more inodes during the build process ;-p
-
That we could change for 19.7 gladly. I think the code is this...
https://github.com/opnsense/tools/blob/master/build/nano.sh#L61-L62
We do not specify a whole lot yet.
Cheers,
Franco
-
Looks like a winner to me :D
Output from my stock NANO install:
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/ufs/OPNsense_Nano 4.0G 1.3G 2.4G 36% 46k 2.5k 95% /
devfs 1.0K 1.0K 0B 100% 0 0 100% /dev
tmpfs 76M 16M 60M 21% 198 2.1G 0% /var
tmpfs 63M 2.5M 60M 4% 73 2.1G 0% /tmp
devfs 1.0K 1.0K 0B 100% 0 0 100% /var/unbound/dev
devfs 1.0K 1.0K 0B 100% 0 0 100% /var/dhcpd/dev
Looks like it has close to 50,000 inodes, and is out. Maybe make it 100,000?
When I reload my install, I will probably do something like 250,000, just for future proofing. You can always expand the partition on a larger disk later in life. You can't increase inodes though, without migrating to a fresh filesystem.
-
Doesn't it decrease the block size?
https://www.freebsd.org/cgi/man.cgi?query=makefs&apropos=0&sektion=8&manpath=FreeBSD+11.2-RELEASE&arch=default&format=html
Looks like something with bsize and density could do the trick. But I don't want to kill speed or longevity for SD cards too much for the sake of being future proof. We'll have to compromise a little, ok?
Cheers,
Franco
-
My lunch break just ended :'( I will read this before I reload my current install, sounds like I will need to review it anyhow.
I completely agree with you, block size for any modern filesystem should never be less than 4K. I have read about people seeing cheap flash drives with 8K blocks. Ick, lol.
Again, I will post here after I get a chance to read what you linked.
Thank you very much for working with me on this!
-
Oh I see. No, it doesn't effect block size. It is simply a calculation of how many bytes of storage you want per inode. So, basically, bytes*$value
This talks about it in more detail:
https://forums.freebsd.org/threads/ufs2-inodes.51236/
The above also states that fragsize is a component of it. More homework is needed on my end ???
Scratch that, frag size is only a component of the default if nothing is specified. If you specify it, it is simplybytes*$value=$inodesTotal
-
Ok, from the ISO installer, default settings, guided install mode, approximate inodes per disk size:
3GB = 400,000 inodes
4GB = 600,000 inodes
8GB = 1,100,000 inodes
So, I think it is probably pretty reasonable to bump the inode count up on the 3GB NANO image. At LEAST roughly 400,000 inodes, as this is what newfs uses by default. newfs is also defaulting to 32k block size & 4k fragment size. Those inode numbers aren't exact, but it's fairly easy to math them out using The default is to create an inode for every (2 * frag-size) bytes of data space.
from https://www.freebsd.org/cgi/man.cgi?newfs(8) (https://www.freebsd.org/cgi/man.cgi?newfs(8))
I decided to use an 8GB disk this time, and keep the default of 1.1m inodes.
-
Hmm, this should do the trick....
https://github.com/opnsense/tools/commit/0657a0ae479b4
Need a test image?
Cheers,
Franco
-
I would personally go with 1,500,000 inodes, or possibly even more. My thinking is that the NANO image is auto expanding. So if you write it to a 250GB SSD, on first boot, the size of the filesystem is going to expand to 250GB, but the inode count will remain the same. I know we can't preplan for every use case, but I do feel like 250,000 isn't enough. Think of the 512GB microSD cards, lol. We also aren't talking about a notable loss of usable space to store inode records. I suppose that might be worth finding out first, the actual size of an inode or block of inodes in bytes.
The only reason I didn't go with more inodes than that, on this particular install, is because it is primarily used as an OpenVPN endpoint. Possibly a few more network services down the road, like DHCP, DNS, NTP, Dynamic DNS, SMTP smart host, etc. But for the most part, this specific install will never be a true edge device, it will live behind the edge device, and as such, won't see a ton of plugins installed.
If I get some time today, I will look around and see if I can find inode size in bytes for UFS2.
-
The size of a single UFS2 inode appears to be 256 bytes.
Reference: http://www.ico.aha.ru/h/The_Design_and_Implementation_of_the_FreeBSD_Operating_System/ch08lev1sec2.htm
Disk space per inode count examples:
5,000,000 = 1221MB
1,500,000 = 366MB
1,000,000 = 244MB
500,000 = 122MB
250,000 = 61MB
I feel like we can get away with a little more than 250,000 ;)
-
growfs and not upping inodes is a problem we cannot solve. We can go to 400-500k inodes, but I don't see the point in doing millions just for the off-chance of growfs happening. The installer is way more effective in this case.
Cheers,
Franco
-
Right, agreed. I like 500,000 better ;)
That definitely makes things better than they were. At around 50k, simply installing a NANO image, and updating to current, will brick the install.
Thank you sir!
PS: If you want me to try a test image, I can do that, but not until the weekend.
-
Here's a snapshot:
https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-201905160745-LibreSSL-nano-amd64.img.bz2
I think it came out around 800k for some reason, but that's ok too I guess.
Cheers,
Franco
-
Looks good to me. Booted, checked inodes, configured a few simple services, nothing broke.
-
Neat, thanks for all the help! Next images will be 19.7.
Cheers,
Franco
-
Not a problem :D
Thank you for all the fixing ;D
-
Ladies, can you enlighten me what the official way to fix this is? :-) Many thanks in advance...
-
Needs a reflash with above image. No fixing this otherwise.
Cheers,
Franco
-
Thanks, Franco!
Due to the problems of LibreSSL + unbound with DNS-over-TLS I use openSSL on these nano installs. Should I use the LibreSSL image and switch afterwards to openSSL or is there a more direct approach?
PS: Could you post the sha256 for the image here? Many, many thanks in advance! :-)
-
Created a vanilla 19.1.8 OpenSSL for you...
e5647314586ab8d321ff6adbfc4842fe135ee4ad23c3a99df0a0b1fd9704a6a6 OPNsense-19.1.8-OpenSSL-nano-amd64.img.bz2
https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-19.1.8-OpenSSL-nano-amd64.img.bz2
Cheers,
Franco
-
pörfect!
:-)
-
@franco: would it be possible/easy for you to build an updated i386 image? (e.g. for use with alix boards)
-
You can use the RC1 image or wait 2 days for 19.7 images.
https://pkg.opnsense.org/releases/19.7/
Cheers,
Franco