Power fails do corrupt SD Cards and MSATA

[jstrebel]

Hi,
I had a power fail with my APU3 running 20.7 Software. The Box did not come up.
The following errors are observed on the serial console.

CYLINDER GROUP 104: BAD MAGIC NUMBER
UNEXPECTED SOFT UPDATE INCONSISTENCY

REBUILD CYLINDER GROUP? yes

CYLINDER GROUP 105: BAD MAGIC NUMBER
UNEXPECTED SOFT UPDATE INCONSISTENCY

REBUILD CYLINDER GROUP? Yes


REBUILD CYLINDER GROUP? yes

** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
UNREF FILE  I=27  OWNER=root MODE=100644
SIZE=785656 MTIME=Oct 29 20:17 2020
RECONNECT? yes

NO lost+found DIRECTORY
CREATE? yes

CYLINDER GROUP 0: BAD MAGIC NUMBER
UNEXPECTED SOFT UPDATE INCONSISTENCY

REBUWARNING:  was not properly dismounted
ILD CYLINDER GROUP? yes

fsck_ufs: bad inode number 0 to ginode
pid 34421 (mktemp), uid 0 inumber 2 on : out of inodes

: create/symlink failed, no inodes free
mkdir: /tmp/.cdrom: No space left on device
mount_unionfs: /tmp/.cdrom: No such file or directory
mkdir: /tmp/.cdrom: No space left on device
mount_unionfs: /tmp/.cdrom: No such file or directory
mkdir: /tmp/.cdrom: No space left on device
mount_unionfs: /tmp/.cdrom: No such file or directory
mkdir: /tmp/.cdrom: No space left on device
mount_unionfs: /tmp/.cdrom: No such file or directory
mkdir: /tmp/.cdrom: No space left on device
mount_unionfs: /tmp/.cdrom: No such file or directory
mkdir: /tmp/.cdrom: No space left on device
mount_unionfs: /tmp/.cdrom: No such file or directory
mkdir: /tmp/.cdrom: No space left on device
mount_unionfs: /tmp/.cdrom: No such file or directory
PHP Warning:  file_put_contents(): Only 0 of 2086 bytes written, possibly out of free disk space in /usr/local/etc/rc.subr.d/recover on line 270
PHP Warning:  file_put_contents(): Only 0 of 820 bytes written, possibly out of free disk space in /usr/local/etc/rc.subr.d/recover on line 274
pwd_mkdb: /etc/pwd.db.tmp: No space left on device
pwd_mkdb: /etc/pwd.db.tmp: No space left on device
Setting hostuuid: e5a54693-1a22-11eb-83e7-000db957347c.
Setting hostid: 0x8878688a.
Configuring vt: blanktime.
pid 26310 (mktemp), uid 0 inumber 2 on : out of inodes

: create/symlink failed, no inodes free
mkdir: /var/etc: No space left on device
mkdir: /var/lib: No space left on device
chown: /var/lib/php/sessions: No such file or directory
chmod: /var/lib/php/sessions: No such file or directory
eval: cannot create /var/run/dmesg.boot: No space left on device
/usr/local/etc/rc.subr.d/crashdump: cannot create /tmp/ddb.conf: No space left on device
Configuring crash dump device: /dev/null
usage: ddb capture [-M core] [-N system] print
       ddb capture [-M core] [-N system] status
       ddb script scriptname
       ddb script scriptname=script
       ddb scripts
       ddb unscript scriptname
       ddb pathname
.ldconfig: mkstemp(/var/run/ld-elf.so.hints.227C1m): No space left on device
ELF ldconfig path: /lib /usr/lib /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.30/mach/CORE
32-bit compatibility ldconfig path:


First I upgraded the APU3 to the latest BIOS. Still the same.
Booting from MSATA or SD Card -> same symptoms

This just happens with Power Loss. Its reproduceable.
Thanks for your hints
Jakob

[jstrebel]

Problem occurs with APU2 as well

[lshantz]

I may be way off base here, but are you caching data? If you are, and power goes out, you will likely have a paperweight OS. I found this out the hard way.

[jstrebel]

It looks to me its a FreeBSD issue. How can we inhibit caching? What do you mean with a paperwight OS?

[patman]

Problem occurs with APU2 as well

I had a similar issue with OPNsense running in a virtual machine that I turned of the hard way due to a hang. Reinstall was the only solution I could find.

[Patrick M. Hausen]

Your filesystem is full when you pull the power. That is the problem. Always monitor disk usage and make sure there is space left.

It is normal for UFS (and every other FS, btw) to lose a couple of transactions that are "in flight" when you just pull the plug. But the reason why the FS cannot be repaired automatically at the next reboot is simply that your media is full. See all the "No space left on device" messages.

[jstrebel]

You are right about the error message.
It seems to me this error message is misleading.
- I have tried with with 64GB SD Card and with 16GB mSATA just after the installation of OPNsense i pull the power plug, to simulate the power outage. No additional options installed. 

[jstrebel]

update:
Tried with version 19.1 APU restarts properly after power fail.
Remark 19.1 runs on FreeBSD 11.2 p9
 

[r.friemer]

I'm running an APU4D4 with 20.7.3 and MSATA.

It ooted up correytly after Power fail.

I can confirm, that it must be an UFS Failure. This can happen, i.e. if the system is writing on the Device in the moment of Power Failure. Normally the System should repiar the error with fsck by booting.

Robert

[jstrebel]

Thanks to the suggestion of Michael Steinmann from PCengines.
He suggested to download the serial image,  boot it from the USB and install it to the mSATA.
Disconnect power and reconnect it do not create the problem as reported.

[Patrick M. Hausen]

What were you running before if not the serial image installed to your MSATA?

[jstrebel]

The Nano image.
"dd gzcat OPNsense-20.7-OpenSSL-nano-amd64.img.bz2 | sudo dd of=/dev/disk2 bs=512”