[resolved] Corrupt rootfs after upgrade to 21.7

[lynix]

After upgrading from 21.1.9 to 21.7 (on a PCEngines APU2) a few days ago I noticed that I have the following in dmesg.boot:

Code Select Expand

ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0:  ATA8-ACS SATA 3.x device
ada0: Serial Number 50026B7268061F0D
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 28626MB (58626288 512 byte sectors)
Trying to mount root from ufs:/dev/gpt/rootfs [rw,noatime]...
WARNING: /mnt was not properly dismounted
WARNING: /mnt: mount pending error: blocks 224 files 3
WARNING: /mnt: reload pending error: blocks 224 files 3
WARNING: /mnt: reload pending error: blocks 224 files 3
WARNING: /mnt: reload pending error: blocks 224 files 3

When running multiple fsck consecutively the results seem to vary randomly:

Code Select Expand

root@opnsense:~ # fsck -f /dev/gpt/rootfs
** /dev/gpt/rootfs (NO WRITE)
** Last Mounted on /mnt
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
71629 files, 512368 used, 6327308 free (14868 frags, 789055 blocks, 0.2% fragmentation)
root@opnsense:~ # fsck -f /dev/gpt/rootfs
** /dev/gpt/rootfs (NO WRITE)
** Last Mounted on /mnt
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
71629 files, 512368 used, 6327308 free (14868 frags, 789055 blocks, 0.2% fragmentation)
root@opnsense:~ # fsck -f /dev/gpt/rootfs
** /dev/gpt/rootfs (NO WRITE)
** Last Mounted on /mnt
** Root file system
** Phase 1 - Check Blocks and Sizes
INCORRECT BLOCK COUNT I=1605346 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=1605347 (120 should be 0)
CORRECT? no

** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? no

SUMMARY INFORMATION BAD
SALVAGE? no

BLK(S) MISSING IN BIT MAPS
SALVAGE? no

71629 files, 512351 used, 6327292 free (14860 frags, 789054 blocks, 0.2% fragmentation)

I don't know much about UFS but shouldn't the results of multiple fsck runs be identical?

Apart from those error messages everything seems to run fine. Any idea what could have gone wrong during the upgrade or what I should do to fix the issue?

[Darkopnsense]

Hello,
Opnsense version 21.7 has a huge gap in disk management.
This will be resolved.
patience.
Cordially,

[Patrick M. Hausen]

@lynix

NEVER run fsck on a live read-write mounted filesystem. NEVER!

You must connect a console, boot into single user mode an run fsck with the filesystem mounted read-only.

[franco]

Opnsense version 21.7 has a huge gap in disk management.

Hm, the disk management code is the same as 21.1 and 20.7... Lots of jumping to conclusions here.


Cheers,
Franco

[lynix]

@lynix

NEVER run fsck on a live read-write mounted filesystem. NEVER!

You must connect a console, boot into single user mode an run fsck with the filesystem mounted read-only.

My naive impression was that, when using UFS, the rootfs is read-only during normal operation. Okay I'll reboot to single user mode and trigger an fsck via serial.

[lynix]

Okay so I booted to single-user mode, re-ran fsck -f 20 times without an error being detected.

Then I rebooted to the default runlevel and dmesg.boot is clean.

So... it seems this all came down to a couple of unclean shutdowns then? Anyway, I'll consider this fixed, sorry for the noise!

[Patrick M. Hausen]

Probably. Glad it worked as expected. That's why I prefer ZFS even on an apu4d4.