[SOLVED] CAM status: ATA Status Error

[roro]

Hi Everyone,
I have 2 Soekris devices installed with CF-card running Opnsense 16.7.2-i386.

On both I get the following messages in the System Log File:
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): Error 5, Retries exhausted
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): RES: 51 04 6f 63 6b 45 45 00 00 01 00
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): DSM TRIM. ACB: 06 01 00 00 00 40 00 00 00 00 01 00
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): Retrying command
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): RES: 51 04 6f 63 6b 45 45 00 00 01 00
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): DSM TRIM. ACB: 06 01 00 00 00 40 00 00 00 00 01 00
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): Retrying command
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): RES: 51 04 6f 63 6b 45 45 00 00 01 00
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): DSM TRIM. ACB: 06 01 00 00 00 40 00 00 00 00 01 00
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): Retrying command
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): RES: 51 04 6f 63 6b 45 45 00 00 01 00
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): DSM TRIM. ACB: 06 01 00 00 00 40 00 00 00 00 01 00
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): Retrying command
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): RES: 51 04 6f 63 6b 45 45 00 00 01 00
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): CAM status: ATA Status Error
Aug 26 12:16:51    kernel: (ada0:ata0:0:0:0): DSM TRIM. ACB: 06 01 00 00 00 40 00 00 00 00 01 00

It has to do with the disk, anyone seen this before and is there a solution for this?

Greets,
Rosie

[franco]

Hi Rosie,

CF card and TRIM in the same post is suspicious.

Assuming this is a Nano image, can you edit your /etc/fstab accordingly:

DISCLAIMER: Editing the fstab should not be done lightly, system bootup may fail. That's why OPNsense code never touches the file beyond image building or the bsdinstaller's installation process.

Add " # notrim" (no quotes) at the very end of the root partition line (the device should be /dev/ufs/opnsense0) and reboot.

It'll disable TRIM and hopefully the errors go away as the CF card is likely unable to handle the command.


Cheers,
Franco

[roro]

Hi Franco,
thanks for the reply.
I have done research as well and found tunefs and fsck.
As I have a CF there are 2 slices.
ad0s1a has trim enabled and ad0s2a has trim disabled.
When I try to disabled it on ad0s1a i'm not able due to the fact that this slice is still readonly.
I thought by booting in second slice i could disable it, but still no success.
Can you tell me how to enable writing to the first slice.
Greetings,
Rosie

[franco]

Didn't you mean it's still read-write? That's when you can't run the tunefs to modify.

If you use the fstab trick the RC system will do the disabling for you on reboot. That's why I mentioned it.


Cheers,
Franco

[roro]

Hi Franco,
i'm still strugling with this one.
What I do:
First reboot and start Opnsense with option 2.
I do:
# cat /etc/fstab
/dev/ufs/OPNsense0 / ufs rw,async,noatime 1 1
# vi /etc/fstab
ex/vi: Error: /var/tmp/vi.recover/: Read-only file system
ex/vi: Modifications not recoverable if the session fails
ex/vi: Error: /etc/fstab: Read-only file system
ex/vi: Error: Unable to create temporary file: Read-only file system
#
Sorry, but I still don't know how to get this partition read write.
I'm missing knowledge to get this slice read write.
What is the trick?

[roro]

Franco,
don't know what happened but system is back to OPNsense 15.7.18_1-i386 and i had to assing nics and ips again.
I was trying with serial connection and I reset the soekris system with reset button.
I will build the system up again and see what happens.
Keep you informed.
Roro

[roro]

Well I booted system with OPNsense1, I will update this slice and see whats happening next.
Fetching the update files now. Upgrade in progress:
Fetching libevent2-2.0.22_1.txz: 100%  254 KiB 260.0kB/s    00:01
Fetching libedit-3.1.20150325_1.txz: 100%  119 KiB 121.8kB/s    00:01
Fetching ldns-1.6.17_5.txz: 100%  379 KiB 388.3kB/s    00:01
Fetching jansson-2.7_1.txz: 100%   39 KiB  40.4kB/s    00:01
Fetching idnkit-1.0_5.txz: 100%  184 KiB 188.5kB/s    00:01
Fetching gmp-5.1.3_2.txz: 100%  474 KiB 484.9kB/s    00:01
Fetching gettext-runtime-0.19.6.txz: 100%  144 KiB 147.7kB/s    00:01
Fetching freetype2-2.6.2.txz: 100%  535 KiB 547.4kB/s    00:01
Fetching easy-rsa-3.0.1.txz: 100%   31 KiB  32.1kB/s    00:01
Fetching dnsmasq-2.75_1,1.txz: 100%  257 KiB 262.7kB/s    00:01
Fetching dhcp6-20080615_5.txz: 100%  104 KiB 106.2kB/s    00:01
Fetching curl-7.46.0_2.txz: 100%    1 MiB   1.5MB/s    00:01
Fetching choparp-20150613.txz: 100%    7 KiB   7.2kB/s    00:01
Fetching ca_root_nss-3.21.txz: 100%  330 KiB 337.7kB/s    00:01
Fetching bind910-9.10.3P2.txz: 100%    6 MiB   5.9MB/s    00:01
Fetching apinger-0.6.1_4.txz: 100%   32 KiB  33.0kB/s    00:01
Fetching libucl-0.7.3_1.txz: 100%   79 KiB  80.7kB/s    00:01
Checking integrity... done (0 conflicting)
[1/74] Upgrading openssl from 1.0.2_4 to 1.0.2_6...
[1/74] Extracting openssl-1.0.2_6:  62%

[franco]

I'm not exactly sure what you're trying to do. The second slice is dormant, so it wasn't up to date. It doesn't have a shared configuration directory, that's also by design. We'd either go ahead and make the second slice of nano usable, or we'd at some point see that the second slice isn't as useful. We've now seen the latter.

Boot from the first slice as usual (not single-user mode), edit /etc/fstab accordingly and reboot and TRIM will be off...


Cheers,
Franco

[roro]

The whole evening upgrading the opnsense cf i386 soekris box from version 15.x.x to 16.7.2 OpenSSL.
1  OPNsense
2  OPNsense

F6 PXE
Boot:  2

Took a long time but no errors at all.
As usuall I go from openssl to libressl and then the messages about:

(ada0:ata0:0:0:0): CAM status: ATA Status Error
etc.

appears again. After that I went to sleep.

Now I will go back to OpenSSL and see whats happening.

System is running now OPNsense 16.7.2 (i386/OpenSSL), but:

 0) Logout                             7) Ping host
 1) Assign Interfaces                  Shell
 2) Set interface(s) IP address        9) pfTop
 3) Reset the root password           10) Filter Logs
 4) Reset to factory defaults         11) Restart web interface
 5) Power off system                  12) Upgrade from console
 6) Reboot system                     13) Restore a configuration

Enter an option: (ada0:ata0:0:0:0): DSM TRIM. ACB: 06 01 00 00 00 40 00 00 00 00
 01 00
(ada0:ata0:0:0:0): CAM status: ATA Status Error
(ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
(ada0:ata0:0:0:0): RES: 51 04 20 3c 2f 74 74 00 00 01 00
(ada0:ata0:0:0:0): Retrying command

So I think the switch from OpenSSL to LibreSSL causes these messages in the system log.
Is this solvable in the installation or otherwise?

Greets.

[franco]

If it's not the TRIM support the SD card's first slice is wearing out. LibreSSL will not cause physical errors on your media. Note that your second slice is pretty young in terms of write cycles as opposed to the first one.

[roro]

Recap:
after a few days of hard try and error.
Soekris net5501-70 serial.
Opnsense i386 cf-card.

Findings:
1. because I could not see much via serial (soekris 9600, opnsense 115200). fixed
2. the nanobsd boot menu. Here you can choose:
   1  OPNsense
   2  OPNsense

   F6 PXE
   Boot:  1

3. then the freebsd/opnsense logo will come, but it doesn't display well on serial output, so you miss the freebsd/opnsense menu with Multiuser, Singleuser etc.

4. then I had issues with VI and /etc/fstab. I choose to use EE to edit this file and I was able to add "# notrim".

5. Finally SUCCES. The trim messages are gone. Thanks to Franco's help.

root@opn01:~ # tunefs -p /dev/ad0s1a
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: soft update journaling: (-j)                       disabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  512
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: space to hold for metadata blocks: (-k)            1032
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)                                 OPNsense0

root@opn01:~ # tunefs -p /dev/ada0s1a
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: soft update journaling: (-j)                       disabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  512
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: space to hold for metadata blocks: (-k)            1032
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)                                 OPNsense0
root@opn01:~ #
root@opn01:~ #

[franco]

Hi Rosie,

Sounds good. I'm not entirely sure why the system things the card is TRIM-capable but there were only ever two cases I heard of. I think chemlud had similar issues, we added # notrim upon his request. Maybe he has more background on this?

You can probably improve the console experience in terms of disabling the second console (new in 16.7.3) and changing the serial speed as well to match Soekris, all under System: Settings: Administration.


Cheers,
Franco

[roro]

Hi Franco,
i'm still getting the messages again.
Is it also possible to turn of:
tunefs: soft updates: (-n)                                 enabled
Greets

[roro]

and it driving crazy.

[franco]

Try " # notrim,nosoft" instead.