Opnsense file corruption

[zackboll]

Hi,

I believe I have a failing SSD so I am currently in the process of buying a new one.

A number of my opnsense packages were corrupted, but I was able to resolve most issues by reinstalling the package.  I am currently stuck on the package (cc) c compiler.  I keep getting pager fault errors, and I cannot figure out how to reinstall the package that provides it.

root@OPNsense:~ # pkg which /usr/bin/cc
/usr/bin/cc was not found in the database

Output from dmesg for example of error:
[87] vm_fault: pager read error, pid 14115 (cc)
[87] pid 14115 (cc), jid 0, uid 0: exited on signal 10 (no core dump - bad address)
[116] vm_fault: pager read error, pid 29661 (cc)
[116] pid 29661 (cc), jid 0, uid 0: exited on signal 10 (no core dump - bad address)
[176] vm_fault: pager read error, pid 75465 (cc)
[176] pid 75465 (cc), jid 0, uid 0: exited on signal 10 (no core dump - bad address)

I was getting many more vm_fault before I reinstalled all packages (with pkg upgrade -f).  Any idea how I can reinstall /usr/bin/cc?

When I try to use executable:
root@OPNsense:~ # cc --version
Bus error

Worse case scenario, I can start from a fresh install (on new SSD) and try to restore from backup configuration.  I am currently running on the latest version 25.x series, the thought crossed my mind about trying to do in-place upgrade to 26.x to see if that fixes my corrupted filesystem.  Since my firewall is currently partly operational, and I have internet, I figure it may be better to wait until my new SSD arrives. 

Any ideas how to resolve the vm_fault errors stemming from the corrupted cc binary would be appreciated.

Thanks,
Zack

[franco]

/usr/bin/cc is part of the "base" "package". You can reinstall it from the firmware GUI packages tab or

# opnsense-update -fb
(reboot)


Cheers,
Franco

[zackboll]

Thank you, that worked!

[nero355]

I believe I have a failing SSD

if you think that then you should check the S.M.A.R.T. data with either smartctl or nvmecontrol first and then look further :)

See :
- https://man.freebsd.org/cgi/man.cgi?query=smartctl
- https://man.freebsd.org/cgi/man.cgi?query=nvmecontrol

[zackboll]

I did install smartmontools and do an extended test.  It says passed.  My new ssd arrived today, and I am considering whether I want to upgrade to it.

Code Select Expand

root@OPNsense:~ # smartctl -a /dev/ada0
smartctl 7.5 2025-04-30 r5714 [FreeBSD 14.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     M900-64
Serial Number:    AA000000000000002405
Firmware Version: U0330A0
User Capacity:    64,023,257,088 bytes [64.0 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      mSATA
TRIM Command:     Available
Device is:        Not in smartctl database 7.5/5706
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 22 20:09:04 2026 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (  120) seconds.
Offline data collection
capabilities: (0x11) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities:            (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: (   2) minutes.
Extended self-test routine
recommended polling time: (  10) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0032   100   100   050    Old_age   Always       -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   050    Old_age   Always       -       9
  9 Power_On_Hours          0x0032   100   100   050    Old_age   Always       -       35721
 12 Power_Cycle_Count       0x0032   100   100   050    Old_age   Always       -       55
160 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       2
161 Unknown_Attribute       0x0033   100   100   050    Pre-fail  Always       -       86
163 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       12
164 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       4550244
165 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       21380
166 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       3296
167 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       4638
168 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       5050
169 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       9
175 Program_Fail_Count_Chip 0x0032   100   100   050    Old_age   Always       -       0
176 Erase_Fail_Count_Chip   0x0032   100   100   050    Old_age   Always       -       0
177 Wear_Leveling_Count     0x0032   100   100   050    Old_age   Always       -       0
178 Used_Rsvd_Blk_Cnt_Chip  0x0032   100   100   050    Old_age   Always       -       9
181 Program_Fail_Cnt_Total  0x0032   100   100   050    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   050    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   050    Old_age   Always       -       48
194 Temperature_Celsius     0x0022   100   100   050    Old_age   Always       -       40
195 Hardware_ECC_Recovered  0x0032   100   100   050    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   100   100   050    Old_age   Always       -       2
197 Current_Pending_Sector  0x0032   100   100   050    Old_age   Always       -       9
198 Offline_Uncorrectable   0x0032   100   100   050    Old_age   Always       -       2
199 UDMA_CRC_Error_Count    0x0032   100   100   050    Old_age   Always       -       19
232 Available_Reservd_Space 0x0032   100   100   050    Old_age   Always       -       86
241 Total_LBAs_Written      0x0030   100   100   050    Old_age   Offline      -       3227517
242 Total_LBAs_Read         0x0030   100   100   050    Old_age   Offline      -       31273
245 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       6532568

SMART Error Log Version: 1
ATA Error Count: 19 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 15 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 51 00 00 00 00 00  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d0 01 00 4f c2 40 08      00:00:00.000  SMART READ DATA
  b0 d1 01 01 4f c2 40 08      00:00:00.000  SMART READ ATTRIBUTE THRESHOLDS [OBS-4]
  b0 da 00 00 4f c2 40 08      00:00:00.000  SMART RETURN STATUS
  b0 d5 01 00 4f c2 40 08      00:00:00.000  SMART READ LOG
  b0 d5 01 01 4f c2 40 08      00:00:00.000  SMART READ LOG

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     35703         -
# 2  Short offline       Completed without error       00%     35698         -

Selective Self-tests/Logging not supported

The above only provides legacy SMART information - try 'smartctl -x' for more


[zackboll]

Code Select Expand

root@OPNsense:~ # smartctl -x /dev/ada0
smartctl 7.5 2025-04-30 r5714 [FreeBSD 14.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     M900-64
Serial Number:    AA000000000000002405
Firmware Version: U0330A0
User Capacity:    64,023,257,088 bytes [64.0 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      mSATA
TRIM Command:     Available
Device is:        Not in smartctl database 7.5/5706
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 22 20:21:33 2026 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM level is:     128 (minimum power consumption without standby)
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, frozen [SEC2]

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (  120) seconds.
Offline data collection
capabilities: (0x11) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities:            (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: (   2) minutes.
Extended self-test routine
recommended polling time: (  10) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     -O--CK   100   100   050    -    0
  5 Reallocated_Sector_Ct   -O--CK   100   100   050    -    9
  9 Power_On_Hours          -O--CK   100   100   050    -    35721
 12 Power_Cycle_Count       -O--CK   100   100   050    -    55
160 Unknown_Attribute       -O--CK   100   100   050    -    2
161 Unknown_Attribute       PO--CK   100   100   050    -    86
163 Unknown_Attribute       -O--CK   100   100   050    -    12
164 Unknown_Attribute       -O--CK   100   100   050    -    4550458
165 Unknown_Attribute       -O--CK   100   100   050    -    21382
166 Unknown_Attribute       -O--CK   100   100   050    -    3296
167 Unknown_Attribute       -O--CK   100   100   050    -    4638
168 Unknown_Attribute       -O--CK   100   100   050    -    5050
169 Unknown_Attribute       -O--CK   100   100   050    -    9
175 Program_Fail_Count_Chip -O--CK   100   100   050    -    0
176 Erase_Fail_Count_Chip   -O--CK   100   100   050    -    0
177 Wear_Leveling_Count     -O--CK   100   100   050    -    0
178 Used_Rsvd_Blk_Cnt_Chip  -O--CK   100   100   050    -    9
181 Program_Fail_Cnt_Total  -O--CK   100   100   050    -    0
182 Erase_Fail_Count_Total  -O--CK   100   100   050    -    0
192 Power-Off_Retract_Count -O--CK   100   100   050    -    48
194 Temperature_Celsius     -O---K   100   100   050    -    40
195 Hardware_ECC_Recovered  -O--CK   100   100   050    -    0
196 Reallocated_Event_Count -O--CK   100   100   050    -    2
197 Current_Pending_Sector  -O--CK   100   100   050    -    9
198 Offline_Uncorrectable   -O--CK   100   100   050    -    2
199 UDMA_CRC_Error_Count    -O--CK   100   100   050    -    19
232 Available_Reservd_Space -O--CK   100   100   050    -    86
241 Total_LBAs_Written      ----CK   100   100   050    -    3227582
242 Total_LBAs_Read         ----CK   100   100   050    -    31273
245 Unknown_Attribute       -O--CK   100   100   050    -    6532769
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      1  Comprehensive SMART error log
0x03       GPL     R/O      1  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xde       GPL     VS       8  Device vendor specific log

SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
Device Error Count: 19 (device log contains only the most recent 4 errors)
CR     = Command Register
FEATR  = Features Register
COUNT  = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
LH     = LBA High (was: Cylinder High) Register    ]   LBA
LM     = LBA Mid (was: Cylinder Low) Register      ] Register
LL     = LBA Low (was: Sector Number) Register     ]
DV     = Device (was: Device/Head) Register
DC     = Device Control Register
ER     = Error register
ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 19 [2] log entry is empty
Error 18 [1] log entry is empty
Error 17 [0] log entry is empty
Error 16 [3] occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 51 00 00 00 00 00 00 00 00 00 00  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  b0 00 d1 01 01 00 00 4f 00 c2 01 40 08     00:00:00.000  SMART READ ATTRIBUTE THRESHOLDS [OBS-4]
  2f 00 00 01 01 00 00 00 00 00 03 40 08     00:00:00.000  READ LOG EXT
  2f 00 00 01 01 00 00 00 00 00 00 40 08     00:00:00.000  READ LOG EXT
  b0 00 d5 01 01 00 00 4f 00 c2 00 40 08     00:00:00.000  SMART READ LOG
  b0 00 da 00 00 00 00 4f 00 c2 00 40 08     00:00:00.000  SMART RETURN STATUS

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     35703         -
# 2  Short offline       Completed without error       00%     35698         -

Selective Self-tests/Logging not supported

SCT Commands not supported

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4              55  ---  Lifetime Power-On Resets
0x01  0x010  4           35721  ---  Power-on Hours
0x01  0x018  6      1069436808  ---  Logical Sectors Written
0x01  0x020  6         7559555  ---  Number of Write Commands
0x01  0x028  6      2049552508  ---  Logical Sectors Read
0x01  0x030  6          323569  ---  Number of Read Commands
0x07  =====  =               =  ===  == Solid State Device Statistics (rev 1) ==
0x07  0x008  1              91  ---  Percentage Used Endurance Indicator
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  4            0  Command failed due to ICRC error
0x0002  4            0  R_ERR response for data FIS
0x0005  4            0  R_ERR response for non-data FIS
0x000a  4           21  Device-to-host register FISes sent due to a COMRESET



[nero355]

I did install smartmontools and do an extended test.  It says passed.

In my experience those are useless : You have to check the output yourself :)

My new ssd arrived today, and I am considering whether I want to upgrade to it.

I would definitely replace the current one :

Code Select Expand

Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0032   100   100   050    Old_age   Always       -       9
178 Used_Rsvd_Blk_Cnt_Chip  0x0032   100   100   050    Old_age   Always       -       9
196 Reallocated_Event_Count 0x0032   100   100   050    Old_age   Always       -       2
197 Current_Pending_Sector  0x0032   100   100   050    Old_age   Always       -       9
198 Offline_Uncorrectable   0x0032   100   100   050    Old_age   Always       -       2
199 UDMA_CRC_Error_Count    0x0032   100   100   050    Old_age   Always       -       19
232 Available_Reservd_Space 0x0032   100   100   050    Old_age   Always       -       86

And also please keep your eyes on UDMA_CRC_Error_Count when you start using the new SSD because this value usually means connection issues, either the cable or the controller side of the SSD and sometimes even the SATA Controller on your Motherboard !!

Since this is a mSATA model I guess you can ignore the cable part :)