Opnsense file corruption

[zackboll]

Hi,

I believe I have a failing SSD so I am currently in the process of buying a new one.

A number of my opnsense packages were corrupted, but I was able to resolve most issues by reinstalling the package.  I am currently stuck on the package (cc) c compiler.  I keep getting pager fault errors, and I cannot figure out how to reinstall the package that provides it.

root@OPNsense:~ # pkg which /usr/bin/cc
/usr/bin/cc was not found in the database

Output from dmesg for example of error:
[87] vm_fault: pager read error, pid 14115 (cc)
[87] pid 14115 (cc), jid 0, uid 0: exited on signal 10 (no core dump - bad address)
[116] vm_fault: pager read error, pid 29661 (cc)
[116] pid 29661 (cc), jid 0, uid 0: exited on signal 10 (no core dump - bad address)
[176] vm_fault: pager read error, pid 75465 (cc)
[176] pid 75465 (cc), jid 0, uid 0: exited on signal 10 (no core dump - bad address)

I was getting many more vm_fault before I reinstalled all packages (with pkg upgrade -f).  Any idea how I can reinstall /usr/bin/cc?

When I try to use executable:
root@OPNsense:~ # cc --version
Bus error

Worse case scenario, I can start from a fresh install (on new SSD) and try to restore from backup configuration.  I am currently running on the latest version 25.x series, the thought crossed my mind about trying to do in-place upgrade to 26.x to see if that fixes my corrupted filesystem.  Since my firewall is currently partly operational, and I have internet, I figure it may be better to wait until my new SSD arrives. 

Any ideas how to resolve the vm_fault errors stemming from the corrupted cc binary would be appreciated.

Thanks,
Zack

[franco]

/usr/bin/cc is part of the "base" "package". You can reinstall it from the firmware GUI packages tab or

# opnsense-update -fb
(reboot)


Cheers,
Franco

[zackboll]

Thank you, that worked!

[nero355]

I believe I have a failing SSD

if you think that then you should check the S.M.A.R.T. data with either smartctl or nvmecontrol first and then look further :)

See :
- https://man.freebsd.org/cgi/man.cgi?query=smartctl
- https://man.freebsd.org/cgi/man.cgi?query=nvmecontrol

[zackboll]

I did install smartmontools and do an extended test.  It says passed.  My new ssd arrived today, and I am considering whether I want to upgrade to it.

Code Select Expand

root@OPNsense:~ # smartctl -a /dev/ada0
smartctl 7.5 2025-04-30 r5714 [FreeBSD 14.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     M900-64
Serial Number:    AA000000000000002405
Firmware Version: U0330A0
User Capacity:    64,023,257,088 bytes [64.0 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      mSATA
TRIM Command:     Available
Device is:        Not in smartctl database 7.5/5706
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 22 20:09:04 2026 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (  120) seconds.
Offline data collection
capabilities: (0x11) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities:            (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: (   2) minutes.
Extended self-test routine
recommended polling time: (  10) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0032   100   100   050    Old_age   Always       -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   050    Old_age   Always       -       9
  9 Power_On_Hours          0x0032   100   100   050    Old_age   Always       -       35721
 12 Power_Cycle_Count       0x0032   100   100   050    Old_age   Always       -       55
160 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       2
161 Unknown_Attribute       0x0033   100   100   050    Pre-fail  Always       -       86
163 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       12
164 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       4550244
165 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       21380
166 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       3296
167 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       4638
168 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       5050
169 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       9
175 Program_Fail_Count_Chip 0x0032   100   100   050    Old_age   Always       -       0
176 Erase_Fail_Count_Chip   0x0032   100   100   050    Old_age   Always       -       0
177 Wear_Leveling_Count     0x0032   100   100   050    Old_age   Always       -       0
178 Used_Rsvd_Blk_Cnt_Chip  0x0032   100   100   050    Old_age   Always       -       9
181 Program_Fail_Cnt_Total  0x0032   100   100   050    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   050    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   050    Old_age   Always       -       48
194 Temperature_Celsius     0x0022   100   100   050    Old_age   Always       -       40
195 Hardware_ECC_Recovered  0x0032   100   100   050    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   100   100   050    Old_age   Always       -       2
197 Current_Pending_Sector  0x0032   100   100   050    Old_age   Always       -       9
198 Offline_Uncorrectable   0x0032   100   100   050    Old_age   Always       -       2
199 UDMA_CRC_Error_Count    0x0032   100   100   050    Old_age   Always       -       19
232 Available_Reservd_Space 0x0032   100   100   050    Old_age   Always       -       86
241 Total_LBAs_Written      0x0030   100   100   050    Old_age   Offline      -       3227517
242 Total_LBAs_Read         0x0030   100   100   050    Old_age   Offline      -       31273
245 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       6532568

SMART Error Log Version: 1
ATA Error Count: 19 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 15 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 51 00 00 00 00 00  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d0 01 00 4f c2 40 08      00:00:00.000  SMART READ DATA
  b0 d1 01 01 4f c2 40 08      00:00:00.000  SMART READ ATTRIBUTE THRESHOLDS [OBS-4]
  b0 da 00 00 4f c2 40 08      00:00:00.000  SMART RETURN STATUS
  b0 d5 01 00 4f c2 40 08      00:00:00.000  SMART READ LOG
  b0 d5 01 01 4f c2 40 08      00:00:00.000  SMART READ LOG

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     35703         -
# 2  Short offline       Completed without error       00%     35698         -

Selective Self-tests/Logging not supported

The above only provides legacy SMART information - try 'smartctl -x' for more


[zackboll]

Code Select Expand

root@OPNsense:~ # smartctl -x /dev/ada0
smartctl 7.5 2025-04-30 r5714 [FreeBSD 14.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     M900-64
Serial Number:    AA000000000000002405
Firmware Version: U0330A0
User Capacity:    64,023,257,088 bytes [64.0 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      mSATA
TRIM Command:     Available
Device is:        Not in smartctl database 7.5/5706
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 22 20:21:33 2026 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM level is:     128 (minimum power consumption without standby)
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, frozen [SEC2]

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (  120) seconds.
Offline data collection
capabilities: (0x11) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities:            (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: (   2) minutes.
Extended self-test routine
recommended polling time: (  10) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     -O--CK   100   100   050    -    0
  5 Reallocated_Sector_Ct   -O--CK   100   100   050    -    9
  9 Power_On_Hours          -O--CK   100   100   050    -    35721
 12 Power_Cycle_Count       -O--CK   100   100   050    -    55
160 Unknown_Attribute       -O--CK   100   100   050    -    2
161 Unknown_Attribute       PO--CK   100   100   050    -    86
163 Unknown_Attribute       -O--CK   100   100   050    -    12
164 Unknown_Attribute       -O--CK   100   100   050    -    4550458
165 Unknown_Attribute       -O--CK   100   100   050    -    21382
166 Unknown_Attribute       -O--CK   100   100   050    -    3296
167 Unknown_Attribute       -O--CK   100   100   050    -    4638
168 Unknown_Attribute       -O--CK   100   100   050    -    5050
169 Unknown_Attribute       -O--CK   100   100   050    -    9
175 Program_Fail_Count_Chip -O--CK   100   100   050    -    0
176 Erase_Fail_Count_Chip   -O--CK   100   100   050    -    0
177 Wear_Leveling_Count     -O--CK   100   100   050    -    0
178 Used_Rsvd_Blk_Cnt_Chip  -O--CK   100   100   050    -    9
181 Program_Fail_Cnt_Total  -O--CK   100   100   050    -    0
182 Erase_Fail_Count_Total  -O--CK   100   100   050    -    0
192 Power-Off_Retract_Count -O--CK   100   100   050    -    48
194 Temperature_Celsius     -O---K   100   100   050    -    40
195 Hardware_ECC_Recovered  -O--CK   100   100   050    -    0
196 Reallocated_Event_Count -O--CK   100   100   050    -    2
197 Current_Pending_Sector  -O--CK   100   100   050    -    9
198 Offline_Uncorrectable   -O--CK   100   100   050    -    2
199 UDMA_CRC_Error_Count    -O--CK   100   100   050    -    19
232 Available_Reservd_Space -O--CK   100   100   050    -    86
241 Total_LBAs_Written      ----CK   100   100   050    -    3227582
242 Total_LBAs_Read         ----CK   100   100   050    -    31273
245 Unknown_Attribute       -O--CK   100   100   050    -    6532769
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      1  Comprehensive SMART error log
0x03       GPL     R/O      1  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xde       GPL     VS       8  Device vendor specific log

SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
Device Error Count: 19 (device log contains only the most recent 4 errors)
CR     = Command Register
FEATR  = Features Register
COUNT  = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
LH     = LBA High (was: Cylinder High) Register    ]   LBA
LM     = LBA Mid (was: Cylinder Low) Register      ] Register
LL     = LBA Low (was: Sector Number) Register     ]
DV     = Device (was: Device/Head) Register
DC     = Device Control Register
ER     = Error register
ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 19 [2] log entry is empty
Error 18 [1] log entry is empty
Error 17 [0] log entry is empty
Error 16 [3] occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  04 -- 51 00 00 00 00 00 00 00 00 00 00  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  b0 00 d1 01 01 00 00 4f 00 c2 01 40 08     00:00:00.000  SMART READ ATTRIBUTE THRESHOLDS [OBS-4]
  2f 00 00 01 01 00 00 00 00 00 03 40 08     00:00:00.000  READ LOG EXT
  2f 00 00 01 01 00 00 00 00 00 00 40 08     00:00:00.000  READ LOG EXT
  b0 00 d5 01 01 00 00 4f 00 c2 00 40 08     00:00:00.000  SMART READ LOG
  b0 00 da 00 00 00 00 4f 00 c2 00 40 08     00:00:00.000  SMART RETURN STATUS

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     35703         -
# 2  Short offline       Completed without error       00%     35698         -

Selective Self-tests/Logging not supported

SCT Commands not supported

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4              55  ---  Lifetime Power-On Resets
0x01  0x010  4           35721  ---  Power-on Hours
0x01  0x018  6      1069436808  ---  Logical Sectors Written
0x01  0x020  6         7559555  ---  Number of Write Commands
0x01  0x028  6      2049552508  ---  Logical Sectors Read
0x01  0x030  6          323569  ---  Number of Read Commands
0x07  =====  =               =  ===  == Solid State Device Statistics (rev 1) ==
0x07  0x008  1              91  ---  Percentage Used Endurance Indicator
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  4            0  Command failed due to ICRC error
0x0002  4            0  R_ERR response for data FIS
0x0005  4            0  R_ERR response for non-data FIS
0x000a  4           21  Device-to-host register FISes sent due to a COMRESET