I’m on my 4th SSD, after the previous 3 all suffered from a large number of uncorrectable SMART errors and then going offline; whereas the 1st SSD served happily under Core 13.3-U2.1 for over 5 years with no errors before dying under Goldeneye.
2 Likes
Hmm, my 10 year old cheap sandisk ssd runs perfectly fine and has been on every version ever released of scale
Were all of there the same make/model/form-factor and what was the capacity and wear level? Also, are you using the boot-pool for something other than a boot drive and possibly the system dataset?
I have been using my SSD for several years now with SCALE. My drive is an old ADATA SP500 128GB drive.
If you are using an NVMe drive, what is the output for smartctl -x /dev/nvme? and look at the Media and Data Integrity Errors. If this shows any value other than 0 then there was an internal issue. If Percent Used is at 100% then that would be bad of course, meaning the drive was all used up.
All of them were Kingston SNS4151S316GD mSATA SSDs. Capacity was 16 GB. The 2nd one had very high time, almost 3000 hours in use, but all the others were low time, below 350 hours. The system dataset is not on the boot pool.
With the 4th one, I ran a conveyance and long SMART test and also a full TRIM before install. I also enabled autotrim for the boot-pool.
That is not high, 125 days?
If you are able to grab the smart data smartctl -x for one of those drive an post it, I’d like to see what might be happening.
Here’s the smartctl -x output for the 3rd SSD that died overnight:
=== START OF INFORMATION SECTION ===
Model Family: SandForce Driven SSDs
Device Model: KINGSTON SNS4151S316GD
Serial Number: 50026B7354014F9F
Firmware Version: S9FM01.6
User Capacity: 16,013,942,784 bytes [16.0 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available
Device is: In smartctl database 7.3/5528
ATA Version is: ACS-3 (minor revision not indicated)
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sun May 17 22:16:03 2026 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM level is: 254 (maximum performance)
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Unavailable
=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers missing
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 30) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 2) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate -O-R-- 100 100 000 - 0/0
2 Throughput_Performance P-S--- 100 100 050 - 0
3 Spin_Up_Time POS--- 100 100 050 - 0
5 Retired_Block_Count PO--C- 100 100 050 - 0
7 Unknown_SSD_Attribute PO-R-- 100 100 050 - 0
8 Unknown_SSD_Attribute P-S--- 100 100 050 - 0
9 Power_On_Hours_and_Msec -O--C- 100 100 000 - 254h+00m+00.000s
10 Unknown_SSD_Attribute PO--C- 100 100 050 - 0
12 Power_Cycle_Count -O--C- 100 100 000 - 57
167 Unknown_Attribute -O---K 100 100 000 - 0
168 Unknown_Attribute -O--C- 100 100 000 - 0
169 Unknown_Attribute ------ 100 100 000 - 55
170 Reserve_Block_Count PO--C- 100 100 010 - 12
172 Erase_Fail_Count -O--CK 100 100 000 - 13
173 Unknown_SandForce_Attr ------ 100 100 000 - 4044947471
175 Program_Fail_Count_Chip PO--C- 100 100 010 - 0
181 Program_Fail_Count -O--C- 100 100 000 - 6
187 Reported_Uncorrect -O--CK 100 100 000 - 144
192 Power-Off_Retract_Count -O--C- 100 100 000 - 4
196 Reallocated_Event_Count ------ 100 100 000 - 22
197 Current_Pending_Sector -O--CK 100 100 000 - 1684300900
199 SATA_CRC_Error_Count -O--CK 100 100 000 - 0
218 Unknown_SandForce_Attr ------ 100 100 000 - 0
233 SandForce_Internal PO--C- 100 100 000 - 121664
240 Unknown_SSD_Attribute PO--C- 100 100 050 - 0
241 Lifetime_Writes_GiB -O--C- 100 100 000 - 117265
242 Lifetime_Reads_GiB -O--C- 100 100 000 - 20470
243 Unknown_Attribute -O--C- 100 100 000 - 110382343808100
244 Unknown_Attribute ------ 100 100 000 - 15
245 Unknown_Attribute ------ 100 100 000 - 61721
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 51 Comprehensive SMART error log
0x03 GPL R/O 64 Ext. Comprehensive SMART error log
0x04 GPL,SL R/O 8 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
SMART Extended Comprehensive Error Log Version: 1 (64 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 14 -
# 2 Short offline Completed without error 00% 52 -
# 3 Extended offline Completed without error 00% 0 -
# 4 Conveyance offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Commands not supported
Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x07 ===== = = === == Solid State Device Statistics (rev 1) ==
0x07 0x008 1 0 --- Percentage Used Endurance Indicator
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value
Pending Defects log (GP Log 0x0c) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 4 3 Transition from drive PhyRdy to drive PhyNRdy
0x000a 4 4 Device-to-host register FISes sent due to a COMRESET
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0010 2 0 R_ERR response for host-to-device data FIS, non-CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x0013 2 0 R_ERR response for host-to-device non-data FIS, non-CRC
This could be off the mark, but I think you’ll see significantly longer life with even slightly larger boot disks. Joe’s and my 128GB pools are doing great; your 16GBs keep having trouble.
As I understand it: Core didn’t write logs to the boot pool, but CE/SCALE does. So wear-leveling becomes an issue.
With a 16GB drive, subtract the size of any boot environments (let’s say you have three of them, totaling just short of 10GB). These cells are claimed, and out of play.
That leaves only 6GB for round-robin wear-leveling. As logs roll through, the same few cells will scrape and scrape until they reach bedrock.
Upgrading to even a 32GB disk would spread out the burden almost 4× before failure.
1 Like
You’re probably right WRT the wear-leveling from systemd journals and middleware audit logs being the cause of my issues. I’ll look into getting larger SSDs.
3 Likes
Good luck! And as a happy not-quite-coincidence, larger SSDs (to a point) will probably also have generationally newer/better wear leveling algorithms and overall durability.
This drive has 117 TB written. The specs for this drive are 150 to 300 TB. So while i feel the drive failed early, i never fully trust the manufacturers specs. They may have been correct for the original batch and then lesser parts were used after that.
1 Like
Also, i don’t particularly like Kingston SSDs. I see a lot of little issues periodically. I prefer Samsung but only because I have not had a bad one to date, and i do have quite a few if them. My ADATA drive is rather old but has been rock solid. Not sure I’d but one today as i know the brand has suffered some. I won’t buy Intel only due to the cost.
2 Likes
Obviously this kind of thing would likely never be a priority for iXSystems since in the enterprise world, the amount of disk wear caused by logs is more-or-less negligible, but on consumer hardware it does add up to a good amount overtime:
I wish there was an option to write logs to RAM only and have them periodically flush to disk a configurable amount, or some reasonable hard-coded amount like every 5-10 minutes, instead of constantly.
Of course this isn’t ideal for when you want to diagnose sudden shutdowns or kernel panics, but it could just be a switch, like “Efficient logs” (with a tooltip that explains what it does), and if you have the need to guarantee logs are written constantly for survivability purposes you could just turn it off.
Previously I’d have just said “whatever I’ll just replace my 500GB boot pool when it dies and it’s inconsequential”; but, now with what’s going on with flash these days… You also can’t even really buy drives less than that capacity anymore either, despite somewhere around 64-256GB being plenty.
Enabling auto-TRIM for the boot pool does seem to help with longevity. Obviously, this can only be done via CLI: zpool set autotrim=on boot-pool.
Hum, I’m not sure why auto-trim would increase longevity. It only erases unused blocks once they have been marked unused vice waiting to the block to be written to later and then it has to erase the block and write to it. Trim speeds up writing.
But, I do not know it all, not by far, and a 16GB SSD going bad as a boot-pool so quickly sounds very wrong. But it did have a lot of TiB written in a very short period of time.
I wish you luck and hope you find the problem.
And you are only using that drive as a boot-pool, correct? Not also as a Swap Drive? I just can’t think of why the drive would have that much data written to it.
Without estimating if those writes are spread across the entire drive or confined to a 20% region.
If the drive controller does what it should, it should be spread across the entire memory region, but the controller may not work very well.
Interesting. Your observation may explain why my 64GB SATADOMs are still fine after all these years while they have generally fallen out of favor. More capacity = more bedrock for wear leveling to grind down.