We’re bringing some SMART options back

Announcements
21

… and at the same time, once management decides that it’s time to make changes to some of the features that are cornerstones to said operational results, it behooves a very careful rollout. This is not libvert vs. incus, or Docker vs. kubernetes.

SMART as a “industry standard” has tremendous improvement potential thanks to OEMs taking significant liberties re: how and what to report, making arbitrary changes, etc. One would think that with only three competitors left that the engineering oligopoly could at least agree on SMART interoperability?

And yet, we have negative disk temperatures and other bugs that may be as annoying as a stuck glove compartment in a car but not as dangerous as cooked brakes. Hence, SMART alarms have always required parsing, usually via experts who had lived with similar drives and know which symptoms were bad (“click of death”) and which could be likely safely ignored.

If ixsystems wants to get into the parsing game by only selectively alarming when a subset of SMART errors crop up, I suggest a fully-fleshed out documentation page that lays out what will and what will not cause a SMART warning with default settings. I’d also document the default cadence of scans, etc. This is not for legal reasons but primarily for customer goodwill.

The team doesn’t have to lay out why it’s made its decisions (ie those may be based on proprietary data) but the end user can then make a informed decision re: whether to solely rely on the ixsystems suggested set of SMART parameters getting monitored (with likely fewer false positives) or whether to rely instead on more inclusive packages like multi-report (that will likely result in more false positives, additional research requirements but also a fuller picture of alleged drive health).

Instead of eliminating GUI scheduling of SMART scans, i suggest starting the likely multi-year process of building and updating a better SMART error parser.

The official support of this SMART error parser could be limited to the series of enterprise drives that ixsystems ships with its server hardware, further aligning engineering efforts with paid customers. WD used to be ixsystems official supplier, and for all I know would be happy to lend engineering help involving their drives.

CE users have to understand that drives that were never shipped by ixsystems may not enjoy 100% parser compatibility. But, CE users can rely on multi-report instead and scrutiny now has a new lifeline (thank you, @pmh for that update).

IIRC, similar efforts were made way back when to bring some bespoke Marvell support to FreeBSD / TrueNAS in order to make the Asrock Rack C2750D4I in the mini series fully reliable. Other chips didn’t get bespoke drivers and hence remained buggy. C’est la vie.

But putting together a better SMART error parser is worlds away from removing a GUI feature and replacing it with an cron job. I’m flummoxed and saddened by the damage left in the wake of this decision, including the banning of at least one user.

4 Likes
22

You worded the comparison in utility of ZFS SMART far better than I did.

I do have issue (as an idiot) with the first part of this quote though. How is the SMART data relevant and up-to-date if (long) SMART tests aren’t being ran on a periodic basis?

My understanding is in 25.10 (prior to any changes following this announcement) there are no built-in automatic SMART tests or schedules defined (I think some users are confused on this point). IIRC when I first installed my SCALE system on 24.x I had to go in and setup the test schedule. There were none out-of-box, and I think this has always been the case.

23

That would be a major investment and its not clear what the return would be. If you have any evidence that this would work, please provide in a feature request. Is there any existing software/project?

We concluded that using SMART and ZFS data is a better solution and provides a catch-all even when drives do not accurately report their own issues.

P.S: We fully support Community use of multi-report, we just want the default behaviour to be simple and reliable.

2 Likes
24

I read this thread and like the comments by everyone. @Captain_Morgan thanks for your explanation, it helps. @Constantin you made a very good point, if there were a document that explained what the program triggered on and just some basics then the group can individually analyze what they feel is better for their use case. I know that I really want to know. And I can see the advantage of analyzing smart and zfs data to do things. I was fiddling with statistics to analyze faults and what I consider “Replace the drive” or do some maintenece, maybe a ZFS issue is responsible. It can be a lot of data to process.

But again, Thanks for the postings, they are helpful.

3 Likes
25

@Captain_Morgan, I am simply repeating back to you what my interpretation of what the announced plan by ixsystems is. If that’s incorrect, I apologize, but some documentation on what your team wants to do, how they will do it, etc. would go a long way to dispel some of my (and likely others) confusion.

If I understand the ixsystems plan correctly, it is to only alarm on a curated subset of SMART errors, perhaps with some inputs from the ZFS subsystem, to determine if a drive is going bad or has gone bad and warrants replacement.

I call that a parser, as it goes beyond just presenting the admin with ZFS and / or SMART data separately and relying on the admin to figure out the issue and make decisions. Instead, the system is proposed to evaluate the seriousness of a SMART issue, followed by curating the disk health / alert GUI output, correct?

Given the usual implementation / debugging / and refinement cycles for new features, I expect this will not be a quick fix but rather take time to verify and document the merits vs. the alternatives, hence the multi-year reference.

1 Like
26

The high level observation is that ZFS is much more reliable at managing drives… it sees data corruption issues, poor drive response times. Regardless of any SMART data it will remove a drive from service. We don’t need to rely on SMART software’s guess on whether there is an issue.

If we were not running ZFS, the equation would be different. SMART data can be useful after a failure to do root-cause analysis.

SMART data can be useful in predicting issues, but it can also be very wrong. We’ve tried to only use the more reliable warnings.. it’s in the source code.

SMART testing is not needed most of the time.. the drives are being well-exercised by ZFS. SMART data is being collected independently of the testing. That is why TrueNAS has not had any SMART tests running by default.

If there are expected drive failures that TrueNAS is missing, we’d like to know about them and improve. The challenge is making sure the data is useful and reliable across vendors and models.

27

All I know is this: since 25.10, two drives in my system (SSD and rust) have gone kaput with no fecking warning whatsoever.

Post failure analysis showed an accumulation of SMART errors and other indicators that signalled imminent failures, yet TrueNAS was apparently oblivious.

Cosmic.

7 Likes
28

I read this comment earlier today and have been trying to think of a gentle way to put this…

…for nearly every point made, it’s not hard to find a user who can easily respond “not in my use case” or “my experience contradicts this”. Some of that has already been commented on by others in this thread.

And again as mentioned by others, the full rationale is not documented. Getting the rationale piecemeal in this thread is not efficient. It needs to be in the docs (“it’s in the source code” is insufficient, but I understand you’re trying to be brief for the purposes of a forum).

I happen to know the documentation repo has been getting a lot of PRs lately, so fingers crossed something is coming.

1 Like
29

That’s great data to have.

Can you share in General channel the examples (and provide a link). What SMART data was warning about and whether it was unique to just those specific drives (out of how many)?

I assume ZFS handled the failures well…if not, what were the issues?

30

SMART data is being continuously collected by each drive..

Whenever a bit flips or a block/sector has to be mapped out, or a retry is needed, the counters get incremented. So, by being in a ZFS pool, there is constant data being collected by the drive.

ZFS also spreads its data out across the drive, so the full breadth of the drive is being tested by normal Reads and Writes.

The Scrutiny App tracks all this data without scheduling tests.

31

Here’s my anecdote. Homelab context, so take it for exactly what it is. I have 9TiB available on the pool. I don’t have hard data, but I’d say I churn a maximum of 25GiB a week. So it’d take me 368 weeks to fill up the pool as it is.

It will take a very long time for block mapping to be a reliable way to “test” whether unallocated space without fault.

“SMART data being continuously collected” IMO is somewhat misleading. Yeah, no kidding it is. But it’s very passive, not active. It’s been said elsewhere that drives are continuously doing testing but it’s not at all clear to me how often that’s being done/tested. If modern drives do “continuous” testing, I’ve never seen that reflected in SMART logs. I have no idea what % of a drive is being tested over any given month. The only logs/evidence I ever have of drives being tested is … you guessed it … SMART self-test logs.

“ZFS also spreads its data out across the drive, so the full breadth of the drive is being tested by normal Reads and Writes.”

A mile wide and an inch deep.

32

We have not said modern that drives do “continuous testing”. They are doing continuous monitoring and reporting of their stats.

Reading and writing to/from the ZFS pool, does indirectly test the drives. Each ZFS drive is split into 200 metaslabs ensuring that data is distributed around the drives.

I agree the SMART logs do report SMART testing. The open question is how often does that help in a TrueNAS system running ZFS? It may provide “confidence”, but does it provide an operational advantage. Can that advantage be measured?

As indicated, we will make it easier to setup SMART tests again. For those that do run it, we’d like to hear where it is providing an advantage.

33

Couple of anecdotes then:

I run two SATADOMs in a boot pool here. Identical 64GB drives, supermicro, click right into the motherboard, as intended. One of them suddenly reportedly 20*C hotter than the neighbor, and also no longer reporting time since last test. A reboot as part of upgrade to 25.10.x.x fixes the issue.

I run helium spinners in my TrueNAS here. SMART reports that the He levels dropped to zero in one drive (Likely determined that indirectly, by monitoring drive motor power being drawn?) Trashed the drive, it was 8 years old already. Reviewed here how it had been constructed / sealed.

In my mobius RAID arrays sits a primitive JM Micro raid controller whose UI currently consist of blinking LEDs and a piezo squeaker. Squeaks and blinking lights indicate that a drive has gone bad. There too, the details of what trigger said squeaks and blinks are undoubtedly in the source code.

But, ixsystems is not shipping inexpensive raid arrays. The target group for ZFS is centered around data integrity. ZFS users likely could have “faster” file systems with the same hardware by avoiding COW file systems, turning off sync writes, implementing multiple “fast” cache levels, etc. But we stick with ZFS because we want the data to be good, not just fast.

That’s why you see multiple users asking for doc details here. Your users are a self selected crowd and when the ixsystems team wants to change what used to be user doctrine (ie SMART test scheduling was included as part of every Free/TrueNAS setup guide ever written) then easily accessible, non-programmer-readable documentation would be helpful.

In other words, it doesn’t matter to the CE crowd that every shipped copy of Free/TrueNAS didn’t have SMART testing enabled by default. What matters is that almost every CE install likely had the smart test schedules enabled and then… management suddenly took away the GUI to do so…. Followed by showering us with anecdotes of how often SMART is not reliable, etc. to justify their decision.

No white papers, no documentation, etc. Instead, we got memes like two admins here running from an enraged Pygmy (?) tribe. Wow. I really felt seen there.

Data integrity is your business, @honeybadger. ZFS is just a means to that end. I suggest the documentation around it should reflect that.

4 Likes
34

But drives do scan in the background. Seagate calls this “Background Media Scan”, and is described here:

See page 11.

That paper is from 2010, so it’s a bit dated.

1 Like
35

I do not have the luxury of time so my comments will be terse/low-quality.

“We have not said modern that drives do “continuous testing”. They are doing continuous monitoring and reporting of their stats.”

A few points in reply:

  • It’s really easy to say that iXsystems hasn’t said anything. We agree. Because the documentation is crap or non-existent.
  • It was said (I’m paraphrasing) in one of the T3 videos that “we want TN to do all the obvious stuff automatically instead of admins needing to configure it”.
  • (Follows the above point) it is not hard to find users confused or misunderstanding the changes in 25.10. I’ve seen people believe that SMART testing is now automatic in 25.10. I never observed iXsystems staff in the wild trying to actively or correct this misconception.
  • Please see iXsystemsChris comments on We’re bringing some SMART options back.
    • “Adding onto this, the concept of “Continuous Background Defect Scanning” was introduced back in SATA spec 2.5. This turns “write” into “write and verify” at the firmware level - so it’s an always-on layer. Similarly, Background Media Scan in SAS drives.”

“Reading and writing to/from the ZFS pool, does indirectly test the drives”

You said it. Indirectly. As I said before, it’s passive and not active.

Your comment is like an automaker saying “driving the car on the public road is testing it”.

“For those that do run it, we’d like to hear where it is providing an advantage.”

I’ve wrote extensively on this. I could probably fill another volume. For the things I link, I encourage you to not just read the OP but dive into the comments where there’s a lot of back-and-forth on the counterarguments. In point form:

  • Who is TrueNAS for, anyway? : truenas
  • A SMART Anecdote : truenas
  • Spare drives should be regularly tested so you know they’re going to fail.
  • All new drives have warranties. Not testing new drives can lead to late discovery of faults/defects and that has a hard $ cost.
  • Here’s a hypothetical I cooked up based on one of the threads I mentioned in the chat:
    • System has a SLOG.
      • SLOG by its nature doesn’t do “deep writes” of the disk.
    • SLOG disks are on the fringe of failing but nothing is testing them to reveal that condition.
    • System has a power outage and writes that need to be played are in the SLOG.
    • System recovers from power outage and the SLOG disks have failed - that’s all it took to push them over the edge.
    • Writes - synchronous writes - that were ack’d back to the storage consumer - can’t be replayed. Data loss.
1 Like
36

I’m short on time. I’ll try to remember to come back and read this. My immediate questions:

  1. How ubiquitous is this technology?
  2. How long does it take to run?
  3. How do we see the results? How do we see statistics/trends in how long it takes for these background scans to complete?
  4. If a disk loses power, does the background scan resume from “checkpoints” akin to ZFS scrubs?
  5. Is it a superficial test across the disk or is it a full surface test like a SMART long?
37

To add to this not only was the GUI taken away, but there is no documentation (that I’ve found) for the parameters/options for calling midclt call disk.smart_test.

38

Page 11 only states that while a BMS is going on, it will not allow a lower power state. This paper is about power, not media scans.

What actually is a BMS? Could it be a S.M.A.R.T. short or long or confidence test which does run in the background and has very low priority over data requests.

Maybe I’m wrong. I’m on the road so i didn’t check my info.

1 Like
39

the term for this is un-shittification.

2 Likes
40

All the to-do about SMART testing and the rest really boils down to one thing, and that’s the importance of proactively, and regularly, testing the unused blocks on the storage media–the used blocks will be addressed by ZFS (during scrubs if not otherwise), and will either throw a “bad sector” SMART attribute (which iX says TrueNAS still monitors), a ZFS error, or perhaps both.

But what process tests the unused sectors, if not SMART self-tests? I’ll admit I haven’t read every post in every thread, but I haven’t seen anyone from iX even begin to address this question until the mention of background media scanning the other day in this thread. Is that your answer?

6 Likes