Multi-Report

joeschmuck · March 29, 2025, 12:39pm

Yes, those values in the SMART data are very strange. Just keep an eye on it. The value I would keep an eye on as well is ID5 (of course). If this value increments at all, then you know you have some bad blocks, likely worn out from erasure.

Another thing you might check on is the firmware, maybe it is old and there is an update?

If something strange keeps happening, reach out, I’d like to know what the values are again.

joeschmuck · March 30, 2025, 2:16pm

This problem has been identified as a SSD Firmware issue. The drive SMART Self-test did record the timestamp as 16445 while Power On Hours was recorded correctly. The problem was raised with the drive manufacturer but who knows if/when they will respond, and if it will be "yes, we are developing a new firmware, or Sorry - so sad).

unseen · March 30, 2025, 2:49pm

I know which option I’d be prepared to bet any money on!

unseen · March 31, 2025, 11:36am

I received a reply from ADATA who simply gave me a link to their drive software and suggested that if the drive is faulty, I contact the place I bought it from.
Completely unhelpful as expected.

joeschmuck · March 31, 2025, 11:40am

Unfortunate but as we both expected.

E_B · March 31, 2025, 6:00pm

TrueNAS sent me an alert today for this SSD, telling me
Device: /dev/sdd [SAT], Failed SMART usage Attribute: 202 Percent_Lifetime_Remain..

So I decided to run multi-report.sh which tells me:
202 Percent_Lifetime_Remain 0x0030 001 001 001 Old_age Offline FAILING_NOW 99

For completeness I looked at the TrueNAS UI for the SMARt tests for this drive, all OK:

and the TrueNAS UI for the SSD itself:

which I think is interesting. I conclude that it means that my SSD is clearly ageing rapidly but hasn’t got any faults as such.

A lot like me (apart from the lack of faults).

joeschmuck · March 31, 2025, 6:21pm

Is it still under warranty? With that few hours, that is definitely a faulty drive.

E_B · March 31, 2025, 8:12pm

It is! I’m glad you asked the question - it hadn’t occurred to me that it might still be but Crucial recognise the serno. as being valid and eligible for a replacement, so I have started the RMA process.

oxyde · April 4, 2025, 3:27pm

i take advantage of the multi report thread to warn multi report users to a possible “breaking change” on the next version of multireport_sendemail.py, that will be released next days.
With the 1.10 release, i consolidated the mechanisms to put an override on the mail.config data about “from email” and “from name” in 2 ways:

passing those input as args to the script (for stand alone users, not rilevant to multi report users)
keeping this data from the default multi report config

This setup make it possible to customize better to fit some scenarios where just relying on the mail.config is limitative, in a centralized way for every provider.
The problem is that if some “unwanted” override (that previously wasn’t handled in a route) in the multi report config is set, the email send can fail.

What you should do to avoid problem BEFORE update the script:

Gmail users: you shouldn’t be impacted anyway, as before incorrect setup will be still overwritten by gmail itself with your account default. In my experience, i love the auto-generated alias (myaddress+something@gmail.com) to keep track of multi report emails
SMTP users: ensure that the From field in your config is an authorized address/alias, because certain provider can deny the connection; also, besides supported, i reccomend anyway to not use different domain to avoid spam problem
Outlook users: ensure that the From field in your config is an authorized address/alias, because i can say for sure that the email will fail

If someone want help testing the new version before the official release, just dm me

Theo · April 27, 2025, 7:40pm

Just wanted to stop by and thank @joeschmuck for this awesome script, I found my drives are all scammed today with the script. The first 8 were purchase in October and the second 8 were purchased in December, hopefully I can recover some money.

Thanks Joe!

joeschmuck · April 28, 2025, 1:24am

Sorry you only found out now.

Okedokey · April 28, 2025, 2:46am

Can you please explain what you mean by “scammed” and how you worked this out?

Theo · April 28, 2025, 3:40am

Multi report added a feature for Seagate drives to check FARM power on time (immutable) against SMART power-on time. If FARM is a lot longer than SMART, the brand new drives you bought were refurbished and the SMART statistics reset,.

Okedokey · April 28, 2025, 3:55am

Thanks for explaining that. I guess that is not a function that can be used for WD drives?

Theo · April 28, 2025, 3:57am

I think this is Seagate only.

joeschmuck · April 28, 2025, 11:37am

It is not. FARM is Seagate only. If WD or any other drive manufacturer has a method to validate the SMART data has not been tampered with, let me know and if I can include that into the script, I most certainly will.

Let’s keep in mind that FARM data can and has been changed on some of the drives reported already, so the check is only as good as the data. Hum… can I automatically check the warranty from a script? That would be a great indicator as well.

joeschmuck · May 15, 2025, 7:38pm

I have posted the Beta versions of Multi-Report v3.17 and Drive-Selftest v1.05.

Multi-Report_v3.17_Beta.zip file. Read the READ-ME file!, it is important.

I will be going on vacation starting Saturday, be gone a week, so if you have any problems, fee free to reach out but I may not answer right away, and you may have to wait until I return from vacation for me to find a solution.

No user manuals, that is the only reason I have not released these versions yet as I need to update the user manuals. My 3D printer has occupied a lot of my time .

Maybe I can get most of those manuals updated while on vacation during some of the slow periods. Use the -help switch, it has a bit of info there. Also read the Change Logs at the front of the scripts, that tells a lot.

Cheers!

StevenMcE · May 15, 2025, 9:18pm

Can you put the readme file outside of the zip?

joeschmuck · May 15, 2025, 9:48pm

READ_ME.txt (592 Bytes)

joeschmuck · June 8, 2025, 3:23pm

I have just uploaded the new versions of Multi-Report v3.17 and Drive-Selftest v1.05, including User Guides and Quick Start Guide to the GitHub account.

What changes were made?
The main change is the ability to possibly test more USB connected type drives, or drives that must use a special interface in smartctl so you can add that parameter now.

In Multi-Report, the Zpool Total Read/Written values were off, just a little bit but it was pointed out to me so it was fixed.

Here are the change logs:
Multi-Report:

# V3.17 (07 June 2025)
#
#  - Added F.A.R.M. data to -dump files.
#  - Added F.A.R.M. Offset to the Custom Drive Configuration.  You may leave at default or ignore the drive.
#  - Updated F.A.R.M. check to now include Reallocated Sectors, Head Flight Hours, Load Cycle Count, and Power Cycle Count, Write Power On Head 0 and Head 1, besides Power on Hours. 
#  - Updated -dump switch to either `-dump` = Send all dump data to user, or `-dump email` which will send all dump data,
#    ---- except the TrueNAS Config file to the user and Joe, and added more Debug data.
#  - Fixed sending dump data SMART '-a' and '-x' data which was absent during data collection.
#  - Added sending smartctl --scan data to dump file.
#  - Changed Seagate "SCAM" to "FARM".
#  - Made some clarifications in the wording of text.
#  - Made adjustments yet again for iXsystems changing sending emails.
#  - Fixed Zpool TDW/TDR values.  Apparently a previous change broke it.
#  - Removed Un-needed TrueNAS Configuration file ZIP encryption.  The file is already encrypted.
#  - Removed installing 7-Zip function.  Retained removal should someone need it still.
#  - Now save Old_multi_report_config.txt in script directory during an upgrade.
#  - Added --debug_enabled for writing the sendemail.py log file always.
#  - Allow an Update is Available message when using '-m' switch.
#  - Added TrueNAS Version Name to report.
#  - Added Update Log file to change the email subject line to report an update is available.
#  - Added Percentage Values for the 30-Day Read/Written Column.  Added Color - Must be manually changed in script variables.
#  - Updated obtaining Power On Hours as some drives (very few) did not post using standardized value.  Hopefully this didn't break anything.
#  - Updated statisticaldatafile to not record "<br>SMR" when an SMR drive is detected.  This did not cause a problem, it just annoyed me once it was discovered.

Drive-Selftest:

# Version 1.05 (07 June 2025)
#
# - Updated the smartctl interface connection to roll through several more variations if the default fails to work.
#   ---- This did impact a few people but it has been fixed now. This also "should" open up options for USB connected devices.
# - Added '--scan' output to a file for data collection.
# - Updated Debugging Data for Troubleshooting and analysis.
# - Updated '-help' information.
# - Fixed potential drive not being Long tested if it had a similar name to a Short test drive (da1 and da11 was the noted problem).
# - Updated Debug to be enabled during a Multi-report -dump switch.
# - Added RESILVER/SCRUB Override for SMART Long tests (by request).
# - Converted options from 'true/false' to 'enable/disable' to make more sense.
# - Fixed reading the multi_report_config.txt file earlier in the script execution.

I will be setting up the notifications shortly so people will see the pending updates in the scripts.

As usual, please let me know if any problems arise.

EDIT The multi_report_config.txt file will have some changes, not backwards compatible, ensure you keep a copy of your old config file. It will be created automatically and as an email attachment, just in-case you need to roll back. Hopefully that will not be nessessary.