Multi-Report

I’ve not done anything weird with drives - for a start I wouldn’t know how to. I’ve just popped them in and added them to pools, and followed the prompts, so unless TrueNas has done something odd itself, it’s probably duff info.

I wouldn’t expect that, but I do expect Dragonfish, or possibly just Debian to be the problem I have to overcome. And partition 4, that is odd, maybe it is the script. Once I have all the data I can determine that.

1 Like

Should be with you any time now. And thanks for your help!

The script works fine from every aspect except on SCALE and then ONLY the text sections “Drives for this pool are listed below:”, specifically the list may list or not list a gptid in the group. “Drives for this pool are listed below:” is a convenience thing to help someone figure out which drive to pull, so you may need to go old school for a day or two. :thinking: It is easy to verify by comparing the list above it to the list below it, the gptid matches or it doesn’t. Someone might say “If you created the first list, why would the second one be wrong?”. The first is a simple output of ‘zpool status -v’, the second tries to identify all the drives by association to the pool using ‘lsblk’. I have yet another way to fail, but I think I have the fix rolling around in my brain.

Again, the script runs fine for everything else, it will let you know of a problem, it can test NVMe drives (provided they support self-test), but I’m hoping to have the next fix out in 2 days. And @Vollans , please let me know of any other fix I need to make before Saturday. But I’m glad some folks are actually noticing things, means it is being used.

Cheers

1 Like

On a different note, I am wondering what is the execution time of other users; mine is 40s.

For my main system (in my sig):
Execution Time: 9 Minutes : 25 Seconds

For my parents’ microserver gen 8 with two drives in a mirror:
Execution Time: 2 Minutes : 52 Seconds

@dan Is that with waiting for the NVMe Self-test to complete? If yes, have you timed how long it takes to run both short and long test and then adjusted the time delays in the config file? It takes me just a little longer to test 4TB. I considered polling the NVMe drives to see when they completed but I figured TrueNAS would implement NVMe drive testing fairly quickly. I was wrong.

My execution time on my NVMe system takes 18 seconds and that is with 1 spinning rust laptop drive and two old SSDs added for testing purposes. Subtract 3 seconds without those drives. I am not waiting for the NVMe drive test to complete which of course causes a delay. 2 Minutes for short, default 10 minutes for long.

My old trusty spinning rust server takes 13 seconds. Only 4 drives.

Both systems running on ESXi with 16GB RAM allocated.

At this point I’m not worried about speed, however I plan to reduce polling the drives to once each, right now I poll then 4 or 5 times.

Here is a heads-up copy of Multi-Report 3.0.6 which fixes the driving issue for this release, the gptid numbers could be incorrect in SCALE. That should be fixed now.

I plan to post this to GitHub tomorrow for those who get notified there is an update and/or chose automatic update. Also I’m adding a Anatomy/Quick Start Guide, hopefully it is useful, if not, I’ll take comments Good or Bad.

Anatomy of Multi-Report Email Github Link

Multi-Report v3.0.6 Github Link

Take care,
-Mark (aka. Joe)

4 Likes

Now thou name is widely known to all instead of being a secretive whisper amongst few chosen ones.

1 Like

So Multi-report issuing a warning is “the bad news (= anti-evangel) according to Mark…” :imp:

1 Like

NVMe Custom Selections - SMARTMONTOOLS Version 7.4

Prior to smartmontools v7.4 the NVMe SMART tests were not
available on ‘smartctl’ and TrueNAS CORE 13.0-U6.1 currently
supports version 7.2. TrueNAS SCALE (23.10.2) does support
version 7.4, however TrueNAS SCALE GUI does not allow the
option to configure running a S.M.A.R.T. test on NVMe drives.

If your version of TrueNAS does not have the option to
setup SMART tests for NVMe drives, Enable this feature.

This section compensates for these issues until they are
corrected in TrueNAS.

Run NVME SMART Test if smartmontools 7.4 is installed
Current: enable
(‘e’ = enable, ‘d’ = disable, Any key to accept

I’m still on version 22.x.x and don’t have smartmontools 7.4. The current settings imply that NVME tests will run only “if smartmontools 7.4 is installed.” However, enabling this setting causes NVME tests to run regardless of the version, leading to errors if version 7.4 is not installed. I suggest either updating the wording to reflect this behavior or implementing a version check before running the NVME tests, which would be beneficial for those of us who update conservatively.

Sorry if the wording is confusing, things in my brain see it clearly through my eyes.

There are two different things going on here:

  1. Smartmontool v7.4 does support NVMe self-tests but anything before does not.
  2. No version of TrueNAS supports running scheduled NVMe self-tests. These are:

Here is the statement in the multi_report_config.txt file:

NVM_Smartmontools_74_Override="enable" # Set to "enable" to override the check for smartmontools 7.4 and force the script to run the SMART tests. Default "enable". And the script does check for version 7.4 and if it does not exist, regardless of the variable value above, it will run the NVMe self-test because version 7.3 and earlier does not support NVMe self-tests. This takes the decision making process out of your hands.

If you do not want the script to run an NVMe self-test while version 7.3 or earlier is on TrueNAS, you will need to set the two values that enable (true)/disable (false) the Short or Long tests.
NVM_Daily_Short_Selftest="true"
NVM_Daily_Long_Selftest="true"

In version 3.0.6 of this script I changed the default value to enable due to the fact the no version of TrueNAS supports scheduled NVMe testing.

“Scheduled NVMe testing is being able to use the GUI and select SMART tests and select an NVMe drive.” However if you already had a config file, whatever the value is in that file will be in the new config file. You would need to change it to enable.

With this value set to “enable” the script will ignore the version 7.4 check and will run NVMe self-tests via this script.

If this value is set to disable then the script will check for version 7.4, if it exists then no NVMe testing will be performed by the script. If version 7.4 does not exist then the script will run the tests.

Following so far?

Maybe you want to run the NVMe self-tests no matter what version of smartmontools you have, then set to enable.

The value above is strictly to ENABLE or DISABLE the override of the results of the smartmontools version number.

Once TrueNAS GUI does support selecting NVMe drives for self-tests, this feature will go away, it will not be needed. I wish there was a way I could check TrueNAS via the script to see if it supported the scheduled testing then I could just make this fully automatic. Maybe the API? I can look but I’m not hopeful.

Exactly what errors? Since you did not give specifics my mind wanders and I’m asking, does your NVMe support self-test? The tests will not run and you will get a message stating this. But maybe there is something unique to your NVMe drive(s), I am using the NVM Express standard for the commands. Please expand on the errors so I can investigate but no one should be getting errors due to the script.

I await your response, I’m intrigued and I would like to duplicate your setup as much as I can to figure out if there is a problem or not.

Cheers!

Hi joe, thanks for the clarification, it has been a while since I updated your script (ntars), so I had to catch up some new features.

These are the current settings:
NVM_Daily_Short_Selftest=“false”
NVM_Weekly_Long_Selftest=“false”
NVM_Smartmontools_74_Override=“disable”

This is the result in the SSD section of the mail report:

Changing to NVM_Smartmontools_74_Override=“enable” does not alter the result in the mail report. Same error.

@mistermanko It appears you have drive failures or maybe it aborted.
These errors are being reported via smartmontools, not the self-test run by the nvme command.

I would appreciate you providing the following, first to send me a dump using -dump emailextra and this is different, it includes some extra data for every drive. Depending on how this goes, I may update the dump routine to add another item or two. Note: running a -dump will bypass any NVMe Self-Tests.

Next run the script from an SSH window, are there any errors listed in the SSH window? You should see information about running Short selftests, then a PASS (hopefully). What does the Text Section in the error reporting section state for errors?

Now to collect some more data and explain it a little bit…
From an SSH window:

The entire output of nvme error-log /dev/nvme0.
This will tell us what errors have occurred. Don’t freak out if there are a lot.

And the entire output of nvme self-test-log /dev/nvme0
For the second command we are looking at the Operation Result which should be “0” for the most recent self-test.

Why not one last command to keep me from asking for it later… (I don’t know which namespace you are using, I never want to assume as I have seen one result with a namespace of 4 being used. Very odd and the user had no idea how that happened.
nvme smart-log /dev/nvme0n1 -H
nvme smart-log /dev/nvme0n2 -H

I’m still learning NVMe commands, the reference material is scattered about. I will try to look up more result codes, so far NVMe Express lists “0” as “Operation completed without error”. If this is not zero then the script adds “nvme0 S.M.A.R.T. Short (or Long) Self-test completed - FAILED ?”, it does not generate the message in the chart.

I hope I find something wrong with the script interpreting the results. You do have two different NVMe models so I too would suspect the script if I were you.

After the data has been collected, lets run a Short test on one of the drives, we will use nvme0 for no real reason.

In the multi_report_config.txt file, edit the lines below to disable all script run self-tests:
NVM_Daily_Short_Selftest=“true” to “false”
NVM_Daily_Long_Selftest=“true” to “false”

FOR SCALE: The command is nvme device-self-test /dev/nvme0 -s 1 and wait 2 minutes. After two minutes run the command nvme self-test-log /dev/nvme0 and look for
“Self Test Result[0]: Operation Result” should = “0”. If it does equal “0” then run the script again normally, do you still have that failure?

I threw a lot of information your way. I will update the -dump command to include some of this new NVMe data to help troubleshoot in the future should I find it needed. I’m hoping the -dump emailextra provides me what I need.

Cheers,
-Joe

Mail dump is on it’s way to you.

Next run the script from an SSH window, are there any errors listed in the SSH window?

I don’t get any extra output with errors. This is all I get:

./multi_report.sh
Multi-Report v3.0.6 dtd:2024-06-02 (TrueNAS Scale 22.12.4.2)
Checking for Updates
Current Version 3.0.6 -- GitHub Version 3.0.6
No Update Required

Now to collect some more data

nvme error-log /dev/nvme0
nvme error-log /dev/nvme0

.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[10]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[11]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[12]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trty
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[22]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[23]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[24]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[25]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed su
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[35]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[36]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[37]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[38]
.................
error_count     : 0
sqid            :
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[42]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[43]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[44]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[45]
.................
error_count
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[49]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[50]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[51]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[52]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[53]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[54]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[55]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[56]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[57]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[58]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[59]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[60]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[61]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[62]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
 Entry[63]
.................
error_count     : 0
sqid            : 0
cmdid           : 0
status_field    : 0(SUCCESS: The command completed successfully)
parm_err_loc    : 0
lba             : 0
nsid            : 0
vs              : 0
trtype          : The transport type is not indicated or the error is not transport related.
cs              : 0
trtype_spec_info: 0
.................
nvme self-test-log /dev/nvme0
nvme self-test-log /dev/nvme0
Device Self Test Log for NVME device:nvme0
Current operation  : 0
Current Completion : 0%
Self Test Result[0]:
  Operation Result             : 0
  Self Test Code               : 1
  Valid Diagnostic Information : 0
  Power on hours (POH)         : 0x18f2
  Vendor Specific              : 0 0
Self Test Result[1]:
  Operation Result             : 0
  Self Test Code               : 1
  Valid Diagnostic Information : 0
  Power on hours (POH)         : 0x18eb
  Vendor Specific              : 0 0
Self Test Result[2]:
  Operation Result             : 0
  Self Test Code               : 1
  Valid Diagnostic Information : 0
  Power on hours (POH)         : 0x18ea
  Vendor Specific              : 0 0
Self Test Result[3]:
  Operation Result             : 0
  Self Test Code               : 1
  Valid Diagnostic Information : 0
  Power on hours (POH)         : 0x18ea
  Vendor Specific              : 0 0
Self Test Result[4]:
  Operation Result             : 0
  Self Test Code               : 1
  Valid Diagnostic Information : 0
  Power on hours (POH)         : 0x18ea
  Vendor Specific              : 0 0
Self Test Result[5]:
  Operation Result             : 0
  Self Test Code               : 1
  Valid Diagnostic Information : 0
  Power on hours (POH)         : 0x18ea
  Vendor Specific              : 0 0
Self Test Result[6]:
  Operation Result             : 0xf
Self Test Result[7]:
  Operation Result             : 0xf
Self Test Result[8]:
  Operation Result             : 0xf
Self Test Result[9]:
  Operation Result             : 0xf
Self Test Result[10]:
  Operation Result             : 0xf
Self Test Result[11]:
  Operation Result             : 0xf
Self Test Result[12]:
  Operation Result             : 0xf
Self Test Result[13]:
  Operation Result             : 0xf
Self Test Result[14]:
  Operation Result             : 0xf
Self Test Result[15]:
  Operation Result             : 0xf
Self Test Result[16]:
  Operation Result             : 0xf
Self Test Result[17]:
  Operation Result             : 0xf
Self Test Result[18]:
  Operation Result             : 0xf
Self Test Result[19]:
  Operation Result             : 0xf
nvme smart-log /dev/nvme0n1 -H
nvme smart-log /dev/nvme0n1 -H
Smart Log for NVME device:nvme0n1 namespace-id:ffffffff
critical_warning                        : 0
      Available Spare[0]             : 0
      Temp. Threshold[1]             : 0
      NVM subsystem Reliability[2]   : 0
      Read-only[3]                   : 0
      Volatile mem. backup failed[4] : 0
      Persistent Mem. RO[5]          : 0
temperature                             : 44 C
available_spare                         : 100%
available_spare_threshold               : 10%
percentage_used                         : 2%
endurance group critical warning summary: 0
data_units_read                         : 71149077
data_units_written                      : 138726195
host_read_commands                      : 3299723727
host_write_commands                     : 3713627987
controller_busy_time                    : 14997
power_cycles                            : 33
power_on_hours                          : 6391
unsafe_shutdowns                        : 22
media_errors                            : 0
num_err_log_entries                     : 27
Warning Temperature Time                : 0
Critical Composite Temperature Time     : 0
Temperature Sensor 1           : 44 C
Temperature Sensor 2           : 49 C
Thermal Management T1 Trans Count       : 0
Thermal Management T2 Trans Count       : 0
Thermal Management T1 Total Time        : 0
Thermal Management T2 Total Time        : 0
nvme smart-log /dev/nvme0n2 -H
/dev/nvme0n2: No such file or directory
Usage: nvme smart-log <device> [OPTIONS]
nvme self-test-log /dev/nvme0
nvme self-test-log /dev/nvme0

  Vendor Specific              : 0 0
Self Test Result[3]:
  Operation Result             : 0
  Self Test Code               : 1
  Valid Diagnostic Information : 0
  Power on hours (POH)         : 0x18ea
  Vendor Specific              : 0 0
Self Test Result[4]:
  Operation Result             : 0
  Self Test Code               : 1
  Valid Diagnostic Information : 0
  Power on hours (POH)         : 0x18ea
  Vendor Specific              : 0 0
Self Test Result[5]:
  Operation Result             : 0
  Self Test Code               : 1
  Valid Diagnostic Information : 0
  Power on hours (POH)         : 0x18ea
  Vendor Specific              : 0 0
Self Test Result[6]:
  Operation Result             : 0
  Self Test Code               : 1
  Valid Diagnostic Information : 0
  Power on hours (POH)         : 0x18ea
  Vendor Specific              : 0 0
Self Test Result[7]:
  Operation Result             : 0xf
Self Test Result[8]:
  Operation Result             : 0xf
Self Test Result[9]:
  Operation Result             : 0xf
Self Test Result[10]:
  Operation Result             : 0xf
Self Test Result[11]:
  Operation Result             : 0xf
Self Test Result[12]:
  Operation Result             : 0xf
Self Test Result[13]:
  Operation Result             : 0xf
Self Test Result[14]:
  Operation Result             : 0xf
Self Test Result[15]:
  Operation Result             : 0xf
Self Test Result[16]:
  Operation Result             : 0xf
Self Test Result[17]:
  Operation Result             : 0xf
Self Test Result[18]:
  Operation Result             : 0xf
Self Test Result[19]:
  Operation Result             : 0xf

After running the selftest, no change in the mail report, still the same error on nvme0.

Failure or Abort (6391 hrs)

If you need anything else, I’d be happy to help.
cheers!

The script is the problem, should be very easy to fix and I suspect it is an upper/lower case issue but I will email you for one more piece of data, similar to what you have provided. More NVMe learning nightmares.

Found the issue, “white space”. @mistermanko provided me a file I needed in the correct format to identify this issue. The fix was very easy and in process of being verified. The drives reported correctly however my drives didn’t have a leading “space” character. This one does. I’m curious now if it is the version of Debian or the drives just report differently.

@mistermanko has verified the fix worked, I thank you kind sir.

Version 3.0.6a is on Github if you need it to resolve NVMe self-test errors like above.
Github Multi-Report Link

I will push 3.0.7 once I add a few more data collection items to the dump which are all related to NVMe troubleshooting.

3 Likes

Hope i dont miss that from prevoius post… But is normal that under Most recent Short & Extended Tests - Listed by test number of any NVME is not listed any smart test overall? In the NVMe Summary Report is listed the last smart as expected, but not there