Midclt call disk.smart_test SHORT '["*"]' Produced the following output: null

TrueNAS General
1

Hello,

I updated my server from Truenas 25.04 to 25.10 and all seems to have gone smoothly with the exception of SMART testing. The update did convert my smart command to cron jobs, but I am now receiving email with ‘midclt call disk.smart_test SHORT ‘[“*”]’ Produced the following output: null’. Can some one please let me know what this means? I though I read that this was fixed (NAS-137472) or maybe I didn’t understand correctly? Does ‘null’ mean that the drives are fine and will the message change if there is an issue with a drive?

Thank you for any help or insight you may offer.

PS: Also, if there is anyone that can bring some simple clarification to this whole SMART testing change in Truenas 25.10? I’m still uncertain if SMART testing is now automatically running in the background on new installs without any cron jobs or does a cron job need to be setup to use SMART testing? Thought I remember seeing Kris Moore mention something on a T3 podcast about automatic smart testing and that the existing cron jobs could be disabled, but I can’t find that video anymore so I’m not sure.

1 Like
2

reading the ticket i think that you can simply check Hide Standard Output and Hide Standard Error in that cronjob entry to suppress this notification. If not something else is happening

1 Like
3

Have you seen https://www.truenas.com/docs/scale/25.10/gettingstarted/versionnotes/#disk-management?

For those specific questions, my understanding is that TrueNAS still automatically monitors for SMART errors and alerts on them, but it doesn’t set up SMART tests on new installs (it only migrates existing tests to cron jobs on update).

And I think @oxyde is right about the notification but I haven’t tested that.

4

Hi @oxyde ,

Thanks for the reply.

If I hide these email notifications does this mean that I will also not receive any notifications when there is an issue?

5

HI @DjP-iX ,

Thank for the reply.

Base on the link you sent me, Truenas does have some sort of built-in drive monitoring as stated ‘TrueNAS continues to automatically monitor critical disk health indicators’.

I was wondering if it was even necessary to keep the converted cron jobs from 25.04 or are they redundant and if I delete the crons, will I still receive notifications when there is a drive issue or is this something i have to setup?

This is all new and, frankly, a little confusing to me.

BTW, I am running 3.5, 2.5 and NVME drives, so SMART testing is still important for my use case.

6

If you mean that also if the command in the cronjob really fails you will not get adviced… yes, you 100% right.

But seems that this Is the only fix provided, just a message suggestion :smile:

Given that all those moves made around smart functions cut, to me, are very questionable decisions… i would suggest you to seriously evaluate the Multi report script, because It offers way more smarter to perform smart tests, and many other advantage (config backup, complete email report, … ); and to be honest, i would have suggested this script also before those removal

7

Sorry for the confusion what I am trying to say is that if I delete the converted cron jobs from 25.04 and rely only on the Truenas built-in disk health monitoring, will the system send notifications if it detects a drive issue?

The Multi report script link doesn’t work for me.

I have Scrutiny installed, but it doesn’t seem to really do anything. How does it notify me if there is a drive issue? Any ideas on a good alternative to Scrutiny?

Thanks again for all the feedback

8

Link fixed, sorry!

For the first question, answer should be yes… but don’t delete the cronjob of you at least don’t replace It with another tool to perform smart :smile:

At this moment Scrutiny Is good just to watch data, i still don’t get why has been mentioned as a viable replace..

9

They have not gotten rid of the smartd monitoring. As far as I am aware, they have only stopped with any SSD (includes NVMe) drive testing. You can still setup CRON Jobs to make a smart test happen, but not in a easily configured GUI. Okay, easy is subjective. Easy for me, not easy for others.

Why did they do this? I’m sure there is some rationale. Think about it, Windows doesn’t run a routine SMART test on drives. I shouldn’t use that as an example however it was the first one I could come up with. But those of use who know drive failure will occur, especially with a system that run 24/7, we know it’s coming, it is a matter of time. We test to see the small things which help us predict a failure. SMART testing isn’t perfect but it is what we have that is non-destructive.

Multi-Report (making a plug for myself, link below) is a very good script which will test al your drives for you, and generate an email report, the first part is a quick look chart section, easy to spot issues, then a detailed section. I won’t tell you that is is a click and install script, it does require a little setup, however I feel the instructions are simple and we have a great support from the folks who use it if you do have a question. Once setup it needs no further interaction, you get daily reports. Take a look at it if you think this might be what you desire.

Scrutiny is a nice program however it does not schedule SMART testing.

10

Most OSs don’t by default. Monitoring is pretty routine, but actively testing not so much. I don’t believe other NAS OSs schedule it by default. Did TrueNAS schedule tests by default before 25.10? I know it hasn’t always done this. If we accept that there’s value to running those tests regularly (something iX now appears to be disputing), it should schedule them automatically, and certainly should provide a good GUI for users to set and adjust those schedules.

“We couldn’t figure out how to do it right, so now we’re going to call its validity into question, and we’re going to stop trying to do it.” I’m sure they wouldn’t care for that paraphrase, but I think it’s fundamentally accurate.

11

No, not that I can ever recall, however it was available on the first FreeNAS version (version 8.0 for those who do not remember that far back) and has always been available since, well until 25.10. During the development of FreeNAS, between 8.0 and 11.0 were some tuff times for SMART testing. Sometimes an update would not transfer over any SMART tests that were scheduled, hence my little script (the original version was little) to let you know when SMART testing was not occuring (Test Age).

Pretty sure you are correct.

While I would like to see SMART test scheduling come back in the GUI, I can understand that the majority of OS’s do not actively test so why should TrueNAS do it. The answer: They are a premium product and they should go the extra mile to include such a useful feature. It might be a “Bell and Whistle” for the product, but it is a good one. “TrueNAS support ‘ACTIVE SMART TESTING’ while other NAS solutions just wait for a failure to occur” would be a good selling point. Just my opinion.

12

on the topic smartD

Not only did TrueNAS™ (at least since 11.* or so) ever automatically added SMART-Tests for Disks from what i can remember but you also have to configure your default smart temperature thresholds as you otherwise won’t get any temperature alerts at all. As i have seen it often enough that somebody forgot those …

midclt call smart.update '{"difference": 10, "informational": 30, "critical": 40}'
13

It’s the “NAS” part in the name “TrueNAS”. A built-in feature to schedule SMART tests is not a farfetched expectation for a NAS server.

Why is there a GUI for scheduling scrubs? Why not relegate it to cron jobs where you can invoke the midclt command for scrubbing a pool?

1 Like
14

So, is the support for smart tests gone? I just created a custom NAS like 1 week ago and got the ‘chance’ to have automated smart tests for like 5 days and now since I’ve updated I can’t see any section in the UI for such tests (and I get the error the OP gets).

First of all, SMART tests are not a nice to have feature for a NAS OS. They might be for Windows but not for a NAS. I come from using Synology which has a lot of issues itself but smart tests are working. They are configurable in a very similar way to what TrueNAS offered.

Secondly ,upgrading should never produce errors, otherwise it means the upgrade is not 100 percent backwards compatible which is bad for engineering.

I do have 2 mirrored disks and I do have periodic backup to a different storage but I still feel like my entire life photos are in more danger now.

Sorry for being too direct, I’m Eastern European :joy:

15

Through the GUI, yes. See also:

TrueNAS upgrades are almost never 100% backwards compatible. Usually the release notes will explain known issues, and there are often other issues that come out later.

Moreover, you’re running software that could only very charitably be described as “beta.” iX recommends it only for “early adopters,” recommending that “general users” stay with 25.04.2 (probably until the release of 25.10.2).

16

Oh, so I shouldn’t have changed the update train yet? :slight_smile: I could not find anywhere if update trains switch automatically when the new train becomes stable and tbh a very popular LLM told me to switch cause they don’t :joy: Hopefully all goes well though, I don’t have any errors except for the smart tests.

17

This sounds like lesson 6.022x10^23 on “why you shouldn’t blindly trust AI chatbots.” It’s true that the update train doesn’t change automatically. It isn’t true that “therefore, you should change the train as soon as a new release is out.”

iX have kind of a split personality in this regard. On the one hand is “a new release is out, go hammer our update servers so you can upgrade ASAP!” On the other hand, they’ll point to their software status page ( Software Status - TrueNAS Roadmap - Open Source NAS Software ), which usually doesn’t recommend a release for general use until X.2 (so in the case of 25.10, if previous patterns hold, it would be shortly before the release of 26.04 that they’ll recommend 25.10 for general use). Some might call this dishonest; iX calls it “transparent.”

Starting with 25.10, once it goes “general use,” you’ll be able to set that as your update profile if you choose. If you do that, it will only notify you of updates that they deem appropriate for whatever profile you set.

The other issue is that major releases of software make major changes. In the case of TrueNAS, it’s not unusual that these will be breaking changes. You really should check the release notes before upgrading, and hope that iX has included all the breaking changes there.

2 Likes
18

I got a similar error after a chron job ran post upgrade from 24.04.1 to 24.10. See below:

The command:
midclt call disk.smart_test SHORT ‘[“*”]’
Produced the following output:
[EFAULT] [‘smartctl’, ‘/dev/sde’, ‘-t’, ‘short’] failed for sde (1):
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py”, line 360, in
process_method_call
result = await method.call(app, id_, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py”, line 57, in call
result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 954, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 782, in call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 665, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/disk/smart.py", line 18, in smart_test
raise CallError(“\n\n”.join(errors))
middlewared.service_exception.CallError: [EFAULT] [‘smartctl’, ‘/dev/sde’, ‘-t’, ‘short’] failed for sde (1):
If you don’t wish to receive these e-mails, please go to your Cron Job options and check “Hide Standard Output” and
“Hide Standard Error” checkboxes.

It looks like it’s for a disk that failed previously that I replaced (maybe) or something else. I attach a screenshot here. I share this in case it helps and to follow this thread and to get any feedback if someone smarter than me sees why this happened and if it’s a bug or something in my config that needs to be adjusted.

image
image1957×375 13.1 KB

19

I have this same issue! This ghost disk is suddenly in my disk list and failing SMART for obvious reasons (it doesn’t exist). I’ve been trying to figure out how to edit the migrated cron job to only scan the real disks, but I haven’t been able to figure out the right bits to replace the “*” with.