I just installed TrueNAS baremetal on UGREEN DXP4800 pro. The initial transefer of data from my old Synology was ok. However, today, during a heavy load - I initially made my collections of different media into separate datasets (for Movies, TV series, music, books etc) and now I move them via rsync to folders in a single dataset, this means TBs of data. My setup is 4x 6Tb HDDs in RAIDZ1 and I got the following error:
Pool tank state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.
Now the copying has finished, zpool status shows
zpool status -v tank pool: tank state: ONLINEstatus: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
with 13 read errors on the 3rd disk. All disks are brand new WD Red Plus 6 TB (WD60EFPX) and before installing TrueNAS I conducted long SMART tests on all HDD and all were ok (this was like 1.5 weeks ago).
What should I do now? I aināt 100% this is HDD issue, maybe a connection or something. Should I run scrub on my pool or a long SMART test on the problem disk first? Where is SMART tests in the newest stable TrueNAS? I donāt see it in web GUI under storage (old videos showed there was such a button but no more)
This is why paranoid people burn-in their HDDs for days (week[s] depending on hdd size) before putting data on them
If you could provide the full output of the smart results smartctl -a /dev/sdPROBLEMDRIVELETTERHERE and zpool status into nice preformatted tables (< at the start with /> at the end of the output) then we can review further.
SMART Test is no longer in the GUI because of reasons that I personally think are stupid, but it was the devās choice (Iām going to paraphrase here, but seems some of their enterprise customers were getting āfalse positivesā so SMART in the GUI was basically gutted). So youāll have to get friendly with CLI.
After looking at the full smart output for the drive in question, hopefully, we can guide you a bit more.
Thanks! Iām (relatively) calm, over 8 years of using Synology I experienced ffailed disks twice and both times after replacement all was fine.
Iām reading your flow chart and on #1 there is scrub mentioned. I asked in the original post should I run scrub or not, havenāt received an answer (yet) and AI replied that in some cares scub may load the remaining disks too much, causing further failures. So,
sudo smartctl -t long /dev/sdX
ā¦not at the same time as a scrub, for the sake of checking data integrity. And if that triggers more errors, that would be a way to investigate.
Whatās the SATA controller/motherboard?
How are the drives connected and powered?
ASMedia 1164 SATA controller running over a PCIe Gen3 x2 link. Motherboard - cannot find info - it seems a proprietary UGREEN design tailored for an embedded Intel Core i3-1315U processor.
I donāt want to cause any undue panic, but ASMedia controllers have historically been looked down upon when working with TrueNAS.
Your pool is currently online & none of the drives have been kicked out of the pool by the system. If you donāt have a lot of data on the pool Iād start with a scrub since that could in theory complete faster than a smartctl -t long.
Either way, youāre going to have to do both. I donāt see anything from your current results that currently points that either step would put your pool in a critical state.
OK, so thatās 4 drives out of a reasonably modern 4-port controller. No multiplier.
No cables; the only change you could make would be to shuffle drives to see whether problems follow the drive or the slot.
CMR Red Plus drives.
That leaves scrub, long SMART testsā¦ and MemTest, just in case
At least this one is a 11xx, not a 10xx.
B550 and newer AMD chipsets are AsMedia, and so are their integrated SATA controllers, so AsMedia controllers have actually been accepted for some time in home builds.
It shouldnāt - ZFS shouldnāt care about the drive order in any way. There are exceptions to this if something else stupid & unexpected is happening.
Did you run a scrub clear before running a scrub, to remove the current alarms? They do not clear on their own. And based on the data you posted earlier where you only had a few read errors, I suspect that once you clear the scrub errors and run a scrub, you will have no errors, providing you are not moving lots of data at the same time. If this happens then you likely overheated something.
Also, there is a list of commands in the Drive Troubleshooting Chart (last pages) if you follow this guide.
If the error does not return, I recommend that you run two burn-in tests:
Memtest86 - Run for at least 5 complete passes, and it that is faster than 12 hours, then keep running the test for maybe 24 hours. This will help solidify one major section of your system, it is not just RAM either.
A CPU Stress Test like Prime 95, run for at least 4 hours. this will heat saturate your system and will generally fail if there is a component issue or solder joint issue. Anything more than 4 hours will likely not provide much more.
Since this is a ZFS error, And the SMART data looks fantastic, I highly doubt there is a physical hard drive issue.
People who build servers for corporations will run these tests for up to 30 consecutive days, as the corporation must depend on the severs working at 100% all the time. I personnel use the values I provided for my own systems.
@TemplarB
If the troubleshooting guide is giving you problems with how to understand what to do, please let me know and be specific with which step it is. I have not had any feedback on this in several months so if after you read all the charts (not many of them), you have a question, please ask.
If for some reason the problem seems to have disappeared and you do not find a cause, try to repeat the operations you were doing when the issue occurred. I still think you have a stability problem, just a gut feeling. But proving it is better.
Last thing, Do Not Use AI to troubleshoot a TrueNAS problem. Unless you are using a paid AI that actually is capable of troubleshooting these problems, the general AIās provide old outdated and incorrect information. If you run an AI provided command and you do not know what it actually does, you could be in for a bad time. I test AI out all the time, some are okay but they lose focus of the original problem and send you down a rabbit hole.
I try not to use it in cases I consider dangerous (ok, I may err it my assumptions) and in something like this, I run the same question through three different free models, compare the answers and if any command is given, ask for defining every item of the command.
About the guide: it is very good, thank you for making it! If I have any coherent feedback, Iāll write it
I look forward to any feedback about the guide, and feel free to be honest. The folks here know that I prefer to improve whatever I create, even if I feel stupid for having an error or just some instructions were not 100% clear. It is easy to write something, but that doesnāt always mean others understand what I am saying.
Your SMART Long/Extended tests will take about 11 hours to complete. Iām not sure exactly what you have going on for data usage with your NAS, meaning how active it is. Whenever you are doing this kind of troubleshooting to validate the drives or data, it is best to minimize the NAS usage. So, no saving/moving of data. Reading data is generally fine as it requires minimal effort on the drives, such as viewing photos or streaming a video that you are not transcoding on the NAS.
SMART tests are the lowest priority tests and are run by the drive electronics. This means that ANY request by the computer will pause the SMART test so the drive can fulfill the request. And you may already know this, but I try to assume nothing, meaning I do not assume a person has specific knowledge. It can lead to a lot of confusion and make life terrible for both people.
When you run a SCRUB, it too has a low priority over other NAS operations, so lots of data requests to read or write will slow the SCRUB down. But it is the OS low level so SMART is a hardware low level event making SMART the lowest level.
If the problem cannot be identified testing like this, then we must try to recreate the circumstances in which the problem appears to have been committed. Repeatability is very important. But before we go down that road, you should run the stability tests, and run them long enough to validate the system stability. All of this should be in the Flow Charts.
Looking forward to hearing what you find out. And of course, post any SMART and SCRUB results, you will get feedback from the folks here on the form.
And AI, I have checked other AIās against each other, but when I use AI, I already have a reasonable understanding about what I want to do and how it is performed. If Iām not certain about something, I read the forums (not the AI created summary), and not just the TrueNAS forums. You know, what we did before AI.
