I’ve searched the forum, and can’t find an answer to if this is even possible, but I seem to recall that it was.
Scenario: striped pool of X device (all files are replaceable, if the hardware costs hadn’t exploded it would be a 12 wide raidz2 vdev in another pool, no point commenting on the downsides of stripe pool, everyone knows). Pool only stores large files in the GiB range.
A single drive has a single read error.
Is there a way to find out which individual file is affected to do a manual deletion and replacement?
That’s what picked up the read error. It does not state which file is affected (zpool status -v), just that there is a read error and “No known data errors” which is…weird.
Take this with a grain of salt, maybe wait someone more expert: probably no file are affected. I think in this case you can clear the pool than run the scrub again, to check if the error raise up again
I have to agree with @etorix, it appears to have been in redundant Metadata.
By default, there are 2 copies of regular Metadata, and 3 copies of more critical Metadata. This applies to BOTH non-redundant pools and redundant pools. Meaning a 2 way Mirror vDev will have 2 copies of regular Data, but 4 copies of regular Metadata. Control is via these 2 dataset parameters:
I think my miniature media server experienced something similar to what you described. Took me a few weeks to realize that it was probably redundant Metadata. That media server’s media pool is non-redundant, (it was too small to have extra storage).
@Krill One comment. It is useful to still regularly scrub non-redundant pools. The below blog describes my experience with this, in a bit too much detail. But, suffice to say, before I instituted scrubs every 2 weeks, I would get perhaps 2-4 bad blocks a year. Easy enough to restore the files. But since then, I have NEVER had to restore a file. In several years.
Is this one of those slim scenarios where setting copies to anything other than default might make sense? (assuming OP has no budget available for a 2nd drive, but has the storage space to support additional space requirements?)
Edit: Might be a good idea to dmesg -T | tail -n 100 & smartctl -a /dev/sdDRIVEHERE (maybe even run a smartctl -t long /dev/sdDRIVEHERE) and review the results to see if maybe is time to copy anything on the drive to a backup…
Edit 2: would setting copies to lets say 2 automatically start filling up additional copies or is this something that needs to be planned in advance? Also is changing it back to default something that is allowed?
It has to be done before the blocks are written. So a re-write of the data and metadata would be needed.
Even if the original user, @Krill, does not have space for 2 copies of everything, the copies attribute is at the dataset / zVol level. Thus, if their is more important, but smaller data in it’s own dataset, that could be set to copies=2 and the data re-written.
I don’t think its the metadata. I’m using a special dev (sata flash drive)…stripe (not like this is any worse than the data vdevs as a stripe). but the error is on a HDD, where there should not be any meta data.
I’m fairly sure the svdev was in place before any data was placed on the pool. I could be wrong though, and that could be the most likely scenario at present.
Either way, at least its novel.
Just to reiterate this is data which is backed up but there is not enough space for copies=2. Its a media data set of about 70TiB in current size
Could have been transient read error from the HDD I guess? ZFS tried to re-read it & it went through? This is a stretch as a CRC or something would have been logged, but this is about the best stretch of imagination I got.
Edit:
The other thing I can imagine is @winnielinnie coming in to recommend memtest
All member devices, (aka single HDDs, SSDs, or NVMes), have metadata on them. Now to be clear, a Special Metadata vDev will have most, (all?), of the more common Metadata. For example, probably all the directory entries, extended file attributes and file access control lists.
However, the fault could have been in Critical Metadata on a HDD, which also has more than 1 copy, (I think 3 copies by default).
Note that their is an OpenZFS feature request to allow single device Special Metadata vDevs, that have only 1 copy of the standard Metadata. Thus, on loss, the regular vDevs can be used for reading the other standard Metadata copy. This also implies that all normal standard Metadata reads would be by preference from the Special Metadata vDev, with the assumption it is faster. (Like using an SSD or NVMe device.)
Basically this merges a Metadata only L2ARC and Special Metadata vDev type.
I don’t know if that will be accepted. And if so, when it will be implemented.
By default, ZFS would put the 2 copies of standard Metadata on different vDevs. Meaning if you have a pool with 2 disks, each would get 1 copy of standard Metadata. Yet the Data its self would be on only 1 disk, (unless copies=2). Something similar if more that 2 vDevs, probably based on which vDevs have the most free space.
However, with a Special Metadata vDev, both copies of standard Metadata are likely only on the Special Metadata vDev.
That is why someone suggested a change. Keep 1 copy in the normal Data vDevs and the preferred copy in the faster Special Metadata vDev. That then also makes the Special Metadata vDev non-critical in the sense it can fail completely and the pool still has 100% functionality. (Excepting if the Special Metadata vDev is also setup for small blocks. In which case, all those could be lost on complete device failure.)
Someone was thinking good thoughts. It hopefully will happen or something similar. Using a special metadata vDev always sounded like a high risk as it works today. I guess that is why we have backups
Ideally, the Cache / L2ARC could be made more useful. In someways, I want it to work as an additional Metadata copy. In my new NAS, the 2 way Mirror 10TB HDD pool has a 500GB NVMe Cache / L2ARC. With persistence on by default, I’d want all the Metadata in it.
But, that is not how Cache / L2ARC devices work. They have both RAM pointers, (96 bytes each), and blocks, (Data or Metadata). While a Special Metadata vDev might have ARC entries, their is no need for a 1:1 ratio of Special Metadata blocks and RAM pointers.
An update. A scheduled scrub started yesterday, and read errors were creeping up slowly (one every 10 or so minutes) on the single drive.
I ran smartctl -x and one result caught my eye: the Sata connection was only working at Sata 1 speeds. I verified that this speed issue was unique to this drive, and then I stopped everything, swapped two sata connectors around and made sure the connection was secure, and started a new scrub.
There are now no read errors on that drive.
Lesson: always check you cables (and hope).
(more detail: the drive in question is in a rack server, in a moveable 15U rack that was recently moved slightly due to electrical work. The backplane had been removed because of the PWDIS issue, new HDD, old backplane, and the HDD is connected directly to the HBA, but the cables are a mess even after being tidied and very close to the 2U sized fans. In hindsight I’m not shocked that there is a cable issue).