Rotating Cold Spares Every 6 Months

[Apollo]

@smione

Apart from that, I will be interested to know more on your thoughts about the bit rot and bad sector scenarios. As per my understanding the scrub would identify bad sectors, mark as bad, allocate a new sector and copy the actual contents from a good copy in the mirror, and remap it. So now the bad sector becomes untouched by any new data. One thing I want to know is does it reflect in the relocated clusters in SMART or is it internally manged by ZFS only?

I think it is incorrect.

From my understanding:

ZFS and SMART don’t interact with each other.

ZFS only looks at the content of the data, ie the Blocks in ZFS terminology, ie the information.

SMART reports statistics about the life/health of the drive and the firmware handles all the checks at the physical level, ie detection and relocation of bad sectors, head actuator related issues….

The nuance here is:

Bit rot is generally caused by the degradation of the magnetic field in the structure that is used to store the state of the bit. The magnetic field will decay over time. The magnetic field can degrade more quickly or change state (North-South (“+”→”-”) to South-North (“-”→”+”) or become unmagnetized, ie no magnetic field) when external forces are applied, ie magnetic heads writing data in the vicinity.

In the old days of the magnetic tapes (still valid today), there were 2 types of degradation, one being the bit rot, the other being the transfer of information from one layer to the other, known as, “print-through” Print-through - Wikipedia (I had to google it).

ZFS won’t know the strength of the magnetic field for each of the bits the HDD is reading from. What ZFS does is to request the HDD controller to fetch the data in form of blocks. Within a block, there is a field (not the magnetic field), known as checksum, its value is calculated from the data that is going to be written within that block. Hence, the “block” contains both the raw data and the checksum.

When ZFS reads a “block”, it will read the raw data as well as the checksum stored in the “Block”.

ZFS is going to run a check on the data to see if it is valid by comparing the value of the checksum read from the “block” and calculate another checksum based on the raw data. If both checksums are identical, then the “Block” is good or healthy and the raw data is trustworthy.

However, if the checksums are different, it means some corruption has occurred within the “block”, either or both of the stored checksum and raw data have change over time. There is still the possibility of an incorrect checksum calculation done in RAM, but this is not what we are concerned about here.

When ZFS detects a corrupted “block”, it will try to generate a new “block” using the redundant data available from another disk (ie within a mirror, or RAIDZ1/2) and write the new “block” into a freely available area of the disk. The existence of the old corrupted “block” which is stored, or pointed to, in the pool metadata (I think) will be released and replaced with the location of the new “block”.

The space used by the old “block” now becomes available to be used by ZFS.

At the hardware/physical level, it is possible physical damage of the magnetic layer can occur, due to wear, impact…, causing the affected area to no longer hold a charge, and this is where the firmware on the HDD react.

The firmware will exclude the damaged area (ie sector) from being used, either read from or written to. Removing a sector from the disk means that the amount of data usable on the disk will shrink as the number of bad sectors are being taken out of service. In order to prevent the loss of capacity, spare sectors exist and are activated to take the place of the old sectors. This allow the disk to maintain its nominal capacity.

If the spare sectors have been all used up, the disk won’t be able to reallocated future failing sector. Anyway, the drive should have been replaced long ago.

If a damaged sector didn’t hold any data, ZFS wouldn’t know about it, but the firmware would have substituted the sector with ZFS being the wiser.

However, if one or more “block” already existed on the damage sector, the HDD could discard the existing data within the damaged sector or could try to copy what it can to a spare sector (I personally don’t know if the disk is handling any type of copy). Either way, there is a high probability that the data copied over is already corrupted. I also do not know if and when a sector is being reallocated, ie does it happen only during short or long SMART test or is the firmware/hardware handle the operation dynamically outside of a SMART test.

In any case, ZFS will detect the corruption or the absence of “block” and will trigger resilvering of those blocks when redundancy is in place. Those “blocks” will be written in a new freely available location as describe earlier in this post.

[Apollo]

@etorix

Aren’t LLMs designed to basically “please the user” and answer back what he wants to hear? So, if the question was sufficiently precise the Artificial Intriguer will manage to give the “right” answer—even while stating the opposite at the same time, as in this case.

I have experienced issues with local LLM as well as internet ones, where AI would categorically refute a fact I provided them stating I was wrong. Simply because the information they were trained on was not available then, AI wouldn’t acknowledge the fact I provided was the truth.

[Apollo]

@DjP-iX

Yeah and I suspect that’s part of what happened there. The frustrating part is that our AI Search instance is supposed to be really locked down against things like that i.e. we give it a bounded list of vetted source materials, we have training rules that instruct it to stop and inform the user if available sources do not reliably answer the question, we specifically run it on the RAG and not agentic model to prevent it from being too creative in trying to answer the questions, and yet it still seems to have gone fully sycophantic here. The user wanted X and it either completely fabricated an answer or misinterpreted some source (that it didn’t cite in its list of sources for the reply) to give them X.

AI's first kills show we're close to disaster. Godfather of AI.

[DjP-iX]

I once had a conversation with ChatGPT in the early days after it was publicly launched where I asked it the founding date of a small organization I’m a part of. It responded with a guess of, lets say 1987. It was wrong, though that date does appear on the website, so I could see where it was coming from. Then the conversation got weird when I told it that was wrong. It was something like:

That’s wrong

You’re right, I was mistaken. It was actually founded in 1988.

No that’s wrong again. If you don’t know the answer just stop guessing.

You’re right, I shouldn’t be guessing. It was founded in 1994.

Wrong. Stop guessing.

I’m sorry. It was actually 1997.

It never got the date right or agreed to just stop guessing incorrectly.

[smione]

wow…That was not an explanation, it was a walk-through - very detailed and visualizing. Thanks.
(I have seen only Allan Jude of Klara Inc doing these kind of walk-through on ZFS before, are you Allan by any chance?)

[smione]

very informative video… now you have given me the motivation to move my Nas to the basement.

I Don’t want it crawling to me in the middle of the night on 4 legs, as seen in the video.

Thank you for the nightmare for tonight in advance…

[Apollo]

@smione

wow…That was not an explanation, it was a walk-through - very detailed and visualizing. Thanks.
(I have seen only Allan Jude of Klara Inc doing these kind of walk-through on ZFS before, are you Allan by any chance?)

I figured it couldn’t hurt to describe the underlying implementation in order to better understand what is happening under the hood and how you can work you way around.

I am not Allan.

[smione]

@Apollo

In any case, ZFS will detect the corruption or the absence of “block” and will trigger resilvering of those blocks when redundancy is in place. Those “blocks” will be written in a new freely available location as describe earlier in this post.

So, this is the part happening during scrub.

Similarly, as per my understanding; when a new data is being written to a new block, it will try to read back after writing to ensure that it has written correctly and is readable. (someone please correct me if wrong here)

Does ZFS also inform the firmware to mark it as bad block in case of failure, or it needs to be handled by firmware during next SMART scan?

[Apollo]

@smione

@Apollo

In any case, ZFS will detect the corruption or the absence of “block” and will trigger resilvering of those blocks when redundancy is in place. Those “blocks” will be written in a new freely available location as describe earlier in this post.

So, this is the part happening during scrub.

Scrub is a dedicated task which ZFS performs. Its purpose is to detect the presence of corruption on the entire data stored within the pool. If the pool contains 1TB of data, but the pool has a capacity of 50TB, then scrubbing will only run on the 1TB of data.

Scrub will read every single “block” that exist in the pool.

However, if the pool is not being scrubbed, ZFS will still check presence of corruption when it access a file on the pool, let’s say you have a video file you want to watch, ZFS will read each of the “block” from that file and pass it to the service that will delives the file to you, ie Plex, Kodi, jellyfin, SMB share….

Each of the “block” will go through the validation process and if a corruption is detected, ZFS will fix the issue by creating a new healthy “block” from the redundancy and will be reported under the Pool Status page of the GUI or running zpool status -v command in the CLI.

@smione

Similarly, as per my understanding; when a new data is being written to a new block, it will try to read back after writing to ensure that it has written correctly and is readable. (someone please correct me if wrong here)

I am not sure. I think the HDD controller/firmware informs ZFS that the transaction was performed successfully. The HDD is most likely performing its own validation on its end.

@smione

Does ZFS also inform the firmware to mark it as bad block in case of failure, or it needs to be handled by firmware during next SMART scan?

A “block” is different from a “sector”.

A “sector” is the area where “block” are written to.

If ZFS detect a corrupted “block”, it doesn’t know if the corruption is due to physical damage or bit rot. It will only attempt to fix the “block”.

If a “sector” is defective, this is the role of the HDD firmware/hardware to handle the “sector” reallocation.

I think the HDD firmware/hardware will inform ZFS if a transaction failed due to a defective “sector”.

The HDD should automatically replace the defective sector with a clean one from the spares and ZFS should attempt to write the data again and possibly perform resilvering on the new “sector”.

[smione]

@Apollo

let’s say you have a video file you want to watch, ZFS will read each of the “block” from that file and pass it to the service that will delives the file to you

In this case, when we consider a 3 way mirror, let’s say disk A,B & C : when ZFS reads the block, it might read from disk A, and if the data is correct it will just pass it. Does this also verify with the blocks from disks B & C to confirm? (especially when the checksum is matching with expected result)

[pmh]

@smione

In this case, when we consider a 3 way mirror, let’s say disk A,B & C : when ZFS reads the block, it might read from disk A, and if the data is correct it will just pass it. Does this also verify with the blocks from disks B & C to confirm? (especially when the checksum is matching with expected result)

As far as I know - no. It will only do that when performing a scrub.

Reads from a mirror vdev are interleaved to increase performance. This would not make any sense if all copies were always read.

[Crow]

If I understand this correctly, you are planning on doing a 3 disk mirror, and having a 4th cold spare. You then want to swap in your cold spare to replace one of the disks on a periodic basis. The point being wear leveling your disks.

This would suggest from a statistical perspective that the drives will all go bad at the same time. At the same time, you will accelerate the wear on all the drives by requiring extra reads and writes.

Let me posit this idea…. Instead of rotating your disks. Use one disk as a cold backup (1 disk Zpool). periodically backup/replicate your pool to this drive but keep it offline otherwise. Then in the event of a disaster (provided the drive is kept safe - maybe in a fire safe or something) you will have an offline backup that is not subject to ransomware attacks or aging due to being on and spinning all the time.

I am not touching the 3-disk mirror idea. Not sure that makes sense for you as a raid z2 may be better and allow for expansion later down the line though a mirror may have a benefit for reads.

[Bradslinux]

FWIW, and IMHO… You are way overthinking this. Build your system, hot spares/cold spares (either are fine) when a disk fails, replace it. One of the great advantages of ZFS systems is the removal of the worry headache through automated system checks and redundancy. Why add more headache into it? just my $.02 worth…

[smione]

@Crow

Use one disk as a cold backup (1 disk Zpool). periodically backup/replicate your pool to this drive but keep it offline otherwise.

Yes, this is the final plan I decided to settle, based on the feedback I received from all of you. A single disk pool as backup which will be either swapped in with a disk in one of the bays (after making it offline first) and leaving it to do the backup once in 6 months.

Thank you for the suggestion on considering RAIDZ1/2, but in my case I feel 3-way mirror is better for the same reason you mentioned - the read performance. Also that would keep the wear out issues to the minimum on the backup disk (4th) and at the same time it will be protected from any power supply related issues that may affect - if remaining connected; even as a hot-spare.

Thanks again; I really appreciate you taking time to go through this long discussion and sharing your inputs.

(Please note : I am repeating some of the technical points any of you mentioned again from my side - NOT as “I know this already….”, but as a reference for someone who may wish to go through this thread and not wonder so what and why did the OP decided to follow a point. Intention is to learn from each others knowledge and experience as always. Also at the same time; I want you all to know that the efforts that each of you put in answering this query- was valuable, has made something better for someone somewhere. Thank you all.)

I could also get an external USB dock for one disk and use that, but not sure if that will have issues with ZFS running over usb.

If any of you have used a setup like this (with usb single pool for backup), please confirm if there were any reliability issues or that is fine.

[smione]

My thoughts exactly, and the method I followed before all AI, memory shortage, disk shortage etc. started.

I would have stayed with the same plan, if things have not changed this way. But; now that is not the case, we need to re-evaluate our plans from time to time as the surroundings change.

So if you are also in a similar situation, I would urge you to think again - especially focusing on unavailability or indefinite (or at lease relatively long) delays in getting a replacement disk in case needed.

[tannisroot]

I thought I would chime in with some bits others didn’t mention

• Warranty for enterprise grade drives is usually like 5 years
• Each time you’re handling the hard drives, which would be every 6 months, you risk dropping the drive, as well as damaging rather fragile connectors and the PCB they attach to.

[CyberGaut]

the new version do bring new functionality… but are also haw TrueNAS gives you security updates and bug fixes.

if you want to be more conservative, that is good. but stay on the train (go with the General or Mission Critical train)

Don’t assume is works so does not need updating

[CyberGaut]

Smione,

You have received a lot of feedback that this is a “Silly” idea.

But it is you and your system so of course you do you. (but that is the general feedback)

As I understand it you are storing your data on a 3 way mirror - 3 drives, and you are looking to rotate in a 4th drive.

As others have said if this is trying to hedge future drive prices well you go and gamble as you see fit.

if you are just that concerned (Paranoid) about your data then backups are the way to go. More drives give you redundancy for zero down time. Data protection only comes from backups.

invest in a bigger back up system rather than drives and time to rotate.

A 2 way mirror is excellent with very fast resilver time, a 3 way is getting pretty bullet proof.

the price of future drives may well be impacted by AI pressure, but also drives keep getting bigger so you may pay more in the future and get bigger drives. (cost per TB ~)

and once all are update you get the extra space.

you may want to run the new drives in production with your older drives as your cold spares as these have been run for a bit to identify any early manufacturing errors.

also are you buying cheap drives so that is why you see a high risk level…

if you have good drives, red, or enterprise (gold etc.) you already have a great set up.

anyway its just a thought.

GL

[smione]

@CyberGaut Thank you for the points mentioned; they are indeed valuable. Here is a quick status on them

• Yes, running new WD Gold drives in 3-way mirror, as suggested.
• One additional new WD Gold drive acts as the single disk pool for backup. This will be the cold spare too. (Need to decide, if to use over USB or not - awaiting feedback from experienced community members on this)
• Old WD Pro disk as another backup copy (They are smaller in size. so can’t be counted as the cold spare for the pool)

Regarding the upgrades : Yes, you are right - I need to get to a newer version eventually. Did not consider it as a priority, as I was hearing some issues reported before on the newer versions and wanted to get them to a stable state. (Also on some changes regarding incus, docker, smartctl etc going back and forth). However the main purpose of this system is only as a NAS for me and its limited to Lan only and is behind firewall. So I thought I will follow a slower pace on upgrades.

Considering the current situation; planning to wait for the 2026 versions to stabilize and move on to it directly - completely skipping the 2025 versions. Hope this makes sense; please let me know in case I missed any critical points.

[Fleshmauler]

@smione

Considering the current situation; planning to wait for the 2026 versions to stabilize and move on to it directly - completely skipping the 2025 versions

IMO I’m in the same boat - unspecified security issues to me are as much as a gamble as unknown new security issues. Would rather wait until something I want is in the update & skip arbitrary vm/smart changes.

@smione

(Need to decide, if to use over USB or not - awaiting feedback from experienced community members on this)

Quick rule of thumb is avoid usb when possible with TrueNAS; exceptions exist.