BLUF
Trying to understand what limitations if any, received replicas have, and how to best use them in changing my main pool layout.
Background and current state
I have been running the pool tank for many years. Its configuration for the last few years has been 6 x 4TB drives as RAIDZ2. It has been so reliable that I haven’t really looked at it beyond periodically upgrading the TrueNAS version, recently to SCALE.
But the set-and-forget reliability tends towards neglect, to the point that I have let the capacity creep up just above 90% rather than keeping it below 80%.
So last pay check I purchased 3 x 8TB drives which I have replaced (spare bays meant the 4TBs remained in place during the resilver). I have just ordered the next 3 drives.
So it now looks like…
NAME STATE READ WRITE CKSUM vendor model size
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
sdl1 ONLINE 0 0 0 ATA ST8000NT001-3ZL101 7.3T
sda ONLINE 0 0 0 ATA WDC WD40EFRX-68N32N0 3.6T
sdh ONLINE 0 0 0 ATA ST4000VN008-2DR166 3.6T
sdg ONLINE 0 0 0 ATA ST4000VN008-2DR166 3.6T
sdj1 ONLINE 0 0 0 ATA ST8000NT001-3ZL101 7.3T
sdk1 ONLINE 0 0 0 ATA ST8000NT001-3ZL101 7.3T
I have also been running on the same server a backup pool of 6 x 3TB drives as RAIDZ1 that I have been replicating snapshots to.
So the 4TB drives after replication have been zeroed (as an attempt to force remapping of sectors) and shuffed down to the backup pool.
Once this process is done, I should have approximately
tank 6 x 8TB RAIDZ2 = 32TB
backup 6 x 4TB RAIDZ1 = 20TB
Contemplating layout change for tank
Used capacity on tank is currently 13.24TiB / 14.56TB
So to keep below 80% and allow for modest growth to 16TB I can afford to have usable capacity at 20TB.
Hence I am considering changing the layout, to one of these topologies. Both options increase risk slightly, from my current lose any two drives.
| VDEV Layout | ~ Capcity | 2 Drive Failures Combinations (6C2=15) | 3 Drive Failures Combinations (6C3=20) |
|---|---|---|---|
| 3 x mirror | 24 TB | 12 safe / 3 unsafe | 8 safe / 12 unsafe |
| 2 x RAIDZ1 | 32 TB | 9 safe / 6 unsafe | 0 safe / 20 unsafe |
I do have two spare bays, so I could also buy an extra two 8TB drives,and go with 2 x RAID 2 vdevs giving approximately 32TB, but would rather not spend the money, and as I will be replicating to another (local) pool I feel I can accept that risk.
80% of 24TB = 19.2TB, so I am leaning towards the 3 mirror option with the greater number of safer drive failure combinations. While I welcome commentary / opinion on those options, it’s not my actual question.
Actual Questions (finally!)
Given I want to change the topology of tank, and that I have snapshots replicated to backup my questions are these…
Are there any limitations on a received replica? My replication settings include:
| Option | Setting |
|---|---|
Full File System Replication |
Checked |
Replication from scratch |
Checked |
Destination Dataset Read-only Policy |
SET |
Part of the reason for this question is for future options. If I had thought of the new layout before beginning the replacements, could I have added the 8TB disks to backup and destroyed tank and renamed backup to tank (via a multistep export import process). Once that’s done I imagine I would need to zfs set readonly=off (is there a GUI way to do this?). If that’s the case I could have effectively just used old backup pool as the new tank.
That ship has sailed unfortunately, and so I’ll proceed with auto expanding both pools. Then, once I am sure my replicated snapshots are complete, then I assume I can blow away tank, recreate it with the new layout, then replicate the other way. If I don’t set the destination policy to read only, then I guess I don’t need to do anything and I can just start using tank again with it’s new layout.
Am I missing any steps? Is there anything else I need to be careful of in terms of “promoting” replicated snapshots (not sure that concept is correct) to function as the original pool?
Thanks for sticking with me. I’m looking forward to learning more from the community.