RAIDZ1 to RAIDZ2 without doubling drives

TrueNAS General
1

I have an 4x8 TB RAIDZ1 array on TrueNAS. I’ve reached 80% capacity, so I’d like to expand capacity, but I don’t want to keep adding drives to a RAIDZ1 vdev beyond 4, so I’d like to migrate to RAIDZ2.

The problem is that I don’t have a spare 16 TB of space available to store my data in order to destroy my existing pool and rebuild it as RADIZ2.

I know the simplest solution would be to buy five more drives, make a RAIDZ2 vdev, destroy my old pool, then add my old disks to the new RAIDZ2 vdev. But it seems like overkill to buy five drives just for this temporary migration.

I have an idea to do it with only three drives, and I wanted to see if it’s sensible, since I don’t see anyone discuss it.

Here’s my plan:

  1. Create a RAIDZ2 vdev with three new 8 TB HDDs
  2. Set my old RAIDZ1 vdev to read-only
  3. Remove the weakest HDD from my existing RAIDZ1 vdev
  4. Add the old HDD as the fourth drive to my new RAIDZ2 vdev (giving new vdev about 16 TB, enough to migrate my data)
  5. Migrate all datasets from my old RAIDZ1 pool to my new RAIDZ2 pool
  6. Delete my old RAIDZ1 pool
  7. Add the remaining three disks to my new RAIDZ2 pool

I know that I’m taking a risk in that if any disks on my old RAIDZ1 vdev fail during steps 3-5, then I lose all data on my old pool, but I have backups via restic on multiple cloud storage providers. It would be a pain to restore from backup, but it would be doable. My understanding is that the risk I’m taking is on par with a drive from my RAIDZ1 vdev dying from natural causes.

Aside from the risk of data loss in steps 3-5 if a drive fails, is there anything wrong with this plan?

2

Create the new Z2 array with 3 new disks PLUS a sparse zvol, removing the zvol after array creation to create a degraded array (acting as a Z1). This will give you 16TB whilst maintaining parity in both arrays

Then use old disks, once data is transferred to:

  1. Replace missing disk to “upgrade” the array back to Z2
  2. Add disks, one by one to the pool to increase capacity
2 Likes
3

Is the purpose of the sparse zvol to allow me to create the RAIDZ2 vdev with <4 disks? I keep seeing people say that the minimum disks for RAIDZ2 is 4 disks, but I’ve tested it on some USB drives, and it creates a vdev fine with just 3:

$ zpool create \
  -f \
  usbpool \
  raidz2 \
  -m /mnt/usbpool \
  /dev/sdc \
  /dev/sdd \
  /dev/sde
$  zpool status usbpool
  pool: usbpool
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        usbpool     ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sde     ONLINE       0     0     0

errors: No known data errors
$ zpool list usbpool
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
usbpool  85.5G   378K  85.5G        -         -     0%     0%  1.00x    ONLINE  -
4

Not really.

The purpose is to create the Z2 with 3 disks and have 16TB (ish) of space

5

The reason you see people recommend against raidz2 for 3 disks is probably because it would make more sense to have a three-way mirror, for performance reasons. However, that’s for situations where you aren’t planning to expand beyond 3 disks.

In your case, you plan to add more disks to the vdev.

6

Oh, gotcha!

That’s clever, thanks! That mitigates the risk of losing data if a single drive on the old vdev fails during the migration.

7

Note that it is recommend to use UUIDs rather than drive letters when creating pool from teh command line.

1 Like
8

Circling back here, this solution works. Thanks to @NugentS and @etorix for the tips!

I wrote a blog post explaining my process in detail:

4 Likes
9

That was interesting to read. However, I do believe you wouldn’t need that kind of somersault with a proper 3-2-1 backup strategy. Well, you would have been fine even with 2-2-0.

1 Like
10

Thanks for reading!

How do you avoid it unless you’re doing ZFS-native backups? I could have blown away my non-media data and restored the files from backups, but then I still have to recreate ACLs, datasets, cron jobs, TrueNAS config, no?

11

I meant ZFS-native backups. Without ZFS, your backup can simply be bit-rotten. Perhaps cloud solutions also have some kind of bit-rot prevention.

Truenas config stores jobs and shares settings. Datasets are stored at “pool level”. AFAIK, ACLs are stored within datasets themselves. So you would need to restore your datasets (in both scenarios) and ACLs in the non-zfs scenario.

On second thought, with non-zfs backups, you could delete all the data in each dataset, temporarily replicate it to some zfs system, destroy the main pool, recreate it with the desirable configuration, replicate the datasets back, and repopulate them from backups. Thus, (dataset level) ACLs and datasets are preserved. It will spare the extension time. This approach is close (by its consequences) to the case when your main NAS got destroyed/yoinked.
Well, I don’t think that it is definitely a better approach than yours. I would call it a somersault as well.

1 Like
12

That’s the issue I ran into here:

I’d rather just zfs send my data to a spare ZFS server for a few days, but I don’t know of any host that lets you rent a ZFS pool for only a few days. The closest is rsync.net, and they make me pay a whole month, ($180 for 18 TB).

13

Yeah, saw it.