Migration path from 1 drive to 3 drives in Raidz1 (using only 2 new drives to the existing 1)

TrueNAS General
1

Hello. I am new to TrueNAS. I have 1 currently used 18TB drive (0 redundancy) and 2 brand new 18TB drives. I am looking for a migration path that would leave me with 3 drives in raidz1

Although it’s totaly silly as any parity calculation would be inferior to a mirror - is it possible to create a raidz1 with only 2 drives (with redundancy already (basically having 1 “data” and 1 “parity” drive (yes, I understand that it’s similar to raid5 and therefor parity is spread across drives, unlike raid4 but you got my drift :slight_smile: )

I will then add the “source” drive to the array making it a true raidz1 (3 drives).

WDYS? I would have started with a mirror and then converted to raidz1 but it seems that this path is not supported.

Thanks.

2

You have two routes forward (but both involve using the CLI)

  1. You can create a 2 drive RAIDZ1 array using the CLI, move your data and then use RAIDZ expansion to add the third drive.

  2. You can create a 3 “drive” RAIDZ1 array using 2 drives and a sparse file on your existing drive using the CLI, then you offline the sparse file and delete it leaving you with a degraded 3-wide RAIDZ1 using just the 2 new disks. When you have moved your data, you then resilver the offline “disk” using the 3rd drive.

Both of these require some technical knowledge because you need to use the CLI and you ideally want to create the pool in the way that the TrueNAS UI would i.e. partitions, and partition-UUIDs. I haven’t done this myself, so I cannot give you detailed guidance on the commands you need to run.

1 Like
3

Method 2. avoids raidz expansion, and its persisting caveats with respect to space reporting.

Both methods involve CLI, messing with dragons, and raidz1 with 18 TB drives is still not very secure (a 4-wide raidz2 would be preferable).

1 Like
4

Method 2 looses redundancy until the 3rd disk is re-silvered.

While method 1 implements redundancy immediately as a 2 disk RAID-Z1.

Both are not for the casual users.

TrueNAS is not perfect. Perhaps HexOS, based on TrueNAS SCALE, will include 2 disk RAID-Z1 as an option. (In addition, the RAID-Zx expansion would also be required…)

5

Thank you all.
Using Linux since the SLS days I am not a stranger to a shell but I do not know zfs and especially - truenas’ boundaries of using it. So I admit I am a bit weary of using an untested methodology and hopes that someone has already done so.

Option 2 (start in degraded mode) is something I’d rather not do.

Btw - as for raidz1 with large drives. Is there anything specific about raidz1 vs raid5 in that regard? The demise of raid5 is more relevant for non enterprise drives where the URE rate is 10^14 bits whereas enterprise drives are rated at 10^15 bits. (All my drives are enterprise grade).

Also. Again - not an expert on zfs - from my understanding a read error during resilver would result in a bad file but the resilvering would continue?
Wouldn’t that mean that if I am willing to accept some data loss in extreme cases raidz1 would be ok?

Thanks!

6

Fair point. There’s still the other side of the argument: More data to resilver, longer time, and the risk that another drive fails under the continuous strain of resilver.

Correct. It is a matter of your backup strategy and the risk your’re willing to take.

7

Yes - up to a point.

If the read error is in a file, you delete the file and scrub and zpool clean and your pool should be fine.

If the read error is in metadata, a scrub may not fix it and you may need to be a serious ZFS expert to roll back to a valid pool with significant loss of recent data. Or worst case your pool may end up being unrecoverable.

8

Except that by default standard Metadata has 2 copies regardless of vDev redundancy. That means 2 specific & separate RAID-Z1 stripes would have to have bad blocks at the same time, for standard Metadata to fail.

Standard Metadata can be part of the file description, like directory entries. ZFS considers loss of standard Metadata worse than a data block because you can loose the whole file, or large part of it.

Here are the ZFS Dataset controls for Metadata:

redundant_metadata    all             default
copies                1               default

Changing “copies=2” bumps up standard Metadata to 3 copies, (because their are now 2 data copies). (Encryption prevents 3 copies…)

As for critical Metadata, it starts off at 3 or more copies…

9

Is this the Copies field in the dataset Advanced Properties?

image

The tooltip talks about “data copies” but I think it should say “metadata copies”.

10

The CLI parameter is copies=N, and this really is about data; metadata copies are adjusted accordingly.

11

Sorry - but I am unclear what you mean?

12

Hi all.
OK, so I made some progress.
The anal in me decided to go for dual parity anyway, I have a Lenovo SA120 JBOD with plenty of place for disks, so I shelled a bit more money and got myself a 3rd (new) 18TB Exos drive.

So I have 3 new drives and 1 old drive that I will add later. I decided to RTFM a bit and did the following:

Created a sparse file:

dd if=/dev/zero of=/tmp/foo.img bs=1 count=0 seek=18T

Created the pool:

zpool create main_pool raidz2 /dev/sda /dev/sdd /dev/sdf /tmp/foo.img -f

I had to use -f to get it to use the file

Mark the sparsefile as faulty:

zpool offline -f main_pool /tmp/foo.img

export so I could import it into truenas:

zpool export main_pool

I then imported it into TrueNAS, it shows:

Data VDEVs 1 x RAIDZ2 | 4 wide | 16.37 TiB
Usable Capacity: 31.59 tiB

And says it is in degraded mode.

Honestly, much easier than I thought, I am now copying all the data from the old drive to the array and I will then plan to add the drive.

Now, here is a question:
During my testing I noticed that I can also do this:
zpool create main_pool raidz2 /dev/sda /dev/sdd /dev/sdf

This will create a 16TB array 3 disks wide with 2 used for parity and not in degraded mode

What would be the safest approach? do this and expand with the 4th drive or keep it in degraded mode and then resilver using the old drive once I finish copying everything (or it doesn’t really matter).
I wonder if extending is more “expensive” than resilvering / replacing a defective drive

Thanks!

13

You have definitely grasped the basics of going each of the two routes I suggested.

Expanding will be slower than resilvering, and you will need to rebalance afterwards to get the best efficiency.

Using device names is not recommended as the order can change on a reboot. It is better to use disk or partition UUIDs. Thus you should try to:

  1. Create partitions and use them, and then do zpool create using /dev/disks/by-partuuid/xxxxxxxx . (You will need to lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID to map device names to partition-UUIDs.)
  2. If not do zpool create using /dev/disks/by-uuid/xxxxxx` instead.
1 Like
14

Not really specific, but mildly relevant. Assessing the Potential for Data Loss | TrueNAS Community

15

You can change the base value of the copies parameter, which affects standard data. Metadata and critical metadata are adjusted proportionally to this base value (but cannot exceed 3 IIRC).

16

Yes, that GUI you referenced is likely the same as the command line’s “copies” attribute.

While the GUI states “data copies”, it really does all 3:

  • Data, defaults to 1
  • Metadata, defaults to 2
  • Critical Metadata, defaults to 3

Each is bumped up by 1 copy if you bump up “copies=” by 1.

2 Likes
17

Thanks @Protopia !

I totally missed the uuid reference!

So I destroyed the old pool and created it anew using partitions and their respective uuid.

18

How do I replace the failed file? There is no replace button like regular drives have?

Thanks.

image
image1049×572 29.5 KB

I guess I need to do it only via shell?

I assume it’s this?

zpool replace main_pool /tmp/foo.img /dev/disk/by-partuuid/new_drive

Do I need to export the pool or can I just run it in the shell?

19

I think you should be able to do it in the UI. If you select the file is there an Offline or Replace button you can click?

20

Nopes. No replace button. It just shows it. Other drives have replace, etc…

It’s simply a one-way street once I do the replace. I’ll be erasing the original source drive so I want to make sure I am not ending up with 0 files in the end…

So I have 2 questions:

  1. Is the command I wrote is the correct procedure or do I need to do something different/in addition? (Previously I had the luxury of experimenting. Not here)
  2. Is it ok to run zfs commands in the CLI with truenas scale while the pool is attached or do I need to export it first? Thanks.