Enhance the user experience during RAIDZ expansion

Over the last two weeks, I tried out the RAIDZ expansion feature with an 18TB drive in my main pool. I encountered a few issues that could be handled better. When performing a RAIDZ expansion, the current process can be optimized to improve user information and system performance.

Estimated Duration Information:
Clearly inform the user of the estimated duration for the RAIDZ expansion process. This can help set realistic expectations and plan accordingly(~50MB/s,https://openzfs.org/w/images/5/5e/RAIDZ_Expansion_2023.pdf).

Automatic Deactivation of Scrubs:
During the expansion process, automatically deactivate scrubs for the affected pool. This is to prevent significant IO WAIT issues, which can severely impact system performance. Once the expansion is completed, scrubs can be reactivated.

Progress Display in Task Manager:
Integrate detailed progress information from zpool status into the Task Manager. This includes the current data transfer rate, percentage completed, and estimated time to completion. For example:

expand: expansion of raidz1-0 in progress since Wed Jun 19 01:12:14 2024
        1.99T / 50.9T copied at 54.8M/s, 3.91% done, 10 days 19:48:35 to go



1 Like

I agree, i couldn’t even find the option, i kept trying from expand and other entry points like the ‘you have 1 disk unused tile’ agree with all your points.

Until TrueNAS 24.10 hits BETA, it should not be assumed its complete.
After it does, please provide your feedback.

Can you provide details of system before and after, how much data was in the pool, and how long RAIDZ expansion took?

That data might be useful for making the UI more informative.

Actually, most/all of the data in the vdev gets shuffled around. It stays at the same offset within the vdev, but it gets shuffled around to match the new geometry.

1 Like

Good catch.

It preserves existing parity - So it keeps the data
Allows for online expansion - so reads and writes can continue immediately
But it shuffles the data to free the space - that takes a while.

Great video here: https://www.youtube.com/watch?app=desktop&v=yF2KgQGmUic

I’ll fix my post.

We do have a tutorial available for the process now, available here: Managing Pools | TrueNAS Documentation Hub

2 Likes

It took ~ 11 days (~55M/s). Raidz 5 pool was filled at 75%, 4x 18TB. Now 5x 18TB. I’m on the nighty train. It would have taken longer If I hadn’t stopped the scrubbing. In the nedata screenshot you can see right what happend after I stoppend the scrubbing.

Also a good video: https://www.youtube.com/watch?v=tqyNHyq0LYM

Out of curiosity, can you post the output of zpool list -v POOLNAME


root@truenas[~]# zpool list -v Media
NAME                                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Media                                     81.8T  51.3T  30.5T        -         -     2%    62%  1.00x    ONLINE  /mnt
  raidz1-0                                81.8T  51.3T  30.5T        -         -     2%  62.7%      -    ONLINE
    3d0bc877-03e7-4c8d-ab07-7fc46a39d179  16.4T      -      -        -         -      -      -      -    ONLINE
    7eeedebb-b6cb-42d9-8c09-a49929d3009d  16.4T      -      -        -         -      -      -      -    ONLINE
    78cb2807-3d85-4ab9-8579-3a87a907a84b  16.4T      -      -        -         -      -      -      -    ONLINE
    6ecb6553-780f-4448-8c5b-4db5204e3bbd  16.4T      -      -        -         -      -      -      -    ONLINE
    15c58918-9e74-4ee0-afdc-7b57f35d1873  16.4T      -      -        -         -      -      -      -    ONLINE
cache                                         -      -      -        -         -      -      -      -         -
  faba655c-d81c-495d-99ad-dd20d28bb30e     466G   466G  68.0M        -         -     0%  100.0%      -    ONLINE

1 Like

The stable build came out today. Unfortunatly, the user experience has not been improved. I have the feeling that TrueNAS is working on expanding my 4x 4 TB to 5x4TB Z1 pool, but the interface got stuck at the same screen as the OP.

Wonder if there is a way to check the intermediate progress manually?

sudo zpool status in the shell

3 Likes

I am running a RAIDz expansion (says it’s going to take 10 days!) and I also had poor UI experiences.

First, the job sat all day at 25%. It happened that I needed to restart the server, and since I had no idea how long this was going to take I just bit the bullet and did it.

When it came back the drive showed as unavailable. Ok, my server is kind of weird, I can accept that.

Upon rebooting again, all drives were visible. The pool showed the same capacity as before, but with all 5 drives. There was no indication that the expansion was still happening.

I confirmed via the shell that the expansion is in fact still happening, though it is going to take a very long time >_>