Adding new disk to Pool

[misterpele]

I have a server that I am using RaidZ2 and have 8 hard drives and want to expand it to 9 now and maybe up to 12 in the next year.

TrueNAS Scale 25.04.2.5
RAM: 64GB
8 x 16TB Hard drives (all CMR, Seagate,) (RaidZ2)
2 x 256GB NVME (boot) 
MSI Pro Z790-A MAX (i5-12600K)
LSI SAS3008 9300-8I IT-Mode HBA JBOD PCI-E 3.0
Case: Fractal 7xl
PSU: Focus GX-850 power supply

This server has been with me for years (Since Truenas 9 Core) so it’s setup may not be the best, but too big now to rebuild.

Here is my pool status.

root@truenas[~]# zpool status MainNAS
  pool: MainNAS
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub canceled on Sat Oct 25 12:15:22 2025
config:

        NAME                                      STATE     READ WRITE CKSUM
        MainNAS                                   ONLINE       0     0     0
          raidz2-0                                ONLINE       0     0     0
            b280a980-dfe3-4fdc-ae6f-c3eea97fe9ac  ONLINE       0     0     0
            22edd432-9194-11ef-9c05-a85e4550970d  ONLINE       0     0     0
            15bb2646-832a-11ec-8e9c-a85e4550970d  ONLINE       0     0     0
            15e4ae7c-832a-11ec-8e9c-a85e4550970d  ONLINE       0     0     0
            0eb843ed-2c1b-4755-b134-f5eda82a893e  ONLINE       0     0     0
            16430264-832a-11ec-8e9c-a85e4550970d  ONLINE       0     0     0
            fce004e2-1ea5-4e85-82b4-8629a74deccd  ONLINE       0     0     0
            d0d7281e-b043-45ef-9335-730c13a2994b  ONLINE       0     0     0

errors: No known data errors

The drive I am adding is the same 16TB drive that’s already in. I am down to my last 1TB left on my pool.

My question is, is there a certain way I need to do this? Do I need to make a VDEV as I don’t have any currently, just MainNAS pool. And if I do, will it incorporate with my others or be a completely different drive om my SMB?

Thank you..

If anything other information is needed I can provide it.

[misterpele]

Or do I use Extend VDEV option in Storage-Datasets?

If I do that it says this if I chose that option “The expanded vdev uses the pre-expanded parity ratio, which reduces the total vdev capacity. To reset the vdev parity ratio and fully use the new capacity, manually rewrite all data in the vdev. This process takes time and is irreversible.”

Is that the option I want and how do I manually rewrite the data (does that entail reslivering/)?

[dan]

@misterpele

Is that the option I want

Yes.

@misterpele

how do I manually rewrite the data

GitHub - markusressel/zfs-inplace-rebalancing: Simple bash script to rebalance pool data between...

Or use the zfs rewrite command–though I don’t think this is very well-documented at this point.

[Arwen]

Note that you main RAID-Z2 is a vDev, (Virtual Device). It is just your only data vDev.

@misterpele

Or do I use Extend VDEV option in Storage-Datasets?

If I do that it says this if I chose that option “The expanded vdev uses the pre-expanded parity ratio, which reduces the total vdev capacity. To reset the vdev parity ratio and fully use the new capacity, manually rewrite all data in the vdev. This process takes time and is irreversible.”

Yes.

@misterpele

Is that the option I want and how do I manually rewrite the data (does that entail reslivering/)?

Yes.

There are multiple ways of re-writing the data. But, they all come with caveats. Basically snapshots, clones, hard links and block clones all hold onto the prior data. Thus, can fill up your pool quite quickly.

No, a re-write does not cause a re-silver. It is just a normal re-write of the files which would then expand any stripes written to use the new data to parity ratio.

Adding just one disk might not make it worth it to re-write all your data, if you plan on adding more disks sooner rather than later.

As for the exact method to use, many people use a re-write script. These forums have a link to one somewhere. Another method could be by ZFS Send & Receive, but again you need more free storage than you likely have.

[misterpele]

@Arwen

Adding just one disk might not make it worth it to re-write all your data, if you plan on adding more disks sooner rather than later.

So if I planned to add 2 more 16TB (3 total), I should do it all at once.

@Arwen

@misterpele

Or do I use Extend VDEV option in Storage-Datasets?

If I do that it says this if I chose that option “The expanded vdev uses the pre-expanded parity ratio, which reduces the total vdev capacity. To reset the vdev parity ratio and fully use the new capacity, manually rewrite all data in the vdev. This process takes time and is irreversible.”

Yes.

Thank you.

@Arwen

As for the exact method to use, many people use a re-write script. These forums have a link to one somewhere. Another method could be by ZFS Send & Receive, but again you need more free storage than you likely have.

I will have to look into those, though tbh a bit nervous about it, but I need more data space..

@Arwen

There are multiple ways of re-writing the data. But, they all come with caveats. Basically snapshots, clones, hard links and block clones all hold onto the prior data. Thus, can fill up your pool quite quickly.

I do have snapshots setup but I can easily turn those off. I don’t do any cloning or hard links, so I should be ok there.

[Arwen]

@misterpele

@Arwen

Adding just one disk might not make it worth it to re-write all your data, if you plan on adding more disks sooner rather than later.

So if I planned to add 2 more 16TB (3 total), I should do it all at once.

You don’t have to do it all at once. But, doing a re-write to reduce overhead of data to parity ratio, each time you add a disk might not be worth it. Waiting until after you add your last disk might be worth it. You have to choose.

By the way, there is an OpenZFS feature extension to RAID-Zx expansion, to allow more than 1 disk to be added at a time. Its in progress, not available yet. At present, you can only add 1 disk at a time. Then after all that work, RAID-Zx expansion still has to run a full scrub of the data. Basically doubles the work for single Data vDev pools. Doing a full re-write of the data to get higher data to parity ratios simply adds even more work for the disks.

[misterpele]

@Arwen

You don’t have to do it all at once. But, doing a re-write to reduce overhead of data to parity ratio, each time you add a disk might not be worth it. Waiting until after you add your last disk might be worth it. You have to choose.

So you are saying if I add the drive, I don’t have to do the re-write? If that is the case, what are the downfalls/consequences of not doing that right away?

[Arwen]

@misterpele

So you are saying if I add the drive, I don’t have to do the re-write?

Correct.

With the existing 6 Data to 2 Parity ratio, you get pretty good disk usage.

@misterpele

If that is the case, what are the downfalls/consequences of not doing that right away?

Little.

In essence, existing data remains at the 6 Data to 2 Parity ratio. That’s it.

Unlike RAID-5/6 which use fixed width stripes, RAID-Zx is variable stripe width depending on the amount of data written, at the time of write.

Meaning if you write an 1KiloByte file, and then append another 1KiloByte, repeating until you get 100GigaBytes.you would get horrible data to parity ratio. Each 1KiloByte write would be it’s own ZFS RAID-Z2 stripe, taking 3 columns. One column for data and 2 for parity. Perhaps not a perfect example…

This implies that RAID-Zx is not really suitable for tons of small files that can’t fill up RAID-Zx stripes. That is true. But, many uses of a NAS are for files larger than the RAID-Zx stripe width. The few files that are smaller, just work with the less than ideal storage ratio.

[misterpele]

@Arwen

With the existing 6 Data to 2 Parity ratio, you get pretty good disk usage.

So if I add the 9th drive the “new data” being written, does that only go on the 9th drive until I do the re-write or does it get put with the other 6 Data and 2 Parity I have? And if it gets put in with the 6 x 2, then what actually is the point of the re-write?

Also thank you for your input/help.

[Arwen]

You are complicating this too much.

New data for a 9 disk RAID-Zx vDev gets up to 7 data columns and 2 parity. It depends on if their are 9 column stripes free AND if the write needs that much storage.

[misterpele]

@Arwen

You are complicating this too much.

New data for a 9 disk RAID-Zx vDev gets up to 7 data columns and 2 parity. It depends on if their are 9 column stripes free AND if the write needs that much storage.

I probably am. So whatever new stuff gets written will be written on all 7 data discs but the old stuff stays on the 6 data discs. Correct?

[Arwen]

Yes.

[misterpele]

When trying it I get this message

[EZFS_BADTARGET] cannot attach /dev/disk/by-partuuid/5f7f8a7f-1462-4c6c-9e60-612ccf77c0cd to raidz2-0: raidz_expansion feature must be enabled in order to attach a device to raidz

Is this because this was made in Core many many years ago and I have never listened to GUI and did any ZFS update (since I didn’t need features) even though I am on Version: 25.04.2.5.
I still get this in gui

 Notice
New ZFS version or feature flags are available for pool 'The 96er'. Upgrading pools is a one-time process that can prevent rolling the system back to an earlier TrueNAS version. It is recommended to read the TrueNAS release notes and confirm you need the new ZFS feature flags

[Arwen]

Yes, you need the RAID-Zx Expansion feature enabled.

It is generally not recommended to update all pool features. This can cause a pool to only be usable with current ZFS, (and TrueNAS CE). No going back to earlier TrueNAS releases, even as short as 6 months ago. (ZFS tends to add new features several times a year.)

One way to deal with this, is to just enable the features you want from the command line:

zpool set feature@raidz_expansion=enabled MainNAS

There may be other prerequisite features required for RAID-Zx Expansion, I just don’t know. But, RAID-Zx Expansion has been available for over 1 year, so it should be supported by a few earlier releases of TrueNAS CE.

Of course, enabling the RAID-Zx Expansion feature will prevent this pool from being imported by earlier versions of ZFS that don’t support RAID-Zx Expansion.

[misterpele]

@Arwen

One way to deal with this, is to just enable the features you want from the command line:

zpool set feature@raidz_expansion=enabled MainNAS

I will do this on MainNAS tonight, it is doing a quick backup of something for me now.

When I try it on another server that has “The 96er” as the pool name it gives me this message (below). Is that because it is 2 names and is there a way around that?

Zpool Status:

  pool: The 96er
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 1 days 04:30:39 with 0 errors on Mon Oct 27 07:30:40 2025
config:

        NAME        STATE     READ WRITE CKSUM
        The 96er    ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            sdb2    ONLINE       0     0     0
            sdd2    ONLINE       0     0     0
            sde2    ONLINE       0     0     0
            sda2    ONLINE       0     0     0
            sdh2    ONLINE       0     0     0
            sdg2    ONLINE       0     0     0
            sdc2    ONLINE       0     0     0
            sdf2    ONLINE       0     0     0

and output from command

root@truenas[~]# zpool set feature@raidz_expansion=enabled The 96er
cannot open 'The': is not a pool

[dan]

Spaces in pool names are a bad idea, as you’re finding out. As a workaround, enclose the pool name in quotes: zpool set feature@raidz_expansion=enabled "The 96er".

[misterpele]

Thanks, that seems to be working or at least going forward LOL.
Yes, I have learned, I started this pool 7 years ago and wasn’t planning on keeping it, but here we are.

[etorix]

HOWTO: Rename a ZFS pool

The basic process to rename a ZFS pool is to export it from the GUI, import it in the CLI with the new name, then export it again, and re-import it in the GUI. I find I normally want to do this after creating a new pool (with perhaps a different set of disks/layout), replicating my old pool to the new pool, and then I want to rename the new pool to the same as the old pool, and then all the shares work correctly, and its fairly transparent. Mostly. (tested with TrueNAS CORE 12U8-1. Works with …

[misterpele]

@HOWTO: Rename a ZFS pool

The basic process to rename a ZFS pool is to export it from the GUI, import it in the CLI with the new name, then export it again, and re-import it in the GUI. I find I normally want to do this after creating a new pool (with perhaps a different set of disks/layout), replicating my old pool to the new pool, and then I want to rename the new pool to the same as the old pool, and then all the shares work correctly, and its fairly transparent. Mostly. (tested with TrueNAS CORE 12U8-1. Works with …

Thank you for this. I will try it as soon as it is done adding the expansion.