Preferred way to mass-replace healthy drives?

Title feels a bit longer than I think I’d prefer, but effectively what I’m looking for.

Backstory:
I’ve had this pool for over 4 years and it’s suited me well.
I’m aware that at around 50% usage, you start seeing pool performance degradation, and you don’t really want to go over 80% because of larger degradation. As well as the further you get to 100%, the more likely you are to have problems with your pool due to ZFS being a CoW filesystem.

I figured that I’d upgrade some drives, and found some 18 TB SATA drives used for Chia at a decent price.
Over the past several days I’ve been burn-in testing them with bad-blocks (these Toshiba MG09s seem to be faster at reading than these Seagate Exos drives).

My current hardware, for what’s potentially relative:
NetApp DS4246
Single IOM6 module
HDD: 12x 3 TB Hitachi SAS HDDs

R730xd
CPU: 2x E5-2643 v4
RAM: 256 GiB (ECC of course)
HBA: LSI 9300-8e

Info on my current pool:
2x RAIDz2 vdevs of 6x drives
76% used space
25% fragmentation

Actual question:
I’d like to replace the 3 TB drives with the 18 TB drives, but I’m curious on the preferred way to do so.

Since I have the NetApp shelf, I can (and am) comfortably running both sets of drives, so regardless all drives would be online.

The way I see it, there’s two potential ways I could go about this:

  • Online-replace one-by-one (potentially two-by-two due to 2 vdevs) and expand at the end
    • Need to deal with a resilver between each addition
      • Resilvers aren’t sequential
    • Not really complicated
  • Create a new pool on the new drives, sync the data, export the pools, rename, and import
    • I’d probably zfs send | zfs recv for this, doing it once for initial data, stopping everything that writes to the pool, re-syncing, and performing the rest
      • Not sure if zfs send | zfs recv is sequential, but probably faster
    • More complicated

I’m not in a massive need of the free space currently, and can easily delete things to free space before it becomes a concern.

I’m not sure if there’s any other pros or cons. I’d imagine that the send | recv would probably reset my free-space fragmentation, but maybe fragmentation would also drop since there’d be more sequential free-space with the new drives?

Just curious to hear thoughts, I’m probably still several days away, seeing as how the Exos are almost done with the write cycle of the second pass, and the Toshibas are around 20% done with the read cycle of the second pass.

Replacing 3TB drives with 18TB drives sounds like over-kill. Such that you may want to start with a new pool. But, it’s your choice.

In theory, you can replace all 12 drives at once. This would be “replace in place” type of replacement. ZFS will create a temporary mirror between the old drive and the new. Whence the new drive is fully synced up, the old drive is detached from the temporary mirror, restoring the pool to how it was before.

Because “replace in place” does not remove the old drives until the new one is fully synced up, it is reasonably safe for your data.

In practice, the CPU capacity and or I/O bandwidth would likely be bottlenecks. So doing 2 per vDev maximum would probably be faster.

Their is a special ZFS tunable that allows more than 1 drive to be replaced at a time. But, if you choose to replace more than 1 drive at once, do it immediately after starting the 1st. It appears that starting another drive replacement causes the 1st drive replacement to start over.

Newer versions of OpenZFS allow sequential re-silvers, which are supposedly faster.

I do agree that it sounds over-kill. Really a lot of it has just come down to pricing and seeing the need for some more storage.
I paid $16.50/drive for these 3TB ones. Looking over $/TB for larger used drives, 18 TB seemed like the sweet spot.
The larger space will also let me expand into various data retention, so it seems fine to me.

I appreciate the information, and I’ll try to take a look into that special tunable as well as keep an eye out for any other information provided.