Slow expansion of RAIDZ1

While I’d prefer to keep the discussion on topic, I do want to explore your suggestion.

Keep in mind that

  • I “just” received the device and purchased the 3 drives
  • I migrated from only 1x8TB on a different SBC to the TrueNAS with the initial 3x8TB Z1 one week prior
  • All the data is ‘just’ recoverable media data
  • keeping some budget in mind since I bought the 3x8TB for this cause

Is the wise decision really to buy 2 more and to follow your guide?
I reasoned that I considered having a 2-disk fail tolerance to be overkill for my use since I’ve accepted that if SHTF and a second disk fails during shipment/resilvering of the replacement then so be it and I’ll have to re-acquire the data.

How could I have included the “donor” disk without expansion or taking down the old system?

Are there other benefits to your suggested 6xZ2 (I don’t even have room for 8x…)

1 Like

I think your rationale for staying on RAIDZ1 is a sound one.

This is all about risk.

My own NAS has 5x 4TB RAIDZ1 holding some backups of personal files, and a lot of re-downloadable media.

  1. At the time I created the pool, RAIDZ1 seemed OK for this.

  2. Later I decided RAIDZ2 would have been better but by then it was too late.

  3. I bought a one-time payment for 500GB of online storage to do offsite back up of my critical files and reduce my risks.

  4. Now I am OK again with RAIDZ1.

1 Like

To stay roughly on-topic before I veer sharply off again: A week to expand a raidz seems acceptable, if it really took that long. That’s also roughly the time to do a full test / burn-in of a drive. So, new drive gets tested, 1 week, then expanded, another week - seems fine.

Free space display not being correct seems a larger annoyance than the time it takes to expand a raidz.

If the cost of acquiring the drive(s) weighs higher than the risk of needing to re-acquire all those media files, then no.

Risk tolerance is highly individual.

Lengthy musing on risk, skip as desired.

In my case, I too use the NAS for space-shifting media files. Everything exists on CDs, DVDs and Blurays. I am strongly opposed to using the -arr apps, “theft is wrong” kinda thing.

It took me probably weeks to rip everything to files - as in weeks of actual time spent, months to actually get it done. I don’t want to do so again. And because I don’t want to do so again, the extra parity drive is worth it. To me.

Space-shifting the media is a flaunting of the law I’m personally OK with, as I do own the physical discs and don’t share the electronic version.

I do not back it up, there’s my current risk tolerance. It’d be reasonable to build a second NAS as a backup target. But even that doesn’t get me to 3-2-1: This is too much stuff to keep an off-site copy, just by the sheer amount of time it takes to move all this crap over a 200Mbit up link.

Thinking further, if I lose the local NAS to fire/flood I also lose the physical media, and that means I need to re-acquire physical media (as well as shelter, clothing, and all my belongings). At that point I have other worries than whether my media library was backed up to cloud.

But I may talk to the husband and see whether he can stomach a second NAS as a send/receive target in the house.

Documents on PCs (not the media stuff) are backed up to cloud ofc, and to the NAS.

2 Likes

I used the exapnsion feature from 5 x 18TB Seagate Exos to 6 in Raidz2 . I tried the raidz_expand_max_copy_bytes=160MB command but it didn’t do anything I don’t know if its right or not but nothing reported back after putting the command in. Anyway I am stuck at 31MB/S . I think it should be a lot faster but if anyone has a suggestion. I am using PCI-E 3.0 x16 expansion card for all the HDDs should be fast enough. Cpu is i3 8th gen says I have a lot of free RAM so I don’t think it’s a hardware limitation.

Details on what card you are using?

Can’t find the email I bought it from but I am doing around 250MB/s doing rsync backups from a synology nas so its not the card bottleneck.

Update found it as it turns out it’s PCI-E 2.0 X2 but that’s still around 800MB so not the problem here is a link.

https://www.delock.com/produkt/89384/merkmale.html

1 Like

That is the kinda card that has most experienced members of ths forum shuddering with dismay.

Mostly these end up costing the user data - they work till it all goes horribly wrong - and its generally not fixable

You should be using a proper HBA (LSI card) for additional SATA ports.

3 Likes

It’s a cold storage server so not so important but it works fine for 3 years now so can’t complain. Anyway the problem for the slow expansion is not the card fault since as I said I am writing through rsync with much higher speeds.

Sure, some cards and controllers are better than others, there is a wide variety in build and design quality. 10 SATA ports points to a expander, which further limits performance, iirc.

I guess part of the danger is that they work o.k. on a low workload, but under heavier workloads the problems begin. Like a resilver.

Even if it is not the cause of your problems, considering replacing it with a SAS HBA might not be a bad idea. Or at least checking e.g. the old forum if people experienced problems with this card/the controller chip used.

That’ll be an uphill battle to prove - too many posts with with port multipliers having various issues. From intermittent nonsense, all the way to data loss.

To clarify, I’m not saying it is the problem, but unless you can replicate the same results without the card in play, I think you’ll be at an impasse in terms of support.

I just changed my 5 hdds 10TB to 5 16TB Resilvering was done 1 disk at a time it took around 18 hours I did the same in my synology NAS around the same time so the card works. I have only 4 SATA ports on the motherboard it’s just and old pc nothing special. I found like 4-5 threads where people have low expansion speeds so it’s not just my use scenario. Anyway If someone has a suggestion to try something I am open to it :slight_smile: . Since it’s taking this long I will just erase everything and do a new fresh pool as I said it’s cold storage so not a problem for the data.

Fair enough, I think IX stated on some of the posts that expansion is expected to double in speed in a future update. Some of the posts though had SMR drives, some also had sata controllers, but to your point - some just confirm that it isn’t a fast process.

Yes, but it reads/writes to all disk simultaneously, even if you “only” add one disk, IIRC.
But sure, raidz exansion is pretty new and rather slow.

A raidz expansion on synology? How does that prove the card works?

Actually that sounds like a good idea, the next thing you will probably stumble on is that pool size is not shown correctly in the gui and the old data will still be at the old parity level!

I mean I will erase eveeything whole pool and put the data back again this should fix the parity ratio right? As for the synology I was talking about resilvering/repairing process of a disk change.

1 Like

Yep

I added a disk to a Synology array (Raid6 of 8 disks initially). It took just under a month to complete.
{just for context}

I just copied my data back did the new pool. I did a performance test just to be sure its going around 240/300Mibs so the card was not the problem. Can memory be the problem though cause since it’s a cold storage I had some 2x4GB ram laying around and just used that its old DDR3 so I can get more for very cheap. I know ZFS rule that 1TB=1GB ram so if that’s a problem it may explain the slow expansion.

1 Like

You would have to take a look at your Memory stats and see if it is starved.

Hi, just to add another datapoint to this thread.

I have an Ugreen DXP6800 Pro with latest TrueNas Scale installed, 64GB of RAM, 5x8TB IronWolf ST8000VN002 CMR disks, expanding after adding a 6th drive sdf in this case

root@saturn[/home/truenas_admin]# zpool status -v titan
  pool: titan
 state: ONLINE
expand: expansion of raidz2-0 in progress since Thu Feb  6 18:56:35 2025
	1.96T / 4.83T copied at 38.9M/s, 40.65% done, 21:29:44 to go
config:

	NAME                                      STATE     READ WRITE CKSUM
	titan                                     ONLINE       0     0     0
	  raidz2-0                                ONLINE       0     0     0
	    c7c91079-5b5c-4002-8423-5fb327acd140  ONLINE       0     0     0
	    ff00ea78-25a3-4998-adbb-ebdd5eb9970b  ONLINE       0     0     0
	    0cfdcdca-ef0e-4a98-a8a5-0acf2a33fca9  ONLINE       0     0     0
	    f802ab1b-e609-41b7-b6a0-60b5aef31e90  ONLINE       0     0     0
	    c8a8ee04-4bfa-49cf-ae1f-6f699875f660  ONLINE       0     0     0
	    c5d6db45-972c-473f-afcf-b9b9fae6d5fa  ONLINE       0     0     0

errors: No known data errors
root@saturn[/home/truenas_admin]# zpool iostat -v titan
                                            capacity     operations     bandwidth 
pool                                      alloc   free   read  write   read  write
----------------------------------------  -----  -----  -----  -----  -----  -----
titan                                     4.83T  31.5T    113     99  8.45M  4.36M
  raidz2-0                                4.83T  31.5T    113     99  8.45M  4.36M
    c7c91079-5b5c-4002-8423-5fb327acd140      -      -     22     16  1.69M   798K
    ff00ea78-25a3-4998-adbb-ebdd5eb9970b      -      -     22     16  1.69M   798K
    0cfdcdca-ef0e-4a98-a8a5-0acf2a33fca9      -      -     22     16  1.69M   798K
    f802ab1b-e609-41b7-b6a0-60b5aef31e90      -      -     22     16  1.69M   798K
    c8a8ee04-4bfa-49cf-ae1f-6f699875f660      -      -     22     16  1.69M   798K
    c5d6db45-972c-473f-afcf-b9b9fae6d5fa      -      -      2    229  97.6K  6.49M
----------------------------------------  -----  -----  -----  -----  -----  -----
root@saturn[/home/truenas_admin]# iostat -x            
Linux 6.6.44-production+truenas (saturn) 	02/07/25 	_x86_64_	(12 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.74    1.55    1.96    2.62    0.00   93.13

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
loop0            0.00      0.00     0.00   0.00    0.11    17.75    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
nvme0n1          0.15      5.69     0.00   0.00    0.51    37.68   10.85    151.52     0.00   0.00    0.86    13.96    0.00      0.00     0.00   0.00    0.00     0.00    0.70    1.04    0.01   0.48
nvme1n1          0.00      0.04     0.00   0.00    0.67    55.08    0.00      0.00     0.00   0.00    0.11    11.90    0.00      0.00     0.00   0.00    0.00     0.00    0.00    1.50    0.00   0.00
nvme2n1          0.30     17.99     0.00   0.00    0.32    60.73    3.19     40.26     0.00   0.00    0.35    12.61    0.00      0.00     0.00   0.00    0.00     0.00    0.10    2.42    0.00   0.07
sda             23.04   1728.94     0.03   0.12    8.08    75.03   17.01    797.93     0.01   0.07    8.25    46.92    0.00      0.00     0.00   0.00    0.00     0.00    0.06  220.33    0.34  12.83
sdb             23.02   1728.97     0.03   0.12    8.14    75.12   17.00    797.96     0.01   0.08    8.26    46.95    0.00      0.00     0.00   0.00    0.00     0.00    0.06  219.39    0.34  12.86
sdc             23.07   1728.97     0.03   0.12    8.03    74.95   16.99    797.96     0.01   0.07    8.27    46.96    0.00      0.00     0.00   0.00    0.00     0.00    0.06  220.59    0.34  12.83
sdd             23.14   1728.83     0.03   0.12    7.89    74.70   17.09    797.91     0.01   0.08    8.17    46.69    0.00      0.00     0.00   0.00    0.00     0.00    0.06  219.52    0.33  12.75
sde             23.14   1728.95     0.03   0.12    7.92    74.73   17.00    797.89     0.01   0.07    8.26    46.94    0.00      0.00     0.00   0.00    0.00     0.00    0.06  220.10    0.34  12.76
sdf              0.15      6.94     0.00   1.97   43.80    45.68   16.50    471.55     0.02   0.14    8.93    28.58    0.00      0.00     0.00   0.00    0.00     0.00    0.01  692.34    0.16   6.77

root@saturn[/home/truenas_admin]# lsblk -o NAME,MODEL,SIZE,TYPE | grep disk      
sda         ST8000VN002-2ZM188           7.3T disk
sdb         ST8000VN002-2ZM188           7.3T disk
sdc         ST8000VN002-2ZM188           7.3T disk
sdd         ST8000VN002-2ZM188           7.3T disk
sde         ST8000VN002-2ZM188           7.3T disk
sdf         ST8000VN002-2ZM188           7.3T disk

Currently writing at around 40MB/s

SATA controller is ASM1164

root@saturn[/home/truenas_admin]# lspci -nn                                
00:00.0 Host bridge [0600]: Intel Corporation Alder Lake-U15 Host and DRAM Controller [8086:4601] (rev 04)
00:02.0 VGA compatible controller [0300]: Intel Corporation Alder Lake-UP3 GT2 [Iris Xe Graphics] [8086:46a8] (rev 0c)
00:06.0 PCI bridge [0604]: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 [8086:464d] (rev 04)
00:06.2 PCI bridge [0604]: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #2 [8086:463d] (rev 04)
00:07.0 PCI bridge [0604]: Intel Corporation Alder Lake-P Thunderbolt 4 PCI Express Root Port #0 [8086:466e] (rev 04)
00:07.2 PCI bridge [0604]: Intel Corporation Alder Lake-P Thunderbolt 4 PCI Express Root Port #2 [8086:462f] (rev 04)
00:0d.0 USB controller [0c03]: Intel Corporation Alder Lake-P Thunderbolt 4 USB Controller [8086:461e] (rev 04)
00:0d.2 USB controller [0c03]: Intel Corporation Alder Lake-P Thunderbolt 4 NHI #0 [8086:463e] (rev 04)
00:0d.3 USB controller [0c03]: Intel Corporation Alder Lake-P Thunderbolt 4 NHI #1 [8086:466d] (rev 04)
00:14.0 USB controller [0c03]: Intel Corporation Alder Lake PCH USB 3.2 xHCI Host Controller [8086:51ed] (rev 01)
00:14.2 RAM memory [0500]: Intel Corporation Alder Lake PCH Shared SRAM [8086:51ef] (rev 01)
00:15.0 Serial bus controller [0c80]: Intel Corporation Alder Lake PCH Serial IO I2C Controller #0 [8086:51e8] (rev 01)
00:15.1 Serial bus controller [0c80]: Intel Corporation Alder Lake PCH Serial IO I2C Controller #1 [8086:51e9] (rev 01)
00:16.0 Communication controller [0780]: Intel Corporation Alder Lake PCH HECI Controller [8086:51e0] (rev 01)
00:17.0 SATA controller [0106]: Intel Corporation Alder Lake-P SATA AHCI Controller [8086:51d3] (rev 01)
00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:51b8] (rev 01)
00:1c.2 PCI bridge [0604]: Intel Corporation Device [8086:51ba] (rev 01)
00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:51bc] (rev 01)
00:1d.0 PCI bridge [0604]: Intel Corporation Alder Lake PCI Express Root Port [8086:51b0] (rev 01)
00:1f.0 ISA bridge [0601]: Intel Corporation Alder Lake PCH eSPI Controller [8086:5182] (rev 01)
00:1f.3 Audio device [0403]: Intel Corporation Alder Lake PCH-P High Definition Audio Controller [8086:51c8] (rev 01)
00:1f.4 SMBus [0c05]: Intel Corporation Alder Lake PCH-P SMBus Host Controller [8086:51a3] (rev 01)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Alder Lake-P PCH SPI Controller [8086:51a4] (rev 01)
01:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology Device [c0a9:5426] (rev 01)
02:00.0 Ethernet controller [0200]: Intel Corporation 82599 10 Gigabit Network Connection [8086:1557] (rev 01)
57:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961/SM963 [144d:a804]
58:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
59:00.0 Ethernet controller [0200]: Aquantia Corp. AQtion AQC113 NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G] [1d6a:04c0] (rev 03)
5a:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1164 Serial ATA AHCI Controller [1b21:1164] (rev 02)

Just in case you missed it - Fangtooth improves the time to expand a raidz, by “up to 10x”. I wouldn’t expect a 10x improvement in every case, and, it should help a ton. Even 2x to 5x is a lot.

3 Likes