Hi there,
I extended my raidz1 3x4tb pool with a new 1x 4tb drive.
The process is still at 25% even after 24 hours, is this normal behavior or should i be worried?
Attaching Images for context:
Extending at 25% since 24 hours:
zpool list -v st4000dm
zpool iostat -vy 1 1
What steps should I take next any help or guidance is most welcomed, thankyou for reading.
What are your drives? Are they SMR or CMR? And what version of TrueNAS are you on?
Also are you sure you want to Extend rather than Expand. I am not 100% certain, but I have a feeling that extending is growing your existing partitions on existing disks, and expanding is adding a new disk to a RAIDZ vDev.
on the TrueNAS UI it’s the opposite of this. Expand (on the storage page) does a partition expand (or it breaks horribly with that “we were unable to inform the kernel” error)
and Extend (found on a vdev of the pool’s disks page, OP’s screenshot), is used for adding disks to a given VDEV of a zpool (mirrors and now Raidz)
hrmm… you should look at a zpool status st4000dm to see the actual progress on the expansion. the TrueNAS UI/middleware is less than perfect.
honestly, yes expansion takes AGES as it re-writes the data in-place. significantly longer than a scrub for example. using it on one of my servers to expand an 8x 8TB pool takes ~2.5 Days.
I’m on
Current Train: TrueNAS-SCALE-ElectricEel - TrueNAS SCALE ElectricEel 24.10 [release]
My pool Originally has 3x SMR drives which were the only ones i could afford 2 years back, now I got 2 more drives which are both CMR drives
Also Im Extending the pool not Expanding.
Thankyou for this
zpool status st4000dm
# sudo zpool status st4000dm
pool: st4000dm
state: ONLINE
scan: resilvered 10.6M in 00:00:11 with 0 errors on Wed Nov 6 19:54:39 2024
expand: expansion of raidz1-0 in progress since Wed Nov 6 01:08:54 2024
2.4T / 9.30T copied at 45.3M/s, 24.81% done, 2 days 08:53:50 to go
config:
NAME STATE READ WRITE CKSUM
st4000dm ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
11e8f5b1-a7b0-4660-bc0e-aa939d9e26d1 ONLINE 0 0 0
a3267cad-1a01-4c90-8144-fb8f12282033 ONLINE 0 0 0
2c9fe82a-69fb-49b0-b94b-d3088b5534ef ONLINE 0 0 0
af7b7464-1db5-417e-aec7-a29f2cd79046 ONLINE 0 0 0
errors: No known data errors
I get the progress is only 24.81%, and 2 days to go. Thats so long but a relief
I the mean while Can I use my pool normally with read/writes ?
or do i need to keep my services down for the entire process?
I guess with 2 drives im looking at an entire week
SMR drives will take forever and then a bit longer. They have bulk write performance issues and all ZFS disk operations (resilvering, extending, expanding) are heavy on writes.
you can use the pool as normal. just don’t expect miracle speeds with it in this state.
The SMR drives are going to slow everything down, especially any post re-write of any data. so be prepared for that as well.
not sure about what particular drives you have, but some SMR disks support TRIM to help the performance recover a little faster, turn on autotrim on that particular ZFS pool when the expansion is done. it will either do nothing or it’ll help a little bit.
I know better now, did not know back then,
when the expansion is done i’ll have some space for a breather, and then i’m planning to replace the SMR disks one by one.
So, I dont need to do the moving manually, the ui message is really confusing me now.
So I dont need to use this:
Lawrence Systems
Lawrence Systems suggests so @06:05
thanks for this suggestion, I will do so.
you will still need to move all the data, post expansion. to regain the space used by all the data blocks using the old data+parity ratio.
specifically what the expand is re-writing in-place is lower level than that, it’s just distributing the existing blocks, it has no ability to change the block as it’s written, just move it around slightly to accommodate a disk being added. which leaves the block non-optimal after expansion.
Just to clarify this a little further, you start with 3x RAIDZ1 where the disks are pretty full. When you first add the 4th disk, all the spare space in on this 4th disk, but to be able to use it as RAIDZ1 you need to have roughly the same amount of free space on all the disks in the pool instead otherwise you cannot spread the data amongst disks as you need to.
So part of the expansion process is to move some of the data blocks (without actually recalculating the parity) so as to equalise the free space across all drives.
But I thought I read that new blocks written after expansion would use the full width of the new vDev and only existing blocks would remain at a narrower width (hence the rebalancing script doing a rewrite of the data).
Good. The bad news is that, for for the sake of safety, you now want to replace all SMR drives by CMR, and then run the rebalancing script…
Yes, one by one thats the plan.
thank you all for all the help and guidance.
