Yes, it can be confusing because some RAID software does not support multiple RAID sets in the same “pool”. Some people only learn after a “mistake” their is a difference between a ZFS vDev, and ZFS pool.
Having used NetApp around 2000, I learned that lesson. It too supports multiple RAID sets. So when ZFS came out with multiple vDevs in a single “pool” a few years later, I understood it.
You can. You just need to create multiple vdevs. It’s still a single pool with 60 drives.
E.g. 7 times 8-wide RAIDZ2 and 4 hot spares. 10 times 6-wide RAIDZ2. Endless possibilities.
I have a couple of 24 disk systems. I use 4 times 6-wide RAIDZ2 in these.
My pools normally contain 90 drives but I use multiple vdevs to construct the pool. Perhaps this is where the confusion was. Hey we live and learn and there is nothing wrong with that.
zpool clear VoxelZ2
Thanks you all 
just to be sure, Johnny, I can safely delete all these files :
errors: Permanent errors have been detected in the following files:
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/netdata-meta.db
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine/journalfile-1-0000001650.njf
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine/journalfile-1-0000001654.njf
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine/journalfile-1-0000001656.njf
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine/datafile-1-0000001660.ndf
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/netdata-meta.db-wal
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine/journalfile-1-0000001655.njf
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine/journalfile-1-0000001657.njf
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine-tier1/datafile-1-0000000129.ndf
VoxelZ2/.system/netdata-ae32c386e13840b2bf9c0083275e7941:/netdata-meta.db-wal
VoxelZ2/.system/netdata-ae32c386e13840b2bf9c0083275e7941:/dbengine-tier1/datafile-1-0000000001.ndf
VoxelZ2/.system/netdata-ae32c386e13840b2bf9c0083275e7941:/dbengine/datafile-1-0000000001.ndf
and then do the clear, and finish safely (for my files I’ve just fully recovered) by another and last scrub of the entire pool, and miracle I’m good ?
lastly all these steps will get rid of the expansion on hold I don’t want anymore too ?
at least it’s what I understand there
Yep.
I’m not sure about the expansion being on hold as I say it’s voodoo to me. If I was guessing then I say the expansion is on hold because your hot-spare kicked in during expansion and until you sort that it may stay on hold but I would do the other stuff first and let’s see how that goes.
Thanks
scrub running
Well, scrub took less than a day, but if there are 10 less errors with the same files, the problem isn’t solved yet 
and the ‘clear’ stuff don’t have done anything more.
BTW, I can import the zpool very quickly and have it show in the dashboard without issues happy.
So, I’m a little puzzled there, I will run a full resilver today, but before that I would like to know from the wizard there, if I can cancel an expansion by one drive that is on hold because of that errors in the apps database (see the report upper) ?
I think I can safely ‘remove’ the ‘unavail’ disk as it has been replaced, but still show up in the zpool disks list while not in the NAS physically anymore since ages…
I’ve already successfully ‘removed’ the spare disk that was shown ‘unavail’ too,
so only the 16 disks from the zpool, the system nvme and L2ARC SSD are now online.
I want to get rid of the lasts errors (all from the apps side and their databases), Please note that the files shown as errored in the report upper are nowhere to be found at the place it are said to be or anywhere on the zpool or system disk…
All the apps that was installed aren’t showing at all too in the dashboard, but I can see the last (and first installed ones) still present in the xiapp zpool folder.
So, looking for the correct turn of commands to issue in the shell here from the zfs wizards
Thanks by advance.
If the RAID-Zx expansion started, it CAN NOT be stopped. On disk failure, it will automatically pause, awaiting disk replacement. But, after failed disk replacement, the RAID-Zx expansion MUST continue. (Immediately followed by a scrub…)
ZFS is not the most flexible file system, volume manager and RAID scheme out their. While it would be nice to have an option to abort RAID-Zx expansion, it is not possible now. Nor is the opposite of RAID-Zx expansion, (aka column remove).
If you supply the full output of zpool status elZ2, we can see the current status of the RAID-Zx expansion.
I don’t mean to derail this thread, but I only just now noticed two things.
Arrg… I tend to read the prior responses when I am alerted. Seems this should have had a 2nd thread…
Hi Arwen ^^)
well here is the report :
min@VoxelNas[~]$ sudo zpool status -v
[sudo] password for truenas_admin:
pool: VoxelZ2
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: Message ID: ZFS-8000-8A — OpenZFS documentation
scan: scrub repaired 0B in 16:38:02 with 31 errors on Fri Feb 13 23:52:23 2026
expand: expansion of raidz2-0 in progress since Thu Feb 27 15:19:37 2025
5.28T / 118T copied at 186K/s, 4.48% done, paused for resilver or clear
config:
NAME STATE READ WRITE CKSUM
VoxelZ2 DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
aab036a6-7e2b-4632-b16d-cd7866845e5e ONLINE 0 0 0
7a8f8da1-836b-4a4c-9eaa-2aa29da2beb7 ONLINE 0 0 0
a10d9ac9-ef2c-4ede-9e6b-eba01e901361 ONLINE 0 0 0
9f2dce88-cde7-4923-b0bb-bf4de890ea52 ONLINE 0 0 0
181fbb56-496c-4540-a107-d35504e0d61b ONLINE 0 0 0
96d445ec-e8b2-42ce-8f1a-63ca963eead9 ONLINE 0 0 0
b4dbc538-a334-417c-9b6a-7265693bc710 ONLINE 0 0 0
fd7f4ef7-237a-48c0-8a72-bebaf1cd3fb5 ONLINE 0 0 0
83e62634-a0a5-4c2f-abb8-10b763833042 ONLINE 0 0 0
ea3b911a-d478-4341-b4cd-4549385fcdde ONLINE 0 0 0
52c0cc19-1576-4186-b41f-8ee005e7ebaf ONLINE 0 0 0
ee9542ae-6266-4615-bcaa-6a0f06dafb87 ONLINE 0 0 0
0b764a40-fd5d-4631-922d-625178717347 ONLINE 0 0 0
e0df314c-cc1e-460e-9591-842264306d5b ONLINE 0 0 0
af112e1e-211c-4061-a0bf-4e7f943903ca ONLINE 0 0 0
15096135188792341576 UNAVAIL 0 0 0 was /dev/disk/by-partuuid/c2c7b568-7eff-4f85-998b-eb4e1ac51897
ebd9e063-ebb9-4bdb-9ac6-3302a748390a ONLINE 0 0 0
cache
b7523e48-e07b-4fbb-94c5-c6de97a7d33c ONLINE 0 0 0
errors: Permanent errors have been detected in the following files:
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/netdata-meta.db
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine/journalfile-1-0000001650.njf
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine/journalfile-1-0000001654.njf
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine/journalfile-1-0000001656.njf
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine/datafile-1-0000001660.ndf
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/netdata-meta.db-wal
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine/journalfile-1-0000001655.njf
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine/journalfile-1-0000001657.njf
VoxelZ2/.system/netdata-5a0a2a47cd884dbcbe527966286bfc29:/dbengine-tier1/datafile-1-0000000129.ndf
as you see, expansion was at the very beginning.
My problem with it is that the disk meant for the expansion is the one shown ‘unavail’ here as it died shortly after the start of the said expansion process… it was brand new so I’m very angry after it.
Btw, if it is not possible to forget it, I just wait for the new Z3 pool to come to life (in the future).
as long as I have my files safe I’m happy
@ winnielinnie : well I’m Jeff but not B**os… it takes time to me to address problem when it are costly… 
btw, I’m a very bright person, hence the smileys all over the place ^^)
and it was 'disks died Question ’
now why I put this in the old thread from last year ? 'cause it’s is direct follow up, so I find easier to have a full grasp of the problems without having to go back and forth between threads. no more than that.
Okay.
The RAID-Zx Expansion is paused, until you replace the UNAVAIL disk.