Hello TrueNAS Community,
I’m seeking urgent help with a RAIDZ3 pool that has become stuck in an inconsistent state after multiple interruptions during a vdev expansion. I have run through several recovery steps but am now at a point where I cannot get a solution.
System Setup:
- OS: TrueNAS SCALE 25.04.2.1
- Pool: RAIDZ3
- Action: Expanding a 5x 4TB vdev to a 6x 4TB vdev.
Background:
The expansion process was interrupted 2x times by server crashes/restarts due to freezing issues. This left the pool online, but with several problems: - The expansion was incomplete. Showing 6 drives in the pool but only available space of a 5 drives in raidz3. At first restart the expansion automatically resumed (sudo zpool status showed it was resumed, and went from 20% to 85%). At second restart the expansion also resumed, but like 8h after, it stopped showing that was fineshed, but it did not fineshed. The same issue showing 6 drives in the pool but now a little bigger, increased something like 500GB
- zpool status reported 4 CKSUM errors on one disk after 2nd restart, and were automatically resolved. A SMART showed no issues with the drive.
Troubleshooting Steps I’ve Already Taken:
- S.M.A.R.T. Check: An extended S.M.A.R.T. test on the affected disk (/dev/sdj) passed, suggesting the CKSUM errors were transient and not a hardware failure.
- zpool clear / offline: All attempts to clear the errors or offline the disk failed with the same contradictory error, even though the disk was listed as ONLINE in zpool status: Error: cannot clear errors for …: no such device in pool. All 6 drives are online. But if I try to clean error of some drives with sudo /sbin/zpool clear EOS_Gemini wwn-0x50014ee60795bf6a, 2 drives show the error: cannot clear… No such device in pool.
- zpool export / import: As a last resort to fix the state inconsistency, I ran an export/import cycle. Did not helped to solve the expansion problem.
- I tried to force the expansion to resume with the following command, but like I sad before; 2 drives showed that were no such device in the pool. And for the drives the command line worked it did not resumed the expansion
sudo /sbin/zpool online -e EOS_Gemini 19513023-8bd2-4f4d-ac30-5180a668255a
sudo /sbin/zpool online -e EOS_Gemini 9ba0dccf-a102-4221-9d33-3389d8994f78
sudo /sbin/zpool online -e EOS_Gemini 502713da-8b26-49ea-9bea-03a3c27b2c1f
sudo /sbin/zpool online -e EOS_Gemini 6181bbcf-4cb0-4d34-8955-7ca86d01caba
sudo /sbin/zpool online -e EOS_Gemini f1bf643c-d29d-427c-abab-0e3053aa05d4
sudo /sbin/zpool online -e EOS_Gemini f88e265e-cd5e-4a6b-a38d-0389ef22ab73
Current Status:
- I started to scrub this pool, it will take a while to see if it solves the expansion problem.
expand: expanded raidz3-0 copied 13.2T in 6 days 17:56:29,
on Tue Aug 19 10:35:04 2025
My Questions:
- What is the correct and safe sequence of steps from this point to get the expansion completed?
Any advice would be greatly appreciated. I’m new at Truenas Scale. So I’m still learning, although I learn quite fast.
I am stuck and want to ensure I don’t risk data loss.
Thank you!
Ps: and please, may the answers be focused on the solutions of my questions.
