URGENT: RAIDZ1 Pool UNAVAIL After Replace Attempt — All Disks Healthy, Labels Intact, Need Help Assembling Pool
EDIT:
Platform: Generic
Edition: Community
Version: 25.04.2.4
Hostname: NAS01
Hi everyone —
Long-time TrueNAS user here, and I’m in a deeply uncomfortable spot and hoping the ZFS gurus can help me recover a RAIDZ1 pool that refuses to import.
I’ll try to present this cleanly, with full logs and no guesswork.
Background
System:
- TrueNAS SCALE
- Dell R620 (LSI 9300 HBA flashed to IT Mode)
- 10 × 4TB SAS (Seagate ST4000NM0034) in a single RAIDZ1 vdev
- 2 × SSD mirror boot pool (PERC controller)
- All drives show in
lsblkwith correct sizes
The pool is called: Storage
What happened
One disk in the RAIDZ1 started showing write errors and was marked REMOVED by ZFS.
I physically replaced the disk with a known-good spare (Seagate IronWolf 6TB).
I attempted to do a Replace operation in the SCALE UI.
During that process, the pool entered SUSPENDED state due to I/O errors.
After a reboot, the pool no longer imports and is stuck in:
Storage UNAVAIL insufficient replicas
Even though:
- all 10 original SAS disks are present
- the two “UNAVAIL” disks respond perfectly to SMART
- all ZFS labels are intact on all 10 members
I have not destroyed the pool, re-created it, wiped any disks, or run any destructive commands.
Current State
Running zpool import shows:
pool: Storage
id: 15453394492121721749
state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
config:
Storage UNAVAIL insufficient replicas
raidz1-0 UNAVAIL insufficient replicas
446ef2de-... ONLINE
d4b65d26-... ONLINE
9f3d9be8-... ONLINE
34b50bb7-... ONLINE
12356ecb-... ONLINE
fac65638-... UNAVAIL
4aa4ffb0-... ONLINE
c1bb301d-... ONLINE
a7ea0820-... UNAVAIL
c6a49122-... ONLINE
Attempts to import:
zpool import -f Storage
zpool import -fF Storage
zpool import -fFX Storage
zpool import -o readonly=on -F -d /dev Storage
All return:
cannot import 'Storage': no such pool or dataset
Destroy and re-create the pool from a backup source.
SMART Results for the two “UNAVAIL” devices
Both disks (sdk and sdp) report:
- SMART Health: OK
- No reallocated sectors
- No pending sectors
- No uncorrected read errors
- Full SMART logs readable
- Normal age for SAS drives
Example excerpt:
Elements in grown defect list: 0
Non-medium error count: 5
SMART Health Status: OK
So neither disk is actually failed at a hardware level.
ZFS Label Mapping (from zdb -l)
sdk1 maps to:
guid: 17687009939021634546
(partuuid fac65638-95e1-4b7b-9add-1a60a9b3a52e)
sdp1 maps to:
guid: 11757704506353456861
(partuuid a7ea0820-a3c0-40c1-8ebd-96a10a4ccee3)
Both labels are intact and readable.
My goal
I need to recover this pool.
It contains critical personal data that is not duplicated elsewhere.
I’m hoping someone experienced with:
- manual vdev assembly
- Uberblock recovery
- txg rewind
zpool import -cworkflows
can help me determine:
- Whether the remaining good members of the RAIDZ1 can be assembled manually
- Whether the two “UNAVAIL” devices contain usable data but ZFS has flagged them incorrectly
- Whether a readonly import is possible
- What the next safe steps are before resorting to imaging the drives and professional recovery
I have not run:
zpool destroyzpool labelclear- any writes to these disks
- or anything destructive whatsoever.
Everything is still in original on-disk condition.
Full system is now powered OFF to prevent further changes.
I can power it back on to run diagnostics under guidance.
Any help would be massively appreciated.
I will happily provide:
- full
zdb -lfor every member zpool history(if recoverable)udevadm info- controller details
- photos of backplane & cabling
I’m trying to avoid a professional lab unless absolutely necessary, but I’m committed to doing whatever is needed to get this pool imported.
Thanks in advance.