I imported a pool containing two RAIDZ1. One RAIDZ1 contains three 10 TB HD and the second one contains three 3 TB HD. But after the import of the Pool, it looks like this:
RAIDZ1
sda 10TB
sdc 10TB
sdd 10TB
RAIDZ1
sdb 3TB
sdc 10TB
sde 3TB
As you can see, sdc is on both RAID.
Any clue why and how to fix it?
Please supply the output of the following, in separate CODE tags:
zpool status
fdisk -l /dev/sdc
zdb -l /dev/sdc
The last might have some garbage because the “sdc” drive will likely be partitioned. If so, add the partition number the the last command. And if more than one ZFS partition, run the last command twice, once on the first ZFS partition and again on the second partition.
Please output bellow:
pool: Pool10
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon May 12 14:53:45 2025
2.21T / 20.8T scanned at 176M/s, 28.9G / 20.8T issued at 2.25M/s
9.07G resilvered, 0.14% done, no estimated completion time
config:
NAME STATE READ WRITE CKSUM
Pool10 DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
51c10af9-8f77-422e-8073-fc22fb0ee56b ONLINE 0 0 0 (resilvering)
f72f8fc2-462c-11eb-b5c3-503eaa0acd6d DEGRADED 0 0 1 too many errors
b35cba61-4c4d-4aaa-ae93-ef7a80d96860 ONLINE 0 0 10
raidz1-1 DEGRADED 0 0 0
f57d6444-5041-11eb-8c68-6805ca5abe38 ONLINE 0 0 0
345aa184-5040-11eb-8c68-6805ca5abe38 ONLINE 0 0 0
9724333599733858192 FAULTED 0 0 0 was /dev/sdc2
cache
14852522-4192-4d8b-91b7-6537d2eeb4d7 UNAVAIL 0 0 0
spares
72251d1a-3cbe-47de-a2d5-aa420dca6112 AVAIL
errors: 1038 data errors, use '-v' for a list
pool: boot-pool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
nvme0n1p3 ONLINE 0 0 0
errors: No known data errors
Disk /dev/sdc: 9.1 TiB, 10009952870400 bytes, 19550689200 sectors
Disk model: ST10000NM0046
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 24C7D519-0EF5-4F8D-B9BD-9082D19BB796
Device Start End Sectors Size Type
/dev/sdc1 2048 4196352 4194305 2G Linux swap
/dev/sdc2 4198400 19532877824 19528679425 9.1T Solaris /usr & Apple ZFS
truenas_admin@truenas:~$ sudo zdb -l /dev/sdc
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3
sudo zdb -l /dev/sdc2 then.
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'Pool10'
state: 0
txg: 6177891
pool_guid: 8596046533860088929
errata: 0
hostid: 945097498
hostname: 'truenas'
top_guid: 3630653426565973403
guid: 2897143672395654520
vdev_children: 2
vdev_tree:
type: 'raidz'
id: 0
guid: 3630653426565973403
nparity: 1
metaslab_array: 256
metaslab_shift: 34
ashift: 12
asize: 29996036653056
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 16937255374696992665
path: '/dev/disk/by-partuuid/51c10af9-8f77-422e-8073-fc22fb0ee56b'
whole_disk: 0
DTL: 14504
create_txg: 4
resilver_txg: 4883770
children[1]:
type: 'disk'
id: 1
guid: 6330288675695004236
path: '/dev/disk/by-partuuid/f72f8fc2-462c-11eb-b5c3-503eaa0acd6d'
phys_path: 'id1,enc@n3061686369656d30/type@0/slot@6/elmdesc@Slot_05/p2'
DTL: 127139
create_txg: 4
degraded: 1
aux_state: 'err_exceeded'
children[2]:
type: 'disk'
id: 2
guid: 2897143672395654520
path: '/dev/disk/by-partuuid/b35cba61-4c4d-4aaa-ae93-ef7a80d96860'
whole_disk: 0
DTL: 545
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
com.klarasystems:vdev_zaps_v2
labels = 0 1 2 3
That is really weird.
Please supply the output of the following, in separate CODE tags:
zpool status -L Pool10
fdisk -l /dev/sda
fdisk -l /dev/sdb
fdisk -l /dev/sdd
fdisk -l /dev/sde
As for how to fix, you have a SPARE drive available. Not sure why it did not take over for the faulty drive. In theory, you could use it. However, their is an on-going Re-Silver taking place that likely has to finish before starting anything else.
If you can supply the output of fdisk -l for that spare drive, it would be helpful too.
Note that you have data loss. There appears to have been multiple problems with raidz1-0 in which the other 2 disks had issues. Including on-going resilver of the first disk in raidz1-0. Because you only have 1 disks worth of redundancy, (aka RAID-Z1), this might lead to total pool loss.
Here is the requested outputs
pool: Pool10
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon May 12 14:53:45 2025
2.21T / 20.8T scanned at 22.7M/s, 207G / 20.8T issued at 2.08M/s
68.9G resilvered, 0.97% done, no estimated completion time
config:
NAME STATE READ WRITE CKSUM
Pool10 DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
sdg1 REMOVED 0 0 0
sdd2 DEGRADED 0 0 1 too many errors
sdc2 ONLINE 0 0 10
raidz1-1 DEGRADED 0 0 0
sde2 ONLINE 0 0 0
sdb2 ONLINE 0 0 0
9724333599733858192 FAULTED 0 0 0 was /dev/sdc2
cache
14852522-4192-4d8b-91b7-6537d2eeb4d7 UNAVAIL 0 0 0
spares
sdf1 AVAIL
errors: 1038 data errors, use '-v' for a list
Here is an fdisk -l for all drives:
Disk /dev/nvme0n1: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Samsung SSD 970 EVO Plus 250GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 1E314372-CB47-45A8-BF96-05C04B3DEBC6
Device Start End Sectors Size Type
/dev/nvme0n1p1 4096 6143 2048 1M BIOS boot
/dev/nvme0n1p2 6144 1054719 1048576 512M EFI System
/dev/nvme0n1p3 1054720 488397134 487342415 232.4G Solaris /usr & Apple ZFS
Disk /dev/sdc: 9.1 TiB, 10009952870400 bytes, 19550689200 sectors
Disk model: ST10000NM0046
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 24C7D519-0EF5-4F8D-B9BD-9082D19BB796
Device Start End Sectors Size Type
/dev/sdc1 2048 4196352 4194305 2G Linux swap
/dev/sdc2 4198400 19532877824 19528679425 9.1T Solaris /usr & Apple ZFS
Disk /dev/sdb: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA DT01ACA3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 340FCD33-5040-11EB-8C68-6805CA5ABE38
Device Start End Sectors Size Type
/dev/sdb1 128 4194431 4194304 2G FreeBSD swap
/dev/sdb2 4194432 5860533127 5856338696 2.7T FreeBSD ZFS
Disk /dev/sdd: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: ST10000NE0008-2J
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: F6DFB56E-462C-11EB-B5C3-503EAA0ACD6D
Device Start End Sectors Size Type
/dev/sdd1 128 4194431 4194304 2G FreeBSD swap
/dev/sdd2 4194432 19532873687 19528679256 9.1T FreeBSD ZFS
Disk /dev/sde: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: TOSHIBA DT01ACA3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 16773120 bytes
Disklabel type: gpt
Disk identifier: F5598DCE-5041-11EB-8C68-6805CA5ABE38
Device Start End Sectors Size Type
/dev/sde1 128 4194431 4194304 2G FreeBSD swap
/dev/sde2 4194432 5860533127 5856338696 2.7T FreeBSD ZFS
Disk /dev/sdf: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: ST3000DM001-1ER1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 16773120 bytes
Disklabel type: gpt
Disk identifier: D7BDE39B-7ED0-4764-B1BA-A5239F9836EB
Device Start End Sectors Size Type
/dev/sdf1 2048 5856339967 5856337920 2.7T Solaris /usr & Apple ZFS
Disk /dev/sdg: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: ST10000NE0008-2J
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 6302FD68-014E-4A01-BE31-4F3A2B7DB8A9
Device Start End Sectors Size Type
/dev/sdg1 2048 19532871679 19532869632 9.1T Solaris /usr & Apple ZFS
Looks like resilvering has been completed. Nothing is displayed running jobs menu.
I guess the problem with the pool is in it’s state is because the 9TB drive in the two raidz configurations.
Thinking maybe to make a JBOD setup and temporary copy everything to it and rebuild the Pool and restoring it from that backup.
Can you point me to any documentation were it is documented on how to create a JBOD?
In ZFS parlance, a JBOD is a stripe of single drive vdevs.