ZFS Label missing how to restore?

squinj · April 30, 2025, 3:19am

Hello,

I am very new to ZFS and have already broken my ZFS pool

This was due to label corruption via improper shutdown on an external USB DAS.
Is it possible to restore the ZFS label from /dev/sda1 or /dev/sdb1 since they seem to be the only valid labels existing for the pool?

zpool import -d /dev/disk/by-partuuid
  pool: zfspool0
    id: 17328721603664047508
state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
   see: 
config:

zfspool0                                  UNAVAIL  insufficient replicas
  mirror-0                                UNAVAIL  insufficient replicas
    66382eb7-820e-3d47-9000-d844c1326d13  UNAVAIL  invalid label
    6b1797a8-26e8-5b41-8fec-18f3977607c5  UNAVAIL  invalid label

root@localhost:~# zdb -l /dev/sda
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

root@localhost:~# zdb -l /dev/sda1
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'zfspool0'
    state: 0
    txg: 29380
    pool_guid: 17328721603664047508
    errata: 0
    hostid: [snip]
    hostname: 'localhost'
    top_guid: 16873280761438265250
    guid: 7788381016733623247
    vdev_children: 1
    vdev_tree:
        type: 'mirror'
        id: 0
        guid: 16873280761438265250
        metaslab_array: 128
        metaslab_shift: 34
        ashift: 12
        asize: 129986248053358592
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 7788381016733623247
            path: '/dev/disk/by-id/usb-External_USB3.0_DISK00_20170331000C3-0:0-part1'
            devid: 'usb-External_USB3.0_DISK00_20170331000C3-0:0-part1'
            phys_path: 'platform-xhci-hcd.1-usb-0:1:1.0-scsi-0:0:0:0'
            whole_disk: 1
            DTL: 5839
            create_txg: 4
            expansion_time: 1745969724
        children[1]:
            type: 'disk'
            id: 1
            guid: 241061445970378625
            path: '/dev/disk/by-id/usb-External_USB3.0_DISK01_20170331000C3-0:1-part1'
            devid: 'usb-External_USB3.0_DISK01_20170331000C3-0:1-part1'
            phys_path: 'platform-xhci-hcd.1-usb-0:1:1.0-scsi-0:0:0:1'
            whole_disk: 1
            DTL: 5838
            create_txg: 4
            expansion_time: 1745969723
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1
------------------------------------
LABEL 2
------------------------------------
    version: 5000
    name: 'zfspool0'
    state: 0
    txg: 29376
    pool_guid: 17328721603664047508
    errata: 0
    hostid: [snip]
    hostname: 'localhost'
    top_guid: 16873280761438265250
    guid: 7788381016733623247
    vdev_children: 1
    vdev_tree:
        type: 'mirror'
        id: 0
        guid: 16873280761438265250
        metaslab_array: 128
        metaslab_shift: 34
        ashift: 12
        asize: 8001548713984
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 7788381016733623247
            path: '/dev/disk/by-id/usb-External_USB3.0_DISK00_20170331000C3-0:0-part1'
            devid: 'usb-External_USB3.0_DISK00_20170331000C3-0:0-part1'
            phys_path: 'platform-xhci-hcd.1-usb-0:1:1.0-scsi-0:0:0:0'
            whole_disk: 1
            DTL: 5839
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 241061445970378625
            path: '/dev/disk/by-id/usb-External_USB3.0_DISK01_20170331000C3-0:1-part1'
            devid: 'usb-External_USB3.0_DISK01_20170331000C3-0:1-part1'
            phys_path: 'platform-xhci-hcd.1-usb-0:1:1.0-scsi-0:0:0:1'
            whole_disk: 1
            DTL: 5838
            create_txg: 4
            expansion_time: 1745969723
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 2 3

root@localhost:~# zdb -l /dev/sdb
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

root@localhost:~# zdb -l /dev/sdb1
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'zfspool0'
    state: 0
    txg: 29380
    pool_guid: 17328721603664047508
    errata: 0
    hostid: [snip]
    hostname: 'localhost'
    top_guid: 16873280761438265250
    guid: 241061445970378625
    vdev_children: 1
    vdev_tree:
        type: 'mirror'
        id: 0
        guid: 16873280761438265250
        metaslab_array: 128
        metaslab_shift: 34
        ashift: 12
        asize: 129986248053358592
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 7788381016733623247
            path: '/dev/disk/by-id/usb-External_USB3.0_DISK00_20170331000C3-0:0-part1'
            devid: 'usb-External_USB3.0_DISK00_20170331000C3-0:0-part1'
            phys_path: 'platform-xhci-hcd.1-usb-0:1:1.0-scsi-0:0:0:0'
            whole_disk: 1
            DTL: 5839
            create_txg: 4
            expansion_time: 1745969724
        children[1]:
            type: 'disk'
            id: 1
            guid: 241061445970378625
            path: '/dev/disk/by-id/usb-External_USB3.0_DISK01_20170331000C3-0:1-part1'
            devid: 'usb-External_USB3.0_DISK01_20170331000C3-0:1-part1'
            phys_path: 'platform-xhci-hcd.1-usb-0:1:1.0-scsi-0:0:0:1'
            whole_disk: 1
            DTL: 5838
            create_txg: 4
            expansion_time: 1745969723
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1
------------------------------------
LABEL 2
------------------------------------
    version: 5000
    name: 'zfspool0'
    state: 0
    txg: 29373
    pool_guid: 17328721603664047508
    errata: 0
    hostid: [snip]
    hostname: 'localhost'
    top_guid: 16873280761438265250
    guid: 241061445970378625
    vdev_children: 1
    vdev_tree:
        type: 'mirror'
        id: 0
        guid: 16873280761438265250
        metaslab_array: 128
        metaslab_shift: 34
        ashift: 12
        asize: 8001548713984
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 7788381016733623247
            path: '/dev/disk/by-id/usb-External_USB3.0_DISK00_20170331000C3-0:0-part1'
            devid: 'usb-External_USB3.0_DISK00_20170331000C3-0:0-part1'
            phys_path: 'platform-xhci-hcd.1-usb-0:1:1.0-scsi-0:0:0:0'
            whole_disk: 1
            DTL: 5839
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 241061445970378625
            path: '/dev/disk/by-id/usb-External_USB3.0_DISK01_20170331000C3-0:1-part1'
            devid: 'usb-External_USB3.0_DISK01_20170331000C3-0:1-part1'
            phys_path: 'platform-xhci-hcd.1-usb-0:1:1.0-scsi-0:0:0:1'
            whole_disk: 1
            DTL: 5838
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 2 3

root@localhost:~# zdb -l /dev/sdb9
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

winnielinnie · May 1, 2025, 3:25am

What is this DAS? How does it present the drives to the OS?

squinj · May 2, 2025, 4:31am

The DAS itself is an Orico 5 disk DAS usb enclosure:

lsusb

Bus 004 Device 002: ID 152d:0567 JMicron Technology Corp. / JMicron USA Technology Corp. JMS567 SATA 6Gb/s bridge

the disks hosted in the DAS can be accessed via their respective device ids and partition ids via fdisk/gparted etc
/dev/sda,
/dev/sdb etc
/dev/sda1
/dev/sdb1

they are also viewable via

ls -al /dev/disk/by-partuuid
total 0
drwxr-xr-x 2 root root 280 Apr 30 15:07 .
drwxr-xr-x 9 root root 180 Apr 30 15:06 …
lrwxrwxrwx 1 root root 10 Apr 30 15:06 027cf62c-72bb-5d4d-8a4d-dd7a11458627 → …/…/sdd1
lrwxrwxrwx 1 root root 10 Apr 30 15:07 2fefb5bd-aeda-3042-87d5-f7e52e1efd41 → …/…/sde1
lrwxrwxrwx 1 root root 10 Apr 30 15:06 3942ceb5-1fd3-b045-a368-67305e57deec → …/…/sdc9
lrwxrwxrwx 1 root root 10 Apr 30 15:06 638ab192-9243-e645-95fe-b7247821f831 → …/…/sdc1
lrwxrwxrwx 1 root root 10 Apr 30 15:06 66382eb7-820e-3d47-9000-d844c1326d13 → …/…/sda1
lrwxrwxrwx 1 root root 10 Apr 30 15:06 6b1797a8-26e8-5b41-8fec-18f3977607c5 → …/…/sdb1
lrwxrwxrwx 1 root root 10 Apr 30 15:06 7624147c-1c4d-bd41-a250-d6d3c39f27f6 → …/…/sdb9
lrwxrwxrwx 1 root root 10 Apr 30 15:06 ba4b4966-804f-b444-8a7e-9cfc2ad2bdeb → …/…/sde9
lrwxrwxrwx 1 root root 10 Apr 30 15:06 e886700f-be68-e648-afaf-ac8f6077e831 → …/…/sda9
lrwxrwxrwx 1 root root 10 Apr 30 15:06 ead0dd76-029f-774d-a5ac-6f4e45b2eefc → …/…/sdd9

squinj · May 2, 2025, 7:28am

When I try and import without the -d option it returns:

zpool import zfspool0
cannot import ‘zfspool0’: invalid vdev configuration

I am really hoping that based on the labels and potential recovery via uberblocks from /dev/sda1 and /dev/sdb1 it is somehow recoverable.

zdb -ul zfspool0

LABEL 0

version: 5000
name: 'zfspool0'
state: 0
txg: 29380
pool_guid: 17328721603664047508
errata: 0
hostid: xxx
hostname: 'localhost'
top_guid: 16873280761438265250
guid: 7788381016733623247
vdev_children: 1
vdev_tree:
    type: 'mirror'
    id: 0
    guid: 16873280761438265250
    metaslab_array: 128
    metaslab_shift: 34
    ashift: 12
    asize: 129986248053358592
    is_log: 0
    create_txg: 4
    children[0]:
        type: 'disk'
        id: 0
        guid: 7788381016733623247
        path: '/dev/disk/by-id/usb-External_USB3.0_DISK00_20170331000C3-0:0-part1'
        devid: 'usb-External_USB3.0_DISK00_20170331000C3-0:0-part1'
        phys_path: 'platform-xhci-hcd.1-usb-0:1:1.0-scsi-0:0:0:0'
        whole_disk: 1
        DTL: 5839
        create_txg: 4
        expansion_time: 1745969724
    children[1]:
        type: 'disk'
        id: 1
        guid: 241061445970378625
        path: '/dev/disk/by-id/usb-External_USB3.0_DISK01_20170331000C3-0:1-part1'
        devid: 'usb-External_USB3.0_DISK01_20170331000C3-0:1-part1'
        phys_path: 'platform-xhci-hcd.1-usb-0:1:1.0-scsi-0:0:0:1'
        whole_disk: 1
        DTL: 5838
        create_txg: 4
        expansion_time: 1745969723
features_for_read:
    com.delphix:hole_birth
    com.delphix:embedded_data
    com.klarasystems:vdev_zaps_v2
labels = 0 1
Uberblock[0]

magic = 0000000000bab10c
version = 5000
txg = 29376
guid_sum = 5337956680387211398
timestamp = 1745969724 UTC = Wed Apr 30 08:35:24 2025
bp = DVA[0]=<0:27400071000:1000> DVA[1]=<0:27800071000:1000> DVA[2]=<0:27c00070000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29376L/29376P fill=212 cksum=0000000274aca46a:0000097cda388c47:00125931eb4db725:17aed84f08a35726
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[1]
magic = 0000000000bab10c
version = 5000
txg = 29345
guid_sum = 5337956680387211398
timestamp = 1745969385 UTC = Wed Apr 30 08:29:45 2025
bp = DVA[0]=<0:26c5dc3c000:1000> DVA[1]=<0:2647ae22000:1000> DVA[2]=<0:270018c1000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29345L/29345P fill=205 cksum=00000002e75b975a:00000b33910487fa:0015a120a981508f:1bdfe960924cf076
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[2]
magic = 0000000000bab10c
version = 5000
txg = 29346
guid_sum = 5337956680387211398
timestamp = 1745969390 UTC = Wed Apr 30 08:29:50 2025
bp = DVA[0]=<0:26c5ddcf000:1000> DVA[1]=<0:2647ae6a000:1000> DVA[2]=<0:270018ce000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29346L/29346P fill=205 cksum=00000004f727495b:0000131976870f48:0024c5e062d758ed:2f4077b90085e6a4
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[3]
magic = 0000000000bab10c
version = 5000
txg = 29379
guid_sum = 5337956680387211398
timestamp = 1745969724 UTC = Wed Apr 30 08:35:24 2025
bp = DVA[0]=<0:27400086000:1000> DVA[1]=<0:27800086000:1000> DVA[2]=<0:27c00085000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29379L/29379P fill=212 cksum=00000003df33371d:00000ee958a690b5:001cbeee08394922:24fc05404b1f212c
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1
Uberblock[4]
magic = 0000000000bab10c
version = 5000
txg = 29380
guid_sum = 5337956680387211398
timestamp = 1745969724 UTC = Wed Apr 30 08:35:24 2025
bp = DVA[0]=<0:2740009a000:1000> DVA[1]=<0:2780009a000:1000> DVA[2]=<0:27c00099000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29380L/29380P fill=212 cksum=0000000433871ef3:0000102c4962d8d4:001f29ef72046334:2813d6a59786361b
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1
Uberblock[5]
magic = 0000000000bab10c
version = 5000
txg = 29349
guid_sum = 5337956680387211398
timestamp = 1745969405 UTC = Wed Apr 30 08:30:05 2025
bp = DVA[0]=<0:26c5e9b9000:1000> DVA[1]=<0:2647af7e000:1000> DVA[2]=<0:2700190a000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29349L/29349P fill=206 cksum=00000004250a7419:00000ff455196f11:001ebdb5e8a9e7c7:2788130a4b9fc891
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[6]
magic = 0000000000bab10c
version = 5000
txg = 29350
guid_sum = 5337956680387211398
timestamp = 1745969410 UTC = Wed Apr 30 08:30:10 2025
bp = DVA[0]=<0:26c5ea59000:1000> DVA[1]=<0:2647afd1000:1000> DVA[2]=<0:27001916000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29350L/29350P fill=206 cksum=0000000327e4a525:00000c2e90f7a274:0017899d7b58c1b9:1e5a30a4d0a50599
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[7]
magic = 0000000000bab10c
version = 5000
txg = 29351
guid_sum = 5337956680387211398
timestamp = 1745969415 UTC = Wed Apr 30 08:30:15 2025
bp = DVA[0]=<0:26c5eaf2000:1000> DVA[1]=<0:2647b01f000:1000> DVA[2]=<0:27001925000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29351L/29351P fill=206 cksum=000000041da01d60:00000fd74b016361:001e84cda37ccc4d:273daae6f12582b9
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[8]
magic = 0000000000bab10c
version = 5000
txg = 29352
guid_sum = 5337956680387211398
timestamp = 1745969420 UTC = Wed Apr 30 08:30:20 2025
bp = DVA[0]=<0:26c5eb31000:1000> DVA[1]=<0:2647b05e000:1000> DVA[2]=<0:27001944000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29352L/29352P fill=206 cksum=00000003b5a75518:00000e4c6cdb3a20:001b9643a69a6cdd:2385b572717a9dcc
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[9]
magic = 0000000000bab10c
version = 5000
txg = 29353
guid_sum = 5337956680387211398
timestamp = 1745969426 UTC = Wed Apr 30 08:30:26 2025
bp = DVA[0]=<0:26c5ebbb000:1000> DVA[1]=<0:2647b09d000:1000> DVA[2]=<0:27001950000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29353L/29353P fill=206 cksum=00000004a7fb4d53:000011e7ffb30f77:002278077c29f364:2c4868b1098e39eb
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[10]
magic = 0000000000bab10c
version = 5000
txg = 29354
guid_sum = 5337956680387211398
timestamp = 1745969431 UTC = Wed Apr 30 08:30:31 2025
bp = DVA[0]=<0:26c5ebf8000:1000> DVA[1]=<0:2647b0da000:1000> DVA[2]=<0:2700195b000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29354L/29354P fill=206 cksum=00000003e41c100f:00000ef97d846c32:001cd8b65cd9c91e:25164f3ec19eb4ee
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[11]
magic = 0000000000bab10c
version = 5000
txg = 29355
guid_sum = 5337956680387211398
timestamp = 1745969436 UTC = Wed Apr 30 08:30:36 2025
bp = DVA[0]=<0:26c5f88a000:1000> DVA[1]=<0:2647b121000:1000> DVA[2]=<0:2700196a000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29355L/29355P fill=206 cksum=00000003483497b0:00000ca672c0946b:00186801836a5718:1f6d35f23d27a9ab
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[12]
magic = 0000000000bab10c
version = 5000
txg = 29356
guid_sum = 5337956680387211398
timestamp = 1745969441 UTC = Wed Apr 30 08:30:41 2025
bp = DVA[0]=<0:26c5f92c000:1000> DVA[1]=<0:2647b178000:1000> DVA[2]=<0:2700198a000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29356L/29356P fill=205 cksum=0000000482612427:0000115c313a167c:002174166945f33c:2b062c6ace38ec57
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[13]
magic = 0000000000bab10c
version = 5000
txg = 29357
guid_sum = 5337956680387211398
timestamp = 1745969446 UTC = Wed Apr 30 08:30:46 2025
bp = DVA[0]=<0:26c5fa59000:1000> DVA[1]=<0:2647b1a5000:1000> DVA[2]=<0:27001996000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29357L/29357P fill=205 cksum=000000036116badf:00000d07596e7c45:001924d3b266a7e8:2062b0c51f964ff3
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[14]
magic = 0000000000bab10c
version = 5000
txg = 29358
guid_sum = 5337956680387211398
timestamp = 1745969451 UTC = Wed Apr 30 08:30:51 2025
bp = DVA[0]=<0:26c5fae6000:1000> DVA[1]=<0:2647b1e7000:1000> DVA[2]=<0:270019a5000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29358L/29358P fill=205 cksum=0000000458696ef2:000010b9a4672605:002039047bb0502e:296eafbd679b66f3
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[15]
magic = 0000000000bab10c
version = 5000
txg = 29359
guid_sum = 5337956680387211398
timestamp = 1745969456 UTC = Wed Apr 30 08:30:56 2025
bp = DVA[0]=<0:26c5fb30000:1000> DVA[1]=<0:2647b231000:1000> DVA[2]=<0:270019c5000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29359L/29359P fill=205 cksum=0000000308fcdf3c:00000bb797d0254f:0016a46513f196aa:1d337ccd2cac6aa6
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[16]
magic = 0000000000bab10c
version = 5000
txg = 29360
guid_sum = 5337956680387211398
timestamp = 1745969461 UTC = Wed Apr 30 08:31:01 2025
bp = DVA[0]=<0:26c5fbb6000:1000> DVA[1]=<0:2647b26c000:1000> DVA[2]=<0:270019d7000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29360L/29360P fill=206 cksum=000000045ebf4948:000010d1e3d6cac7:002067756b29c0c3:29aa08af472f1307
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[17]
magic = 0000000000bab10c
version = 5000
txg = 29361
guid_sum = 5337956680387211398
timestamp = 1745969467 UTC = Wed Apr 30 08:31:07 2025
bp = DVA[0]=<0:26c60749000:1000> DVA[1]=<0:2647b2b4000:1000> DVA[2]=<0:270019e3000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29361L/29361P fill=206 cksum=00000003e3312b80:00000ef58ee6f17b:001cd06153f8175b:250aa470b8f16f88
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[18]
magic = 0000000000bab10c
version = 5000
txg = 29362
guid_sum = 5337956680387211398
timestamp = 1745969472 UTC = Wed Apr 30 08:31:12 2025
bp = DVA[0]=<0:26c6098c000:1000> DVA[1]=<0:2647b2f7000:1000> DVA[2]=<0:270019f2000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29362L/29362P fill=206 cksum=00000003ca91122d:00000e988a425258:001c208c2586b799:242cdd980ec76ed1
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[19]
magic = 0000000000bab10c
version = 5000
txg = 29363
guid_sum = 5337956680387211398
timestamp = 1745969477 UTC = Wed Apr 30 08:31:17 2025
bp = DVA[0]=<0:26c60a24000:1000> DVA[1]=<0:2647b344000:1000> DVA[2]=<0:27001a12000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29363L/29363P fill=205 cksum=000000043a24b24e:00001046e53fadee:001f5f50486d6532:285b11b05c73347d
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[20]
magic = 0000000000bab10c
version = 5000
txg = 29364
guid_sum = 5337956680387211398
timestamp = 1745969482 UTC = Wed Apr 30 08:31:22 2025
bp = DVA[0]=<0:26c612ca000:1000> DVA[1]=<0:2647b39f000:1000> DVA[2]=<0:27001a24000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29364L/29364P fill=205 cksum=0000000499e0cf89:000011b4a444faf9:00221aac1c27eac6:2bd77ceebc70abe8
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[21]
magic = 0000000000bab10c
version = 5000
txg = 29365
guid_sum = 5337956680387211398
timestamp = 1745969487 UTC = Wed Apr 30 08:31:27 2025
bp = DVA[0]=<0:26c61409000:1000> DVA[1]=<0:2647b3de000:1000> DVA[2]=<0:27001a34000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29365L/29365P fill=206 cksum=00000004439efd61:0000106aee34a9b8:001fa3e0526016b7:28b21c829f62e3e7
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[22]
magic = 0000000000bab10c
version = 5000
txg = 29366
guid_sum = 5337956680387211398
timestamp = 1745969492 UTC = Wed Apr 30 08:31:32 2025
bp = DVA[0]=<0:26c61492000:1000> DVA[1]=<0:2647b41c000:1000> DVA[2]=<0:27001a40000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29366L/29366P fill=206 cksum=00000004a5ae6577:000011e00bef0d6d:00226a6aa84c358f:2c3910591991c80a
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[23]
magic = 0000000000bab10c
version = 5000
txg = 29367
guid_sum = 5337956680387211398
timestamp = 1745969498 UTC = Wed Apr 30 08:31:38 2025
bp = DVA[0]=<0:26c62432000:1000> DVA[1]=<0:2647b4d5000:1000> DVA[2]=<0:27001a60000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29367L/29367P fill=206 cksum=00000004c8234c32:000012635cc4fcca:002364e629635d09:2d77eb1ba7fc1cbd
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[24]
magic = 0000000000bab10c
version = 5000
txg = 29368
guid_sum = 5337956680387211398
timestamp = 1745969502 UTC = Wed Apr 30 08:31:42 2025
bp = DVA[0]=<0:26c6245e000:1000> DVA[1]=<0:2647b501000:1000> DVA[2]=<0:27001a76000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29368L/29368P fill=206 cksum=000000040101bd60:00000f67b0519f42:001dab0e24450295:26224203ae165d29
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[25]
magic = 0000000000bab10c
version = 5000
txg = 29369
guid_sum = 5337956680387211398
timestamp = 1745969508 UTC = Wed Apr 30 08:31:48 2025
bp = DVA[0]=<0:26c62598000:1000> DVA[1]=<0:2647b53b000:1000> DVA[2]=<0:27001a85000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29369L/29369P fill=206 cksum=00000003e00e2415:00000eecce8d1416:001cc5d9dac1f60e:25052780bd660181
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[26]
magic = 0000000000bab10c
version = 5000
txg = 29370
guid_sum = 5337956680387211398
timestamp = 1745969723 UTC = Wed Apr 30 08:35:23 2025
bp = DVA[0]=<0:27400005000:1000> DVA[1]=<0:27800005000:1000> DVA[2]=<0:27c00004000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29370L/29370P fill=209 cksum=0000000413e9555a:00000fb37d3c0046:001e42edc9788c61:26ed06177cc752e1
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[27]
magic = 0000000000bab10c
version = 5000
txg = 29371
guid_sum = 5337956680387211398
timestamp = 1745969723 UTC = Wed Apr 30 08:35:23 2025
bp = DVA[0]=<0:27400026000:1000> DVA[1]=<0:27800026000:1000> DVA[2]=<0:27c00025000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29371L/29371P fill=210 cksum=000000047e2376a3:0000114b80482f69:0021534d8b47e0b2:2adb513510f14f06
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[28]
magic = 0000000000bab10c
version = 5000
txg = 29340
guid_sum = 5337956680387211398
timestamp = 1745969359 UTC = Wed Apr 30 08:29:19 2025
bp = DVA[0]=<0:26c5ccf3000:1000> DVA[1]=<0:2647acba000:1000> DVA[2]=<0:2700185c000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29340L/29340P fill=205 cksum=00000003d30b0017:00000eb73474be81:001c57e100a10a4d:246f3b284b4d3739
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[29]
magic = 0000000000bab10c
version = 5000
txg = 29373
guid_sum = 5337956680387211398
timestamp = 1745969723 UTC = Wed Apr 30 08:35:23 2025
bp = DVA[0]=<0:2740003f000:1000> DVA[1]=<0:2780003f000:1000> DVA[2]=<0:27c0003e000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29373L/29373P fill=211 cksum=00000004706467b3:000011157e537c10:0020e92708b9a9f6:2a5026af2ff7f995
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[30]
magic = 0000000000bab10c
version = 5000
txg = 29342
guid_sum = 5337956680387211398
timestamp = 1745969369 UTC = Wed Apr 30 08:29:29 2025
bp = DVA[0]=<0:26c5cdd0000:1000> DVA[1]=<0:2647ad4c000:1000> DVA[2]=<0:27001889000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29342L/29342P fill=205 cksum=000000045c141c27:000010c5694c5871:00204b34b086d08c:29805824d734bcea
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[31]
magic = 0000000000bab10c
version = 5000
txg = 29375
guid_sum = 5337956680387211398
timestamp = 1745969723 UTC = Wed Apr 30 08:35:23 2025
bp = DVA[0]=<0:27400051000:1000> DVA[1]=<0:27800051000:1000> DVA[2]=<0:27c00050000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29375L/29375P fill=212 cksum=0000000447134442:00001079787d98ba:001fc252191aaa02:28dc6ce26702cacf
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3

LABEL 2

version: 5000
name: 'zfspool0'
state: 0
txg: 29376
pool_guid: 17328721603664047508
errata: 0
hostid: xxx
hostname: localhost
top_guid: 16873280761438265250
guid: 7788381016733623247
vdev_children: 1
vdev_tree:
    type: 'mirror'
    id: 0
    guid: 16873280761438265250
    metaslab_array: 128
    metaslab_shift: 34
    ashift: 12
    asize: 8001548713984
    is_log: 0
    create_txg: 4
    children[0]:
        type: 'disk'
        id: 0
        guid: 7788381016733623247
        path: '/dev/disk/by-id/usb-External_USB3.0_DISK00_20170331000C3-0:0-part1'
        devid: 'usb-External_USB3.0_DISK00_20170331000C3-0:0-part1'
        phys_path: 'platform-xhci-hcd.1-usb-0:1:1.0-scsi-0:0:0:0'
        whole_disk: 1
        DTL: 5839
        create_txg: 4
    children[1]:
        type: 'disk'
        id: 1
        guid: 241061445970378625
        path: '/dev/disk/by-id/usb-External_USB3.0_DISK01_20170331000C3-0:1-part1'
        devid: 'usb-External_USB3.0_DISK01_20170331000C3-0:1-part1'
        phys_path: 'platform-xhci-hcd.1-usb-0:1:1.0-scsi-0:0:0:1'
        whole_disk: 1
        DTL: 5838
        create_txg: 4
        expansion_time: 1745969723
features_for_read:
    com.delphix:hole_birth
    com.delphix:embedded_data
    com.klarasystems:vdev_zaps_v2
labels = 2 3
Uberblock[3]

magic = 0000000000bab10c
version = 5000
txg = 29347
guid_sum = 5337956680387211398
timestamp = 1745969395 UTC = Wed Apr 30 08:29:55 2025
bp = DVA[0]=<0:26c5de18000:1000> DVA[1]=<0:2647aea8000:1000> DVA[2]=<0:270018de000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29347L/29347P fill=205 cksum=00000003cb1c2112:00000e9e01a8e345:001c3184ed3f4cca:244ae2e38bae8cfb
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 2 3
Uberblock[4]
magic = 0000000000bab10c
version = 5000
txg = 29348
guid_sum = 5337956680387211398
timestamp = 1745969400 UTC = Wed Apr 30 08:30:00 2025
bp = DVA[0]=<0:26c5e026000:1000> DVA[1]=<0:2647af2c000:1000> DVA[2]=<0:270018fe000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=29348L/29348P fill=206 cksum=000000039e4153c8:00000def83a1656e:001addd70a89dfd9:2291b1ee959b880c
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 2 3

squinj · May 6, 2025, 5:13am

also fyi import is still broken upon moving the zfs cache file out of the way:

mv /etc/zfs/zpool.cache /etc/zfs/zpool.cache.20250430

zpool import

pool: zfspool2
id: 12795995150218211739
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

zfspool2 ONLINE
usb-External_USB3.0_DISK04_20170331000C3-0:4 ONLINE

pool: zfspool1
id: 1315775537458570594
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

zfspool1 ONLINE
mirror-0 ONLINE
usb-External_USB3.0_DISK02_20170331000C3-0:2 ONLINE
usb-External_USB3.0_DISK03_20170331000C3-0:3 ONLINE

pool: zfspool0
id: 17328721603664047508
state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
see: link snipped
config:

zfspool0 UNAVAIL insufficient replicas
mirror-0 UNAVAIL insufficient replicas
usb-External_USB3.0_DISK00_20170331000C3-0:0 UNAVAIL invalid label
usb-External_USB3.0_DISK01_20170331000C3-0:1 UNAVAIL invalid label

squinj · May 6, 2025, 5:18am

further info on the disks from fdisk

Disk /dev/sda: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: USB3.0 DISK00
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 54BAE8D7-DBEF-2C4D-832B-62DC038F88E2

Device Start End Sectors Size Type
/dev/sda1 2048 15628036095 15628034048 7.3T Solaris /usr & Apple ZFS
/dev/sda9 15628036096 15628052479 16384 8M Solaris reserved 1

Command (m for help): q

fdisk /dev/sdb

Welcome to fdisk (util-linux 2.38.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): p
Disk /dev/sdb: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: USB3.0 DISK01
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 14A1C265-D868-6D4F-949A-654BA154A094

Device Start End Sectors Size Type
/dev/sdb1 2048 15628036095 15628034048 7.3T Solaris /usr & Apple ZFS
/dev/sdb9 15628036096 15628052479 16384 8M Solaris reserved 1

Command (m for help): q

squinj · May 6, 2025, 5:53am

further testing:

mkdir /tmp/zfspool0

ln -s /dev/sda1 /tmp/zfspool0

ln -s /dev/sdb1 /tmp/zfspool0

zpool import -d /tmp/zfspool0

pool: zfspool0
id: 17328721603664047508
state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
see: xxx
config:

zfspool0 UNAVAIL insufficient replicas
mirror-0 UNAVAIL insufficient replicas
/tmp/zfspool0/sda1 UNAVAIL invalid label
/tmp/zfspool0/sdb1 UNAVAIL invalid label

squinj · May 6, 2025, 6:51am

zhack label repair -cu /dev/sda

Calculated filesize to be 8001563222016
error: label 0: Expected the nvlist checksum magic number to not be zero
There should already be a checksum for the label.
error: label 1: Expected the nvlist checksum magic number to not be zero
There should already be a checksum for the label.
error: label 2: Expected the nvlist checksum magic number to not be zero
There should already be a checksum for the label.
error: label 3: Expected the nvlist checksum magic number to not be zero
There should already be a checksum for the label.
label 0: uberblock: skipped checksum: skipped
label 1: uberblock: skipped checksum: skipped
label 2: uberblock: skipped checksum: skipped
label 3: uberblock: skipped checksum: skipped

zhack label repair -cu /dev/sdb

Calculated filesize to be 8001563222016
error: label 0: Expected the nvlist checksum magic number to not be zero
There should already be a checksum for the label.
error: label 1: Expected the nvlist checksum magic number to not be zero
There should already be a checksum for the label.
error: label 2: Expected the nvlist checksum magic number to not be zero
There should already be a checksum for the label.
error: label 3: Expected the nvlist checksum magic number to not be zero
There should already be a checksum for the label.
label 0: uberblock: skipped checksum: skipped
label 1: uberblock: skipped checksum: skipped
label 2: uberblock: skipped checksum: skipped
label 3: uberblock: skipped checksum: skipped

zpool import zfspool0

cannot import ‘zfspool0’: invalid vdev configuration

zhack label repair -cu /dev/sda1

Calculated filesize to be 8001553432576
error: label 0: UB TXG of 0 expected, but got 29376
It would appear the device was not properly removed.
error: label 1: UB TXG of 0 expected, but got 29376
It would appear the device was not properly removed.
error: label 2: UB TXG of 0 expected, but got 29376
It would appear the device was not properly removed.
error: label 3: UB TXG of 0 expected, but got 29376
It would appear the device was not properly removed.
label 0: uberblock: skipped checksum: skipped
label 1: uberblock: skipped checksum: skipped
label 2: uberblock: skipped checksum: skipped
label 3: uberblock: skipped checksum: skipped

zhack label repair -cu /dev/sdb1

Calculated filesize to be 8001553432576
error: label 0: UB TXG of 0 expected, but got 29376
It would appear the device was not properly removed.
error: label 1: UB TXG of 0 expected, but got 29376
It would appear the device was not properly removed.
error: label 2: UB TXG of 0 expected, but got 29344
It would appear the device was not properly removed.
error: label 3: UB TXG of 0 expected, but got 29344
It would appear the device was not properly removed.
label 0: uberblock: skipped checksum: skipped
label 1: uberblock: skipped checksum: skipped
label 2: uberblock: skipped checksum: skipped
label 3: uberblock: skipped checksum: skipped

zpool import zfspool0

cannot import ‘zfspool0’: invalid vdev configuration

pmh · May 6, 2025, 7:25am

USB enclosures are unsupported and prone to lead to failure as you just experienced.

etorix · May 6, 2025, 9:40am

Please use the formatted text </> button when pasting big blocks of terminal output.