Cannot import Pool I/O Error after UDMA CRC Errors and replaced HBA

TrueNAS General
,
21

Thanks @NickF1227 for your detailed analysis on the current states of my disks.

OP its your call. We can dig in deeper and try to figure out what happened or we can try to get you up and running.

Either pull the sdi disk first and try and see if it just works,

Or we can check the disk speeds to confirm sdi is a problem, but maybe more.

My preference is to dig in deeper to figure out what happened and try to determine what are the best disks to use for running the import.

I have three queries:

I now want to run the 10G read across each disk as you have advised. What is the best mechanism to send you the resulting log files?

I’m also concerned that drive sde (serial JEGSBJDM) cannot be trusted as it was in the process of being resilvered 12th May using the old HBA and this task never completed. Should this drive be removed from any import?

Is there any value in providing you with a history of the previous Multi_Reports detailing the propagation of these ATA Errors?

22

Here is the output after running this command against the ZFS partition of each disk in the hd01 pool.

Disk - sda (serial: JEJJXMEX)

root@emnas[~]# sudo zdb -l /dev/sda2
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'hd01'
    state: 0
    txg: 27697225
    pool_guid: 3338235958544899494
    errata: 0
    hostid: 1226022578
    hostname: 'emnas'
    top_guid: 16803706522893222247
    guid: 14889606364094946993
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 16803706522893222247
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 59992073306112
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4397673188761218932
            path: '/dev/sdc2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@1/elmdesc@Slot00/p2'
            not_present: 1
            DTL: 17136
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 5530636865394825414
            path: '/dev/sdb2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@a/elmdesc@Slot09/p2'
            DTL: 17135
            create_txg: 4
        children[2]:
            type: 'replacing'
            id: 2
            guid: 16396119398788283152
            whole_disk: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 17618637945733147505
                path: '/dev/sdc2'
                phys_path: 'id1,enc@n500304800191893d/type@0/slot@7/elmdesc@Slot06/p2'
                DTL: 17134
                create_txg: 4
                degraded: 1
            children[1]:
                type: 'disk'
                id: 1
                guid: 8397950867974713209
                path: '/dev/disk/by-partuuid/b381495c-244a-4bba-a33c-a83e2bcc4fba'
                whole_disk: 0
                DTL: 282
                create_txg: 4
                resilver_txg: 27695154
        children[3]:
            type: 'disk'
            id: 3
            guid: 14889606364094946993
            path: '/dev/sda2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@3/elmdesc@Slot02/p2'
            DTL: 1034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3449743511316939874
            path: '/dev/sdd2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@9/elmdesc@Slot08/p2'
            DTL: 17132
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15234466540208390303
            path: '/dev/sdh2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@c/elmdesc@Slot11/p2'
            DTL: 17131
            create_txg: 4
            degraded: 1
            removed: 1
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3

Disk - sdb (serial: 2TJZET2D)

root@emnas[~]# sudo zdb -l /dev/sdb2
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'hd01'
    state: 0
    txg: 27697225
    pool_guid: 3338235958544899494
    errata: 0
    hostid: 1226022578
    hostname: 'emnas'
    top_guid: 16803706522893222247
    guid: 3449743511316939874
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 16803706522893222247
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 59992073306112
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4397673188761218932
            path: '/dev/sdc2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@1/elmdesc@Slot00/p2'
            not_present: 1
            DTL: 17136
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 5530636865394825414
            path: '/dev/sdb2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@a/elmdesc@Slot09/p2'
            DTL: 17135
            create_txg: 4
        children[2]:
            type: 'replacing'
            id: 2
            guid: 16396119398788283152
            whole_disk: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 17618637945733147505
                path: '/dev/sdc2'
                phys_path: 'id1,enc@n500304800191893d/type@0/slot@7/elmdesc@Slot06/p2'
                DTL: 17134
                create_txg: 4
                degraded: 1
            children[1]:
                type: 'disk'
                id: 1
                guid: 8397950867974713209
                path: '/dev/disk/by-partuuid/b381495c-244a-4bba-a33c-a83e2bcc4fba'
                whole_disk: 0
                DTL: 282
                create_txg: 4
                resilver_txg: 27695154
        children[3]:
            type: 'disk'
            id: 3
            guid: 14889606364094946993
            path: '/dev/sda2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@3/elmdesc@Slot02/p2'
            DTL: 1034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3449743511316939874
            path: '/dev/sdd2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@9/elmdesc@Slot08/p2'
            DTL: 17132
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15234466540208390303
            path: '/dev/sdh2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@c/elmdesc@Slot11/p2'
            DTL: 17131
            create_txg: 4
            degraded: 1
            removed: 1
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3

Disk - sdd (serial: 2YK2ZP4D)

root@emnas[~]# sudo zdb -l /dev/sdd2
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'hd01'
    state: 0
    txg: 27583278
    pool_guid: 3338235958544899494
    errata: 0
    hostid: 1226022578
    hostname: 'emnas'
    top_guid: 16803706522893222247
    guid: 17618637945733147505
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 16803706522893222247
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 59992073306112
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4397673188761218932
            path: '/dev/sdc2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@1/elmdesc@Slot00/p2'
            DTL: 17136
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 5530636865394825414
            path: '/dev/sdi2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@a/elmdesc@Slot09/p2'
            DTL: 17135
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 17618637945733147505
            path: '/dev/sdd2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@7/elmdesc@Slot06/p2'
            DTL: 17134
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 14889606364094946993
            path: '/dev/sdf2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@3/elmdesc@Slot02/p2'
            DTL: 1034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3449743511316939874
            path: '/dev/sda2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@9/elmdesc@Slot08/p2'
            DTL: 17132
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15234466540208390303
            path: '/dev/sdj2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@c/elmdesc@Slot11/p2'
            DTL: 17131
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3

Disk - sde (serial: JEGSBJDM)

Please note there is only one partition available ‘sde1’ and it is reported as type “Solaris /usr & Apple ZFS” and all the other disks in the hd01 pool are partition type “FreeBSD ZFS”. This disk was in the process of being resilvered and this task never completed.

root@emnas[~]# sudo zdb -l /dev/sde1
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'hd01'
    state: 0
    txg: 27697225
    pool_guid: 3338235958544899494
    errata: 0
    hostid: 1226022578
    hostname: 'emnas'
    top_guid: 16803706522893222247
    guid: 8397950867974713209
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 16803706522893222247
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 59992073306112
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4397673188761218932
            path: '/dev/sdc2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@1/elmdesc@Slot00/p2'
            not_present: 1
            DTL: 17136
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 5530636865394825414
            path: '/dev/sdb2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@a/elmdesc@Slot09/p2'
            DTL: 17135
            create_txg: 4
        children[2]:
            type: 'replacing'
            id: 2
            guid: 16396119398788283152
            whole_disk: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 17618637945733147505
                path: '/dev/sdc2'
                phys_path: 'id1,enc@n500304800191893d/type@0/slot@7/elmdesc@Slot06/p2'
                DTL: 17134
                create_txg: 4
                degraded: 1
            children[1]:
                type: 'disk'
                id: 1
                guid: 8397950867974713209
                path: '/dev/disk/by-partuuid/b381495c-244a-4bba-a33c-a83e2bcc4fba'
                whole_disk: 0
                DTL: 282
                create_txg: 4
                resilver_txg: 27695154
        children[3]:
            type: 'disk'
            id: 3
            guid: 14889606364094946993
            path: '/dev/sda2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@3/elmdesc@Slot02/p2'
            DTL: 1034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3449743511316939874
            path: '/dev/sdd2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@9/elmdesc@Slot08/p2'
            DTL: 17132
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15234466540208390303
            path: '/dev/sdh2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@c/elmdesc@Slot11/p2'
            DTL: 17131
            create_txg: 4
            degraded: 1
            removed: 1
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3

Disk - sdf (serial: 2YK3H7HD)

root@emnas[~]# sudo zdb -l /dev/sdf2
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'hd01'
    state: 0
    txg: 27583278
    pool_guid: 3338235958544899494
    errata: 0
    hostid: 1226022578
    hostname: 'emnas'
    top_guid: 16803706522893222247
    guid: 4397673188761218932
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 16803706522893222247
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 59992073306112
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4397673188761218932
            path: '/dev/sdc2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@1/elmdesc@Slot00/p2'
            DTL: 17136
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 5530636865394825414
            path: '/dev/sdi2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@a/elmdesc@Slot09/p2'
            DTL: 17135
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 17618637945733147505
            path: '/dev/sdd2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@7/elmdesc@Slot06/p2'
            DTL: 17134
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 14889606364094946993
            path: '/dev/sdf2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@3/elmdesc@Slot02/p2'
            DTL: 1034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3449743511316939874
            path: '/dev/sda2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@9/elmdesc@Slot08/p2'
            DTL: 17132
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15234466540208390303
            path: '/dev/sdj2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@c/elmdesc@Slot11/p2'
            DTL: 17131
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3

Disk - sdh (serial: JEK9190N)

root@emnas[~]# sudo zdb -l /dev/sdh2
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'hd01'
    state: 0
    txg: 27697225
    pool_guid: 3338235958544899494
    errata: 0
    hostid: 1226022578
    hostname: 'emnas'
    top_guid: 16803706522893222247
    guid: 5530636865394825414
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 16803706522893222247
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 59992073306112
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4397673188761218932
            path: '/dev/sdc2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@1/elmdesc@Slot00/p2'
            not_present: 1
            DTL: 17136
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 5530636865394825414
            path: '/dev/sdb2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@a/elmdesc@Slot09/p2'
            DTL: 17135
            create_txg: 4
        children[2]:
            type: 'replacing'
            id: 2
            guid: 16396119398788283152
            whole_disk: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 17618637945733147505
                path: '/dev/sdc2'
                phys_path: 'id1,enc@n500304800191893d/type@0/slot@7/elmdesc@Slot06/p2'
                DTL: 17134
                create_txg: 4
                degraded: 1
            children[1]:
                type: 'disk'
                id: 1
                guid: 8397950867974713209
                path: '/dev/disk/by-partuuid/b381495c-244a-4bba-a33c-a83e2bcc4fba'
                whole_disk: 0
                DTL: 282
                create_txg: 4
                resilver_txg: 27695154
        children[3]:
            type: 'disk'
            id: 3
            guid: 14889606364094946993
            path: '/dev/sda2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@3/elmdesc@Slot02/p2'
            DTL: 1034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3449743511316939874
            path: '/dev/sdd2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@9/elmdesc@Slot08/p2'
            DTL: 17132
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15234466540208390303
            path: '/dev/sdh2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@c/elmdesc@Slot11/p2'
            DTL: 17131
            create_txg: 4
            degraded: 1
            removed: 1
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3

Disk - sdi (serial: 2YK3RZLD)

root@emnas[~]# sudo zdb -l /dev/sdi2
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'hd01'
    state: 0
    txg: 27695185
    pool_guid: 3338235958544899494
    errata: 0
    hostid: 1226022578
    hostname: 'emnas'
    top_guid: 16803706522893222247
    guid: 15234466540208390303
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 16803706522893222247
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 59992073306112
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 4397673188761218932
            path: '/dev/sdc2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@1/elmdesc@Slot00/p2'
            not_present: 1
            DTL: 17136
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 5530636865394825414
            path: '/dev/sdb2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@a/elmdesc@Slot09/p2'
            DTL: 17135
            create_txg: 4
        children[2]:
            type: 'replacing'
            id: 2
            guid: 16396119398788283152
            whole_disk: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 17618637945733147505
                path: '/dev/sdc2'
                phys_path: 'id1,enc@n500304800191893d/type@0/slot@7/elmdesc@Slot06/p2'
                not_present: 1
                DTL: 17134
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 8397950867974713209
                path: '/dev/disk/by-partuuid/b381495c-244a-4bba-a33c-a83e2bcc4fba'
                whole_disk: 0
                DTL: 282
                create_txg: 4
                resilver_txg: 27695154
        children[3]:
            type: 'disk'
            id: 3
            guid: 14889606364094946993
            path: '/dev/sda2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@3/elmdesc@Slot02/p2'
            DTL: 1034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 3449743511316939874
            path: '/dev/sdd2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@9/elmdesc@Slot08/p2'
            DTL: 17132
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15234466540208390303
            path: '/dev/sdh2'
            phys_path: 'id1,enc@n500304800191893d/type@0/slot@c/elmdesc@Slot11/p2'
            DTL: 17131
            create_txg: 4
            degraded: 1
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3
23

You can just copy and paste here, it will output to the shell as well as the log file.

24

I don’t have reference to /dev/nvme*n1. What disk should I replace this with? Should I replace this with /dev/sdk or /dev/sdj which are 2 SSD in a separate pool called ‘sd01’?

root@emnas[~]# for d in /dev/sd? /dev/sd?? /dev/nvme*n1; do [ -b "$d" ] && echo -e "\n=== Reading 10G from $d ===" && dd if="$d" of=/dev/null bs=1M count=10240 status=progress; done 2>&1 | tee /root/read_10g_from_disks.log
zsh: no matches found: /dev/nvme*n1
root@emnas[~]#

My output from running - sblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAME

root@emnas[~]# lsblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAME
NAME   LABEL       MAJ:MIN TRAN   ROTA ZONED VENDOR  MODEL          SERIAL          PARTUUID                               START           SIZE PARTTYPENAME
sda                  8:0   sas       1 none  ATA     WDC WD100EFAX- JEJJXMEX                                                     10000831348736 
├─sda1 emnas:swap1   8:1             1 none                                         495687fd-2444-11ee-ad54-002590faf4a4     128     2147483648 FreeBSD swap
└─sda2 hd01          8:2             1 none                                         496602b9-2444-11ee-ad54-002590faf4a4 4194432  9998683779072 FreeBSD ZFS
sdb                  8:16  sas       1 none  ATA     WDC WD100EFAX- 2TJZET2D                                                     10000831348736 
├─sdb1 emnas:swap0   8:17            1 none                                         0e4b99e0-2840-11eb-9ccb-002590faf4a4     128     2147483648 FreeBSD swap
└─sdb2 hd01          8:18            1 none                                         0e822397-2840-11eb-9ccb-002590faf4a4 4194432  9998683779072 FreeBSD ZFS
sdc                  8:32  sas       1 none  ATA     WDC WD4000F9YZ WD-WCC131817132                                               4000787030016 
├─sdc1               8:33            1 none                                         4d8d9ffa-232d-11ed-8552-441ea13edfdc     128     2147483648 FreeBSD swap
└─sdc2 zd04          8:34            1 none                                         4da00c6f-232d-11ed-8552-441ea13edfdc 4194432  3998639460352 FreeBSD ZFS
sdd                  8:48  sas       1 none  ATA     WDC WD100EFAX- 2YK2ZP4D                                                     10000831348736 
├─sdd1 emnas:swap1   8:49            1 none                                         0daa3a2e-2840-11eb-9ccb-002590faf4a4     128     2147483648 FreeBSD swap
└─sdd2 hd01          8:50            1 none                                         0ded02af-2840-11eb-9ccb-002590faf4a4 4194432  9998683779072 FreeBSD ZFS
sde                  8:64  sas       1 none  ATA     WDC WD100EFAX- JEGSBJDM                                                     10000831348736 
└─sde1 hd01          8:65            1 none                                         b381495c-244a-4bba-a33c-a83e2bcc4fba    2048 10000829251584 Solaris /usr & Apple ZFS
sdf                  8:80  sas       1 none  ATA     WDC WD100EFAX- 2YK3H7HD                                                     10000831348736 
├─sdf1               8:81            1 none                                         0da13c60-2840-11eb-9ccb-002590faf4a4     128     2147483648 FreeBSD swap
└─sdf2 hd01          8:82            1 none                                         0de232dd-2840-11eb-9ccb-002590faf4a4 4194432  9998683779072 FreeBSD ZFS
sdg                  8:96  sas       1 none  ATA     WDC WD100EFAX- JEGSM56M                                                     10000831348736 
├─sdg1               8:97            1 none                                         13f664cd-71b3-11ec-8f6a-002590faf4a4     128     2147483648 FreeBSD swap
└─sdg2 zd01          8:98            1 none                                         141cfead-71b3-11ec-8f6a-002590faf4a4 4194432  9998683779072 FreeBSD ZFS
sdh                  8:112 sas       1 none  ATA     WDC WD100EFAX- JEK9190N                                                     10000831348736 
├─sdh1 emnas:swap1   8:113           1 none                                         0dac76b3-2840-11eb-9ccb-002590faf4a4     128     2147483648 FreeBSD swap
└─sdh2 hd01          8:114           1 none                                         0debcf35-2840-11eb-9ccb-002590faf4a4 4194432  9998683779072 FreeBSD ZFS
sdi                  8:128 sas       1 none  ATA     WDC WD100EFAX- 2YK3RZLD                                                     10000831348736 
├─sdi1               8:129           1 none                                         0e5440b6-2840-11eb-9ccb-002590faf4a4     128     2147483648 FreeBSD swap
└─sdi2 hd01          8:130           1 none                                         0e7cb1d5-2840-11eb-9ccb-002590faf4a4 4194432  9998683779072 FreeBSD ZFS
sdj                  8:144 sata      0 none  ATA     SAMSUNG MZ7LH9 S45NNA0N804048                                                 960197124096 
├─sdj1 emnas:swap0   8:145           0 none                                         71bda442-2840-11eb-9ccb-002590faf4a4     128     2147483648 FreeBSD swap
└─sdj2 sd01          8:146           0 none                                         71c2abe9-2840-11eb-9ccb-002590faf4a4 4194432   958049554432 FreeBSD ZFS
sdk                  8:160 sata      0 none  ATA     SAMSUNG MZ7LH9 S45NNA0N804030                                                 960197124096 
├─sdk1 emnas:swap0   8:161           0 none                                         71b8aa9a-2840-11eb-9ccb-002590faf4a4     128     2147483648 FreeBSD swap
└─sdk2 sd01          8:162           0 none                                         71bfab11-2840-11eb-9ccb-002590faf4a4 4194432   958049554432 FreeBSD ZFS
sdl                  8:176 sata      0 none  ATA     SATADOM-SL 3IE BCA116101402316                                                 64023257088 
├─sdl1               8:177           0 none                                         26b330c0-6d48-4fd9-b0ef-5141d208626e    4096        1048576 BIOS boot
├─sdl2 EFI           8:178           0 none                                         e7805f5e-2ee2-48d6-a37d-19e915cf9cc4    6144      536870912 EFI System
└─sdl3 boot-pool     8:179           0 none                                         091de296-1cff-4018-abdf-3633f0c3de79 1054720    63483223552 Solaris /usr & Apple ZFS
sdm                  8:192 sata      0 none  ATA     SATADOM-SL 3IE BCA116101402312                                                 64023257088 
├─sdm1               8:193           0 none                                         4a8c4733-1081-4a04-8117-57d99ca86883    4096        1048576 BIOS boot
├─sdm2 EFI           8:194           0 none                                         c7f74448-a2a5-4b28-8a43-1f41f15da007    6144      536870912 EFI System
└─sdm3 boot-pool     8:195           0 none                                         db5f6636-9705-408e-8554-fc5ce9dd63fd 1054720    63483223552 Solaris /usr & Apple ZFS
zd0                230:0             0 none                                                                                        107374198784 
zd16               230:16            0 none                                                                                         34359738368 
zd32               230:32            0 none                                                                                         21474852864 
zd48               230:48            0 none                                                                                         21474852864 
zd64               230:64            0 none                                                                                         10737418240 
zd80               230:80            0 none                                                                                         10737418240
25

That depends on the drive firmware. You know I get a lot of data from people and some drives report errors in this format, even using -a, however I would concede that -x is the catch all. I will send you a message so we can discuss possibly tracking this value. I don’t want to take away from the OP’s thread.

26

OK, that’s clear, thank you. I’m struggling with the script as it didn’t run so hopefully you can modify it based on the disks I have access to.

My result:

root@emnas[~]# for d in /dev/sd? /dev/sd?? /dev/nvme*n1; do [ -b "$d" ] && echo -e "\n=== Reading 10G from $d ===" && dd if="$d" of=/dev/null bs=1M count=10240 status=progress; done 2>&1 | tee /root/read_10g_from_disks.log
zsh: no matches found: /dev/nvme*n1
27 
for d in /dev/sd? /dev/sd??; do [ -b "$d" ] && echo -e "\n=== Reading 10G from $d ===" && dd if="$d" of=/dev/null bs=1M count=10240 status=progress; done 2>&1 | tee /root/read_10g_from_disks.log
28

sde is the disk which was being resilvered to and TrueNAS no longer creates a swap partition at the start of the disk - hence partition 1 rather than 2.

So if we look at the zdb output we can see why the pool isn’t importing - the txg numbers are not all the same:

/dev/sda2: txg: 27697225
/dev/sdb2: txg: 27697225
/dev/sdd2: txg: 27583278
/dev/sde1: txg: 27697225
/dev/sdf2: txg: 27583278
/dev/sdh2: txg: 27697225
/dev/sdi2: txg: 27695185

so the only way we will be able to import this with integrity will be to do so by rolling back to the first common transaction and lose any data written after that.

The fact that -fFn succeeded suggests that it may well be able to be mounted read-write from the earliest txg 27583278 assuming that it can achieve consistency at that txg (but the difference between 27697225 and 27583278 is 113,947 txgs, and this is a lot to expect to roll-back).

However, given that it is RAIDZ2, and it may not be able to use sdd/sde for import, then it may be able to ignore sdf and roll back to the earliest transaction ignoring sdd/sde and sdf which is sdi 27695185 and that is “only” [sic.] 2,040 txgs earlier. It is not clear whether this will also be too many txgs.

But my guess is that you are going to have a problem restoring read-write access to this pool - and the best you will achieve is to get read-only access to enable you to copy most of the files off so you can recreate the pool and restore.

So my next suggestion would be for you to attempt to import it readonly using

sudo zpool import -R /mnt -fF -o readonly=on hd01

and see what happens.

If you can import it, then you will have read access to your data, and you will then need to decide whether to attempt to mount it read-write or to backup the data whilst you have access, destroy and recreate the pool and then restire your data.

If you cannot import it with this command, then you will need to use -X as well in which case getting read-write access is going to be much more unlikely.

So my advice is to run the command above, see what happens, and at that point we can ask HoneyBadger to take a look and give a second opinion.

29

Here is the output from the log file for the 7 disks that make up the hd01 pool as produced by the script.

/dev/sda

=== Reading 10G from /dev/sda ===

2097152 bytes (2.1 MB, 2.0 MiB) copied, 1 s, 2.1 MB/s
213909504 bytes (214 MB, 204 MiB) copied, 2 s, 107 MB/s
429916160 bytes (430 MB, 410 MiB) copied, 3 s, 143 MB/s
642777088 bytes (643 MB, 613 MiB) copied, 4 s, 161 MB/s
857735168 bytes (858 MB, 818 MiB) copied, 5 s, 172 MB/s
1072693248 bytes (1.1 GB, 1023 MiB) copied, 6 s, 179 MB/s
1298137088 bytes (1.3 GB, 1.2 GiB) copied, 7 s, 185 MB/s 
1521483776 bytes (1.5 GB, 1.4 GiB) copied, 8 s, 190 MB/s
1746927616 bytes (1.7 GB, 1.6 GiB) copied, 9 s, 194 MB/s
1959788544 bytes (2.0 GB, 1.8 GiB) copied, 10 s, 196 MB/s
2166358016 bytes (2.2 GB, 2.0 GiB) copied, 11 s, 197 MB/s
2367684608 bytes (2.4 GB, 2.2 GiB) copied, 12 s, 197 MB/s
2575302656 bytes (2.6 GB, 2.4 GiB) copied, 13 s, 198 MB/s
2787115008 bytes (2.8 GB, 2.6 GiB) copied, 14 s, 199 MB/s
3011510272 bytes (3.0 GB, 2.8 GiB) copied, 15 s, 201 MB/s
3238002688 bytes (3.2 GB, 3.0 GiB) copied, 16 s, 202 MB/s
3451912192 bytes (3.5 GB, 3.2 GiB) copied, 17 s, 203 MB/s
3674210304 bytes (3.7 GB, 3.4 GiB) copied, 18 s, 204 MB/s
3893362688 bytes (3.9 GB, 3.6 GiB) copied, 19 s, 205 MB/s
4112515072 bytes (4.1 GB, 3.8 GiB) copied, 20 s, 206 MB/s
4332716032 bytes (4.3 GB, 4.0 GiB) copied, 21 s, 206 MB/s
4550819840 bytes (4.6 GB, 4.2 GiB) copied, 22 s, 207 MB/s
4775215104 bytes (4.8 GB, 4.4 GiB) copied, 23 s, 208 MB/s
5000658944 bytes (5.0 GB, 4.7 GiB) copied, 24 s, 208 MB/s
5226102784 bytes (5.2 GB, 4.9 GiB) copied, 25 s, 209 MB/s
5437915136 bytes (5.4 GB, 5.1 GiB) copied, 26 s, 209 MB/s
5637144576 bytes (5.6 GB, 5.2 GiB) copied, 27 s, 209 MB/s
5842665472 bytes (5.8 GB, 5.4 GiB) copied, 28 s, 209 MB/s
6048186368 bytes (6.0 GB, 5.6 GiB) copied, 29 s, 209 MB/s
6264193024 bytes (6.3 GB, 5.8 GiB) copied, 30 s, 209 MB/s
6486491136 bytes (6.5 GB, 6.0 GiB) copied, 31 s, 209 MB/s
6704594944 bytes (6.7 GB, 6.2 GiB) copied, 32 s, 209 MB/s
6928990208 bytes (6.9 GB, 6.5 GiB) copied, 33 s, 210 MB/s
7144996864 bytes (7.1 GB, 6.7 GiB) copied, 34 s, 210 MB/s
7355760640 bytes (7.4 GB, 6.9 GiB) copied, 35 s, 210 MB/s
7568621568 bytes (7.6 GB, 7.0 GiB) copied, 36 s, 210 MB/s
7772045312 bytes (7.8 GB, 7.2 GiB) copied, 37 s, 210 MB/s
7985954816 bytes (8.0 GB, 7.4 GiB) copied, 38 s, 210 MB/s
8213495808 bytes (8.2 GB, 7.6 GiB) copied, 39 s, 211 MB/s
8441036800 bytes (8.4 GB, 7.9 GiB) copied, 40 s, 211 MB/s
8659140608 bytes (8.7 GB, 8.1 GiB) copied, 41 s, 211 MB/s
8877244416 bytes (8.9 GB, 8.3 GiB) copied, 42 s, 211 MB/s
9082765312 bytes (9.1 GB, 8.5 GiB) copied, 43 s, 211 MB/s
9280946176 bytes (9.3 GB, 8.6 GiB) copied, 44 s, 211 MB/s
9487515648 bytes (9.5 GB, 8.8 GiB) copied, 45 s, 211 MB/s
9693036544 bytes (9.7 GB, 9.0 GiB) copied, 46 s, 211 MB/s
9917431808 bytes (9.9 GB, 9.2 GiB) copied, 47 s, 211 MB/s
10146021376 bytes (10 GB, 9.4 GiB) copied, 48 s, 211 MB/s
10362028032 bytes (10 GB, 9.7 GiB) copied, 49 s, 211 MB/s
10588520448 bytes (11 GB, 9.9 GiB) copied, 50 s, 212 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 51.2807 s, 209 MB/s

/dev/sdb

=== Reading 10G from /dev/sdb ===

213909504 bytes (214 MB, 204 MiB) copied, 1 s, 214 MB/s
434110464 bytes (434 MB, 414 MiB) copied, 2 s, 217 MB/s
654311424 bytes (654 MB, 624 MiB) copied, 3 s, 218 MB/s
872415232 bytes (872 MB, 832 MiB) copied, 4 s, 218 MB/s
1090519040 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 218 MB/s
1304428544 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 217 MB/s
1522532352 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 217 MB/s
1737490432 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 217 MB/s
1953497088 bytes (2.0 GB, 1.8 GiB) copied, 9 s, 217 MB/s
2170552320 bytes (2.2 GB, 2.0 GiB) copied, 10 s, 217 MB/s
2380267520 bytes (2.4 GB, 2.2 GiB) copied, 11 s, 216 MB/s
2596274176 bytes (2.6 GB, 2.4 GiB) copied, 12 s, 216 MB/s
2811232256 bytes (2.8 GB, 2.6 GiB) copied, 13 s, 216 MB/s
3032481792 bytes (3.0 GB, 2.8 GiB) copied, 14 s, 217 MB/s
3254779904 bytes (3.3 GB, 3.0 GiB) copied, 15 s, 217 MB/s
3467640832 bytes (3.5 GB, 3.2 GiB) copied, 16 s, 217 MB/s
3686793216 bytes (3.7 GB, 3.4 GiB) copied, 17 s, 217 MB/s
3906994176 bytes (3.9 GB, 3.6 GiB) copied, 18 s, 217 MB/s
4128243712 bytes (4.1 GB, 3.8 GiB) copied, 19 s, 217 MB/s
4349493248 bytes (4.3 GB, 4.1 GiB) copied, 20 s, 217 MB/s
4562354176 bytes (4.6 GB, 4.2 GiB) copied, 21 s, 217 MB/s
4783603712 bytes (4.8 GB, 4.5 GiB) copied, 22 s, 217 MB/s
5005901824 bytes (5.0 GB, 4.7 GiB) copied, 23 s, 218 MB/s
5228199936 bytes (5.2 GB, 4.9 GiB) copied, 24 s, 218 MB/s
5422186496 bytes (5.4 GB, 5.0 GiB) copied, 25 s, 217 MB/s
5603590144 bytes (5.6 GB, 5.2 GiB) copied, 26 s, 216 MB/s
5794430976 bytes (5.8 GB, 5.4 GiB) copied, 27 s, 215 MB/s
5985271808 bytes (6.0 GB, 5.6 GiB) copied, 28 s, 214 MB/s
6179258368 bytes (6.2 GB, 5.8 GiB) copied, 29 s, 213 MB/s
6402605056 bytes (6.4 GB, 6.0 GiB) copied, 30 s, 213 MB/s
6613368832 bytes (6.6 GB, 6.2 GiB) copied, 31 s, 213 MB/s
6836715520 bytes (6.8 GB, 6.4 GiB) copied, 32 s, 214 MB/s
7056916480 bytes (7.1 GB, 6.6 GiB) copied, 33 s, 214 MB/s
7273971712 bytes (7.3 GB, 6.8 GiB) copied, 34 s, 214 MB/s
7493124096 bytes (7.5 GB, 7.0 GiB) copied, 35 s, 214 MB/s
7711227904 bytes (7.7 GB, 7.2 GiB) copied, 36 s, 214 MB/s
7930380288 bytes (7.9 GB, 7.4 GiB) copied, 37 s, 214 MB/s
8157921280 bytes (8.2 GB, 7.6 GiB) copied, 38 s, 215 MB/s
8385462272 bytes (8.4 GB, 7.8 GiB) copied, 39 s, 215 MB/s
8610906112 bytes (8.6 GB, 8.0 GiB) copied, 40 s, 215 MB/s
8825864192 bytes (8.8 GB, 8.2 GiB) copied, 41 s, 215 MB/s
9042919424 bytes (9.0 GB, 8.4 GiB) copied, 42 s, 215 MB/s
9258926080 bytes (9.3 GB, 8.6 GiB) copied, 43 s, 215 MB/s
9474932736 bytes (9.5 GB, 8.8 GiB) copied, 44 s, 215 MB/s
9695133696 bytes (9.7 GB, 9.0 GiB) copied, 45 s, 215 MB/s
9910091776 bytes (9.9 GB, 9.2 GiB) copied, 46 s, 215 MB/s
10131341312 bytes (10 GB, 9.4 GiB) copied, 47 s, 216 MB/s
10353639424 bytes (10 GB, 9.6 GiB) copied, 48 s, 216 MB/s
10570694656 bytes (11 GB, 9.8 GiB) copied, 49 s, 216 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 50.3289 s, 213 MB/s

/dev/sdd

=== Reading 10G from /dev/sdd ===

210763776 bytes (211 MB, 201 MiB) copied, 1 s, 210 MB/s
425721856 bytes (426 MB, 406 MiB) copied, 2 s, 213 MB/s
641728512 bytes (642 MB, 612 MiB) copied, 3 s, 214 MB/s
851443712 bytes (851 MB, 812 MiB) copied, 4 s, 213 MB/s
1070596096 bytes (1.1 GB, 1021 MiB) copied, 5 s, 214 MB/s
1293942784 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 216 MB/s 
1517289472 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 217 MB/s
1740636160 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 217 MB/s
1954545664 bytes (2.0 GB, 1.8 GiB) copied, 9 s, 217 MB/s
2168455168 bytes (2.2 GB, 2.0 GiB) copied, 10 s, 217 MB/s
2384461824 bytes (2.4 GB, 2.2 GiB) copied, 11 s, 217 MB/s
2600468480 bytes (2.6 GB, 2.4 GiB) copied, 12 s, 217 MB/s
2820669440 bytes (2.8 GB, 2.6 GiB) copied, 13 s, 217 MB/s
3038773248 bytes (3.0 GB, 2.8 GiB) copied, 14 s, 217 MB/s
3265265664 bytes (3.3 GB, 3.0 GiB) copied, 15 s, 218 MB/s
3489660928 bytes (3.5 GB, 3.2 GiB) copied, 16 s, 218 MB/s
3703570432 bytes (3.7 GB, 3.4 GiB) copied, 17 s, 218 MB/s
3905945600 bytes (3.9 GB, 3.6 GiB) copied, 18 s, 217 MB/s
4103077888 bytes (4.1 GB, 3.8 GiB) copied, 19 s, 216 MB/s
4305453056 bytes (4.3 GB, 4.0 GiB) copied, 20 s, 215 MB/s
4508876800 bytes (4.5 GB, 4.2 GiB) copied, 21 s, 215 MB/s
4738514944 bytes (4.7 GB, 4.4 GiB) copied, 22 s, 215 MB/s
4968153088 bytes (5.0 GB, 4.6 GiB) copied, 23 s, 216 MB/s
5190451200 bytes (5.2 GB, 4.8 GiB) copied, 24 s, 216 MB/s
5416943616 bytes (5.4 GB, 5.0 GiB) copied, 25 s, 217 MB/s
5630853120 bytes (5.6 GB, 5.2 GiB) copied, 26 s, 217 MB/s
5844762624 bytes (5.8 GB, 5.4 GiB) copied, 27 s, 216 MB/s
6058672128 bytes (6.1 GB, 5.6 GiB) copied, 28 s, 216 MB/s
6256852992 bytes (6.3 GB, 5.8 GiB) copied, 29 s, 216 MB/s
6480199680 bytes (6.5 GB, 6.0 GiB) copied, 30 s, 216 MB/s
6705643520 bytes (6.7 GB, 6.2 GiB) copied, 31 s, 216 MB/s
6933184512 bytes (6.9 GB, 6.5 GiB) copied, 32 s, 217 MB/s
7155482624 bytes (7.2 GB, 6.7 GiB) copied, 33 s, 217 MB/s
7354712064 bytes (7.4 GB, 6.8 GiB) copied, 34 s, 216 MB/s
7562330112 bytes (7.6 GB, 7.0 GiB) copied, 35 s, 216 MB/s
7768899584 bytes (7.8 GB, 7.2 GiB) copied, 36 s, 216 MB/s
7975469056 bytes (8.0 GB, 7.4 GiB) copied, 37 s, 215 MB/s
8193572864 bytes (8.2 GB, 7.6 GiB) copied, 38 s, 216 MB/s
8413773824 bytes (8.4 GB, 7.8 GiB) copied, 39 s, 216 MB/s
8638169088 bytes (8.6 GB, 8.0 GiB) copied, 40 s, 216 MB/s
8862564352 bytes (8.9 GB, 8.3 GiB) copied, 41 s, 216 MB/s
9070182400 bytes (9.1 GB, 8.4 GiB) copied, 42 s, 216 MB/s
9274654720 bytes (9.3 GB, 8.6 GiB) copied, 43 s, 216 MB/s
9474932736 bytes (9.5 GB, 8.8 GiB) copied, 44 s, 215 MB/s
9680453632 bytes (9.7 GB, 9.0 GiB) copied, 45 s, 215 MB/s
9890168832 bytes (9.9 GB, 9.2 GiB) copied, 46 s, 215 MB/s
10116661248 bytes (10 GB, 9.4 GiB) copied, 47 s, 215 MB/s
10344202240 bytes (10 GB, 9.6 GiB) copied, 48 s, 215 MB/s
10563354624 bytes (11 GB, 9.8 GiB) copied, 49 s, 216 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 50.3527 s, 213 MB/s

/dev/sde

=== Reading 10G from /dev/sde ===

209715200 bytes (210 MB, 200 MiB) copied, 1 s, 209 MB/s
426770432 bytes (427 MB, 407 MiB) copied, 2 s, 213 MB/s
643825664 bytes (644 MB, 614 MiB) copied, 3 s, 214 MB/s
859832320 bytes (860 MB, 820 MiB) copied, 4 s, 214 MB/s
1071644672 bytes (1.1 GB, 1022 MiB) copied, 5 s, 214 MB/s
1278214144 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 213 MB/s 
1491075072 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 213 MB/s
1704984576 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 213 MB/s
1925185536 bytes (1.9 GB, 1.8 GiB) copied, 9 s, 214 MB/s
2149580800 bytes (2.1 GB, 2.0 GiB) copied, 10 s, 215 MB/s
2365587456 bytes (2.4 GB, 2.2 GiB) copied, 11 s, 215 MB/s
2587885568 bytes (2.6 GB, 2.4 GiB) copied, 12 s, 216 MB/s
2797600768 bytes (2.8 GB, 2.6 GiB) copied, 13 s, 215 MB/s
3007315968 bytes (3.0 GB, 2.8 GiB) copied, 14 s, 215 MB/s
3217031168 bytes (3.2 GB, 3.0 GiB) copied, 15 s, 214 MB/s
3416260608 bytes (3.4 GB, 3.2 GiB) copied, 16 s, 213 MB/s
3621781504 bytes (3.6 GB, 3.4 GiB) copied, 17 s, 213 MB/s
3827302400 bytes (3.8 GB, 3.6 GiB) copied, 18 s, 213 MB/s
4034920448 bytes (4.0 GB, 3.8 GiB) copied, 19 s, 212 MB/s
4238344192 bytes (4.2 GB, 3.9 GiB) copied, 20 s, 212 MB/s
4431282176 bytes (4.4 GB, 4.1 GiB) copied, 21 s, 211 MB/s
4630511616 bytes (4.6 GB, 4.3 GiB) copied, 22 s, 210 MB/s
4829741056 bytes (4.8 GB, 4.5 GiB) copied, 23 s, 210 MB/s
5028970496 bytes (5.0 GB, 4.7 GiB) copied, 24 s, 210 MB/s
5230297088 bytes (5.2 GB, 4.9 GiB) copied, 25 s, 209 MB/s
5442109440 bytes (5.4 GB, 5.1 GiB) copied, 26 s, 209 MB/s
5654970368 bytes (5.7 GB, 5.3 GiB) copied, 27 s, 209 MB/s
5868879872 bytes (5.9 GB, 5.5 GiB) copied, 28 s, 210 MB/s
6083837952 bytes (6.1 GB, 5.7 GiB) copied, 29 s, 210 MB/s
6276775936 bytes (6.3 GB, 5.8 GiB) copied, 30 s, 209 MB/s
6467616768 bytes (6.5 GB, 6.0 GiB) copied, 31 s, 209 MB/s
6660554752 bytes (6.7 GB, 6.2 GiB) copied, 32 s, 208 MB/s
6855589888 bytes (6.9 GB, 6.4 GiB) copied, 33 s, 208 MB/s
7046430720 bytes (7.0 GB, 6.6 GiB) copied, 34 s, 207 MB/s
7235174400 bytes (7.2 GB, 6.7 GiB) copied, 35 s, 207 MB/s
7419723776 bytes (7.4 GB, 6.9 GiB) copied, 36 s, 206 MB/s
7608467456 bytes (7.6 GB, 7.1 GiB) copied, 37 s, 206 MB/s
7811891200 bytes (7.8 GB, 7.3 GiB) copied, 38 s, 206 MB/s
8022654976 bytes (8.0 GB, 7.5 GiB) copied, 39 s, 206 MB/s
8235515904 bytes (8.2 GB, 7.7 GiB) copied, 40 s, 206 MB/s
8438939648 bytes (8.4 GB, 7.9 GiB) copied, 41 s, 206 MB/s
8654946304 bytes (8.7 GB, 8.1 GiB) copied, 42 s, 206 MB/s
8870952960 bytes (8.9 GB, 8.3 GiB) copied, 43 s, 206 MB/s
9086959616 bytes (9.1 GB, 8.5 GiB) copied, 44 s, 206 MB/s
9300869120 bytes (9.3 GB, 8.7 GiB) copied, 45 s, 207 MB/s
9500098560 bytes (9.5 GB, 8.8 GiB) copied, 46 s, 206 MB/s
9704570880 bytes (9.7 GB, 9.0 GiB) copied, 47 s, 206 MB/s
9911140352 bytes (9.9 GB, 9.2 GiB) copied, 48 s, 206 MB/s
10118758400 bytes (10 GB, 9.4 GiB) copied, 49 s, 206 MB/s
10323230720 bytes (10 GB, 9.6 GiB) copied, 50 s, 206 MB/s
10522460160 bytes (11 GB, 9.8 GiB) copied, 51 s, 206 MB/s
10726932480 bytes (11 GB, 10 GiB) copied, 52 s, 206 MB/s 
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 52.6231 s, 204 MB/s

/dev/sdf

=== Reading 10G from /dev/sdf ===

199229440 bytes (199 MB, 190 MiB) copied, 1 s, 198 MB/s
417333248 bytes (417 MB, 398 MiB) copied, 2 s, 209 MB/s
637534208 bytes (638 MB, 608 MiB) copied, 3 s, 212 MB/s
855638016 bytes (856 MB, 816 MiB) copied, 4 s, 214 MB/s
1074790400 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 215 MB/s
1289748480 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 215 MB/s
1513095168 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 216 MB/s
1736441856 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 217 MB/s
1958739968 bytes (2.0 GB, 1.8 GiB) copied, 9 s, 218 MB/s
2183135232 bytes (2.2 GB, 2.0 GiB) copied, 10 s, 218 MB/s
2404384768 bytes (2.4 GB, 2.2 GiB) copied, 11 s, 219 MB/s
2629828608 bytes (2.6 GB, 2.4 GiB) copied, 12 s, 219 MB/s
2853175296 bytes (2.9 GB, 2.7 GiB) copied, 13 s, 219 MB/s
3074424832 bytes (3.1 GB, 2.9 GiB) copied, 14 s, 220 MB/s
3296722944 bytes (3.3 GB, 3.1 GiB) copied, 15 s, 220 MB/s
3512729600 bytes (3.5 GB, 3.3 GiB) copied, 16 s, 219 MB/s
3732930560 bytes (3.7 GB, 3.5 GiB) copied, 17 s, 220 MB/s
3961520128 bytes (4.0 GB, 3.7 GiB) copied, 18 s, 220 MB/s
4189061120 bytes (4.2 GB, 3.9 GiB) copied, 19 s, 220 MB/s
4416602112 bytes (4.4 GB, 4.1 GiB) copied, 20 s, 221 MB/s
4636803072 bytes (4.6 GB, 4.3 GiB) copied, 21 s, 221 MB/s
4859101184 bytes (4.9 GB, 4.5 GiB) copied, 22 s, 221 MB/s
5080350720 bytes (5.1 GB, 4.7 GiB) copied, 23 s, 221 MB/s
5300551680 bytes (5.3 GB, 4.9 GiB) copied, 24 s, 221 MB/s
5521801216 bytes (5.5 GB, 5.1 GiB) copied, 25 s, 221 MB/s
5735710720 bytes (5.7 GB, 5.3 GiB) copied, 26 s, 221 MB/s
5955911680 bytes (6.0 GB, 5.5 GiB) copied, 27 s, 221 MB/s
6177161216 bytes (6.2 GB, 5.8 GiB) copied, 28 s, 221 MB/s
6396313600 bytes (6.4 GB, 6.0 GiB) copied, 29 s, 221 MB/s
6611271680 bytes (6.6 GB, 6.2 GiB) copied, 30 s, 220 MB/s
6819938304 bytes (6.8 GB, 6.4 GiB) copied, 31 s, 220 MB/s
7035944960 bytes (7.0 GB, 6.6 GiB) copied, 32 s, 220 MB/s
7253000192 bytes (7.3 GB, 6.8 GiB) copied, 33 s, 220 MB/s
7469006848 bytes (7.5 GB, 7.0 GiB) copied, 34 s, 220 MB/s
7688159232 bytes (7.7 GB, 7.2 GiB) copied, 35 s, 220 MB/s
7904165888 bytes (7.9 GB, 7.4 GiB) copied, 36 s, 220 MB/s
8123318272 bytes (8.1 GB, 7.6 GiB) copied, 37 s, 219 MB/s
8338276352 bytes (8.3 GB, 7.8 GiB) copied, 38 s, 219 MB/s
8557428736 bytes (8.6 GB, 8.0 GiB) copied, 39 s, 219 MB/s
8776581120 bytes (8.8 GB, 8.2 GiB) copied, 40 s, 219 MB/s
8992587776 bytes (9.0 GB, 8.4 GiB) copied, 41 s, 219 MB/s
9210691584 bytes (9.2 GB, 8.6 GiB) copied, 42 s, 219 MB/s
9424601088 bytes (9.4 GB, 8.8 GiB) copied, 43 s, 219 MB/s
9638510592 bytes (9.6 GB, 9.0 GiB) copied, 44 s, 219 MB/s
9854517248 bytes (9.9 GB, 9.2 GiB) copied, 45 s, 219 MB/s
10063183872 bytes (10 GB, 9.4 GiB) copied, 46 s, 219 MB/s
10282336256 bytes (10 GB, 9.6 GiB) copied, 47 s, 219 MB/s
10505682944 bytes (11 GB, 9.8 GiB) copied, 48 s, 219 MB/s
10729029632 bytes (11 GB, 10 GiB) copied, 49 s, 219 MB/s 
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 49.618 s, 216 MB/s

/dev/sdh

=== Reading 10G from /dev/sdh ===

2097152 bytes (2.1 MB, 2.0 MiB) copied, 1 s, 2.0 MB/s
204472320 bytes (204 MB, 195 MiB) copied, 2 s, 102 MB/s
413138944 bytes (413 MB, 394 MiB) copied, 3 s, 138 MB/s
622854144 bytes (623 MB, 594 MiB) copied, 4 s, 156 MB/s
832569344 bytes (833 MB, 794 MiB) copied, 5 s, 166 MB/s
1038090240 bytes (1.0 GB, 990 MiB) copied, 6 s, 173 MB/s
1261436928 bytes (1.3 GB, 1.2 GiB) copied, 7 s, 180 MB/s
1486880768 bytes (1.5 GB, 1.4 GiB) copied, 8 s, 186 MB/s
1711276032 bytes (1.7 GB, 1.6 GiB) copied, 9 s, 190 MB/s
1930428416 bytes (1.9 GB, 1.8 GiB) copied, 10 s, 193 MB/s
2151677952 bytes (2.2 GB, 2.0 GiB) copied, 11 s, 196 MB/s
2363490304 bytes (2.4 GB, 2.2 GiB) copied, 12 s, 197 MB/s
2581594112 bytes (2.6 GB, 2.4 GiB) copied, 13 s, 199 MB/s
2801795072 bytes (2.8 GB, 2.6 GiB) copied, 14 s, 200 MB/s
3032481792 bytes (3.0 GB, 2.8 GiB) copied, 15 s, 202 MB/s
3261071360 bytes (3.3 GB, 3.0 GiB) copied, 16 s, 204 MB/s
3483369472 bytes (3.5 GB, 3.2 GiB) copied, 17 s, 205 MB/s
3703570432 bytes (3.7 GB, 3.4 GiB) copied, 18 s, 206 MB/s
3915382784 bytes (3.9 GB, 3.6 GiB) copied, 19 s, 206 MB/s
4127195136 bytes (4.1 GB, 3.8 GiB) copied, 20 s, 206 MB/s
4339007488 bytes (4.3 GB, 4.0 GiB) copied, 21 s, 207 MB/s
4546625536 bytes (4.5 GB, 4.2 GiB) copied, 22 s, 207 MB/s
4764729344 bytes (4.8 GB, 4.4 GiB) copied, 23 s, 207 MB/s
4994367488 bytes (5.0 GB, 4.7 GiB) copied, 24 s, 208 MB/s
5222957056 bytes (5.2 GB, 4.9 GiB) copied, 25 s, 209 MB/s
5451546624 bytes (5.5 GB, 5.1 GiB) copied, 26 s, 210 MB/s
5664407552 bytes (5.7 GB, 5.3 GiB) copied, 27 s, 210 MB/s
5867831296 bytes (5.9 GB, 5.5 GiB) copied, 28 s, 210 MB/s
6073352192 bytes (6.1 GB, 5.7 GiB) copied, 29 s, 209 MB/s
6276775936 bytes (6.3 GB, 5.8 GiB) copied, 30 s, 209 MB/s
6480199680 bytes (6.5 GB, 6.0 GiB) copied, 31 s, 209 MB/s
6687817728 bytes (6.7 GB, 6.2 GiB) copied, 32 s, 209 MB/s
6916407296 bytes (6.9 GB, 6.4 GiB) copied, 33 s, 209 MB/s
7140802560 bytes (7.1 GB, 6.7 GiB) copied, 34 s, 210 MB/s
7367294976 bytes (7.4 GB, 6.9 GiB) copied, 35 s, 210 MB/s
7586447360 bytes (7.6 GB, 7.1 GiB) copied, 36 s, 211 MB/s
7795113984 bytes (7.8 GB, 7.3 GiB) copied, 37 s, 211 MB/s
8009023488 bytes (8.0 GB, 7.5 GiB) copied, 38 s, 211 MB/s
8227127296 bytes (8.2 GB, 7.7 GiB) copied, 39 s, 211 MB/s
8441036800 bytes (8.4 GB, 7.9 GiB) copied, 40 s, 211 MB/s
8671723520 bytes (8.7 GB, 8.1 GiB) copied, 41 s, 211 MB/s
8890875904 bytes (8.9 GB, 8.3 GiB) copied, 42 s, 212 MB/s
9121562624 bytes (9.1 GB, 8.5 GiB) copied, 43 s, 212 MB/s
9342812160 bytes (9.3 GB, 8.7 GiB) copied, 44 s, 212 MB/s
9560915968 bytes (9.6 GB, 8.9 GiB) copied, 45 s, 212 MB/s
9776922624 bytes (9.8 GB, 9.1 GiB) copied, 46 s, 213 MB/s
9990832128 bytes (10 GB, 9.3 GiB) copied, 47 s, 213 MB/s 
10206838784 bytes (10 GB, 9.5 GiB) copied, 48 s, 213 MB/s
10433331200 bytes (10 GB, 9.7 GiB) copied, 49 s, 213 MB/s
10661920768 bytes (11 GB, 9.9 GiB) copied, 50 s, 213 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 50.9011 s, 211 MB/s

/dev/sdi

=== Reading 10G from /dev/sdi ===

5242880 bytes (5.2 MB, 5.0 MiB) copied, 1 s, 5.2 MB/s
212860928 bytes (213 MB, 203 MiB) copied, 2 s, 106 MB/s
419430400 bytes (419 MB, 400 MiB) copied, 3 s, 140 MB/s
627048448 bytes (627 MB, 598 MiB) copied, 4 s, 157 MB/s
834666496 bytes (835 MB, 796 MiB) copied, 5 s, 167 MB/s
1043333120 bytes (1.0 GB, 995 MiB) copied, 6 s, 174 MB/s
1260388352 bytes (1.3 GB, 1.2 GiB) copied, 7 s, 180 MB/s
1478492160 bytes (1.5 GB, 1.4 GiB) copied, 8 s, 185 MB/s
1695547392 bytes (1.7 GB, 1.6 GiB) copied, 9 s, 188 MB/s
1908408320 bytes (1.9 GB, 1.8 GiB) copied, 10 s, 191 MB/s
2114977792 bytes (2.1 GB, 2.0 GiB) copied, 11 s, 192 MB/s
2329935872 bytes (2.3 GB, 2.2 GiB) copied, 12 s, 194 MB/s
2543845376 bytes (2.5 GB, 2.4 GiB) copied, 13 s, 196 MB/s
2762997760 bytes (2.8 GB, 2.6 GiB) copied, 14 s, 197 MB/s
2984247296 bytes (3.0 GB, 2.8 GiB) copied, 15 s, 199 MB/s
3206545408 bytes (3.2 GB, 3.0 GiB) copied, 16 s, 200 MB/s
3421503488 bytes (3.4 GB, 3.2 GiB) copied, 17 s, 201 MB/s
3642753024 bytes (3.6 GB, 3.4 GiB) copied, 18 s, 202 MB/s
3862953984 bytes (3.9 GB, 3.6 GiB) copied, 19 s, 203 MB/s
4081057792 bytes (4.1 GB, 3.8 GiB) copied, 20 s, 204 MB/s
4301258752 bytes (4.3 GB, 4.0 GiB) copied, 21 s, 205 MB/s
4506779648 bytes (4.5 GB, 4.2 GiB) copied, 22 s, 205 MB/s
4722786304 bytes (4.7 GB, 4.4 GiB) copied, 23 s, 205 MB/s
4938792960 bytes (4.9 GB, 4.6 GiB) copied, 24 s, 206 MB/s
5150605312 bytes (5.2 GB, 4.8 GiB) copied, 25 s, 206 MB/s
5366611968 bytes (5.4 GB, 5.0 GiB) copied, 26 s, 206 MB/s
5584715776 bytes (5.6 GB, 5.2 GiB) copied, 27 s, 207 MB/s
5802819584 bytes (5.8 GB, 5.4 GiB) copied, 28 s, 207 MB/s
6020923392 bytes (6.0 GB, 5.6 GiB) copied, 29 s, 208 MB/s
6241124352 bytes (6.2 GB, 5.8 GiB) copied, 30 s, 208 MB/s
6457131008 bytes (6.5 GB, 6.0 GiB) copied, 31 s, 208 MB/s
6673137664 bytes (6.7 GB, 6.2 GiB) copied, 32 s, 208 MB/s
6891241472 bytes (6.9 GB, 6.4 GiB) copied, 33 s, 209 MB/s
7105150976 bytes (7.1 GB, 6.6 GiB) copied, 34 s, 209 MB/s
7316963328 bytes (7.3 GB, 6.8 GiB) copied, 35 s, 209 MB/s
7528775680 bytes (7.5 GB, 7.0 GiB) copied, 36 s, 209 MB/s
7734296576 bytes (7.7 GB, 7.2 GiB) copied, 37 s, 209 MB/s
7946108928 bytes (7.9 GB, 7.4 GiB) copied, 38 s, 209 MB/s
8168407040 bytes (8.2 GB, 7.6 GiB) copied, 39 s, 209 MB/s
8390705152 bytes (8.4 GB, 7.8 GiB) copied, 40 s, 210 MB/s
8609857536 bytes (8.6 GB, 8.0 GiB) copied, 41 s, 210 MB/s
8816427008 bytes (8.8 GB, 8.2 GiB) copied, 42 s, 210 MB/s
9021947904 bytes (9.0 GB, 8.4 GiB) copied, 43 s, 210 MB/s
9228517376 bytes (9.2 GB, 8.6 GiB) copied, 44 s, 210 MB/s
9435086848 bytes (9.4 GB, 8.8 GiB) copied, 45 s, 210 MB/s
9642704896 bytes (9.6 GB, 9.0 GiB) copied, 46 s, 210 MB/s
9861857280 bytes (9.9 GB, 9.2 GiB) copied, 47 s, 210 MB/s
10078912512 bytes (10 GB, 9.4 GiB) copied, 48 s, 210 MB/s
10289676288 bytes (10 GB, 9.6 GiB) copied, 49 s, 210 MB/s
10504634368 bytes (11 GB, 9.8 GiB) copied, 50 s, 210 MB/s
10720641024 bytes (11 GB, 10 GiB) copied, 51 s, 210 MB/s 
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 51.657 s, 208 MB/s

/dev/sda1 & /dev/sda2

=== Reading 10G from /dev/sda1 ===

212860928 bytes (213 MB, 203 MiB) copied, 1 s, 213 MB/s
427819008 bytes (428 MB, 408 MiB) copied, 2 s, 214 MB/s
641728512 bytes (642 MB, 612 MiB) copied, 3 s, 214 MB/s
857735168 bytes (858 MB, 818 MiB) copied, 4 s, 214 MB/s
1077936128 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 216 MB/s
1293942784 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 216 MB/s
1520435200 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 217 MB/s
1744830464 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 218 MB/s
1954545664 bytes (2.0 GB, 1.8 GiB) copied, 9 s, 217 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 10.0716 s, 213 MB/s

=== Reading 10G from /dev/sda2 ===

205520896 bytes (206 MB, 196 MiB) copied, 1 s, 205 MB/s
415236096 bytes (415 MB, 396 MiB) copied, 2 s, 207 MB/s
624951296 bytes (625 MB, 596 MiB) copied, 3 s, 208 MB/s
849346560 bytes (849 MB, 810 MiB) copied, 4 s, 212 MB/s
1075838976 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 215 MB/s
1289748480 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 215 MB/s
1512046592 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 216 MB/s
1732247552 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 216 MB/s
1950351360 bytes (2.0 GB, 1.8 GiB) copied, 9 s, 217 MB/s
2170552320 bytes (2.2 GB, 2.0 GiB) copied, 10 s, 217 MB/s
2386558976 bytes (2.4 GB, 2.2 GiB) copied, 11 s, 217 MB/s
2613051392 bytes (2.6 GB, 2.4 GiB) copied, 12 s, 218 MB/s
2837446656 bytes (2.8 GB, 2.6 GiB) copied, 13 s, 218 MB/s
3061841920 bytes (3.1 GB, 2.9 GiB) copied, 14 s, 219 MB/s
3274702848 bytes (3.3 GB, 3.0 GiB) copied, 15 s, 218 MB/s
3474980864 bytes (3.5 GB, 3.2 GiB) copied, 16 s, 217 MB/s
3680501760 bytes (3.7 GB, 3.4 GiB) copied, 17 s, 216 MB/s
3883925504 bytes (3.9 GB, 3.6 GiB) copied, 18 s, 216 MB/s
4097835008 bytes (4.1 GB, 3.8 GiB) copied, 19 s, 216 MB/s
4322230272 bytes (4.3 GB, 4.0 GiB) copied, 20 s, 216 MB/s
4538236928 bytes (4.5 GB, 4.2 GiB) copied, 21 s, 216 MB/s
4762632192 bytes (4.8 GB, 4.4 GiB) copied, 22 s, 216 MB/s
4977590272 bytes (5.0 GB, 4.6 GiB) copied, 23 s, 216 MB/s
5190451200 bytes (5.2 GB, 4.8 GiB) copied, 24 s, 216 MB/s
5402263552 bytes (5.4 GB, 5.0 GiB) copied, 25 s, 216 MB/s
5604638720 bytes (5.6 GB, 5.2 GiB) copied, 26 s, 216 MB/s
5816451072 bytes (5.8 GB, 5.4 GiB) copied, 27 s, 215 MB/s
6043992064 bytes (6.0 GB, 5.6 GiB) copied, 28 s, 216 MB/s
6270484480 bytes (6.3 GB, 5.8 GiB) copied, 29 s, 216 MB/s
6496976896 bytes (6.5 GB, 6.1 GiB) copied, 30 s, 217 MB/s
6704594944 bytes (6.7 GB, 6.2 GiB) copied, 31 s, 216 MB/s
6912212992 bytes (6.9 GB, 6.4 GiB) copied, 32 s, 216 MB/s
7115636736 bytes (7.1 GB, 6.6 GiB) copied, 33 s, 216 MB/s
7322206208 bytes (7.3 GB, 6.8 GiB) copied, 34 s, 215 MB/s
7528775680 bytes (7.5 GB, 7.0 GiB) copied, 35 s, 215 MB/s
7748976640 bytes (7.7 GB, 7.2 GiB) copied, 36 s, 215 MB/s
7975469056 bytes (8.0 GB, 7.4 GiB) copied, 37 s, 216 MB/s
8204058624 bytes (8.2 GB, 7.6 GiB) copied, 38 s, 216 MB/s
8426356736 bytes (8.4 GB, 7.8 GiB) copied, 39 s, 216 MB/s
8642363392 bytes (8.6 GB, 8.0 GiB) copied, 40 s, 216 MB/s
8851030016 bytes (8.9 GB, 8.2 GiB) copied, 41 s, 216 MB/s
9065988096 bytes (9.1 GB, 8.4 GiB) copied, 42 s, 216 MB/s
9281994752 bytes (9.3 GB, 8.6 GiB) copied, 43 s, 216 MB/s
9499049984 bytes (9.5 GB, 8.8 GiB) copied, 44 s, 216 MB/s
9723445248 bytes (9.7 GB, 9.1 GiB) copied, 45 s, 216 MB/s
9940500480 bytes (9.9 GB, 9.3 GiB) copied, 46 s, 216 MB/s
10164895744 bytes (10 GB, 9.5 GiB) copied, 47 s, 216 MB/s
10378805248 bytes (10 GB, 9.7 GiB) copied, 48 s, 216 MB/s
10593763328 bytes (11 GB, 9.9 GiB) copied, 49 s, 216 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 50.2427 s, 214 MB/s

/dev/sdb1 & /dev/sdb2

=== Reading 10G from /dev/sdb1 ===

216006656 bytes (216 MB, 206 MiB) copied, 1 s, 215 MB/s
436207616 bytes (436 MB, 416 MiB) copied, 2 s, 218 MB/s
656408576 bytes (656 MB, 626 MiB) copied, 3 s, 218 MB/s
874512384 bytes (875 MB, 834 MiB) copied, 4 s, 218 MB/s
1092616192 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 218 MB/s
1299185664 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 216 MB/s
1516240896 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 217 MB/s
1734344704 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 217 MB/s
1948254208 bytes (1.9 GB, 1.8 GiB) copied, 9 s, 216 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 10.0697 s, 213 MB/s

=== Reading 10G from /dev/sdb2 ===

213909504 bytes (214 MB, 204 MiB) copied, 1 s, 213 MB/s
427819008 bytes (428 MB, 408 MiB) copied, 2 s, 214 MB/s
645922816 bytes (646 MB, 616 MiB) copied, 3 s, 215 MB/s
866123776 bytes (866 MB, 826 MiB) copied, 4 s, 216 MB/s
1088421888 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 217 MB/s
1302331392 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 217 MB/s
1522532352 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 217 MB/s
1742733312 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 218 MB/s
1962934272 bytes (2.0 GB, 1.8 GiB) copied, 9 s, 218 MB/s
2183135232 bytes (2.2 GB, 2.0 GiB) copied, 10 s, 218 MB/s
2394947584 bytes (2.4 GB, 2.2 GiB) copied, 11 s, 218 MB/s
2618294272 bytes (2.6 GB, 2.4 GiB) copied, 12 s, 218 MB/s
2840592384 bytes (2.8 GB, 2.6 GiB) copied, 13 s, 218 MB/s
3061841920 bytes (3.1 GB, 2.9 GiB) copied, 14 s, 219 MB/s
3258974208 bytes (3.3 GB, 3.0 GiB) copied, 15 s, 217 MB/s
3443523584 bytes (3.4 GB, 3.2 GiB) copied, 16 s, 215 MB/s
3632267264 bytes (3.6 GB, 3.4 GiB) copied, 17 s, 214 MB/s
3823108096 bytes (3.8 GB, 3.6 GiB) copied, 18 s, 212 MB/s
4017094656 bytes (4.0 GB, 3.7 GiB) copied, 19 s, 211 MB/s
4237295616 bytes (4.2 GB, 3.9 GiB) copied, 20 s, 212 MB/s
4451205120 bytes (4.5 GB, 4.1 GiB) copied, 21 s, 212 MB/s
4673503232 bytes (4.7 GB, 4.4 GiB) copied, 22 s, 212 MB/s
4892655616 bytes (4.9 GB, 4.6 GiB) copied, 23 s, 213 MB/s
5111808000 bytes (5.1 GB, 4.8 GiB) copied, 24 s, 213 MB/s
5328863232 bytes (5.3 GB, 5.0 GiB) copied, 25 s, 213 MB/s
5544869888 bytes (5.5 GB, 5.2 GiB) copied, 26 s, 213 MB/s
5765070848 bytes (5.8 GB, 5.4 GiB) copied, 27 s, 214 MB/s
5993660416 bytes (6.0 GB, 5.6 GiB) copied, 28 s, 214 MB/s
6218055680 bytes (6.2 GB, 5.8 GiB) copied, 29 s, 214 MB/s
6446645248 bytes (6.4 GB, 6.0 GiB) copied, 30 s, 215 MB/s
6659506176 bytes (6.7 GB, 6.2 GiB) copied, 31 s, 215 MB/s
6876561408 bytes (6.9 GB, 6.4 GiB) copied, 32 s, 215 MB/s
7094665216 bytes (7.1 GB, 6.6 GiB) copied, 33 s, 215 MB/s
7310671872 bytes (7.3 GB, 6.8 GiB) copied, 34 s, 215 MB/s
7529824256 bytes (7.5 GB, 7.0 GiB) copied, 35 s, 215 MB/s
7744782336 bytes (7.7 GB, 7.2 GiB) copied, 36 s, 215 MB/s
7966031872 bytes (8.0 GB, 7.4 GiB) copied, 37 s, 215 MB/s
8187281408 bytes (8.2 GB, 7.6 GiB) copied, 38 s, 215 MB/s
8406433792 bytes (8.4 GB, 7.8 GiB) copied, 39 s, 215 MB/s
8623489024 bytes (8.6 GB, 8.0 GiB) copied, 40 s, 216 MB/s
8836349952 bytes (8.8 GB, 8.2 GiB) copied, 41 s, 216 MB/s
9053405184 bytes (9.1 GB, 8.4 GiB) copied, 42 s, 216 MB/s
9262071808 bytes (9.3 GB, 8.6 GiB) copied, 43 s, 215 MB/s
9470738432 bytes (9.5 GB, 8.8 GiB) copied, 44 s, 215 MB/s
9677307904 bytes (9.7 GB, 9.0 GiB) copied, 45 s, 215 MB/s
9877585920 bytes (9.9 GB, 9.2 GiB) copied, 46 s, 215 MB/s
10093592576 bytes (10 GB, 9.4 GiB) copied, 47 s, 215 MB/s
10311696384 bytes (10 GB, 9.6 GiB) copied, 48 s, 215 MB/s
10531897344 bytes (11 GB, 9.8 GiB) copied, 49 s, 215 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 50.5037 s, 213 MB/s

/dev/sdd1 & /dev/sdd2

=== Reading 10G from /dev/sdd1 ===

213909504 bytes (214 MB, 204 MiB) copied, 1 s, 213 MB/s
429916160 bytes (430 MB, 410 MiB) copied, 2 s, 215 MB/s
645922816 bytes (646 MB, 616 MiB) copied, 3 s, 215 MB/s
861929472 bytes (862 MB, 822 MiB) copied, 4 s, 215 MB/s
1081081856 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 216 MB/s
1299185664 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 216 MB/s
1521483776 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 217 MB/s
1744830464 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 218 MB/s
1960837120 bytes (2.0 GB, 1.8 GiB) copied, 9 s, 218 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 10.0247 s, 214 MB/s

=== Reading 10G from /dev/sdd2 ===

218103808 bytes (218 MB, 208 MiB) copied, 1 s, 217 MB/s
434110464 bytes (434 MB, 414 MiB) copied, 2 s, 217 MB/s
652214272 bytes (652 MB, 622 MiB) copied, 3 s, 217 MB/s
876609536 bytes (877 MB, 836 MiB) copied, 4 s, 219 MB/s
1102053376 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 220 MB/s
1319108608 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 220 MB/s
1535115264 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 219 MB/s
1738539008 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 217 MB/s
1941962752 bytes (1.9 GB, 1.8 GiB) copied, 9 s, 216 MB/s
2145386496 bytes (2.1 GB, 2.0 GiB) copied, 10 s, 214 MB/s
2338324480 bytes (2.3 GB, 2.2 GiB) copied, 11 s, 212 MB/s
2564816896 bytes (2.6 GB, 2.4 GiB) copied, 12 s, 214 MB/s
2794455040 bytes (2.8 GB, 2.6 GiB) copied, 13 s, 215 MB/s
3024093184 bytes (3.0 GB, 2.8 GiB) copied, 14 s, 216 MB/s
3252682752 bytes (3.3 GB, 3.0 GiB) copied, 15 s, 217 MB/s
3456106496 bytes (3.5 GB, 3.2 GiB) copied, 16 s, 216 MB/s
3671064576 bytes (3.7 GB, 3.4 GiB) copied, 17 s, 216 MB/s
3886022656 bytes (3.9 GB, 3.6 GiB) copied, 18 s, 216 MB/s
4086300672 bytes (4.1 GB, 3.8 GiB) copied, 19 s, 215 MB/s
4308598784 bytes (4.3 GB, 4.0 GiB) copied, 20 s, 215 MB/s
4531945472 bytes (4.5 GB, 4.2 GiB) copied, 21 s, 216 MB/s
4758437888 bytes (4.8 GB, 4.4 GiB) copied, 22 s, 216 MB/s
4987027456 bytes (5.0 GB, 4.6 GiB) copied, 23 s, 217 MB/s
5193596928 bytes (5.2 GB, 4.8 GiB) copied, 24 s, 216 MB/s
5398069248 bytes (5.4 GB, 5.0 GiB) copied, 25 s, 216 MB/s
5601492992 bytes (5.6 GB, 5.2 GiB) copied, 26 s, 215 MB/s
5807013888 bytes (5.8 GB, 5.4 GiB) copied, 27 s, 215 MB/s
6024069120 bytes (6.0 GB, 5.6 GiB) copied, 28 s, 215 MB/s
6248464384 bytes (6.2 GB, 5.8 GiB) copied, 29 s, 215 MB/s
6472859648 bytes (6.5 GB, 6.0 GiB) copied, 30 s, 216 MB/s
6687817728 bytes (6.7 GB, 6.2 GiB) copied, 31 s, 216 MB/s
6897532928 bytes (6.9 GB, 6.4 GiB) copied, 32 s, 216 MB/s
7103053824 bytes (7.1 GB, 6.6 GiB) copied, 33 s, 215 MB/s
7308574720 bytes (7.3 GB, 6.8 GiB) copied, 34 s, 215 MB/s
7511998464 bytes (7.5 GB, 7.0 GiB) copied, 35 s, 215 MB/s
7717519360 bytes (7.7 GB, 7.2 GiB) copied, 36 s, 214 MB/s
7942963200 bytes (7.9 GB, 7.4 GiB) copied, 37 s, 215 MB/s
8168407040 bytes (8.2 GB, 7.6 GiB) copied, 38 s, 215 MB/s
8394899456 bytes (8.4 GB, 7.8 GiB) copied, 39 s, 215 MB/s
8618246144 bytes (8.6 GB, 8.0 GiB) copied, 40 s, 215 MB/s
8827961344 bytes (8.8 GB, 8.2 GiB) copied, 41 s, 215 MB/s
9046065152 bytes (9.0 GB, 8.4 GiB) copied, 42 s, 215 MB/s
9265217536 bytes (9.3 GB, 8.6 GiB) copied, 43 s, 215 MB/s
9483321344 bytes (9.5 GB, 8.8 GiB) copied, 44 s, 216 MB/s
9709813760 bytes (9.7 GB, 9.0 GiB) copied, 45 s, 216 MB/s
9931063296 bytes (9.9 GB, 9.2 GiB) copied, 46 s, 216 MB/s
10156507136 bytes (10 GB, 9.5 GiB) copied, 47 s, 216 MB/s
10376708096 bytes (10 GB, 9.7 GiB) copied, 48 s, 216 MB/s
10592714752 bytes (11 GB, 9.9 GiB) copied, 49 s, 216 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 50.2502 s, 214 MB/s

/dev/sde1

=== Reading 10G from /dev/sde1 ===

217055232 bytes (217 MB, 207 MiB) copied, 1 s, 217 MB/s
432013312 bytes (432 MB, 412 MiB) copied, 2 s, 215 MB/s
648019968 bytes (648 MB, 618 MiB) copied, 3 s, 216 MB/s
864026624 bytes (864 MB, 824 MiB) copied, 4 s, 216 MB/s
1075838976 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 215 MB/s
1279262720 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 213 MB/s
1492123648 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 213 MB/s
1704984576 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 213 MB/s
1927282688 bytes (1.9 GB, 1.8 GiB) copied, 9 s, 214 MB/s
2149580800 bytes (2.1 GB, 2.0 GiB) copied, 10 s, 215 MB/s
2365587456 bytes (2.4 GB, 2.2 GiB) copied, 11 s, 215 MB/s
2585788416 bytes (2.6 GB, 2.4 GiB) copied, 12 s, 215 MB/s
2794455040 bytes (2.8 GB, 2.6 GiB) copied, 13 s, 215 MB/s
3005218816 bytes (3.0 GB, 2.8 GiB) copied, 14 s, 215 MB/s
3214934016 bytes (3.2 GB, 3.0 GiB) copied, 15 s, 214 MB/s
3420454912 bytes (3.4 GB, 3.2 GiB) copied, 16 s, 214 MB/s
3623878656 bytes (3.6 GB, 3.4 GiB) copied, 17 s, 213 MB/s
3830448128 bytes (3.8 GB, 3.6 GiB) copied, 18 s, 213 MB/s
4037017600 bytes (4.0 GB, 3.8 GiB) copied, 19 s, 212 MB/s
4240441344 bytes (4.2 GB, 3.9 GiB) copied, 20 s, 212 MB/s
4437573632 bytes (4.4 GB, 4.1 GiB) copied, 21 s, 211 MB/s
4636803072 bytes (4.6 GB, 4.3 GiB) copied, 22 s, 211 MB/s
4836032512 bytes (4.8 GB, 4.5 GiB) copied, 23 s, 210 MB/s
5037359104 bytes (5.0 GB, 4.7 GiB) copied, 24 s, 210 MB/s
5236588544 bytes (5.2 GB, 4.9 GiB) copied, 25 s, 209 MB/s
5445255168 bytes (5.4 GB, 5.1 GiB) copied, 26 s, 209 MB/s
5658116096 bytes (5.7 GB, 5.3 GiB) copied, 27 s, 210 MB/s
5872025600 bytes (5.9 GB, 5.5 GiB) copied, 28 s, 210 MB/s
6085935104 bytes (6.1 GB, 5.7 GiB) copied, 29 s, 210 MB/s
6278873088 bytes (6.3 GB, 5.8 GiB) copied, 30 s, 209 MB/s
6465519616 bytes (6.5 GB, 6.0 GiB) copied, 31 s, 209 MB/s
6660554752 bytes (6.7 GB, 6.2 GiB) copied, 32 s, 208 MB/s
6855589888 bytes (6.9 GB, 6.4 GiB) copied, 33 s, 208 MB/s
7045382144 bytes (7.0 GB, 6.6 GiB) copied, 34 s, 207 MB/s
7235174400 bytes (7.2 GB, 6.7 GiB) copied, 35 s, 207 MB/s
7417626624 bytes (7.4 GB, 6.9 GiB) copied, 36 s, 206 MB/s
7606370304 bytes (7.6 GB, 7.1 GiB) copied, 37 s, 206 MB/s
7809794048 bytes (7.8 GB, 7.3 GiB) copied, 38 s, 206 MB/s
8021606400 bytes (8.0 GB, 7.5 GiB) copied, 39 s, 206 MB/s
8233418752 bytes (8.2 GB, 7.7 GiB) copied, 40 s, 206 MB/s
8442085376 bytes (8.4 GB, 7.9 GiB) copied, 41 s, 206 MB/s
8657043456 bytes (8.7 GB, 8.1 GiB) copied, 42 s, 206 MB/s
8870952960 bytes (8.9 GB, 8.3 GiB) copied, 43 s, 206 MB/s
9085911040 bytes (9.1 GB, 8.5 GiB) copied, 44 s, 206 MB/s
9300869120 bytes (9.3 GB, 8.7 GiB) copied, 45 s, 207 MB/s
9502195712 bytes (9.5 GB, 8.8 GiB) copied, 46 s, 207 MB/s
9709813760 bytes (9.7 GB, 9.0 GiB) copied, 47 s, 207 MB/s
9915334656 bytes (9.9 GB, 9.2 GiB) copied, 48 s, 207 MB/s
10120855552 bytes (10 GB, 9.4 GiB) copied, 49 s, 207 MB/s
10328473600 bytes (10 GB, 9.6 GiB) copied, 50 s, 207 MB/s
10526654464 bytes (11 GB, 9.8 GiB) copied, 51 s, 206 MB/s
10733223936 bytes (11 GB, 10 GiB) copied, 52 s, 206 MB/s 
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 52.5982 s, 204 MB/s

/dev/sdf1 & /dev/sdf2

=== Reading 10G from /dev/sdf1 ===

201326592 bytes (201 MB, 192 MiB) copied, 1 s, 200 MB/s
419430400 bytes (419 MB, 400 MiB) copied, 2 s, 210 MB/s
639631360 bytes (640 MB, 610 MiB) copied, 3 s, 213 MB/s
857735168 bytes (858 MB, 818 MiB) copied, 4 s, 214 MB/s
1076887552 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 215 MB/s
1293942784 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 216 MB/s
1518338048 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 217 MB/s
1742733312 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 218 MB/s
1965031424 bytes (2.0 GB, 1.8 GiB) copied, 9 s, 218 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 9.97138 s, 215 MB/s

=== Reading 10G from /dev/sdf2 ===

224395264 bytes (224 MB, 214 MiB) copied, 1 s, 224 MB/s
449839104 bytes (450 MB, 429 MiB) copied, 2 s, 225 MB/s
674234368 bytes (674 MB, 643 MiB) copied, 3 s, 225 MB/s
895483904 bytes (895 MB, 854 MiB) copied, 4 s, 223 MB/s
1115684864 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 223 MB/s
1331691520 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 222 MB/s
1552941056 bytes (1.6 GB, 1.4 GiB) copied, 7 s, 222 MB/s
1778384896 bytes (1.8 GB, 1.7 GiB) copied, 8 s, 222 MB/s
2006974464 bytes (2.0 GB, 1.9 GiB) copied, 9 s, 223 MB/s
2233466880 bytes (2.2 GB, 2.1 GiB) copied, 10 s, 223 MB/s
2451570688 bytes (2.5 GB, 2.3 GiB) copied, 11 s, 223 MB/s
2672820224 bytes (2.7 GB, 2.5 GiB) copied, 12 s, 223 MB/s
2894069760 bytes (2.9 GB, 2.7 GiB) copied, 13 s, 223 MB/s
3116367872 bytes (3.1 GB, 2.9 GiB) copied, 14 s, 223 MB/s
3338665984 bytes (3.3 GB, 3.1 GiB) copied, 15 s, 222 MB/s
3554672640 bytes (3.6 GB, 3.3 GiB) copied, 16 s, 222 MB/s
3774873600 bytes (3.8 GB, 3.5 GiB) copied, 17 s, 222 MB/s
3996123136 bytes (4.0 GB, 3.7 GiB) copied, 18 s, 222 MB/s
4216324096 bytes (4.2 GB, 3.9 GiB) copied, 19 s, 222 MB/s
4433379328 bytes (4.4 GB, 4.1 GiB) copied, 20 s, 222 MB/s
4644143104 bytes (4.6 GB, 4.3 GiB) copied, 21 s, 221 MB/s
4860149760 bytes (4.9 GB, 4.5 GiB) copied, 22 s, 221 MB/s
5077204992 bytes (5.1 GB, 4.7 GiB) copied, 23 s, 221 MB/s
5293211648 bytes (5.3 GB, 4.9 GiB) copied, 24 s, 221 MB/s
5511315456 bytes (5.5 GB, 5.1 GiB) copied, 25 s, 220 MB/s
5723127808 bytes (5.7 GB, 5.3 GiB) copied, 26 s, 220 MB/s
5941231616 bytes (5.9 GB, 5.5 GiB) copied, 27 s, 220 MB/s
6157238272 bytes (6.2 GB, 5.7 GiB) copied, 28 s, 220 MB/s
6376390656 bytes (6.4 GB, 5.9 GiB) copied, 29 s, 220 MB/s
6596591616 bytes (6.6 GB, 6.1 GiB) copied, 30 s, 220 MB/s
6809452544 bytes (6.8 GB, 6.3 GiB) copied, 31 s, 220 MB/s
7028604928 bytes (7.0 GB, 6.5 GiB) copied, 32 s, 220 MB/s
7242514432 bytes (7.2 GB, 6.7 GiB) copied, 33 s, 219 MB/s
7457472512 bytes (7.5 GB, 6.9 GiB) copied, 34 s, 219 MB/s
7671382016 bytes (7.7 GB, 7.1 GiB) copied, 35 s, 219 MB/s
7879000064 bytes (7.9 GB, 7.3 GiB) copied, 36 s, 219 MB/s
8097103872 bytes (8.1 GB, 7.5 GiB) copied, 37 s, 219 MB/s
8319401984 bytes (8.3 GB, 7.7 GiB) copied, 38 s, 219 MB/s
8542748672 bytes (8.5 GB, 8.0 GiB) copied, 39 s, 219 MB/s
8763998208 bytes (8.8 GB, 8.2 GiB) copied, 40 s, 219 MB/s
8981053440 bytes (9.0 GB, 8.4 GiB) copied, 41 s, 219 MB/s
9200205824 bytes (9.2 GB, 8.6 GiB) copied, 42 s, 219 MB/s
9420406784 bytes (9.4 GB, 8.8 GiB) copied, 43 s, 219 MB/s
9640607744 bytes (9.6 GB, 9.0 GiB) copied, 44 s, 219 MB/s
9848225792 bytes (9.8 GB, 9.2 GiB) copied, 45 s, 219 MB/s
10060038144 bytes (10 GB, 9.4 GiB) copied, 46 s, 219 MB/s
10279190528 bytes (10 GB, 9.6 GiB) copied, 47 s, 219 MB/s
10498342912 bytes (10 GB, 9.8 GiB) copied, 48 s, 219 MB/s
10715398144 bytes (11 GB, 10 GiB) copied, 49 s, 219 MB/s 
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 49.6737 s, 216 MB/s

/dev/sdh1 & /dev/sdh2

=== Reading 10G from /dev/sdh1 ===

203423744 bytes (203 MB, 194 MiB) copied, 1 s, 203 MB/s
413138944 bytes (413 MB, 394 MiB) copied, 2 s, 206 MB/s
622854144 bytes (623 MB, 594 MiB) copied, 3 s, 207 MB/s
831520768 bytes (832 MB, 793 MiB) copied, 4 s, 208 MB/s
1042284544 bytes (1.0 GB, 994 MiB) copied, 5 s, 208 MB/s
1266679808 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 211 MB/s
1491075072 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 213 MB/s
1717567488 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 215 MB/s
1935671296 bytes (1.9 GB, 1.8 GiB) copied, 9 s, 215 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 10.1084 s, 212 MB/s

=== Reading 10G from /dev/sdh2 ===

222298112 bytes (222 MB, 212 MiB) copied, 1 s, 221 MB/s
443547648 bytes (444 MB, 423 MiB) copied, 2 s, 222 MB/s
661651456 bytes (662 MB, 631 MiB) copied, 3 s, 220 MB/s
893386752 bytes (893 MB, 852 MiB) copied, 4 s, 223 MB/s
1123024896 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 225 MB/s
1348468736 bytes (1.3 GB, 1.3 GiB) copied, 6 s, 225 MB/s
1566572544 bytes (1.6 GB, 1.5 GiB) copied, 7 s, 224 MB/s
1780482048 bytes (1.8 GB, 1.7 GiB) copied, 8 s, 222 MB/s
1990197248 bytes (2.0 GB, 1.9 GiB) copied, 9 s, 221 MB/s
2202009600 bytes (2.2 GB, 2.1 GiB) copied, 10 s, 220 MB/s
2405433344 bytes (2.4 GB, 2.2 GiB) copied, 11 s, 219 MB/s
2625634304 bytes (2.6 GB, 2.4 GiB) copied, 12 s, 219 MB/s
2854223872 bytes (2.9 GB, 2.7 GiB) copied, 13 s, 220 MB/s
3082813440 bytes (3.1 GB, 2.9 GiB) copied, 14 s, 220 MB/s
3313500160 bytes (3.3 GB, 3.1 GiB) copied, 15 s, 221 MB/s
3519021056 bytes (3.5 GB, 3.3 GiB) copied, 16 s, 220 MB/s
3724541952 bytes (3.7 GB, 3.5 GiB) copied, 17 s, 219 MB/s
3927965696 bytes (3.9 GB, 3.7 GiB) copied, 18 s, 218 MB/s
4133486592 bytes (4.1 GB, 3.8 GiB) copied, 19 s, 217 MB/s
4336910336 bytes (4.3 GB, 4.0 GiB) copied, 20 s, 217 MB/s
4542431232 bytes (4.5 GB, 4.2 GiB) copied, 21 s, 216 MB/s
4768923648 bytes (4.8 GB, 4.4 GiB) copied, 22 s, 217 MB/s
4996464640 bytes (5.0 GB, 4.7 GiB) copied, 23 s, 217 MB/s
5221908480 bytes (5.2 GB, 4.9 GiB) copied, 24 s, 218 MB/s
5441060864 bytes (5.4 GB, 5.1 GiB) copied, 25 s, 218 MB/s
5653921792 bytes (5.7 GB, 5.3 GiB) copied, 26 s, 217 MB/s
5870977024 bytes (5.9 GB, 5.5 GiB) copied, 27 s, 217 MB/s
6085935104 bytes (6.1 GB, 5.7 GiB) copied, 28 s, 217 MB/s
6301941760 bytes (6.3 GB, 5.9 GiB) copied, 29 s, 217 MB/s
6530531328 bytes (6.5 GB, 6.1 GiB) copied, 30 s, 218 MB/s
6754926592 bytes (6.8 GB, 6.3 GiB) copied, 31 s, 218 MB/s
6983516160 bytes (7.0 GB, 6.5 GiB) copied, 32 s, 218 MB/s
7206862848 bytes (7.2 GB, 6.7 GiB) copied, 33 s, 218 MB/s
7424966656 bytes (7.4 GB, 6.9 GiB) copied, 34 s, 218 MB/s
7640973312 bytes (7.6 GB, 7.1 GiB) copied, 35 s, 218 MB/s
7849639936 bytes (7.8 GB, 7.3 GiB) copied, 36 s, 218 MB/s
8067743744 bytes (8.1 GB, 7.5 GiB) copied, 37 s, 218 MB/s
8294236160 bytes (8.3 GB, 7.7 GiB) copied, 38 s, 218 MB/s
8521777152 bytes (8.5 GB, 7.9 GiB) copied, 39 s, 218 MB/s
8751415296 bytes (8.8 GB, 8.2 GiB) copied, 40 s, 219 MB/s
8967421952 bytes (9.0 GB, 8.4 GiB) copied, 41 s, 219 MB/s
9175040000 bytes (9.2 GB, 8.5 GiB) copied, 42 s, 218 MB/s
9378463744 bytes (9.4 GB, 8.7 GiB) copied, 43 s, 218 MB/s
9586081792 bytes (9.6 GB, 8.9 GiB) copied, 44 s, 218 MB/s
9791602688 bytes (9.8 GB, 9.1 GiB) copied, 45 s, 218 MB/s
9999220736 bytes (10 GB, 9.3 GiB) copied, 46 s, 217 MB/s 
10229907456 bytes (10 GB, 9.5 GiB) copied, 47 s, 218 MB/s
10458497024 bytes (10 GB, 9.7 GiB) copied, 48 s, 218 MB/s
10682892288 bytes (11 GB, 9.9 GiB) copied, 49 s, 218 MB/s
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 49.8296 s, 215 MB/s

/dev/sdi1 & /dev/sdi2

=== Reading 10G from /dev/sdi1 ===

204472320 bytes (204 MB, 195 MiB) copied, 1 s, 204 MB/s
411041792 bytes (411 MB, 392 MiB) copied, 2 s, 205 MB/s
619708416 bytes (620 MB, 591 MiB) copied, 3 s, 207 MB/s
826277888 bytes (826 MB, 788 MiB) copied, 4 s, 206 MB/s
1040187392 bytes (1.0 GB, 992 MiB) copied, 5 s, 208 MB/s
1249902592 bytes (1.2 GB, 1.2 GiB) copied, 6 s, 208 MB/s
1468006400 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 210 MB/s
1686110208 bytes (1.7 GB, 1.6 GiB) copied, 8 s, 211 MB/s
1897922560 bytes (1.9 GB, 1.8 GiB) copied, 9 s, 211 MB/s
2111832064 bytes (2.1 GB, 2.0 GiB) copied, 10 s, 211 MB/s
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 10.3572 s, 207 MB/s

=== Reading 10G from /dev/sdi2 ===

220200960 bytes (220 MB, 210 MiB) copied, 1 s, 220 MB/s
435159040 bytes (435 MB, 415 MiB) copied, 2 s, 217 MB/s
656408576 bytes (656 MB, 626 MiB) copied, 3 s, 218 MB/s
877658112 bytes (878 MB, 837 MiB) copied, 4 s, 219 MB/s
1092616192 bytes (1.1 GB, 1.0 GiB) copied, 5 s, 218 MB/s
1314914304 bytes (1.3 GB, 1.2 GiB) copied, 6 s, 219 MB/s
1534066688 bytes (1.5 GB, 1.4 GiB) copied, 7 s, 219 MB/s
1754267648 bytes (1.8 GB, 1.6 GiB) copied, 8 s, 219 MB/s
1974468608 bytes (2.0 GB, 1.8 GiB) copied, 9 s, 219 MB/s
2190475264 bytes (2.2 GB, 2.0 GiB) copied, 10 s, 219 MB/s
2404384768 bytes (2.4 GB, 2.2 GiB) copied, 11 s, 219 MB/s
2619342848 bytes (2.6 GB, 2.4 GiB) copied, 12 s, 218 MB/s
2833252352 bytes (2.8 GB, 2.6 GiB) copied, 13 s, 218 MB/s
3049259008 bytes (3.0 GB, 2.8 GiB) copied, 14 s, 218 MB/s
3261071360 bytes (3.3 GB, 3.0 GiB) copied, 15 s, 217 MB/s
3480223744 bytes (3.5 GB, 3.2 GiB) copied, 16 s, 218 MB/s
3697278976 bytes (3.7 GB, 3.4 GiB) copied, 17 s, 217 MB/s
3917479936 bytes (3.9 GB, 3.6 GiB) copied, 18 s, 218 MB/s
4135583744 bytes (4.1 GB, 3.9 GiB) copied, 19 s, 218 MB/s
4347396096 bytes (4.3 GB, 4.0 GiB) copied, 20 s, 217 MB/s
4567597056 bytes (4.6 GB, 4.3 GiB) copied, 21 s, 217 MB/s
4785700864 bytes (4.8 GB, 4.5 GiB) copied, 22 s, 217 MB/s
4996464640 bytes (5.0 GB, 4.7 GiB) copied, 23 s, 217 MB/s
5209325568 bytes (5.2 GB, 4.9 GiB) copied, 24 s, 217 MB/s
5416943616 bytes (5.4 GB, 5.0 GiB) copied, 25 s, 217 MB/s
5625610240 bytes (5.6 GB, 5.2 GiB) copied, 26 s, 216 MB/s
5840568320 bytes (5.8 GB, 5.4 GiB) copied, 27 s, 216 MB/s
6060769280 bytes (6.1 GB, 5.6 GiB) copied, 28 s, 216 MB/s
6283067392 bytes (6.3 GB, 5.9 GiB) copied, 29 s, 217 MB/s
6485442560 bytes (6.5 GB, 6.0 GiB) copied, 30 s, 216 MB/s
6698303488 bytes (6.7 GB, 6.2 GiB) copied, 31 s, 216 MB/s
6904872960 bytes (6.9 GB, 6.4 GiB) copied, 32 s, 216 MB/s
7109345280 bytes (7.1 GB, 6.6 GiB) copied, 33 s, 215 MB/s
7316963328 bytes (7.3 GB, 6.8 GiB) copied, 34 s, 215 MB/s
7526678528 bytes (7.5 GB, 7.0 GiB) copied, 35 s, 215 MB/s
7740588032 bytes (7.7 GB, 7.2 GiB) copied, 36 s, 215 MB/s
7957643264 bytes (8.0 GB, 7.4 GiB) copied, 37 s, 215 MB/s
8176795648 bytes (8.2 GB, 7.6 GiB) copied, 38 s, 215 MB/s
8394899456 bytes (8.4 GB, 7.8 GiB) copied, 39 s, 215 MB/s
8606711808 bytes (8.6 GB, 8.0 GiB) copied, 40 s, 215 MB/s
8816427008 bytes (8.8 GB, 8.2 GiB) copied, 41 s, 215 MB/s
9030336512 bytes (9.0 GB, 8.4 GiB) copied, 42 s, 215 MB/s
9244246016 bytes (9.2 GB, 8.6 GiB) copied, 43 s, 215 MB/s
9463398400 bytes (9.5 GB, 8.8 GiB) copied, 44 s, 215 MB/s
9681502208 bytes (9.7 GB, 9.0 GiB) copied, 45 s, 215 MB/s
9892265984 bytes (9.9 GB, 9.2 GiB) copied, 46 s, 215 MB/s
10108272640 bytes (10 GB, 9.4 GiB) copied, 47 s, 215 MB/s
10312744960 bytes (10 GB, 9.6 GiB) copied, 48 s, 215 MB/s
10517217280 bytes (11 GB, 9.8 GiB) copied, 49 s, 215 MB/s
10726932480 bytes (11 GB, 10 GiB) copied, 50 s, 214 MB/s 
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 50.6145 s, 212 MB/s
30

These results look pretty okay. All of the averages are within 10%ish so thats solid.
Insert AI Generated Table

Table: Stable Read Performance (Ignoring First 3 Seconds)

Disk Avg Stable Speed (MB/s) Max Speed (MB/s) Time (s) Data Read After 3s
/dev/sda 209 211 48.28 10.00 GB
/dev/sdb 216 221 47.33 10.00 GB
/dev/sdd 215 219 47.35 10.00 GB
/dev/sde 214 216 49.62 10.00 GB
/dev/sdf 219 225 46.62 10.00 GB
/dev/sdh 212 218 47.90 10.00 GB
/dev/sdi 210 213 48.66 10.00 GB
Average 214.3 217.6

Digging a little deeper tho
Insert AI generated table

Table 1: High-Variability Disks (Slow Start + Speed Ramp-up)

Disk Avg Speed (MB/s) Std Dev (MB/s) Min Speed (MB/s) Max Speed (MB/s) Time to Stabilize (s)
/dev/sdi 208 48.2 5.2 (s1) 210 (s51) 3
/dev/sda 209 45.1 2.1 (s1) 211 (s2) 2
/dev/sdh 211 44.8 2.0 (s1) 211 (s2) 2

This is certainly relevant. Some of the disks were likely parked and so it took longer to start, but it could also indicate some weirdness going on in the inner tracks of the drive. Multiple runs could confirm.

NOT NOW, but when you plan next steps.
I would test these disks further with like badblocks or something like dban or even just dd a bunch of /dev/urandom junk to it for a few days, because thats basically how dban works anyway just less good, but available natively on *NIX.

AI Sorted by oldest to newest and added partuuid to make my eyes not cross
:grimacing:

Here’s the TXG-sorted list from oldest to newest, now augmented with the corresponding PARTUUID for each ZFS partition (like sda2, sdb2, etc.):

Device TXG PARTUUID
/dev/sdd2 27583278 0ded02af-2840-11eb-9ccb-002590faf4a4
/dev/sdf2 27583278 0de232dd-2840-11eb-9ccb-002590faf4a4
/dev/sdi2 27695185 0e7cb1d5-2840-11eb-9ccb-002590faf4a4
/dev/sda2 27697225 496602b9-2444-11ee-ad54-002590faf4a4
/dev/sdb2 27697225 0e822397-2840-11eb-9ccb-002590faf4a4
/dev/sde1 27697225 b381495c-244a-4bba-a33c-a83e2bcc4fba
/dev/sdh2 27697225 0debcf35-2840-11eb-9ccb-002590faf4a4

so TXG 27583278 would indeed unfortunately be the one to look at @Protopia sir

I concur, but lets not use F, lets use the TXG with -T

sudo zpool import -R /mnt -T 27583278 -o readonly=on hd01
31

But with RAIDZ2, you could lose two drives (or in this case 3 because one is resilvering) and still import as a fully degraded zero-redundancy pool. BUT we have 4 drives that have problems - 3 with bad TXGs and 1 which is only partially resilvered.

The problem with trying to import at 27697225 is that it is highly likely that the uber-blocks have cycled round and so highly unlikely that you can import it read-write and perhaps not at all.

Or you could try to import at 27695185 which is a much smaller number of TXGs behind - though probably still with wrapped uber-blocks (and you might need to remove sdd/sde/sdf).

But in theory you should be able to import at 27697225 (though you might have to remove sdd/sde/sdf/sdi drives to achieve it) but then non-redundant and so you would be at risk of another drive failure. Update: No - I was wrong - you have too few drives with valid data to achieve this.

There is no great option here - but if you import readonly then you can always try a different option later.

NOTE: All of the above is theoretical guesswork on my part - we definitely need greater expertise to have a better idea of what might be achievable, whether you need to physically remove disks and whether it is possible to ever get this pool back read-write.

32

Yes, exactly :slight_smile:

33

Per the man page, -T requires -FX.

         -T      Specify the txg to use for rollback.  Implies -FX.  
                 For more details about pool recovery mode, see the -X option, above.
                 WARNING: This option can be extremely hazardous to the health of 
                 your pool and should only be used as a last resort.

27695185 could be importable, without sdd and sdf. But there are only 32 txg in überblocks.

This is another unfortunate case of label mismatch for @HoneyBadger 's collection. :fearful:
There’s something rotten in OpenZFS…

34

It “Implies -FX”. In other words -T is -FX but with a specific predetermined target in mind. In this case, the only known txg that all disks would have in common.

We could also just try and import readonly without any flags so the risk is lowest and the problems arent masked, but I don’t think in this siutation it will import.

I’m still fairly confident this is a hardware issue, not a ZFS one. Given @ericm has replaced hardware already, I cannot prove it.

I’m also still pretty confident booting the system without sdi could help.

35

HoneyBadger talked about this in today’s T3 vLog on YouTube.

In his view it could be hardware issues on every occasion, perhaps SMR drives or non-ECC memory.

His theory with SMR drives is that when the data is destaged by the drive from CMR cache to SMR, then if you get a power failure you can get data corruption.

Here we do have ECC memory and although these are WD Red drives (and not Red Plus or Red Pro) and are EFAX models, according to NasCompares these are actually CMR drives (EFAX drives are SMR drives <= 6TB and CMR > 6TB).

So, (unless WD did it again and made these drives SMR as well without telling anyone) as far as I can see this is good quality hardware without any obvious causes for hardware-related TXG mismatches.

And I simply cannot fathom how 2 drives could become 110,000 TXGs behind unless there is some sort of ZFS bug which allows that to happen e.g. perhaps because somehow a drive stops working and ZFS fails to spot it. I mean at 1 TXG literally every 5 seconds, 113,947 TXGs is equivalent to just over 6.5 days (more if there are no disk writes). (Given the size of an SMR drive’s CMR cache, I couldn’t even see an SMR drive getting corrupted so far back.)

36

At this point I am tagging @HoneyBadger as I think we now need to bring him into this.

37

The Actual ATA bus itelf is telling us that there is an issue, I provided the documentation for this already. I am confused by your reticence here.

Have we validated we can read from the disks mostly fine? Yes
Have we validated the data that was read is being corrupted by it’s controller? No
We also have not confirmed whether or not the old HBA was the issue or not.

@Protopia have you been around in the forums long enough to remember all of the 2/3TB WD Green drives people had been using like 10 years ago? People were using wdidle3 to STOP those drives from head parking. Since that time industry trends towards lower power utilization haven’t stopped.

5 years and that many head parks is alot. Its not as bad as it was in those days, but thats not a neglible amount of use. The fact that we proved with dd that some drives may spin up slower than others is also interesting.

# Drive Power On Hours Approx. Years Head Parks (Load_Cycle_Count) UDMA CRC Errors (199) ATA Error Count
1 /dev/sdi 39,257 hrs 4.48 years 118,383 117,392 65,535 (maxed)
2 /dev/sdh 39,257 hrs 4.48 years 116,774 704 704
3 /dev/sdf 39,167 hrs 4.47 years 114,617 395 395
4 /dev/sdd 39,167 hrs 4.47 years 115,988 6,014 6,014

@joeschmuck I noticed only after putting them together in a table that these are the same two values, ata errors must be a superset of 199.

Each of the 4 drives I posted have errors
Triggered by: READ FPDMA QUEUED (data reads) and READ LOG EXT (SMART checks), drives are aborting a command and resetting itself.

ER: 0x84 → ABRT (Command Aborted) + ICRC (Interface CRC Error)  
ST: 0x41 → ERR bit set + DRDY (Drive Ready)

https://linux.die.net/man/8/smartctl

personal opinion:

  1. sudo zpool import -R /mnt -o readonly=on hd01
    if that does not work:

  2. sudo zpool import -R /mnt -T 27583278 -o readonly=on hd01
    if that does work:

  3. Power down, remove sdi and reboot to see if it imports read write

38

Yes, UDMA_CRC_Errors are generally ATA Errors. But not all ATA errors are UDMA_CRC_Errors.

I have not been following this thread today, been busy with other things. But as I can see, current troubleshooting is to get the pool back online, hopefully without rolling back to an earlier transaction, or at least limit that damage.

Here is what I have for the UDMA_CRC_Error troubleshooting:

  1. The backplane has not been ruled out. In my opinion, it is the most suspect item. I recommend hooking the drives up directly to the HBA if possible to remove the BP as a suspect.
  2. A replacement HBA is coming?
  3. The MiniSAS HD Cable has not been replaced as well.
  4. the motherboard has 10 SATA ports, and those have not been tried (hint hint).

Aside from the mounting of a confused pool, the original problem has not been resolved, correct? Because of this, I recommend mounting as read-only if possible and backing up any important data. I would not try to mount normally until you can solve the million dollar questions, what component is faulty.

@ericm You have 10 SATA ports on that computer motherboard, I highly recommend you use those and rule out the HBA, Backplane, and SCSI cable. This means you need power connectors and a few SATA cables. Remove the HBA when you do this test.

If you cannot do that, do you have a different computer, it doesn’t need to be fancy, just have 8GB+ RAM and have six SATA data connections and power connections. You can boot from a USB Flash drive for the moment while troubleshooting or use another SATA drive. We are only trying to make your system function, we are troubleshooting and looking for the failing item.

If you have a computer which could temporarily support this effort, I would:

  1. Install TrueNAS Scale on to a USB Flash drive, or any other drive you have handy.
  2. Next, transplant the six WD drives into the computer.
  3. Boot the system up, it will be basically a virgin system. Let’s keep it as minimal as possible. Configure the NIC and that should be good.
  4. Record all the UDMA_CRC_Error counts for each drive. Use this to monitor if they continue to grow or have stopped.
  5. While I doubt you will be able to import your pool, you can try it. And at this point I’d start figuring out how to get the pool mounted. And you watch the CRC Error counts. With some luck, they will not increment at all. I don’t expect them to increment actually.

@ericm you have been told to do a lot of things. Read all of these posts and see what you want to do. I can only tell you what I would do. But I do have a spare computer that I can move 6 hard drives into, you may not have that luxury. Solve the original problem or when you finally mount your pool, more damage could occur, it is just an opinion and you know what they say, everyone has one.

Best of luck. This day is rolling to an end for me. Be back in about 10 hours.

39

Thanks @Protopia, @NickF1227, and @etorix for all your analysis, discussion and feedback.

It’s worth stating that disk ‘sdd’ is the device I requested to be replaced. Would I be correct in thinking this explains why it has the lowest TXG of 27583278?

I triggering the resilvering to disk ‘sde’. If it will be possible to roll back to TXG 27583278 on import (as read only) then I can check and compare the data against the last replicated backup held on separate mirrored pool (on the backup server).

From my memory of the resilvering event (dating back to 5th May), I can recall that not all of the disks were visible in the GUI to support resilvering! However, I only spotted this after triggering the disk replacement starting the resilvering to disk ‘sde’. At no stage was I ever prevented from triggering the disk replacement but I was very much aware that resilvering had no hope of being successful.

I have no proper record or way of checking back on this. Could this theory help to explain how disk ‘sdf’ would end up with a matching TXG of 27583278 to disk ‘sdd’? Could a bad HBA prevent one or more disks from appearing in the GUI?

Based on what I have read so far I think it makes sense to proceed with step 1 as indicated by @NickF1227 to at least attempt an initial read only import on the server to see if this works or fails.

Here is the result:

root@emnas[~]# sudo zpool import -R /mnt -o readonly=on hd01
cannot import 'hd01': I/O error
        Destroy and re-create the pool from
        a backup source.

No joy there. :frowning_face:

What is the collective consensus on proceeding with step 2 as this is also a read-only import?

I agree @joeschmuck, I will only attempt read-only imports on this server through the new HBA until I can see what further hardware tests can be performed to validate the existing backplane with the new HBA and the new MiniSAS that are already installed.

Thanks @joeschmuck for making this suggestion. I need a fresh head (after some sleep) to check if I have all the necessary power connectors and required SATA cables in order to do this.

40

I am not surprised that an import without any flags failed (even if readonly).

There needs to be some flags and IMO it is better to try with -F first and only fall back to -T if it fails.

  • sudo zpool import -R /mnt -fF -o readonly=on hd01

This will try to import at any txg that gives integrity i.e. the last few transactions before the uber-blocks get overwritten. I don’t expect it to work, but you should probably try it before moving to -T.

And if that fails and you need to use -T then you should try TXG numbers in descending order because the older the TXGs the more data you lose and the greater the likelihood that you won’t have consistency because e.g. a block used in an early transaction was released and has then been reused - and let’s not forget that losing 2,040 TXGs is about 3 hours of data loss, whilst 113,000 TXGs is about 6.5 days of data loss. So I would suggest the following sequence:

  1. sudo zpool import -R /mnt -f -T 27695185 -o readonly=on hd01
  2. sudo zpool import -R /mnt -f -T 27583278 -o readonly=on hd01
  3. sudo zpool import -R /mnt -fFX -o readonly=on hd01 and let ZFS see if it can find any TXG it can mount at.