HELP Truenas scale constantly marking new drives as failed

CatchyGeo · April 18, 2024, 7:30pm

'm running a dell power edge r530 with 196gb ecc ddr4, 2x xeon e5 2698 v3 cpus, a perc h330 flashed into IT mode, and a lsi sas 9207-8e connected to a DS4246. Now with that out of the way, the issue im having is that a specific vDev keeps being marked as failed. I’ve tried reformatting the drives by replacing them one at a time with one another but formatted in 4kn (the drives support it) but the same thing happens. Smart data looks fine on the drives and there are no uncorrected errors. There are 4 other vDevs in the pool with no issues. Any help would be awesome! The drives being used are HGST H7280A520SUN8.0T drives. If there’s any more info needed let me know! Also running TrueNAS-SCALE-22.12.4.2 as I need hardware transcoding with an older nvidia gpu.

root@truenas[~]# zpool status
 

  pool: Yanni Studio
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: resilvered 804G in 09:16:02 with 188 errors on Thu Apr  4 15:06:25 2024
config:

        NAME                                      STATE     READ WRITE CKSUM
        Yanni Studio                              DEGRADED     0     0     0
          raidz1-0                                ONLINE       0     0     0
            7c3f6eec-6b41-41b6-911a-ed873b909fec  ONLINE       0     0     0
            1b37e351-d33e-4fc5-b622-c0228c1b44f1  ONLINE       0     0     0
            0bfbd6c4-d0f5-4490-a857-9ca5bdc2580b  ONLINE       0     0     0
            604da0dc-5b62-4210-bb52-714482f333c5  ONLINE       0     0     0
            92f3db72-24d3-44e2-af8b-17951a72f33b  ONLINE       0     0     0
          raidz1-1                                ONLINE       0     0     0
            f41d8a1d-f052-456d-9343-5280a3192c26  ONLINE       0     0     0
            2363d5b3-89f9-4586-9291-e2781962e79e  ONLINE       0     0     0
            db0f3488-3475-43cf-89c8-3cd27b960e41  ONLINE       0     0     0
            aaf62f30-53c7-4944-8fec-048303c45def  ONLINE       0     0     0
            148ff0cd-c3d9-4534-aafb-d955871f9319  ONLINE       0     0     0
          raidz1-2                                ONLINE       0     0     0
            b0564f03-24cc-4f06-83d2-7f71256d54ce  ONLINE       0     0     0
            ba56aaae-aa21-480f-9816-9fe1685cf8ac  ONLINE       0     0     0
            84812f61-b7b1-4775-aec0-effa72d5b345  ONLINE       0     0     0
            965195a9-ff64-41bf-946b-da4639583537  ONLINE       0     0     0
            4cce06cf-a844-484f-9b2f-4b3d51aeda62  ONLINE       0     0     0
          raidz1-3                                DEGRADED   164     0     0
            2095c23f-1486-4b1e-9bbc-33ffddf8c11c  ONLINE       0     0   916
            a44577db-861a-49ec-9825-02cd1368e5c2  DEGRADED   190     0   916  too many errors
            79e0dbf9-79d2-45a5-8399-4417dc61fe1e  ONLINE       0     0   916
            4dca8d3d-887f-4b5d-b968-72f3d2a3a872  ONLINE       0     0     0
            47aa4bda-41de-4f2d-bada-defaf1b8617b  FAULTED     57     0   916  too many errors
        spares
          e0fcc478-707b-46ef-826f-d2a1e7233829    AVAIL  

errors: 188 data errors, use '-v' for a list

NickF1227 · April 18, 2024, 8:13pm

Can you provide the output of smartctl -a /dev/DISKNUMBER

CatchyGeo · April 21, 2024, 2:46pm

sdx

root@truenas[~]# smartctl -a /dev/sdx
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              H7280A520SUN8.0T
Revision:             PD51
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   4096 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca2549f8b6c
Serial number:        001621PUS1XV        VKJUS1XV
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sun Apr 21 05:41:44 2024 EDT
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     50 C
Drive Trip Temperature:        85 C

Accumulated power on time, hours:minutes 54541:07
Manufactured in week 21 of year 2016
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  56
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  2319
Elements in grown defect list: 0

Vendor (Seagate Cache) information
  Blocks sent to initiator = 9685610713317376

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0      632         0       632    7726130    1890317.107           0
write:         0       43         0        43    4228836     420348.236           0
verify:        0        0         0         0      99145          0.177           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Completed                   -   54503                 - [-   -    -]
# 2  Background short  Completed                   -   54439                 - [-   -    -]
# 3  Background short  Completed                   -   54391                 - [-   -    -]
# 4  Background short  Completed                   -   54319                 - [-   -    -]
# 5  Background short  Completed                   -   54271                 - [-   -    -]
# 6  Background short  Completed                   -   54223                 - [-   -    -]
# 7  Background short  Completed                   -   54151                 - [-   -    -]
# 8  Background short  Completed                   -   54103                 - [-   -    -]
# 9  Background long   Completed                   -   54096                 - [-   -    -]
#10  Background short  Completed                   -   54055                 - [-   -    -]
#11  Background short  Completed                   -   53983                 - [-   -    -]
#12  Background short  Completed                   -   53935                 - [-   -    -]
#13  Background short  Completed                   -   53887                 - [-   -    -]
#14  Background short  Completed                   -   53815                 - [-   -    -]
#15  Background short  Completed                   -   53767                 - [-   -    -]
#16  Background long   Completed                   -   53759                 - [-   -    -]
#17  Background short  Completed                   -   53719                 - [-   -    -]
#18  Background short  Completed                   -   53647                 - [-   -    -]
#19  Background short  Completed                   -   53599                 - [-   -    -]
#20  Background short  Completed                   -   53551                 - [-   -    -]

Long (extended) Self-test duration: 65535 seconds [1092.2 minutes]

sdt

root@truenas[~]# smartctl -a /dev/sdt
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              H7280A520SUN8.0T
Revision:             PD51
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   4096 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca2549f8b50
Serial number:        001621PUS1PV        VKJUS1PV
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sun Apr 21 05:41:41 2024 EDT
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     52 C
Drive Trip Temperature:        85 C

Accumulated power on time, hours:minutes 54593:45
Manufactured in week 21 of year 2016
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  63
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  2323
Elements in grown defect list: 0

Vendor (Seagate Cache) information
  Blocks sent to initiator = 16801591046701056

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0       38         0        38    7418279    1874932.278           0
write:         0        0         0         0    3996648     449626.817           0
verify:        0        0         0         0      55851          0.173           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Completed                   -   54556                 - [-   -    -]
# 2  Background short  Completed                   -   54492                 - [-   -    -]
# 3  Background short  Completed                   -   54444                 - [-   -    -]
# 4  Background short  Completed                   -   54372                 - [-   -    -]
# 5  Background short  Completed                   -   54324                 - [-   -    -]
# 6  Background short  Completed                   -   54276                 - [-   -    -]
# 7  Background short  Completed                   -   54204                 - [-   -    -]
# 8  Background short  Completed                   -   54156                 - [-   -    -]
# 9  Background long   Completed                   -   54148                 - [-   -    -]
#10  Background short  Completed                   -   54108                 - [-   -    -]
#11  Background short  Completed                   -   54036                 - [-   -    -]
#12  Background short  Completed                   -   53988                 - [-   -    -]
#13  Background short  Completed                   -   53940                 - [-   -    -]
#14  Background short  Completed                   -   53868                 - [-   -    -]
#15  Background short  Completed                   -   53820                 - [-   -    -]
#16  Background long   Completed                   -   53812                 - [-   -    -]
#17  Background short  Completed                   -   53772                 - [-   -    -]
#18  Background short  Completed                   -   53700                 - [-   -    -]
#19  Background short  Completed                   -   53652                 - [-   -    -]
#20  Background short  Completed                   -   53604                 - [-   -    -]

Long (extended) Self-test duration: 65535 seconds [1092.2 minutes]

sdo

root@truenas[~]# smartctl -a /dev/sdo
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              H7280A520SUN8.0T
Revision:             PD51
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   4096 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca254975bc0
Serial number:        001620PP7HBV        VKJP7HBV
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sun Apr 21 05:41:36 2024 EDT
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     53 C
Drive Trip Temperature:        85 C

Accumulated power on time, hours:minutes 54153:12
Manufactured in week 20 of year 2016
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  61
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  2314
Elements in grown defect list: 3

Vendor (Seagate Cache) information
  Blocks sent to initiator = 7500054629711872

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0     3506         0      3506    4460938    1521840.315           0
write:         0      708         0       708    7707019     304447.055           0
verify:        0        1         0         1     623826          0.185           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Completed                   -   54115                 - [-   -    -]
# 2  Background short  Completed                   -   54051                 - [-   -    -]
# 3  Background short  Completed                   -   54003                 - [-   -    -]
# 4  Background short  Completed                   -   53931                 - [-   -    -]
# 5  Background short  Completed                   -   53883                 - [-   -    -]
# 6  Background short  Completed                   -   53835                 - [-   -    -]
# 7  Background short  Completed                   -   53763                 - [-   -    -]
# 8  Background short  Completed                   -   53716                 - [-   -    -]
# 9  Background long   Completed                   -   53708                 - [-   -    -]
#10  Background short  Completed                   -   53668                 - [-   -    -]
#11  Background short  Completed                   -   53596                 - [-   -    -]
#12  Background short  Completed                   -   53548                 - [-   -    -]
#13  Background short  Completed                   -   53499                 - [-   -    -]
#14  Background short  Completed                   -   53427                 - [-   -    -]
#15  Background short  Completed                   -   53379                 - [-   -    -]
#16  Background long   Completed                   -   53371                 - [-   -    -]
#17  Background short  Completed                   -   53331                 - [-   -    -]
#18  Background short  Completed                   -   53259                 - [-   -    -]
#19  Background short  Completed                   -   53211                 - [-   -    -]
#20  Background short  Completed                   -   53163                 - [-   -    -]

Long (extended) Self-test duration: 64971 seconds [1082.8 minutes]

sdj

root@truenas[~]# smartctl -a /dev/sdj 
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              H7280A520SUN8.0T
Revision:             PD51
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   4096 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca254971330
Serial number:        001620PP2NKV        VKJP2NKV
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sun Apr 21 05:41:27 2024 EDT
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     48 C
Drive Trip Temperature:        85 C

Accumulated power on time, hours:minutes 54149:28
Manufactured in week 20 of year 2016
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  85
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  2326
Elements in grown defect list: 0

Vendor (Seagate Cache) information
  Blocks sent to initiator = 16508391899791360

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0      244         0       244    4230586    1879151.109           0
write:         0       47         0        47    3191185     446335.228           0
verify:        0        1         0         1      60186         14.256           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Completed                   -   54112                 - [-   -    -]
# 2  Background short  Completed                   -   54048                 - [-   -    -]
# 3  Background short  Completed                   -   54000                 - [-   -    -]
# 4  Background short  Completed                   -   53928                 - [-   -    -]
# 5  Background short  Completed                   -   53880                 - [-   -    -]
# 6  Background short  Completed                   -   53832                 - [-   -    -]
# 7  Background short  Completed                   -   53760                 - [-   -    -]
# 8  Background short  Completed                   -   53712                 - [-   -    -]
# 9  Background long   Completed                   -   53704                 - [-   -    -]
#10  Background short  Completed                   -   53664                 - [-   -    -]
#11  Background short  Completed                   -   53592                 - [-   -    -]
#12  Background short  Completed                   -   53544                 - [-   -    -]
#13  Background short  Completed                   -   53495                 - [-   -    -]
#14  Background short  Completed                   -   53424                 - [-   -    -]
#15  Background short  Completed                   -   53375                 - [-   -    -]
#16  Background long   Completed                   -   53368                 - [-   -    -]
#17  Background short  Completed                   -   53327                 - [-   -    -]
#18  Background short  Completed                   -   53255                 - [-   -    -]
#19  Background short  Completed                   -   53207                 - [-   -    -]
#20  Background short  Completed                   -   53159                 - [-   -    -]

Long (extended) Self-test duration: 65535 seconds [1092.2 minutes]

sdu

root@truenas[~]# smartctl -a /dev/sdu
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HP
Product:              MB8000JEQVA
Revision:             HPD7
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca254adf804
Serial number:        VKK2NZAV
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sun Apr 21 05:45:44 2024 EDT
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     52 C
Drive Trip Temperature:        60 C

Accumulated power on time, hours:minutes 52065:56
Manufactured in week 23 of year 2016
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  11
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  2081
Elements in grown defect list: 0

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0       23         0         0          0      71991.718           0
write:         0        4         0         4          0      29307.013           0

Non-medium error count:     2034

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Completed                   -   52028                 - [-   -    -]
# 2  Background short  Completed                   -   51964                 - [-   -    -]
# 3  Background short  Completed                   -   51916                 - [-   -    -]
# 4  Background short  Completed                   -   51844                 - [-   -    -]
# 5  Background short  Completed                   -   51796                 - [-   -    -]
# 6  Background short  Completed                   -   51748                 - [-   -    -]
# 7  Background short  Completed                   -   51676                 - [-   -    -]
# 8  Background short  Completed                   -   51628                 - [-   -    -]
# 9  Background long   Completed                   -   51620                 - [-   -    -]
#10  Background short  Completed                   -   51580                 - [-   -    -]
#11  Background short  Completed                   -   51508                 - [-   -    -]
#12  Background short  Completed                   -   51460                 - [-   -    -]
#13  Background short  Completed                   -   51412                 - [-   -    -]
#14  Background short  Completed                   -   51340                 - [-   -    -]
#15  Background short  Completed                   -   51292                 - [-   -    -]
#16  Background long   Completed                   -   51284                 - [-   -    -]
#17  Background short  Completed                   -   51244                 - [-   -    -]
#18  Background short  Completed                   -   51172                 - [-   -    -]
#19  Background short  Completed                   -   51124                 - [-   -    -]
#20  Background short  Completed                   -   51076                 - [-   -    -]

Long (extended) Self-test duration: 65535 seconds [1092.2 minutes]

ericloewe · April 22, 2024, 4:22pm

I don’t like the look of this one… Are all the disks in the same disk shelf, and attached to the same HBA?

Is the SMART data you’ve posted from the disks in the offending vdev? I don’t see any smoking guns, but perhaps -x would show more useful info.

CatchyGeo · April 24, 2024, 3:03pm

Sorry for the delayed response! I’ve been caught up with finals and such at school. All of the disks are in the DS4246 connected externally. I’ve tried swapping positions and even swapping disks from other VDevs within the pool and form outside of the pool. I’ve had this going on since december so I just set everything to the side and made it a later problem. Here’s the same commands with the -x flag

sdx

root@truenas[~]# smartctl -x /dev/sdx
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              H7280A520SUN8.0T
Revision:             PD51
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   4096 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca2549f8b6c
Serial number:        001621PUS1XV        VKJUS1XV
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Wed Apr 24 06:00:12 2024 EDT
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled
Read Cache is:        Enabled
Writeback Cache is:   Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     50 C
Drive Trip Temperature:        85 C

Manufactured in week 21 of year 2016
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  56
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  2322
Elements in grown defect list: 0

Vendor (Seagate Cache) information
  Blocks sent to initiator = 9696379706277888

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0      632         0       632    7727037    1890372.401           0
write:         0       43         0        43    4229663     420355.966           0
verify:        0        0         0         0      99290          0.177           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background short  Completed                   -   54607                 - [-   -    -]
# 2  Background short  Completed                   -   54559                 - [-   -    -]
# 3  Background long   Completed                   -   54503                 - [-   -    -]
# 4  Background short  Completed                   -   54439                 - [-   -    -]
# 5  Background short  Completed                   -   54391                 - [-   -    -]
# 6  Background short  Completed                   -   54319                 - [-   -    -]
# 7  Background short  Completed                   -   54271                 - [-   -    -]
# 8  Background short  Completed                   -   54223                 - [-   -    -]
# 9  Background short  Completed                   -   54151                 - [-   -    -]
#10  Background short  Completed                   -   54103                 - [-   -    -]
#11  Background long   Completed                   -   54096                 - [-   -    -]
#12  Background short  Completed                   -   54055                 - [-   -    -]
#13  Background short  Completed                   -   53983                 - [-   -    -]
#14  Background short  Completed                   -   53935                 - [-   -    -]
#15  Background short  Completed                   -   53887                 - [-   -    -]
#16  Background short  Completed                   -   53815                 - [-   -    -]
#17  Background short  Completed                   -   53767                 - [-   -    -]
#18  Background long   Completed                   -   53759                 - [-   -    -]
#19  Background short  Completed                   -   53719                 - [-   -    -]
#20  Background short  Completed                   -   53647                 - [-   -    -]

Long (extended) Self-test duration: 65535 seconds [1092.2 minutes]

Background scan results log
  Status: scan is active
    Accumulated power on time, hours:minutes 54613:25 [3276805 minutes]
    Number of background scans performed: 329,  scan progress: 55.32%
    Number of background medium scans performed: 329

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 1
  number of phys = 1
  phy identifier = 0
    attached device type: expander device
    attached reason: power on
    reason: unknown
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=0 stp=0 smp=1
    attached target port: ssp=0 stp=0 smp=1
    SAS address = 0x5000cca2549f8b6d
    attached SAS address = 0x500a09800638aeff
    attached phy identifier = 20
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0
relative target port id = 2
  generation code = 1
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: power on
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca2549f8b6e
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0

sdt

root@truenas[~]# smartctl -x /dev/sdt
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              H7280A520SUN8.0T
Revision:             PD51
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   4096 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca2549f8b50
Serial number:        001621PUS1PV        VKJUS1PV
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Wed Apr 24 06:01:07 2024 EDT
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled
Read Cache is:        Enabled
Writeback Cache is:   Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     51 C
Drive Trip Temperature:        85 C

Manufactured in week 21 of year 2016
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  63
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  2326
Elements in grown defect list: 0

Vendor (Seagate Cache) information
  Blocks sent to initiator = 16812310110666752

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0       38         0        38    7419204    1874987.416           0
write:         0        0         0         0    3997772     449634.546           0
verify:        0        0         0         0      55885          0.173           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background short  Completed                   -   54660                 - [-   -    -]
# 2  Background short  Completed                   -   54612                 - [-   -    -]
# 3  Background long   Completed                   -   54556                 - [-   -    -]
# 4  Background short  Completed                   -   54492                 - [-   -    -]
# 5  Background short  Completed                   -   54444                 - [-   -    -]
# 6  Background short  Completed                   -   54372                 - [-   -    -]
# 7  Background short  Completed                   -   54324                 - [-   -    -]
# 8  Background short  Completed                   -   54276                 - [-   -    -]
# 9  Background short  Completed                   -   54204                 - [-   -    -]
#10  Background short  Completed                   -   54156                 - [-   -    -]
#11  Background long   Completed                   -   54148                 - [-   -    -]
#12  Background short  Completed                   -   54108                 - [-   -    -]
#13  Background short  Completed                   -   54036                 - [-   -    -]
#14  Background short  Completed                   -   53988                 - [-   -    -]
#15  Background short  Completed                   -   53940                 - [-   -    -]
#16  Background short  Completed                   -   53868                 - [-   -    -]
#17  Background short  Completed                   -   53820                 - [-   -    -]
#18  Background long   Completed                   -   53812                 - [-   -    -]
#19  Background short  Completed                   -   53772                 - [-   -    -]
#20  Background short  Completed                   -   53700                 - [-   -    -]

Long (extended) Self-test duration: 65535 seconds [1092.2 minutes]

Background scan results log
  Status: scan is active
    Accumulated power on time, hours:minutes 54666:05 [3279965 minutes]
    Number of background scans performed: 329,  scan progress: 57.79%
    Number of background medium scans performed: 329

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 1
  number of phys = 1
  phy identifier = 0
    attached device type: expander device
    attached reason: power on
    reason: unknown
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=0 stp=0 smp=1
    attached target port: ssp=0 stp=0 smp=1
    SAS address = 0x5000cca2549f8b51
    attached SAS address = 0x500a09800638aeff
    attached phy identifier = 16
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0
relative target port id = 2
  generation code = 1
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: power on
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca2549f8b52
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0

sdo

root@truenas[~]# smartctl -x /dev/sdo
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              H7280A520SUN8.0T
Revision:             PD51
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   4096 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca254975bc0
Serial number:        001620PP7HBV        VKJP7HBV
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Wed Apr 24 06:01:32 2024 EDT
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled
Read Cache is:        Enabled
Writeback Cache is:   Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     52 C
Drive Trip Temperature:        85 C

Manufactured in week 20 of year 2016
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  61
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  2317
Elements in grown defect list: 3

Vendor (Seagate Cache) information
  Blocks sent to initiator = 7510762520051712

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0     3506         0      3506    4462833    1521896.386           0
write:         0      763         0       763    7710604     304454.781           0
verify:        0        1         0         1     623826          0.185           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background short  Completed                   -   54219                 - [-   -    -]
# 2  Background short  Completed                   -   54171                 - [-   -    -]
# 3  Background long   Completed                   -   54115                 - [-   -    -]
# 4  Background short  Completed                   -   54051                 - [-   -    -]
# 5  Background short  Completed                   -   54003                 - [-   -    -]
# 6  Background short  Completed                   -   53931                 - [-   -    -]
# 7  Background short  Completed                   -   53883                 - [-   -    -]
# 8  Background short  Completed                   -   53835                 - [-   -    -]
# 9  Background short  Completed                   -   53763                 - [-   -    -]
#10  Background short  Completed                   -   53716                 - [-   -    -]
#11  Background long   Completed                   -   53708                 - [-   -    -]
#12  Background short  Completed                   -   53668                 - [-   -    -]
#13  Background short  Completed                   -   53596                 - [-   -    -]
#14  Background short  Completed                   -   53548                 - [-   -    -]
#15  Background short  Completed                   -   53499                 - [-   -    -]
#16  Background short  Completed                   -   53427                 - [-   -    -]
#17  Background short  Completed                   -   53379                 - [-   -    -]
#18  Background long   Completed                   -   53371                 - [-   -    -]
#19  Background short  Completed                   -   53331                 - [-   -    -]
#20  Background short  Completed                   -   53259                 - [-   -    -]

Long (extended) Self-test duration: 64971 seconds [1082.8 minutes]

Background scan results log
  Status: waiting until BMS interval timer expires
    Accumulated power on time, hours:minutes 54225:32 [3253532 minutes]
    Number of background scans performed: 329,  scan progress: 0.00%
    Number of background medium scans performed: 329

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 1
  number of phys = 1
  phy identifier = 0
    attached device type: expander device
    attached reason: power on
    reason: unknown
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=0 stp=0 smp=1
    attached target port: ssp=0 stp=0 smp=1
    SAS address = 0x5000cca254975bc1
    attached SAS address = 0x500a09800638aeff
    attached phy identifier = 14
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0
relative target port id = 2
  generation code = 1
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: power on
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca254975bc2
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0

sdj

root@truenas[~]# smartctl -x /dev/sdj
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              H7280A520SUN8.0T
Revision:             PD51
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   4096 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca254971330
Serial number:        001620PP2NKV        VKJP2NKV
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Wed Apr 24 06:01:56 2024 EDT
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled
Read Cache is:        Enabled
Writeback Cache is:   Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     47 C
Drive Trip Temperature:        85 C

Manufactured in week 20 of year 2016
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  85
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  2329
Elements in grown defect list: 0

Vendor (Seagate Cache) information
  Blocks sent to initiator = 16517960868823040

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0      244         0       244    4230849    1879161.720           0
write:         0       47         0        47    3192013     446342.154           0
verify:        0        1         0         1      60186         14.256           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background short  Completed                   -   54216                 - [-   -    -]
# 2  Background short  Completed                   -   54168                 - [-   -    -]
# 3  Background long   Completed                   -   54112                 - [-   -    -]
# 4  Background short  Completed                   -   54048                 - [-   -    -]
# 5  Background short  Completed                   -   54000                 - [-   -    -]
# 6  Background short  Completed                   -   53928                 - [-   -    -]
# 7  Background short  Completed                   -   53880                 - [-   -    -]
# 8  Background short  Completed                   -   53832                 - [-   -    -]
# 9  Background short  Completed                   -   53760                 - [-   -    -]
#10  Background short  Completed                   -   53712                 - [-   -    -]
#11  Background long   Completed                   -   53704                 - [-   -    -]
#12  Background short  Completed                   -   53664                 - [-   -    -]
#13  Background short  Completed                   -   53592                 - [-   -    -]
#14  Background short  Completed                   -   53544                 - [-   -    -]
#15  Background short  Completed                   -   53495                 - [-   -    -]
#16  Background short  Completed                   -   53424                 - [-   -    -]
#17  Background short  Completed                   -   53375                 - [-   -    -]
#18  Background long   Completed                   -   53368                 - [-   -    -]
#19  Background short  Completed                   -   53327                 - [-   -    -]
#20  Background short  Completed                   -   53255                 - [-   -    -]

Long (extended) Self-test duration: 65535 seconds [1092.2 minutes]

Background scan results log
  Status: waiting until BMS interval timer expires
    Accumulated power on time, hours:minutes 54221:48 [3253308 minutes]
    Number of background scans performed: 326,  scan progress: 0.00%
    Number of background medium scans performed: 326

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 1
  number of phys = 1
  phy identifier = 0
    attached device type: expander device
    attached reason: power on
    reason: unknown
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=0 stp=0 smp=1
    attached target port: ssp=0 stp=0 smp=1
    SAS address = 0x5000cca254971331
    attached SAS address = 0x500a09800638aeff
    attached phy identifier = 9
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0
relative target port id = 2
  generation code = 1
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: power on
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca254971332
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0

sdu

root@truenas[~]# smartctl -x /dev/sdu
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HP
Product:              MB8000JEQVA
Revision:             HPD7
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca254adf804
Serial number:        VKK2NZAV
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Wed Apr 24 06:02:10 2024 EDT
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled
Read Cache is:        Enabled
Writeback Cache is:   Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     51 C
Drive Trip Temperature:        60 C

Manufactured in week 23 of year 2016
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  11
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  2084
Elements in grown defect list: 0

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0       23         0         0          0      71991.762           0
write:         0        5         0         5          0      29336.627           0

Non-medium error count:     2051

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background short  Completed                   -   52132                 - [-   -    -]
# 2  Background short  Completed                   -   52084                 - [-   -    -]
# 3  Background long   Completed                   -   52028                 - [-   -    -]
# 4  Background short  Completed                   -   51964                 - [-   -    -]
# 5  Background short  Completed                   -   51916                 - [-   -    -]
# 6  Background short  Completed                   -   51844                 - [-   -    -]
# 7  Background short  Completed                   -   51796                 - [-   -    -]
# 8  Background short  Completed                   -   51748                 - [-   -    -]
# 9  Background short  Completed                   -   51676                 - [-   -    -]
#10  Background short  Completed                   -   51628                 - [-   -    -]
#11  Background long   Completed                   -   51620                 - [-   -    -]
#12  Background short  Completed                   -   51580                 - [-   -    -]
#13  Background short  Completed                   -   51508                 - [-   -    -]
#14  Background short  Completed                   -   51460                 - [-   -    -]
#15  Background short  Completed                   -   51412                 - [-   -    -]
#16  Background short  Completed                   -   51340                 - [-   -    -]
#17  Background short  Completed                   -   51292                 - [-   -    -]
#18  Background long   Completed                   -   51284                 - [-   -    -]
#19  Background short  Completed                   -   51244                 - [-   -    -]
#20  Background short  Completed                   -   51172                 - [-   -    -]

Long (extended) Self-test duration: 65535 seconds [1092.2 minutes]

Background scan results log
  Status: scan is active
    Accumulated power on time, hours:minutes 52138:13 [3128293 minutes]
    Number of background scans performed: 1982,  scan progress: 66.69%
    Number of background medium scans performed: 1982

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 1
  number of phys = 1
  phy identifier = 0
    attached device type: expander device
    attached reason: power on
    reason: unknown
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=0 stp=0 smp=1
    attached target port: ssp=0 stp=0 smp=1
    SAS address = 0x5000cca254adf805
    attached SAS address = 0x500a09800638aeff
    attached phy identifier = 15
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0
relative target port id = 2
  generation code = 1
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: power on
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca254adf806
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0

ericloewe · April 24, 2024, 10:23pm

Yeah, the disks aren’t complaining about much of anything. It’s hard to say more at this point, without rigorously testing the disks - but if all other disks are fine, you may have a particularly nasty batch of disks on your hands (e.g. with a firmware bug).

CatchyGeo · April 24, 2024, 10:35pm

The most confusing part has been that the issue follows that VDev. I’ve tried replacing these drives with others used in other VDevs in different pools and same thing. The issue follows the VDev.

Stux · April 24, 2024, 10:49pm

Cabling issue?

I’m not used to seeing SAS smart output, but I do see a lot of ECC corrected errors.

ericloewe · April 24, 2024, 10:59pm

Those are internal, not interface errors. It’s hard to say what’s normal, HDDs are a ridiculous physics experiment on a good day, and my understanding is that errors are common as mud - with ECC taking care of keeping user-facing bit error rates under control.
Sounds crazy, but it’s not that insane.

Wait, do you get new errors or the same errors? The damage has been done and data is corrupted, so replacing disks won’t solve that - but continuing to see more errors would be very bizarre.

CatchyGeo · April 26, 2024, 12:57am

I’ve tried a different cable but I can try using the secondary IOMM6 controller in the DS4246 instead of the primary if that’s the issue

CatchyGeo · April 26, 2024, 12:58am

As far as I know it’s the same errors. I’ve been on and off trying to figure this out since about January. I removed the three files that were shows as corrupt using the -v flag and i’m running another scrub to see what happens. Also ran a -clear before the scrub

NickF1227 · April 26, 2024, 1:00am

Can you type:
less /var/log/messages | grep "kernel: scsi"
And paste the output here?

What we’re looking for is messages like these kinda things, maybe some other similar output:

Apr 25 19:47:00 prod.fusco.me kernel: scsi 0:0:23:0: Power-on or device reset o

but corresponding to a time that was not a recent reboot.

CatchyGeo · April 26, 2024, 1:05am

Only logs that are showing are from my reboot on february 11th. I removed earlier entries because it was too many characters

root@truenas[~]# less /var/log/messages | grep "kernel: scsi"  
Feb 10 08:44:14 truenas kernel: scsi host1: ahci
Feb 10 08:44:14 truenas kernel: scsi host2: ahci
Feb 10 08:44:14 truenas kernel: scsi host3: ahci
Feb 10 08:44:14 truenas kernel: scsi host4: ahci
Feb 10 08:44:14 truenas kernel: scsi host5: ahciem
Feb 10 08:44:14 truenas kernel: scsi 5:0:0:0: Enclosure         AHCI     SGPIO Enclosure  2.00 PQ: 0 ANSI: 7
Feb 10 08:44:14 truenas kernel: scsi host6: ahci
Feb 10 08:44:14 truenas kernel: scsi host7: ahci
Feb 10 08:44:14 truenas kernel: scsi host8: ahci
Feb 10 08:44:14 truenas kernel: scsi host9: ahci
Feb 10 08:44:14 truenas kernel: scsi host10: ahci
Feb 10 08:44:14 truenas kernel: scsi host11: ahci
Feb 10 08:44:14 truenas kernel: scsi host12: ahciem
Feb 10 08:44:14 truenas kernel: scsi 12:0:0:0: Enclosure         AHCI     SGPIO Enclosure  2.00 PQ: 0 ANSI: 7
Feb 10 08:44:14 truenas kernel: scsi host13: usb-storage 2-3:1.0
Feb 10 08:44:14 truenas kernel: scsi host0: Fusion MPT SAS Host
Feb 10 08:44:14 truenas kernel: scsi 0:0:0:0: Direct-Access     ATA      SPCC Solid State 6D.5 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 0:0:0:0: SATA: handle(0x000a), sas_addr(0x4433221100000000), phy(0), device_name(0x9f7ca896b7e1c007)
Feb 10 08:44:14 truenas kernel: scsi 0:0:0:0: enclosure logical id (0x5d09466071559e00), slot(1) 
Feb 10 08:44:14 truenas kernel: scsi 0:0:0:0: enclosure level(0x0001), connector name(     )
Feb 10 08:44:14 truenas kernel: scsi 0:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Feb 10 08:44:14 truenas kernel: scsi 0:0:0:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 0:0:1:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 0:0:1:0: SSP: handle(0x000c), sas_addr(0x5000cca27d024221), phy(1), device_name(0x5000cca27d024223)
Feb 10 08:44:14 truenas kernel: scsi 0:0:1:0: enclosure logical id (0x5d09466071559e00), slot(5) 
Feb 10 08:44:14 truenas kernel: scsi 0:0:1:0: enclosure level(0x0001), connector name(     )
Feb 10 08:44:14 truenas kernel: scsi 0:0:1:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 0:0:2:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 0:0:2:0: SSP: handle(0x000d), sas_addr(0x5000cca27d0240b1), phy(2), device_name(0x5000cca27d0240b3)
Feb 10 08:44:14 truenas kernel: scsi 0:0:2:0: enclosure logical id (0x5d09466071559e00), slot(3) 
Feb 10 08:44:14 truenas kernel: scsi 0:0:2:0: enclosure level(0x0001), connector name(     )
Feb 10 08:44:14 truenas kernel: scsi 0:0:2:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 0:0:3:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 0:0:3:0: SSP: handle(0x000e), sas_addr(0x5000cca27d0272c1), phy(3), device_name(0x5000cca27d0272c3)
Feb 10 08:44:14 truenas kernel: scsi 0:0:3:0: enclosure logical id (0x5d09466071559e00), slot(7) 
Feb 10 08:44:14 truenas kernel: scsi 0:0:3:0: enclosure level(0x0001), connector name(     )
Feb 10 08:44:14 truenas kernel: scsi 0:0:3:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi: waiting for bus probes to complete ...
Feb 10 08:44:14 truenas kernel: scsi host14: Fusion MPT SAS Host
Feb 10 08:44:14 truenas kernel: scsi 13:0:0:0: Direct-Access     hp       v165w            3276 PQ: 0 ANSI: 4
Feb 10 08:44:14 truenas kernel: scsi 0:0:4:0: Direct-Access     ATA      SPCC Solid State 6D.5 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 0:0:4:0: SATA: handle(0x000b), sas_addr(0x4433221104000000), phy(4), device_name(0x9f7ca897aee0bc00)
Feb 10 08:44:14 truenas kernel: scsi 0:0:4:0: enclosure logical id (0x5d09466071559e00), slot(0) 
Feb 10 08:44:14 truenas kernel: scsi 0:0:4:0: enclosure level(0x0001), connector name(     )
Feb 10 08:44:14 truenas kernel: scsi 0:0:4:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Feb 10 08:44:14 truenas kernel: scsi 0:0:4:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi: waiting for bus probes to complete ...
Feb 10 08:44:14 truenas kernel: scsi: waiting for bus probes to complete ...
Feb 10 08:44:14 truenas kernel: scsi 14:0:0:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:0:0: SSP: handle(0x000a), sas_addr(0x5000cca2609ec9b5), phy(8), device_name(0x5000cca2609ec9b7)
Feb 10 08:44:14 truenas kernel: scsi 14:0:0:0: enclosure logical id (0x500a098006c93b00), slot(0) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:0:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:1:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:1:0: SSP: handle(0x000b), sas_addr(0x5000cca254971331), phy(9), device_name(0x5000cca254971333)
Feb 10 08:44:14 truenas kernel: scsi 14:0:1:0: enclosure logical id (0x500a098006c93b00), slot(1) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:1:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:2:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:2:0: SSP: handle(0x000c), sas_addr(0x5000cca254977c39), phy(10), device_name(0x5000cca254977c3b)
Feb 10 08:44:14 truenas kernel: scsi 14:0:2:0: enclosure logical id (0x500a098006c93b00), slot(2) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:2:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:3:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:3:0: SSP: handle(0x000d), sas_addr(0x5000cca254acb561), phy(11), device_name(0x5000cca254acb563)
Feb 10 08:44:14 truenas kernel: scsi 14:0:3:0: enclosure logical id (0x500a098006c93b00), slot(3) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:3:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:4:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:4:0: SSP: handle(0x000e), sas_addr(0x5000cca254a83b65), phy(12), device_name(0x5000cca254a83b67)
Feb 10 08:44:14 truenas kernel: scsi 14:0:4:0: enclosure logical id (0x500a098006c93b00), slot(4) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:4:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:5:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:5:0: SSP: handle(0x000f), sas_addr(0x5000cca2549874fd), phy(13), device_name(0x5000cca2549874ff)
Feb 10 08:44:14 truenas kernel: scsi 14:0:5:0: enclosure logical id (0x500a098006c93b00), slot(5) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:5:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:6:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:6:0: SSP: handle(0x0010), sas_addr(0x5000cca254975bc1), phy(14), device_name(0x5000cca254975bc3)
Feb 10 08:44:14 truenas kernel: scsi 14:0:6:0: enclosure logical id (0x500a098006c93b00), slot(6) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:6:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:7:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:7:0: SSP: handle(0x0011), sas_addr(0x5000cca254adf805), phy(15), device_name(0x5000cca254adf807)
Feb 10 08:44:14 truenas kernel: scsi 14:0:7:0: enclosure logical id (0x500a098006c93b00), slot(7) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:7:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:8:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:8:0: SSP: handle(0x0012), sas_addr(0x5000cca2549f8b51), phy(16), device_name(0x5000cca2549f8b53)
Feb 10 08:44:14 truenas kernel: scsi 14:0:8:0: enclosure logical id (0x500a098006c93b00), slot(8) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:8:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:9:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:9:0: SSP: handle(0x0013), sas_addr(0x5000cca2549f1fe1), phy(17), device_name(0x5000cca2549f1fe3)
Feb 10 08:44:14 truenas kernel: scsi 14:0:9:0: enclosure logical id (0x500a098006c93b00), slot(9) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:9:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:10:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:10:0: SSP: handle(0x0014), sas_addr(0x5000cca2549728a1), phy(18), device_name(0x5000cca2549728a3)
Feb 10 08:44:14 truenas kernel: scsi 14:0:10:0: enclosure logical id (0x500a098006c93b00), slot(10) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:10:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:11:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:11:0: SSP: handle(0x0015), sas_addr(0x5000cca254b0cd31), phy(19), device_name(0x5000cca254b0cd33)
Feb 10 08:44:14 truenas kernel: scsi 14:0:11:0: enclosure logical id (0x500a098006c93b00), slot(11) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:11:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:12:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:12:0: SSP: handle(0x0016), sas_addr(0x5000cca2549f8b6d), phy(20), device_name(0x5000cca2549f8b6f)
Feb 10 08:44:14 truenas kernel: scsi 14:0:12:0: enclosure logical id (0x500a098006c93b00), slot(12) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:12:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:13:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:13:0: SSP: handle(0x0017), sas_addr(0x5000cca2609eba1d), phy(21), device_name(0x5000cca2609eba1f)
Feb 10 08:44:14 truenas kernel: scsi 14:0:13:0: enclosure logical id (0x500a098006c93b00), slot(13) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:13:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:14:0: Direct-Access     ATA      ST8000DM004-2CX1 0001 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:14:0: SATA: handle(0x0018), sas_addr(0x500a09800638aed6), phy(22), device_name(0x0000000000000000)
Feb 10 08:44:14 truenas kernel: scsi 14:0:14:0: enclosure logical id (0x500a098006c93b00), slot(14) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:14:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Feb 10 08:44:14 truenas kernel: scsi 14:0:14:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:15:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:15:0: SSP: handle(0x0019), sas_addr(0x5000cca254b04881), phy(23), device_name(0x5000cca254b04883)
Feb 10 08:44:14 truenas kernel: scsi 14:0:15:0: enclosure logical id (0x500a098006c93b00), slot(15) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:15:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:16:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:16:0: SSP: handle(0x001a), sas_addr(0x5000cca260a56871), phy(24), device_name(0x5000cca260a56873)
Feb 10 08:44:14 truenas kernel: scsi 14:0:16:0: enclosure logical id (0x500a098006c93b00), slot(16) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:16:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:17:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:17:0: SSP: handle(0x001b), sas_addr(0x5000cca260a55ef5), phy(25), device_name(0x5000cca260a55ef7)
Feb 10 08:44:14 truenas kernel: scsi 14:0:17:0: enclosure logical id (0x500a098006c93b00), slot(17) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:17:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:18:0: Direct-Access     SEAGATE  ST6000NM0054     NE03 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:18:0: SSP: handle(0x001c), sas_addr(0x5000c500849f8e15), phy(26), device_name(0x5000c500849f8e14)
Feb 10 08:44:14 truenas kernel: scsi 14:0:18:0: enclosure logical id (0x500a098006c93b00), slot(18) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:18:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:19:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:19:0: SSP: handle(0x001d), sas_addr(0x5000cca254b059f1), phy(27), device_name(0x5000cca254b059f3)
Feb 10 08:44:14 truenas kernel: scsi 14:0:19:0: enclosure logical id (0x500a098006c93b00), slot(19) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:19:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:20:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:20:0: SSP: handle(0x001e), sas_addr(0x5000cca2549da429), phy(28), device_name(0x5000cca2549da42b)
Feb 10 08:44:14 truenas kernel: scsi 14:0:20:0: enclosure logical id (0x500a098006c93b00), slot(20) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:20:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:21:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:21:0: SSP: handle(0x001f), sas_addr(0x5000cca2609eb619), phy(29), device_name(0x5000cca2609eb61b)
Feb 10 08:44:14 truenas kernel: scsi 14:0:21:0: enclosure logical id (0x500a098006c93b00), slot(21) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:21:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:22:0: Direct-Access     SEAGATE  ST6000NM0054     NE03 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:22:0: SSP: handle(0x0020), sas_addr(0x5000c500849f906d), phy(30), device_name(0x5000c500849f906c)
Feb 10 08:44:14 truenas kernel: scsi 14:0:22:0: enclosure logical id (0x500a098006c93b00), slot(22) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:22:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:23:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 14:0:23:0: SSP: handle(0x0021), sas_addr(0x5000cca254cc1c9d), phy(31), device_name(0x5000cca254cc1c9f)
Feb 10 08:44:14 truenas kernel: scsi 14:0:23:0: enclosure logical id (0x500a098006c93b00), slot(23) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:23:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 14:0:24:0: Enclosure         NETAPP   DS424IOM6        0191 PQ: 0 ANSI: 5
Feb 10 08:44:14 truenas kernel: scsi 14:0:24:0: set ignore_delay_remove for handle(0x0022)
Feb 10 08:44:14 truenas kernel: scsi 14:0:24:0: SES: handle(0x0022), sas_addr(0x500a09800638aefe), phy(36), device_name(0x0000000000000000)
Feb 10 08:44:14 truenas kernel: scsi 14:0:24:0: enclosure logical id (0x500a098006c93b00), slot(0) 
Feb 10 08:44:14 truenas kernel: scsi 14:0:24:0: qdepth(254), tagged(1), scsi_level(6), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 11:0:0:0: Direct-Access     ATA      PNY CS900 120GB  0E12 PQ: 0 ANSI: 5
Feb 10 08:44:14 truenas kernel: scsi 0:0:5:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 0:0:5:0: SSP: handle(0x000f), sas_addr(0x5000cca27d0220e5), phy(5), device_name(0x5000cca27d0220e7)
Feb 10 08:44:14 truenas kernel: scsi 0:0:5:0: enclosure logical id (0x5d09466071559e00), slot(4) 
Feb 10 08:44:14 truenas kernel: scsi 0:0:5:0: enclosure level(0x0001), connector name(     )
Feb 10 08:44:14 truenas kernel: scsi 0:0:5:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 0:0:6:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 0:0:6:0: SSP: handle(0x0010), sas_addr(0x5000cca27d064b85), phy(6), device_name(0x5000cca27d064b87)
Feb 10 08:44:14 truenas kernel: scsi 0:0:6:0: enclosure logical id (0x5d09466071559e00), slot(2) 
Feb 10 08:44:14 truenas kernel: scsi 0:0:6:0: enclosure level(0x0001), connector name(     )
Feb 10 08:44:14 truenas kernel: scsi 0:0:6:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 0:0:7:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 10 08:44:14 truenas kernel: scsi 0:0:7:0: SSP: handle(0x0011), sas_addr(0x5000cca27d0649f5), phy(7), device_name(0x5000cca27d0649f7)
Feb 10 08:44:14 truenas kernel: scsi 0:0:7:0: enclosure logical id (0x5d09466071559e00), slot(6) 
Feb 10 08:44:14 truenas kernel: scsi 0:0:7:0: enclosure level(0x0001), connector name(     )
Feb 10 08:44:14 truenas kernel: scsi 0:0:7:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 10 08:44:14 truenas kernel: scsi 13:0:0:0: Embedded Enclosure Device
Feb 11 08:20:38 truenas kernel: scsi host1: ahci
Feb 11 08:20:38 truenas kernel: scsi host2: ahci
Feb 11 08:20:38 truenas kernel: scsi host3: ahci
Feb 11 08:20:38 truenas kernel: scsi host4: ahci
Feb 11 08:20:38 truenas kernel: scsi host5: ahciem
Feb 11 08:20:38 truenas kernel: scsi 5:0:0:0: Enclosure         AHCI     SGPIO Enclosure  2.00 PQ: 0 ANSI: 7
Feb 11 08:20:38 truenas kernel: scsi host6: ahci
Feb 11 08:20:38 truenas kernel: scsi host7: ahci
Feb 11 08:20:38 truenas kernel: scsi host8: ahci
Feb 11 08:20:38 truenas kernel: scsi host9: ahci
Feb 11 08:20:38 truenas kernel: scsi host10: ahci
Feb 11 08:20:38 truenas kernel: scsi host11: ahci
Feb 11 08:20:38 truenas kernel: scsi host12: ahciem
Feb 11 08:20:38 truenas kernel: scsi 12:0:0:0: Enclosure         AHCI     SGPIO Enclosure  2.00 PQ: 0 ANSI: 7
Feb 11 08:20:38 truenas kernel: scsi host0: Fusion MPT SAS Host
Feb 11 08:20:38 truenas kernel: scsi 0:0:0:0: Direct-Access     ATA      SPCC Solid State 6D.5 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 0:0:0:0: SATA: handle(0x000b), sas_addr(0x4433221104000000), phy(4), device_name(0x9f7ca897aee0bc00)
Feb 11 08:20:38 truenas kernel: scsi 0:0:0:0: enclosure logical id (0x5d09466071559e00), slot(0) 
Feb 11 08:20:38 truenas kernel: scsi 0:0:0:0: enclosure level(0x0001), connector name(     )
Feb 11 08:20:38 truenas kernel: scsi 0:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Feb 11 08:20:38 truenas kernel: scsi 0:0:0:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 0:0:1:0: Direct-Access     ATA      SPCC Solid State 6D.5 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 0:0:1:0: SATA: handle(0x000a), sas_addr(0x4433221100000000), phy(0), device_name(0x9f7ca896b7e1c007)
Feb 11 08:20:38 truenas kernel: scsi 0:0:1:0: enclosure logical id (0x5d09466071559e00), slot(1) 
Feb 11 08:20:38 truenas kernel: scsi 0:0:1:0: enclosure level(0x0001), connector name(     )
Feb 11 08:20:38 truenas kernel: scsi 0:0:1:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Feb 11 08:20:38 truenas kernel: scsi 0:0:1:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 0:0:2:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 0:0:2:0: SSP: handle(0x000c), sas_addr(0x5000cca27d024221), phy(1), device_name(0x5000cca27d024223)
Feb 11 08:20:38 truenas kernel: scsi 0:0:2:0: enclosure logical id (0x5d09466071559e00), slot(5) 
Feb 11 08:20:38 truenas kernel: scsi 0:0:2:0: enclosure level(0x0001), connector name(     )
Feb 11 08:20:38 truenas kernel: scsi 0:0:2:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 0:0:3:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 0:0:3:0: SSP: handle(0x000d), sas_addr(0x5000cca27d0240b1), phy(2), device_name(0x5000cca27d0240b3)
Feb 11 08:20:38 truenas kernel: scsi 0:0:3:0: enclosure logical id (0x5d09466071559e00), slot(3) 
Feb 11 08:20:38 truenas kernel: scsi 0:0:3:0: enclosure level(0x0001), connector name(     )
Feb 11 08:20:38 truenas kernel: scsi 0:0:3:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 0:0:4:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 0:0:4:0: SSP: handle(0x000e), sas_addr(0x5000cca27d0272c1), phy(3), device_name(0x5000cca27d0272c3)
Feb 11 08:20:38 truenas kernel: scsi 0:0:4:0: enclosure logical id (0x5d09466071559e00), slot(7) 
Feb 11 08:20:38 truenas kernel: scsi 0:0:4:0: enclosure level(0x0001), connector name(     )
Feb 11 08:20:38 truenas kernel: scsi 0:0:4:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi host14: usb-storage 2-3:1.0
Feb 11 08:20:38 truenas kernel: scsi 0:0:5:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 0:0:5:0: SSP: handle(0x000f), sas_addr(0x5000cca27d0220e5), phy(5), device_name(0x5000cca27d0220e7)
Feb 11 08:20:38 truenas kernel: scsi 0:0:5:0: enclosure logical id (0x5d09466071559e00), slot(4) 
Feb 11 08:20:38 truenas kernel: scsi 0:0:5:0: enclosure level(0x0001), connector name(     )
Feb 11 08:20:38 truenas kernel: scsi 0:0:5:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 0:0:6:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 0:0:6:0: SSP: handle(0x0010), sas_addr(0x5000cca27d064b85), phy(6), device_name(0x5000cca27d064b87)
Feb 11 08:20:38 truenas kernel: scsi 0:0:6:0: enclosure logical id (0x5d09466071559e00), slot(2) 
Feb 11 08:20:38 truenas kernel: scsi 0:0:6:0: enclosure level(0x0001), connector name(     )
Feb 11 08:20:38 truenas kernel: scsi 0:0:6:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 0:0:7:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 0:0:7:0: SSP: handle(0x0011), sas_addr(0x5000cca27d0649f5), phy(7), device_name(0x5000cca27d0649f7)
Feb 11 08:20:38 truenas kernel: scsi 0:0:7:0: enclosure logical id (0x5d09466071559e00), slot(6) 
Feb 11 08:20:38 truenas kernel: scsi 0:0:7:0: enclosure level(0x0001), connector name(     )
Feb 11 08:20:38 truenas kernel: scsi 0:0:7:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi host13: Fusion MPT SAS Host
Feb 11 08:20:38 truenas kernel: scsi: waiting for bus probes to complete ...
Feb 11 08:20:38 truenas kernel: scsi 14:0:0:0: Direct-Access     hp       v165w            3276 PQ: 0 ANSI: 4
Feb 11 08:20:38 truenas kernel: scsi 13:0:0:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:0:0: SSP: handle(0x000a), sas_addr(0x5000cca2609ec9b5), phy(8), device_name(0x5000cca2609ec9b7)
Feb 11 08:20:38 truenas kernel: scsi 13:0:0:0: enclosure logical id (0x500a098006c93b00), slot(0) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:0:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:1:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:1:0: SSP: handle(0x000b), sas_addr(0x5000cca254971331), phy(9), device_name(0x5000cca254971333)
Feb 11 08:20:38 truenas kernel: scsi 13:0:1:0: enclosure logical id (0x500a098006c93b00), slot(1) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:1:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:2:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:2:0: SSP: handle(0x000c), sas_addr(0x5000cca254977c39), phy(10), device_name(0x5000cca254977c3b)
Feb 11 08:20:38 truenas kernel: scsi 13:0:2:0: enclosure logical id (0x500a098006c93b00), slot(2) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:2:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:3:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:3:0: SSP: handle(0x000d), sas_addr(0x5000cca254acb561), phy(11), device_name(0x5000cca254acb563)
Feb 11 08:20:38 truenas kernel: scsi 13:0:3:0: enclosure logical id (0x500a098006c93b00), slot(3) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:3:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:4:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:4:0: SSP: handle(0x000e), sas_addr(0x5000cca254a83b65), phy(12), device_name(0x5000cca254a83b67)
Feb 11 08:20:38 truenas kernel: scsi 13:0:4:0: enclosure logical id (0x500a098006c93b00), slot(4) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:4:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:5:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:5:0: SSP: handle(0x000f), sas_addr(0x5000cca2549874fd), phy(13), device_name(0x5000cca2549874ff)
Feb 11 08:20:38 truenas kernel: scsi 13:0:5:0: enclosure logical id (0x500a098006c93b00), slot(5) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:5:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:6:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:6:0: SSP: handle(0x0010), sas_addr(0x5000cca254975bc1), phy(14), device_name(0x5000cca254975bc3)
Feb 11 08:20:38 truenas kernel: scsi 13:0:6:0: enclosure logical id (0x500a098006c93b00), slot(6) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:6:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:7:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:7:0: SSP: handle(0x0011), sas_addr(0x5000cca254adf805), phy(15), device_name(0x5000cca254adf807)
Feb 11 08:20:38 truenas kernel: scsi 13:0:7:0: enclosure logical id (0x500a098006c93b00), slot(7) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:7:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:8:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:8:0: SSP: handle(0x0012), sas_addr(0x5000cca2549f8b51), phy(16), device_name(0x5000cca2549f8b53)
Feb 11 08:20:38 truenas kernel: scsi 13:0:8:0: enclosure logical id (0x500a098006c93b00), slot(8) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:8:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:9:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:9:0: SSP: handle(0x0013), sas_addr(0x5000cca2549f1fe1), phy(17), device_name(0x5000cca2549f1fe3)
Feb 11 08:20:38 truenas kernel: scsi 13:0:9:0: enclosure logical id (0x500a098006c93b00), slot(9) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:9:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:10:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:10:0: SSP: handle(0x0014), sas_addr(0x5000cca2549728a1), phy(18), device_name(0x5000cca2549728a3)
Feb 11 08:20:38 truenas kernel: scsi 13:0:10:0: enclosure logical id (0x500a098006c93b00), slot(10) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:10:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:11:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:11:0: SSP: handle(0x0015), sas_addr(0x5000cca254b0cd31), phy(19), device_name(0x5000cca254b0cd33)
Feb 11 08:20:38 truenas kernel: scsi 13:0:11:0: enclosure logical id (0x500a098006c93b00), slot(11) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:11:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:12:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:12:0: SSP: handle(0x0016), sas_addr(0x5000cca2549f8b6d), phy(20), device_name(0x5000cca2549f8b6f)
Feb 11 08:20:38 truenas kernel: scsi 13:0:12:0: enclosure logical id (0x500a098006c93b00), slot(12) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:12:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:13:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:13:0: SSP: handle(0x0017), sas_addr(0x5000cca2609eba1d), phy(21), device_name(0x5000cca2609eba1f)
Feb 11 08:20:38 truenas kernel: scsi 13:0:13:0: enclosure logical id (0x500a098006c93b00), slot(13) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:13:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:14:0: Direct-Access     ATA      ST8000DM004-2CX1 0001 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:14:0: SATA: handle(0x0018), sas_addr(0x500a09800638aed6), phy(22), device_name(0x0000000000000000)
Feb 11 08:20:38 truenas kernel: scsi 13:0:14:0: enclosure logical id (0x500a098006c93b00), slot(14) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:14:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Feb 11 08:20:38 truenas kernel: scsi 13:0:14:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:15:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:15:0: SSP: handle(0x0019), sas_addr(0x5000cca254b04881), phy(23), device_name(0x5000cca254b04883)
Feb 11 08:20:38 truenas kernel: scsi 13:0:15:0: enclosure logical id (0x500a098006c93b00), slot(15) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:15:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:16:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:16:0: SSP: handle(0x001a), sas_addr(0x5000cca260a56871), phy(24), device_name(0x5000cca260a56873)
Feb 11 08:20:38 truenas kernel: scsi 13:0:16:0: enclosure logical id (0x500a098006c93b00), slot(16) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:16:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:17:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:17:0: SSP: handle(0x001b), sas_addr(0x5000cca260a55ef5), phy(25), device_name(0x5000cca260a55ef7)
Feb 11 08:20:38 truenas kernel: scsi 13:0:17:0: enclosure logical id (0x500a098006c93b00), slot(17) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:17:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:18:0: Direct-Access     SEAGATE  ST6000NM0054     NE03 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:18:0: SSP: handle(0x001c), sas_addr(0x5000c500849f8e15), phy(26), device_name(0x5000c500849f8e14)
Feb 11 08:20:38 truenas kernel: scsi 13:0:18:0: enclosure logical id (0x500a098006c93b00), slot(18) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:18:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:19:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:19:0: SSP: handle(0x001d), sas_addr(0x5000cca254b059f1), phy(27), device_name(0x5000cca254b059f3)
Feb 11 08:20:38 truenas kernel: scsi 13:0:19:0: enclosure logical id (0x500a098006c93b00), slot(19) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:19:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:20:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:20:0: SSP: handle(0x001e), sas_addr(0x5000cca2549da429), phy(28), device_name(0x5000cca2549da42b)
Feb 11 08:20:38 truenas kernel: scsi 13:0:20:0: enclosure logical id (0x500a098006c93b00), slot(20) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:20:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:21:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:21:0: SSP: handle(0x001f), sas_addr(0x5000cca2609eb619), phy(29), device_name(0x5000cca2609eb61b)
Feb 11 08:20:38 truenas kernel: scsi 13:0:21:0: enclosure logical id (0x500a098006c93b00), slot(21) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:21:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:22:0: Direct-Access     SEAGATE  ST6000NM0054     NE03 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:22:0: SSP: handle(0x0020), sas_addr(0x5000c500849f906d), phy(30), device_name(0x5000c500849f906c)
Feb 11 08:20:38 truenas kernel: scsi 13:0:22:0: enclosure logical id (0x500a098006c93b00), slot(22) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:22:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:23:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 11 08:20:38 truenas kernel: scsi 13:0:23:0: SSP: handle(0x0021), sas_addr(0x5000cca254cc1c9d), phy(31), device_name(0x5000cca254cc1c9f)
Feb 11 08:20:38 truenas kernel: scsi 13:0:23:0: enclosure logical id (0x500a098006c93b00), slot(23) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:23:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 13:0:24:0: Enclosure         NETAPP   DS424IOM6        0191 PQ: 0 ANSI: 5
Feb 11 08:20:38 truenas kernel: scsi 13:0:24:0: set ignore_delay_remove for handle(0x0022)
Feb 11 08:20:38 truenas kernel: scsi 13:0:24:0: SES: handle(0x0022), sas_addr(0x500a09800638aefe), phy(36), device_name(0x0000000000000000)
Feb 11 08:20:38 truenas kernel: scsi 13:0:24:0: enclosure logical id (0x500a098006c93b00), slot(0) 
Feb 11 08:20:38 truenas kernel: scsi 13:0:24:0: qdepth(254), tagged(1), scsi_level(6), cmd_que(1)
Feb 11 08:20:38 truenas kernel: scsi 11:0:0:0: Direct-Access     ATA      PNY CS900 120GB  0E12 PQ: 0 ANSI: 5
Feb 11 17:29:08 truenas kernel: scsi host1: ahci
Feb 11 17:29:08 truenas kernel: scsi host2: ahci
Feb 11 17:29:08 truenas kernel: scsi host3: ahci
Feb 11 17:29:08 truenas kernel: scsi host4: ahci
Feb 11 17:29:08 truenas kernel: scsi host5: ahciem
Feb 11 17:29:08 truenas kernel: scsi 5:0:0:0: Enclosure         AHCI     SGPIO Enclosure  2.00 PQ: 0 ANSI: 7
Feb 11 17:29:08 truenas kernel: scsi host6: ahci
Feb 11 17:29:08 truenas kernel: scsi host7: ahci
Feb 11 17:29:08 truenas kernel: scsi host8: ahci
Feb 11 17:29:08 truenas kernel: scsi host9: ahci
Feb 11 17:29:08 truenas kernel: scsi host10: ahci
Feb 11 17:29:08 truenas kernel: scsi host11: ahci
Feb 11 17:29:08 truenas kernel: scsi host12: ahciem
Feb 11 17:29:08 truenas kernel: scsi 12:0:0:0: Enclosure         AHCI     SGPIO Enclosure  2.00 PQ: 0 ANSI: 7
Feb 11 17:29:08 truenas kernel: scsi host0: Fusion MPT SAS Host
Feb 11 17:29:08 truenas kernel: scsi 0:0:0:0: Direct-Access     ATA      SPCC Solid State 6D.5 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 0:0:0:0: SATA: handle(0x000a), sas_addr(0x4433221100000000), phy(0), device_name(0x9f7ca896b7e1c007)
Feb 11 17:29:08 truenas kernel: scsi 0:0:0:0: enclosure logical id (0x5d09466071559e00), slot(1) 
Feb 11 17:29:08 truenas kernel: scsi 0:0:0:0: enclosure level(0x0001), connector name(     )
Feb 11 17:29:08 truenas kernel: scsi 0:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Feb 11 17:29:08 truenas kernel: scsi 0:0:0:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 0:0:1:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 0:0:1:0: SSP: handle(0x000c), sas_addr(0x5000cca27d024221), phy(1), device_name(0x5000cca27d024223)
Feb 11 17:29:08 truenas kernel: scsi 0:0:1:0: enclosure logical id (0x5d09466071559e00), slot(5) 
Feb 11 17:29:08 truenas kernel: scsi 0:0:1:0: enclosure level(0x0001), connector name(     )
Feb 11 17:29:08 truenas kernel: scsi 0:0:1:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 0:0:2:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 0:0:2:0: SSP: handle(0x000d), sas_addr(0x5000cca27d0240b1), phy(2), device_name(0x5000cca27d0240b3)
Feb 11 17:29:08 truenas kernel: scsi 0:0:2:0: enclosure logical id (0x5d09466071559e00), slot(3) 
Feb 11 17:29:08 truenas kernel: scsi 0:0:2:0: enclosure level(0x0001), connector name(     )
Feb 11 17:29:08 truenas kernel: scsi 0:0:2:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi host14: usb-storage 2-3:1.0
Feb 11 17:29:08 truenas kernel: scsi 0:0:3:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 0:0:3:0: SSP: handle(0x000e), sas_addr(0x5000cca27d0272c1), phy(3), device_name(0x5000cca27d0272c3)
Feb 11 17:29:08 truenas kernel: scsi 0:0:3:0: enclosure logical id (0x5d09466071559e00), slot(7) 
Feb 11 17:29:08 truenas kernel: scsi 0:0:3:0: enclosure level(0x0001), connector name(     )
Feb 11 17:29:08 truenas kernel: scsi 0:0:3:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 0:0:4:0: Direct-Access     ATA      SPCC Solid State 6D.5 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 0:0:4:0: SATA: handle(0x000b), sas_addr(0x4433221104000000), phy(4), device_name(0x9f7ca897aee0bc00)
Feb 11 17:29:08 truenas kernel: scsi 0:0:4:0: enclosure logical id (0x5d09466071559e00), slot(0) 
Feb 11 17:29:08 truenas kernel: scsi 0:0:4:0: enclosure level(0x0001), connector name(     )
Feb 11 17:29:08 truenas kernel: scsi 0:0:4:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Feb 11 17:29:08 truenas kernel: scsi 0:0:4:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 0:0:5:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 0:0:5:0: SSP: handle(0x000f), sas_addr(0x5000cca27d0220e5), phy(5), device_name(0x5000cca27d0220e7)
Feb 11 17:29:08 truenas kernel: scsi 0:0:5:0: enclosure logical id (0x5d09466071559e00), slot(4) 
Feb 11 17:29:08 truenas kernel: scsi 0:0:5:0: enclosure level(0x0001), connector name(     )
Feb 11 17:29:08 truenas kernel: scsi 0:0:5:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 0:0:6:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 0:0:6:0: SSP: handle(0x0010), sas_addr(0x5000cca27d064b85), phy(6), device_name(0x5000cca27d064b87)
Feb 11 17:29:08 truenas kernel: scsi 0:0:6:0: enclosure logical id (0x5d09466071559e00), slot(2) 
Feb 11 17:29:08 truenas kernel: scsi 0:0:6:0: enclosure level(0x0001), connector name(     )
Feb 11 17:29:08 truenas kernel: scsi 0:0:6:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 0:0:7:0: Direct-Access     HGST     HUH721008AL4200  A3Z4 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 0:0:7:0: SSP: handle(0x0011), sas_addr(0x5000cca27d0649f5), phy(7), device_name(0x5000cca27d0649f7)
Feb 11 17:29:08 truenas kernel: scsi 0:0:7:0: enclosure logical id (0x5d09466071559e00), slot(6) 
Feb 11 17:29:08 truenas kernel: scsi 0:0:7:0: enclosure level(0x0001), connector name(     )
Feb 11 17:29:08 truenas kernel: scsi 0:0:7:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi host13: Fusion MPT SAS Host
Feb 11 17:29:08 truenas kernel: scsi: waiting for bus probes to complete ...
Feb 11 17:29:08 truenas kernel: scsi 14:0:0:0: Direct-Access     hp       v165w            3276 PQ: 0 ANSI: 4
Feb 11 17:29:08 truenas kernel: scsi 13:0:0:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:0:0: SSP: handle(0x000a), sas_addr(0x5000cca2609ec9b5), phy(8), device_name(0x5000cca2609ec9b7)
Feb 11 17:29:08 truenas kernel: scsi 13:0:0:0: enclosure logical id (0x500a098006c93b00), slot(0) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:0:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:1:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:1:0: SSP: handle(0x000b), sas_addr(0x5000cca254971331), phy(9), device_name(0x5000cca254971333)
Feb 11 17:29:08 truenas kernel: scsi 13:0:1:0: enclosure logical id (0x500a098006c93b00), slot(1) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:1:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:2:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:2:0: SSP: handle(0x000c), sas_addr(0x5000cca254977c39), phy(10), device_name(0x5000cca254977c3b)
Feb 11 17:29:08 truenas kernel: scsi 13:0:2:0: enclosure logical id (0x500a098006c93b00), slot(2) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:2:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:3:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:3:0: SSP: handle(0x000d), sas_addr(0x5000cca254acb561), phy(11), device_name(0x5000cca254acb563)
Feb 11 17:29:08 truenas kernel: scsi 13:0:3:0: enclosure logical id (0x500a098006c93b00), slot(3) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:3:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:4:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:4:0: SSP: handle(0x000e), sas_addr(0x5000cca254a83b65), phy(12), device_name(0x5000cca254a83b67)
Feb 11 17:29:08 truenas kernel: scsi 13:0:4:0: enclosure logical id (0x500a098006c93b00), slot(4) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:4:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:5:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:5:0: SSP: handle(0x000f), sas_addr(0x5000cca2549874fd), phy(13), device_name(0x5000cca2549874ff)
Feb 11 17:29:08 truenas kernel: scsi 13:0:5:0: enclosure logical id (0x500a098006c93b00), slot(5) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:5:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:6:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:6:0: SSP: handle(0x0010), sas_addr(0x5000cca254975bc1), phy(14), device_name(0x5000cca254975bc3)
Feb 11 17:29:08 truenas kernel: scsi 13:0:6:0: enclosure logical id (0x500a098006c93b00), slot(6) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:6:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:7:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:7:0: SSP: handle(0x0011), sas_addr(0x5000cca254adf805), phy(15), device_name(0x5000cca254adf807)
Feb 11 17:29:08 truenas kernel: scsi 13:0:7:0: enclosure logical id (0x500a098006c93b00), slot(7) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:7:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:8:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:8:0: SSP: handle(0x0012), sas_addr(0x5000cca2549f8b51), phy(16), device_name(0x5000cca2549f8b53)
Feb 11 17:29:08 truenas kernel: scsi 13:0:8:0: enclosure logical id (0x500a098006c93b00), slot(8) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:8:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:9:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:9:0: SSP: handle(0x0013), sas_addr(0x5000cca2549f1fe1), phy(17), device_name(0x5000cca2549f1fe3)
Feb 11 17:29:08 truenas kernel: scsi 13:0:9:0: enclosure logical id (0x500a098006c93b00), slot(9) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:9:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:10:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:10:0: SSP: handle(0x0014), sas_addr(0x5000cca2549728a1), phy(18), device_name(0x5000cca2549728a3)
Feb 11 17:29:08 truenas kernel: scsi 13:0:10:0: enclosure logical id (0x500a098006c93b00), slot(10) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:10:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:11:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:11:0: SSP: handle(0x0015), sas_addr(0x5000cca254b0cd31), phy(19), device_name(0x5000cca254b0cd33)
Feb 11 17:29:08 truenas kernel: scsi 13:0:11:0: enclosure logical id (0x500a098006c93b00), slot(11) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:11:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:12:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:12:0: SSP: handle(0x0016), sas_addr(0x5000cca2549f8b6d), phy(20), device_name(0x5000cca2549f8b6f)
Feb 11 17:29:08 truenas kernel: scsi 13:0:12:0: enclosure logical id (0x500a098006c93b00), slot(12) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:12:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:13:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:13:0: SSP: handle(0x0017), sas_addr(0x5000cca2609eba1d), phy(21), device_name(0x5000cca2609eba1f)
Feb 11 17:29:08 truenas kernel: scsi 13:0:13:0: enclosure logical id (0x500a098006c93b00), slot(13) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:13:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:14:0: Direct-Access     ATA      ST8000DM004-2CX1 0001 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:14:0: SATA: handle(0x0018), sas_addr(0x500a09800638aed6), phy(22), device_name(0x0000000000000000)
Feb 11 17:29:08 truenas kernel: scsi 13:0:14:0: enclosure logical id (0x500a098006c93b00), slot(14) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:14:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Feb 11 17:29:08 truenas kernel: scsi 13:0:14:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:15:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:15:0: SSP: handle(0x0019), sas_addr(0x5000cca254b04881), phy(23), device_name(0x5000cca254b04883)
Feb 11 17:29:08 truenas kernel: scsi 13:0:15:0: enclosure logical id (0x500a098006c93b00), slot(15) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:15:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:16:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:16:0: SSP: handle(0x001a), sas_addr(0x5000cca260a56871), phy(24), device_name(0x5000cca260a56873)
Feb 11 17:29:08 truenas kernel: scsi 13:0:16:0: enclosure logical id (0x500a098006c93b00), slot(16) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:16:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:17:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:17:0: SSP: handle(0x001b), sas_addr(0x5000cca260a55ef5), phy(25), device_name(0x5000cca260a55ef7)
Feb 11 17:29:08 truenas kernel: scsi 13:0:17:0: enclosure logical id (0x500a098006c93b00), slot(17) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:17:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:18:0: Direct-Access     SEAGATE  ST6000NM0054     NE03 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:18:0: SSP: handle(0x001c), sas_addr(0x5000c500849f8e15), phy(26), device_name(0x5000c500849f8e14)
Feb 11 17:29:08 truenas kernel: scsi 13:0:18:0: enclosure logical id (0x500a098006c93b00), slot(18) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:18:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:19:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:19:0: SSP: handle(0x001d), sas_addr(0x5000cca254b059f1), phy(27), device_name(0x5000cca254b059f3)
Feb 11 17:29:08 truenas kernel: scsi 13:0:19:0: enclosure logical id (0x500a098006c93b00), slot(19) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:19:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:20:0: Direct-Access     HGST     H7280A520SUN8.0T PD51 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:20:0: SSP: handle(0x001e), sas_addr(0x5000cca2549da429), phy(28), device_name(0x5000cca2549da42b)
Feb 11 17:29:08 truenas kernel: scsi 13:0:20:0: enclosure logical id (0x500a098006c93b00), slot(20) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:20:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:21:0: Direct-Access     NETAPP   X318_HARHE08TA07 NA00 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:21:0: SSP: handle(0x001f), sas_addr(0x5000cca2609eb619), phy(29), device_name(0x5000cca2609eb61b)
Feb 11 17:29:08 truenas kernel: scsi 13:0:21:0: enclosure logical id (0x500a098006c93b00), slot(21) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:21:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:22:0: Direct-Access     SEAGATE  ST6000NM0054     NE03 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:22:0: SSP: handle(0x0020), sas_addr(0x5000c500849f906d), phy(30), device_name(0x5000c500849f906c)
Feb 11 17:29:08 truenas kernel: scsi 13:0:22:0: enclosure logical id (0x500a098006c93b00), slot(22) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:22:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:23:0: Direct-Access     HP       MB8000JEQVA      HPD7 PQ: 0 ANSI: 6
Feb 11 17:29:08 truenas kernel: scsi 13:0:23:0: SSP: handle(0x0021), sas_addr(0x5000cca254cc1c9d), phy(31), device_name(0x5000cca254cc1c9f)
Feb 11 17:29:08 truenas kernel: scsi 13:0:23:0: enclosure logical id (0x500a098006c93b00), slot(23) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:23:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 13:0:24:0: Enclosure         NETAPP   DS424IOM6        0191 PQ: 0 ANSI: 5
Feb 11 17:29:08 truenas kernel: scsi 13:0:24:0: set ignore_delay_remove for handle(0x0022)
Feb 11 17:29:08 truenas kernel: scsi 13:0:24:0: SES: handle(0x0022), sas_addr(0x500a09800638aefe), phy(36), device_name(0x0000000000000000)
Feb 11 17:29:08 truenas kernel: scsi 13:0:24:0: enclosure logical id (0x500a098006c93b00), slot(0) 
Feb 11 17:29:08 truenas kernel: scsi 13:0:24:0: qdepth(254), tagged(1), scsi_level(6), cmd_que(1)
Feb 11 17:29:08 truenas kernel: scsi 11:0:0:0: Direct-Access     ATA      PNY CS900 120GB  0E12 PQ: 0 ANSI: ```

CatchyGeo · May 1, 2024, 1:00pm

Update. Still having drives marked as degraded and failed after removing two files that showed up under zpool status -v