Possibly lost pool, all advice appreciated

TrueNAS General
,
1

Running ElectricEel-24.10.0
I have two main raidz1 pools storage1 and storage2. 1vdev and 4 disks each. I came home one evening to find storage1 had a disk failure but otherwise was fine and storage2 though apparently online had its vdev ‘exported’. This was before christmas just gone and I didn’t have the disks or funds for them or time to dig into it until recently, so I shut down the server and just now have added a few more disks and moved everything to a better case. storage1 is sorted.

image
image1409×784 95.5 KB

On storage2 I’ve run SMART long on all of the disks and all are reporting fine.
The pool looks to be FUBAR but I think (hope) there’s more to it…
image
image706×262 14.1 KB

storage2 does not show up using zpool status -v but the labels look ok to me

image
image642×1007 38 KB

image
image646×1012 37.2 KB

image
image639×1003 36.9 KB

image
image647×1007 37.3 KB

My thinking here is that a glitch made it appear as if two disks went bad on storage2 and so the pool is in this state. But the disks look good.

What I want to do if possible is:

  • mount it read only
  • copy the data to a temp pool
  • re-burn in the 4 disks with a round of badblocks
  • re-create pool and move data back if all is well.

Granting that the disks are actually ok, what is the best way to achieve this?
REM this is a media library, too big (expensive) to have backups unfortunately.

Many thanks in advance for your input!

2

The disks are UNAVAILABLE i.e. ZFS cannot see them.

This could be because they catastrophically failed or perhaps there is some other reason (cables loose, bad PSU etc.).

Please power cycle your NAS and when it comes back up run the following commands and post the results here with the results from each command inside a separate </> box:

  • lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID
  • sudo zpool status -v
  • sudo zpool import
  • lspci
  • sudo sas2flash -list
  • sudo sas3flash -list
  • sudo storcli show all
  • sudo smartctl -x /dev/sdX replacing sdX with the device name for each of the disks in the storage2 pool.

Thx.

1 Like
3

Thanks for your help Protopia.

image
image891×820 24 KB

root@truenas[~]# lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID
NAME        MODEL                     ROTA PTTYPE TYPE    START           SIZE PARTTYPENAME             PARTUUID
sda         ST18000NM000J-2TV103         1 gpt    disk          18000207937536                          
├─sda1                                   1 gpt    part      128     2147418624 Linux swap               de66399f-0974-4692-bf00-99ccf0e6e9fd
└─sda2                                   1 gpt    part  4194432 17998060371456 Solaris /usr & Apple ZFS 43133b2a-0ae2-4104-b4dc-686d734cbb42
sdb         ST18000NM000J-2TV103         1        disk          18000207937536                          
sdc         ST18000NM000J-2TV103         1 gpt    disk          18000207937536                          
├─sdc1                                   1 gpt    part      128     2147418624 Linux swap               840940a7-ffbd-45c2-9dfa-8671d02bfe17
└─sdc2                                   1 gpt    part  4194432 17998060371456 Solaris /usr & Apple ZFS 9cdf928f-7d07-4e26-a103-05f58bfc7171
sdd         ST18000NM000J-2TV103         1 gpt    disk          18000207937536                          
├─sdd1                                   1 gpt    part      128     2147418624 Linux swap               612983c6-22d2-41bb-ae68-2730dd37d93a
└─sdd2                                   1 gpt    part  4194432 17998060371456 Solaris /usr & Apple ZFS 77292dd5-47c2-4bb9-87bd-7c9d8b276a06
sde         ST16000NM001G-2KK103         1 gpt    disk          16000900661248                          
└─sde1                                   1 gpt    part     2048 16000898564096 Solaris /usr & Apple ZFS 76bd052b-d1fb-43d4-a3f0-a08b0e96d8a4
sdf         ST16000NM001G-2KK103         1 gpt    disk          16000900661248                          
└─sdf1                                   1 gpt    part     2048 16000898564096                          
sdg         ST16000NM001G-2KK103         1 gpt    disk          16000900661248                          
└─sdg1                                   1 gpt    part     2048 16000898564096                          
sdh         ST20000NT001-3LT101          1        disk          20000588955648                          
sdi         INTENSO                      0 gpt    disk            128035676160                          
├─sdi1                                   0 gpt    part     4096        1048576 BIOS boot                89b220c5-4ac7-4956-87f9-42bcebd218b9
├─sdi2                                   0 gpt    part     6144      536870912 EFI System               b3106da1-2354-4860-80f3-168058816b61
├─sdi3                                   0 gpt    part 34609152   110315773440 Solaris /usr & Apple ZFS cb0bf26f-a5a5-436a-9875-12e46e880711
└─sdi4                                   0 gpt    part  1054720    17179869184 Linux swap               b18d5629-c191-465b-9e53-8df71365a6a0
sdj         ST16000NM001G-2KK103         1 gpt    disk          16000900661248                          
└─sdj1                                   1 gpt    part     2048 16000898564096 Solaris /usr & Apple ZFS 92f5e246-b303-4c06-9fa1-d4ac188577bd
sdk         Verbatim Vi550 S3            0 gpt    disk            128035676160                          
├─sdk1                                   0 gpt    part     4096        1048576 BIOS boot                2e7974ac-494a-44d4-9b6c-e235ecdef046
├─sdk2                                   0 gpt    part     6144      536870912 EFI System               9da7b734-e3f4-4e8e-a05e-0e4cbfb98005
├─sdk3                                   0 gpt    part 34609152   110315773440 Solaris /usr & Apple ZFS c2c4a03d-3643-42e2-bc1b-354d7c09cf1b
└─sdk4                                   0 gpt    part  1054720    17179869184 Linux swap               fbe85d2e-cc8c-4f29-b3f5-275a6852f06e
sdl         ST20000NT001-3LT101          1 gpt    disk          20000588955648                          
└─sdl1                                   1 gpt    part     2048 20000586858496 Solaris /usr & Apple ZFS 7912023c-7e6b-4bef-8445-3cef8f64fefe
nvme0n1     Patriot M.2 P300 256GB       0 gpt    disk            256060514304                          
├─nvme0n1p1                              0 gpt    part      128     2147418624 Linux swap               3d1ae95e-5f7f-4b5b-aee8-c066bccece1f
└─nvme0n1p2                              0 gpt    part  4194432   253912948224 Solaris /usr & Apple ZFS 91de82e0-1bb0-4243-bda9-a8dd7fe767b9
nvme1n1     Samsung SSD 960 EVO 250GB    0 gpt    disk            250059350016                          
├─nvme1n1p1                              0 gpt    part      128     2147418624 Linux swap               14dab88a-d357-440c-b2ca-3eac736c28a0
└─nvme1n1p2                              0 gpt    part  4194432   247911783936 Solaris /usr & Apple ZFS 9696510a-3062-4d0c-84cd-9011608ecc36
root@truenas[~]# 

root@truenas[~]# zpool status -v
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:21:40 with 0 errors on Sun Apr 20 04:06:42 2025
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdk3    ONLINE       0     0     0
            sdi3    ONLINE       0     0     0

errors: No known data errors

  pool: fastaccess
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:05:07 with 0 errors on Sun Apr 20 00:05:08 2025
config:

        NAME                                      STATE     READ WRITE CKSUM
        fastaccess                                ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            9696510a-3062-4d0c-84cd-9011608ecc36  ONLINE       0     0     0
            91de82e0-1bb0-4243-bda9-a8dd7fe767b9  ONLINE       0     0     0

errors: No known data errors

  pool: storage1
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: resilvered 6.68T in 13:03:57 with 0 errors on Tue Apr 22 04:42:12 2025
config:

        NAME                                      STATE     READ WRITE CKSUM
        storage1                                  ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            7912023c-7e6b-4bef-8445-3cef8f64fefe  ONLINE       0     0     0
            43133b2a-0ae2-4104-b4dc-686d734cbb42  ONLINE       0     0     0
            9cdf928f-7d07-4e26-a103-05f58bfc7171  ONLINE       0     0     0
            77292dd5-47c2-4bb9-87bd-7c9d8b276a06  ONLINE       0     0     0

errors: No known data errors

root@truenas[~]# zpool import
  pool: storage2
    id: 8231419487065136889
 state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E
config:

        storage2                                  UNAVAIL  insufficient replicas
          raidz1-0                                UNAVAIL  insufficient replicas
            091ae219-2f40-4697-900b-7b1ac769488b  UNAVAIL
            76bd052b-d1fb-43d4-a3f0-a08b0e96d8a4  ONLINE
            bbee6ac7-da9c-4c68-8212-b9afde9e6e98  UNAVAIL
            92f5e246-b303-4c06-9fa1-d4ac188577bd  ONLINE

root@truenas[~]# lspci
00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 (rev 03)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 03)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 03)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 03)
00:03.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 03)
00:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 03)
00:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 03)
00:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 03)
00:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC (rev 03)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:16.1 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #2 (rev 04)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5)
00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C224 Series Chipset Family Server Standard SKU LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
02:00.0 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 0
02:00.1 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 1
02:00.2 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 2
02:00.3 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 3
04:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
07:00.0 Non-Volatile memory controller: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 (rev 01)
08:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961/SM963
09:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 12)
0a:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
0a:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
ff:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 03)
ff:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 03)
ff:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 03)
ff:0b.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link Debug (rev 03)
ff:0c.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0c.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0c.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0c.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0c.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0c.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0c.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0c.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0f.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0f.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:0f.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 03)
ff:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 03)
ff:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 03)
ff:10.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 03)
ff:10.6 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 03)
ff:10.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 03)
ff:12.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 03)
ff:12.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 03)
ff:13.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 03)
ff:13.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 03)
ff:13.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 03)
ff:13.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 03)
ff:13.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 03)
ff:13.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 03)
ff:13.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast (rev 03)
ff:13.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 03)
ff:14.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control (rev 03)
ff:14.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control (rev 03)
ff:14.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error (rev 03)
ff:14.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error (rev 03)
ff:14.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)
ff:14.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)
ff:14.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)
ff:14.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 03)
ff:15.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Thermal Control (rev 03)
ff:15.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Thermal Control (rev 03)
ff:15.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Error (rev 03)
ff:15.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Error (rev 03)
ff:1e.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
ff:1e.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
ff:1e.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
ff:1e.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
ff:1e.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
ff:1f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)
ff:1f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 03)

root@truenas[~]# sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18) 
Copyright (c) 2008-2014 LSI Corporation. All rights reserved 

        Adapter Selected is a LSI SAS: SAS2308_2(D1) 

        Controller Number              : 0
        Controller                     : SAS2308_2(D1) 
        PCI Address                    : 00:01:00:00
        SAS Address                    : 500605b-0-0b13-3220
        NVDATA Version (Default)       : 14.01.00.06
        NVDATA Version (Persistent)    : 14.01.00.06
        Firmware Product ID            : 0x2214 (IT)
        Firmware Version               : 20.00.07.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9207-8i
        BIOS Version                   : 07.39.02.00
        UEFI BSD Version               : 07.27.01.00
        FCODE Version                  : N/A
        Board Name                     : SAS9207-8i
        Board Assembly                 : H3-25412-00K
        Board Tracer Number            : SP55109174

        Finished Processing Commands Successfully.
        Exiting SAS2Flash.

root@truenas[~]# sas3flash -list  
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02) 
Copyright 2008-2017 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

        Controller Number              : 0
        Controller                     : SAS3008(C0)
        PCI Address                    : 00:04:00:00
        SAS Address                    : 5d05099-0-0005-2908
        NVDATA Version (Default)       : 0e.01.00.07
        NVDATA Version (Persistent)    : 0e.01.00.07
        Firmware Product ID            : 0x2221 (IT)
        Firmware Version               : 16.00.12.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9300-8i
        BIOS Version                   : 08.37.02.00
        UEFI BSD Version               : 18.00.00.00
        FCODE Version                  : N/A
        Board Name                     : ASR3008
        Board Assembly                 : N/A
        Board Tracer Number            : N/A

        Finished Processing Commands Successfully.
        Exiting SAS3Flash.

root@truenas[~]# storcli show all
CLI Version = 007.2807.0000.0000 Dec 22, 2023
Operating system = Linux 6.6.44-production+truenas
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = truenas
Operating System  = Linux 6.6.44-production+truenas
StoreLib IT Version = 07.2900.0200.0100
4 
root@truenas[~]# smartctl -x /dev/sde
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Exos X16
Device Model:     ST16000NM001G-2KK103
Serial Number:    ZL2NY19T
LU WWN Device Id: 5 000c50 0e3fe6bb1
Firmware Version: SB40
User Capacity:    16,000,900,661,248 bytes [16.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5671
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 23 17:36:35 2025 IST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Disabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Write SCT (Get) Feature Control Command failed: scsi error aborted command
Wt Cache Reorder: Unknown (SCT Feature Control command failed)

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (  567) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (1439) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   078   064   044    -    66876978
  3 Spin_Up_Time            PO----   090   090   000    -    0
  4 Start_Stop_Count        -O--CK   100   100   020    -    15
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR--   074   060   045    -    26458506
  9 Power_On_Hours          -O--CK   099   099   000    -    1011
 10 Spin_Retry_Count        PO--C-   100   100   097    -    0
 12 Power_Cycle_Count       -O--CK   100   100   020    -    10
 18 Head_Health             PO-R--   100   100   050    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   070   050   040    -    30 (Min/Max 27/40)
192 Power-Off_Retract_Count -O--CK   100   100   000    -    4
193 Load_Cycle_Count        -O--CK   099   099   000    -    2152
194 Temperature_Celsius     -O---K   030   048   000    -    30 (0 18 0 0 0)
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   200   200   000    -    0
200 Pressure_Limit          PO---K   100   100   001    -    0
240 Head_Flying_Hours       ------   100   253   000    -    129h+44m+34.276s
241 Total_LBAs_Written      ------   100   253   000    -    15226709496
242 Total_LBAs_Read         ------   100   253   000    -    31746443369
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      5  Ext. Comprehensive SMART error log
0x04       GPL     R/O    256  Device Statistics log
0x04       SL      R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x0a       GPL     R/W      8  Device Statistics Notification
0x0c       GPL     R/O   2048  Pending Defects log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x13       GPL     R/O      1  SATA NCQ Send and Receive log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    768  Current Device Internal Status Data log
0x2f       GPL     R/O      1  Set Sector Configuration
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa1       GPL,SL  VS     160  Device vendor specific log
0xa2       GPL     VS   16320  Device vendor specific log
0xa4       GPL,SL  VS     160  Device vendor specific log
0xa6       GPL     VS     192  Device vendor specific log
0xa8-0xa9  GPL,SL  VS     136  Device vendor specific log
0xab       GPL     VS       1  Device vendor specific log
0xad       GPL     VS      16  Device vendor specific log
0xbe-0xbf  GPL     VS   65535  Device vendor specific log
0xc1       GPL,SL  VS       8  Device vendor specific log
0xc3       GPL,SL  VS      56  Device vendor specific log
0xc7       GPL,SL  VS       8  Device vendor specific log
0xc9       GPL,SL  VS       8  Device vendor specific log
0xca       GPL,SL  VS      16  Device vendor specific log
0xcd       GPL,SL  VS       1  Device vendor specific log
0xce       GPL     VS       1  Device vendor specific log
0xcf       GPL     VS     512  Device vendor specific log
0xd1       GPL     VS     656  Device vendor specific log
0xd2       GPL     VS   10000  Device vendor specific log
0xd4       GPL     VS    2048  Device vendor specific log
0xda       GPL,SL  VS       1  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer
0xe2       GPL,SL  -      160  Reserved

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%       991         -
# 2  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       522 (0x020a)
Device State:                        Active (0)
Current Temperature:                    30 Celsius
Power Cycle Min/Max Temperature:     27/40 Celsius
Lifetime    Min/Max Temperature:     18/49 Celsius
Under/Over Temperature Limit Count:   0/0
SMART Status:                        0xc24f (PASSED)
Vendor specific:
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         4 minutes
Temperature Logging Interval:        59 minutes
Min/Max recommended Temperature:     10/40 Celsius
Min/Max Temperature Limit:            5/60 Celsius
Temperature History Size (Index):    128 (80)

Index    Estimated Time   Temperature Celsius
  81    2025-04-18 12:29    35  ****************
  82    2025-04-18 13:28    35  ****************
  83    2025-04-18 14:27    34  ***************
 ...    ..(  2 skipped).    ..  ***************
  86    2025-04-18 17:24    34  ***************
  87    2025-04-18 18:23    33  **************
  88    2025-04-18 19:22    33  **************
  89    2025-04-18 20:21    33  **************
  90    2025-04-18 21:20    34  ***************
  91    2025-04-18 22:19    35  ****************
  92    2025-04-18 23:18    35  ****************
  93    2025-04-19 00:17    35  ****************
  94    2025-04-19 01:16    34  ***************
  95    2025-04-19 02:15    35  ****************
  96    2025-04-19 03:14    35  ****************
  97    2025-04-19 04:13    35  ****************
  98    2025-04-19 05:12    34  ***************
  99    2025-04-19 06:11    34  ***************
 100    2025-04-19 07:10    34  ***************
 101    2025-04-19 08:09    33  **************
 102    2025-04-19 09:08    34  ***************
 ...    ..(  6 skipped).    ..  ***************
 109    2025-04-19 16:01    34  ***************
 110    2025-04-19 17:00    33  **************
 ...    ..(  3 skipped).    ..  **************
 114    2025-04-19 20:56    33  **************
 115    2025-04-19 21:55    34  ***************
 116    2025-04-19 22:54    34  ***************
 117    2025-04-19 23:53    35  ****************
 118    2025-04-20 00:52     ?  -
 119    2025-04-20 01:51    18  -
 120    2025-04-20 02:50    28  *********
 121    2025-04-20 03:49     ?  -
 122    2025-04-20 04:48    19  -
 123    2025-04-20 05:47    30  ***********
 124    2025-04-20 06:46    30  ***********
 125    2025-04-20 07:45    29  **********
 ...    ..(  2 skipped).    ..  **********
   0    2025-04-20 10:42    29  **********
   1    2025-04-20 11:41    28  *********
 ...    ..(  2 skipped).    ..  *********
   4    2025-04-20 14:38    28  *********
   5    2025-04-20 15:37    29  **********
   6    2025-04-20 16:36    28  *********
   7    2025-04-20 17:35    29  **********
   8    2025-04-20 18:34    31  ************
   9    2025-04-20 19:33    30  ***********
  10    2025-04-20 20:32    29  **********
  11    2025-04-20 21:31    29  **********
  12    2025-04-20 22:30    27  ********
  13    2025-04-20 23:29    27  ********
  14    2025-04-21 00:28    27  ********
  15    2025-04-21 01:27    26  *******
 ...    ..(  3 skipped).    ..  *******
  19    2025-04-21 05:23    26  *******
  20    2025-04-21 06:22    27  ********
  21    2025-04-21 07:21    26  *******
  22    2025-04-21 08:20    27  ********
 ...    ..(  2 skipped).    ..  ********
  25    2025-04-21 11:17    27  ********
  26    2025-04-21 12:16    26  *******
 ...    ..(  2 skipped).    ..  *******
  29    2025-04-21 15:13    26  *******
  30    2025-04-21 16:12    27  ********
  31    2025-04-21 17:11    28  *********
 ...    ..(  2 skipped).    ..  *********
  34    2025-04-21 20:08    28  *********
  35    2025-04-21 21:07     ?  -
  36    2025-04-21 22:06    27  ********
  37    2025-04-21 23:05    32  *************
  38    2025-04-22 00:04    31  ************
  39    2025-04-22 01:03    30  ***********
  40    2025-04-22 02:02    30  ***********
  41    2025-04-22 03:01    29  **********
  42    2025-04-22 04:00    29  **********
  43    2025-04-22 04:59    28  *********
  44    2025-04-22 05:58    28  *********
  45    2025-04-22 06:57    33  **************
  46    2025-04-22 07:56    38  *******************
  47    2025-04-22 08:55    40  *********************
 ...    ..(  2 skipped).    ..  *********************
  50    2025-04-22 11:52    40  *********************
  51    2025-04-22 12:51    39  ********************
 ...    ..(  2 skipped).    ..  ********************
  54    2025-04-22 15:48    39  ********************
  55    2025-04-22 16:47    38  *******************
 ...    ..(  8 skipped).    ..  *******************
  64    2025-04-23 01:38    38  *******************
  65    2025-04-23 02:37    37  ******************
  66    2025-04-23 03:36    33  **************
  67    2025-04-23 04:35    31  ************
  68    2025-04-23 05:34    31  ************
  69    2025-04-23 06:33    30  ***********
  70    2025-04-23 07:32    30  ***********
  71    2025-04-23 08:31    30  ***********
  72    2025-04-23 09:30    29  **********
 ...    ..(  7 skipped).    ..  **********
  80    2025-04-23 17:22    29  **********

SCT Error Recovery Control:
           Read:    100 (10.0 seconds)
          Write:    100 (10.0 seconds)

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4              10  ---  Lifetime Power-On Resets
0x01  0x010  4            1011  ---  Power-on Hours
0x01  0x018  6     15226709496  ---  Logical Sectors Written
0x01  0x020  6        22868482  ---  Number of Write Commands
0x01  0x028  6     31746431699  ---  Logical Sectors Read
0x01  0x030  6       248249994  ---  Number of Read Commands
0x01  0x038  6               -  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4            1010  ---  Spindle Motor Power-on Hours
0x03  0x010  4             161  ---  Head Flying Hours
0x03  0x018  4            2152  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x03  0x038  4               0  ---  Number of Realloc. Candidate Logical Sectors
0x03  0x040  4               4  ---  Number of High Priority Unload Events
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x04  0x018  4               0  -D-  Physical Element Status Changed
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              30  ---  Current Temperature
0x05  0x010  1              35  ---  Average Short Term Temperature
0x05  0x018  1               -  ---  Average Long Term Temperature
0x05  0x020  1              48  ---  Highest Temperature
0x05  0x028  1              26  ---  Lowest Temperature
0x05  0x030  1              45  ---  Highest Average Short Term Temperature
0x05  0x038  1              27  ---  Lowest Average Short Term Temperature
0x05  0x040  1               -  ---  Highest Average Long Term Temperature
0x05  0x048  1               -  ---  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              60  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               5  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4              35  ---  Number of Hardware Resets
0x06  0x010  4              27  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
0xff  =====  =               =  ===  == Vendor Specific Statistics (rev 1) ==
0xff  0x008  7               0  ---  Vendor Specific
0xff  0x010  7               0  ---  Vendor Specific
0xff  0x018  7               0  ---  Vendor Specific
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x000a  2            8  Device-to-host register FISes sent due to a COMRESET
0x0001  2            0  Command failed due to ICRC error
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS

Seagate FARM log (GP Log 0xa6) supported [try: -l farm]

root@truenas[~]# smartctl -x /dev/sdf
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Exos X16
Device Model:     ST16000NM001G-2KK103
Serial Number:    ZL2HQ05L
LU WWN Device Id: 5 000c50 0db55bed8
Firmware Version: SN03
User Capacity:    16,000,900,661,248 bytes [16.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5671
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 23 17:37:06 2025 IST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Disabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Write SCT (Get) Feature Control Command failed: scsi error aborted command
Wt Cache Reorder: Unknown (SCT Feature Control command failed)

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (  567) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (1404) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   083   065   044    -    188215554
  3 Spin_Up_Time            PO----   090   090   000    -    0
  4 Start_Stop_Count        -O--CK   100   100   020    -    13
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR--   070   061   045    -    12919272999
  9 Power_On_Hours          -O--CK   099   099   000    -    1082
 10 Spin_Retry_Count        PO--C-   100   100   097    -    0
 12 Power_Cycle_Count       -O--CK   100   100   020    -    8
 18 Head_Health             PO-R--   100   100   050    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   070   051   040    -    30 (Min/Max 27/33)
192 Power-Off_Retract_Count -O--CK   100   100   000    -    5
193 Load_Cycle_Count        -O--CK   099   099   000    -    2114
194 Temperature_Celsius     -O---K   030   049   000    -    30 (0 18 0 0 0)
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   200   200   000    -    0
200 Pressure_Limit          PO---K   100   100   001    -    0
240 Head_Flying_Hours       ------   100   253   000    -    148h+56m+09.538s
241 Total_LBAs_Written      ------   100   253   000    -    15532818896
242 Total_LBAs_Read         ------   100   253   000    -    31343996038
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      5  Ext. Comprehensive SMART error log
0x04       GPL     R/O    256  Device Statistics log
0x04       SL      R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x0a       GPL     R/W      8  Device Statistics Notification
0x0c       GPL     R/O   2048  Pending Defects log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x13       GPL     R/O      1  SATA NCQ Send and Receive log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    768  Current Device Internal Status Data log
0x2f       GPL     R/O      1  Set Sector Configuration
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa1       GPL,SL  VS     160  Device vendor specific log
0xa2       GPL     VS   16320  Device vendor specific log
0xa4       GPL,SL  VS     160  Device vendor specific log
0xa6       GPL     VS     192  Device vendor specific log
0xa8-0xa9  GPL,SL  VS     136  Device vendor specific log
0xab       GPL     VS       1  Device vendor specific log
0xad       GPL     VS      16  Device vendor specific log
0xbe-0xbf  GPL     VS   65535  Device vendor specific log
0xc1       GPL,SL  VS       8  Device vendor specific log
0xc3       GPL,SL  VS      56  Device vendor specific log
0xc7       GPL,SL  VS       8  Device vendor specific log
0xc9       GPL,SL  VS       8  Device vendor specific log
0xca       GPL,SL  VS      16  Device vendor specific log
0xcd       GPL,SL  VS       1  Device vendor specific log
0xce       GPL     VS       1  Device vendor specific log
0xcf       GPL     VS     512  Device vendor specific log
0xd1       GPL     VS     656  Device vendor specific log
0xd2       GPL     VS   10000  Device vendor specific log
0xd4       GPL     VS    2048  Device vendor specific log
0xda       GPL,SL  VS       1  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer
0xe2       GPL,SL  -      160  Reserved

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      1011         -
# 2  Conveyance offline  Completed without error       00%       990         -
# 3  Short offline       Completed without error       00%       990         -
# 4  Extended offline    Interrupted (host reset)      30%        58         -
# 5  Extended offline    Aborted by host               90%        42         -
# 6  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       522 (0x020a)
Device State:                        Active (0)
Current Temperature:                    30 Celsius
Power Cycle Min/Max Temperature:     27/33 Celsius
Lifetime    Min/Max Temperature:     18/49 Celsius
Under/Over Temperature Limit Count:   0/0
SMART Status:                        0xc24f (PASSED)
Vendor specific:
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         4 minutes
Temperature Logging Interval:        59 minutes
Min/Max recommended Temperature:     10/40 Celsius
Min/Max Temperature Limit:            5/60 Celsius
Temperature History Size (Index):    128 (15)

Index    Estimated Time   Temperature Celsius
  16    2025-04-18 12:29    32  *************
 ...    ..(  2 skipped).    ..  *************
  19    2025-04-18 15:26    32  *************
  20    2025-04-18 16:25    33  **************
  21    2025-04-18 17:24    33  **************
  22    2025-04-18 18:23    34  ***************
  23    2025-04-18 19:22    34  ***************
  24    2025-04-18 20:21    33  **************
  25    2025-04-18 21:20    33  **************
  26    2025-04-18 22:19    34  ***************
  27    2025-04-18 23:18    33  **************
  28    2025-04-19 00:17    33  **************
  29    2025-04-19 01:16    33  **************
  30    2025-04-19 02:15    32  *************
 ...    ..(  3 skipped).    ..  *************
  34    2025-04-19 06:11    32  *************
  35    2025-04-19 07:10    33  **************
 ...    ..(  2 skipped).    ..  **************
  38    2025-04-19 10:07    33  **************
  39    2025-04-19 11:06    32  *************
 ...    ..(  5 skipped).    ..  *************
  45    2025-04-19 17:00    32  *************
  46    2025-04-19 17:59    33  **************
  47    2025-04-19 18:58    34  ***************
  48    2025-04-19 19:57     ?  -
  49    2025-04-19 20:56    18  -
  50    2025-04-19 21:55    27  ********
  51    2025-04-19 22:54     ?  -
  52    2025-04-19 23:53    19  -
  53    2025-04-20 00:52    30  ***********
  54    2025-04-20 01:51    37  ******************
  55    2025-04-20 02:50    37  ******************
  56    2025-04-20 03:49    37  ******************
  57    2025-04-20 04:48    36  *****************
 ...    ..(  2 skipped).    ..  *****************
  60    2025-04-20 07:45    36  *****************
  61    2025-04-20 08:44    35  ****************
  62    2025-04-20 09:43    35  ****************
  63    2025-04-20 10:42    36  *****************
  64    2025-04-20 11:41    36  *****************
  65    2025-04-20 12:40    37  ******************
  66    2025-04-20 13:39    38  *******************
  67    2025-04-20 14:38    36  *****************
  68    2025-04-20 15:37    35  ****************
  69    2025-04-20 16:36    34  ***************
  70    2025-04-20 17:35    34  ***************
  71    2025-04-20 18:34    34  ***************
  72    2025-04-20 19:33    33  **************
  73    2025-04-20 20:32    33  **************
  74    2025-04-20 21:31    27  ********
  75    2025-04-20 22:30    25  ******
  76    2025-04-20 23:29    24  *****
  77    2025-04-21 00:28    27  ********
 ...    ..(  6 skipped).    ..  ********
  84    2025-04-21 07:21    27  ********
  85    2025-04-21 08:20    26  *******
  86    2025-04-21 09:19    26  *******
  87    2025-04-21 10:18    26  *******
  88    2025-04-21 11:17    27  ********
  89    2025-04-21 12:16    28  *********
  90    2025-04-21 13:15    28  *********
  91    2025-04-21 14:14    28  *********
  92    2025-04-21 15:13    29  **********
  93    2025-04-21 16:12     ?  -
  94    2025-04-21 17:11    27  ********
  95    2025-04-21 18:10    32  *************
  96    2025-04-21 19:09    31  ************
  97    2025-04-21 20:08    31  ************
  98    2025-04-21 21:07    31  ************
  99    2025-04-21 22:06    30  ***********
 100    2025-04-21 23:05    29  **********
 101    2025-04-22 00:04    29  **********
 102    2025-04-22 01:03    28  *********
 103    2025-04-22 02:02    28  *********
 104    2025-04-22 03:01    29  **********
 105    2025-04-22 04:00    31  ************
 106    2025-04-22 04:59    31  ************
 107    2025-04-22 05:58    31  ************
 108    2025-04-22 06:57    30  ***********
 109    2025-04-22 07:56    30  ***********
 110    2025-04-22 08:55    30  ***********
 111    2025-04-22 09:54    29  **********
 112    2025-04-22 10:53    30  ***********
 113    2025-04-22 11:52    29  **********
 ...    ..(  4 skipped).    ..  **********
 118    2025-04-22 16:47    29  **********
 119    2025-04-22 17:46    30  ***********
 ...    ..(  2 skipped).    ..  ***********
 122    2025-04-22 20:43    30  ***********
 123    2025-04-22 21:42    29  **********
 124    2025-04-22 22:41    29  **********
 125    2025-04-22 23:40    30  ***********
 126    2025-04-23 00:39    31  ************
 127    2025-04-23 01:38    31  ************
   0    2025-04-23 02:37    30  ***********
   1    2025-04-23 03:36    30  ***********
   2    2025-04-23 04:35    30  ***********
   3    2025-04-23 05:34    29  **********
 ...    ..(  3 skipped).    ..  **********
   7    2025-04-23 09:30    29  **********
   8    2025-04-23 10:29    28  *********
 ...    ..(  4 skipped).    ..  *********
  13    2025-04-23 15:24    28  *********
  14    2025-04-23 16:23    29  **********
  15    2025-04-23 17:22    29  **********

SCT Error Recovery Control:
           Read:    100 (10.0 seconds)
          Write:    100 (10.0 seconds)

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4               8  ---  Lifetime Power-On Resets
0x01  0x010  4            1082  ---  Power-on Hours
0x01  0x018  6     15532818896  ---  Logical Sectors Written
0x01  0x020  6        49185577  ---  Number of Write Commands
0x01  0x028  6     31343978759  ---  Logical Sectors Read
0x01  0x030  6       245103402  ---  Number of Read Commands
0x01  0x038  6               -  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4            1025  ---  Spindle Motor Power-on Hours
0x03  0x010  4             197  ---  Head Flying Hours
0x03  0x018  4            2114  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x03  0x038  4               0  ---  Number of Realloc. Candidate Logical Sectors
0x03  0x040  4               5  ---  Number of High Priority Unload Events
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x04  0x018  4               0  -D-  Physical Element Status Changed
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              30  ---  Current Temperature
0x05  0x010  1              29  ---  Average Short Term Temperature
0x05  0x018  1               -  ---  Average Long Term Temperature
0x05  0x020  1              49  ---  Highest Temperature
0x05  0x028  1              24  ---  Lowest Temperature
0x05  0x030  1              40  ---  Highest Average Short Term Temperature
0x05  0x038  1              26  ---  Lowest Average Short Term Temperature
0x05  0x040  1               -  ---  Highest Average Long Term Temperature
0x05  0x048  1               -  ---  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              60  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               5  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4              16  ---  Number of Hardware Resets
0x06  0x010  4              10  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
0xff  =====  =               =  ===  == Vendor Specific Statistics (rev 1) ==
0xff  0x008  7               0  ---  Vendor Specific
0xff  0x010  7               0  ---  Vendor Specific
0xff  0x018  7               0  ---  Vendor Specific
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x000a  2            8  Device-to-host register FISes sent due to a COMRESET
0x0001  2            0  Command failed due to ICRC error
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS

Seagate FARM log (GP Log 0xa6) supported [try: -l farm]

root@truenas[~]# smartctl -x /dev/sdg
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Exos X16
Device Model:     ST16000NM001G-2KK103
Serial Number:    ZL2H1691
LU WWN Device Id: 5 000c50 0db6e4ec9
Firmware Version: SN03
User Capacity:    16,000,900,661,248 bytes [16.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5671
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 23 17:37:36 2025 IST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Disabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Write SCT (Get) Feature Control Command failed: scsi error aborted command
Wt Cache Reorder: Unknown (SCT Feature Control command failed)

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (  559) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (1367) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   082   067   044    -    175547746
  3 Spin_Up_Time            PO----   092   092   000    -    0
  4 Start_Stop_Count        -O--CK   100   100   020    -    6
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR--   074   061   045    -    27766200
  9 Power_On_Hours          -O--CK   099   099   000    -    1008
 10 Spin_Retry_Count        PO--C-   100   100   097    -    0
 12 Power_Cycle_Count       -O--CK   100   100   020    -    6
 18 Head_Health             PO-R--   100   100   050    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   069   052   040    -    31 (Min/Max 27/43)
192 Power-Off_Retract_Count -O--CK   100   100   000    -    3
193 Load_Cycle_Count        -O--CK   099   099   000    -    2111
194 Temperature_Celsius     -O---K   031   048   000    -    31 (0 18 0 0 0)
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   200   200   000    -    0
200 Pressure_Limit          PO---K   100   100   001    -    0
240 Head_Flying_Hours       ------   100   253   000    -    133h+50m+27.521s
241 Total_LBAs_Written      ------   100   253   000    -    15532804168
242 Total_LBAs_Read         ------   100   253   000    -    31345654982
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      5  Ext. Comprehensive SMART error log
0x04       GPL     R/O    256  Device Statistics log
0x04       SL      R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x0a       GPL     R/W      8  Device Statistics Notification
0x0c       GPL     R/O   2048  Pending Defects log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x13       GPL     R/O      1  SATA NCQ Send and Receive log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    768  Current Device Internal Status Data log
0x2f       GPL     R/O      1  Set Sector Configuration
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa1       GPL,SL  VS     160  Device vendor specific log
0xa2       GPL     VS   16320  Device vendor specific log
0xa4       GPL,SL  VS     160  Device vendor specific log
0xa6       GPL     VS     192  Device vendor specific log
0xa8-0xa9  GPL,SL  VS     136  Device vendor specific log
0xab       GPL     VS       1  Device vendor specific log
0xad       GPL     VS      16  Device vendor specific log
0xbe-0xbf  GPL     VS   65535  Device vendor specific log
0xc1       GPL,SL  VS       8  Device vendor specific log
0xc3       GPL,SL  VS      56  Device vendor specific log
0xc7       GPL,SL  VS       8  Device vendor specific log
0xc9       GPL,SL  VS       8  Device vendor specific log
0xca       GPL,SL  VS      16  Device vendor specific log
0xcd       GPL,SL  VS       1  Device vendor specific log
0xce       GPL     VS       1  Device vendor specific log
0xcf       GPL     VS     512  Device vendor specific log
0xd1       GPL     VS     656  Device vendor specific log
0xd2       GPL     VS   10000  Device vendor specific log
0xd4       GPL     VS    2048  Device vendor specific log
0xda       GPL,SL  VS       1  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer
0xe2       GPL,SL  -      160  Reserved

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%       987         -
# 2  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       522 (0x020a)
Device State:                        Active (0)
Current Temperature:                    31 Celsius
Power Cycle Min/Max Temperature:     27/43 Celsius
Lifetime    Min/Max Temperature:     18/48 Celsius
Under/Over Temperature Limit Count:   0/19
SMART Status:                        0xc24f (PASSED)
Vendor specific:
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         4 minutes
Temperature Logging Interval:        59 minutes
Min/Max recommended Temperature:     10/40 Celsius
Min/Max Temperature Limit:            5/60 Celsius
Temperature History Size (Index):    128 (62)

Index    Estimated Time   Temperature Celsius
  63    2025-04-18 12:29    34  ***************
  64    2025-04-18 13:28    33  **************
  65    2025-04-18 14:27    33  **************
  66    2025-04-18 15:26    33  **************
  67    2025-04-18 16:25    34  ***************
  68    2025-04-18 17:24    34  ***************
  69    2025-04-18 18:23    35  ****************
 ...    ..(  4 skipped).    ..  ****************
  74    2025-04-18 23:18    35  ****************
  75    2025-04-19 00:17    34  ***************
  76    2025-04-19 01:16    34  ***************
  77    2025-04-19 02:15    34  ***************
  78    2025-04-19 03:14    33  **************
  79    2025-04-19 04:13    33  **************
  80    2025-04-19 05:12    33  **************
  81    2025-04-19 06:11    34  ***************
 ...    ..(  4 skipped).    ..  ***************
  86    2025-04-19 11:06    34  ***************
  87    2025-04-19 12:05    33  **************
 ...    ..(  4 skipped).    ..  **************
  92    2025-04-19 17:00    33  **************
  93    2025-04-19 17:59    34  ***************
  94    2025-04-19 18:58    35  ****************
  95    2025-04-19 19:57     ?  -
  96    2025-04-19 20:56    18  -
  97    2025-04-19 21:55    28  *********
  98    2025-04-19 22:54     ?  -
  99    2025-04-19 23:53    19  -
 100    2025-04-20 00:52    31  ************
 101    2025-04-20 01:51    30  ***********
 102    2025-04-20 02:50    30  ***********
 103    2025-04-20 03:49    29  **********
 ...    ..(  2 skipped).    ..  **********
 106    2025-04-20 06:46    29  **********
 107    2025-04-20 07:45    28  *********
 108    2025-04-20 08:44    28  *********
 109    2025-04-20 09:43    28  *********
 110    2025-04-20 10:42    29  **********
 111    2025-04-20 11:41    29  **********
 112    2025-04-20 12:40    29  **********
 113    2025-04-20 13:39    32  *************
 114    2025-04-20 14:38    30  ***********
 115    2025-04-20 15:37    29  **********
 116    2025-04-20 16:36    29  **********
 117    2025-04-20 17:35    28  *********
 118    2025-04-20 18:34    28  *********
 119    2025-04-20 19:33    27  ********
 120    2025-04-20 20:32    27  ********
 121    2025-04-20 21:31    28  *********
 122    2025-04-20 22:30    25  ******
 123    2025-04-20 23:29    24  *****
 124    2025-04-21 00:28    27  ********
 125    2025-04-21 01:27    27  ********
 126    2025-04-21 02:26    27  ********
 127    2025-04-21 03:25    28  *********
   0    2025-04-21 04:24    27  ********
   1    2025-04-21 05:23    27  ********
   2    2025-04-21 06:22    28  *********
   3    2025-04-21 07:21    27  ********
 ...    ..(  3 skipped).    ..  ********
   7    2025-04-21 11:17    27  ********
   8    2025-04-21 12:16    28  *********
   9    2025-04-21 13:15    29  **********
  10    2025-04-21 14:14    29  **********
  11    2025-04-21 15:13    30  ***********
  12    2025-04-21 16:12    29  **********
  13    2025-04-21 17:11     ?  -
  14    2025-04-21 18:10    27  ********
  15    2025-04-21 19:09    33  **************
  16    2025-04-21 20:08    32  *************
  17    2025-04-21 21:07    32  *************
  18    2025-04-21 22:06    31  ************
  19    2025-04-21 23:05    30  ***********
  20    2025-04-22 00:04    29  **********
  21    2025-04-22 01:03    29  **********
  22    2025-04-22 02:02    28  *********
  23    2025-04-22 03:01    31  ************
  24    2025-04-22 04:00    41  **********************
  25    2025-04-22 04:59    43  ************************
 ...    ..(  3 skipped).    ..  ************************
  29    2025-04-22 08:55    43  ************************
  30    2025-04-22 09:54    42  ***********************
 ...    ..(  3 skipped).    ..  ***********************
  34    2025-04-22 13:50    42  ***********************
  35    2025-04-22 14:49    41  **********************
 ...    ..(  6 skipped).    ..  **********************
  42    2025-04-22 21:42    41  **********************
  43    2025-04-22 22:41    36  *****************
  44    2025-04-22 23:40    32  *************
  45    2025-04-23 00:39    32  *************
  46    2025-04-23 01:38    31  ************
  47    2025-04-23 02:37    31  ************
  48    2025-04-23 03:36    31  ************
  49    2025-04-23 04:35    30  ***********
  50    2025-04-23 05:34    30  ***********
  51    2025-04-23 06:33    30  ***********
  52    2025-04-23 07:32    29  **********
 ...    ..(  7 skipped).    ..  **********
  60    2025-04-23 15:24    29  **********
  61    2025-04-23 16:23    30  ***********
  62    2025-04-23 17:22    30  ***********

SCT Error Recovery Control:
           Read:    100 (10.0 seconds)
          Write:    100 (10.0 seconds)

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4               6  ---  Lifetime Power-On Resets
0x01  0x010  4            1008  ---  Power-on Hours
0x01  0x018  6     15532804168  ---  Logical Sectors Written
0x01  0x020  6        49312987  ---  Number of Write Commands
0x01  0x028  6     31345640390  ---  Logical Sectors Read
0x01  0x030  6       245103554  ---  Number of Read Commands
0x01  0x038  6               -  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4            1008  ---  Spindle Motor Power-on Hours
0x03  0x010  4             182  ---  Head Flying Hours
0x03  0x018  4            2111  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x03  0x038  4               0  ---  Number of Realloc. Candidate Logical Sectors
0x03  0x040  4               3  ---  Number of High Priority Unload Events
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x04  0x018  4               0  -D-  Physical Element Status Changed
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              31  ---  Current Temperature
0x05  0x010  1              35  ---  Average Short Term Temperature
0x05  0x018  1               -  ---  Average Long Term Temperature
0x05  0x020  1              48  ---  Highest Temperature
0x05  0x028  1              24  ---  Lowest Temperature
0x05  0x030  1              45  ---  Highest Average Short Term Temperature
0x05  0x038  1              27  ---  Lowest Average Short Term Temperature
0x05  0x040  1               -  ---  Highest Average Long Term Temperature
0x05  0x048  1               -  ---  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              60  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               5  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4              16  ---  Number of Hardware Resets
0x06  0x010  4               8  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
0xff  =====  =               =  ===  == Vendor Specific Statistics (rev 1) ==
0xff  0x008  7               0  ---  Vendor Specific
0xff  0x010  7               0  ---  Vendor Specific
0xff  0x018  7               0  ---  Vendor Specific
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x000a  2            8  Device-to-host register FISes sent due to a COMRESET
0x0001  2            0  Command failed due to ICRC error
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS

Seagate FARM log (GP Log 0xa6) supported [try: -l farm]
5 
root@truenas[~]# smartctl -x /dev/sdj
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.44-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Exos X16
Device Model:     ST16000NM001G-2KK103
Serial Number:    ZL2NXNAA
LU WWN Device Id: 5 000c50 0e3fcc8c2
Firmware Version: SB40
User Capacity:    16,000,900,661,248 bytes [16.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5671
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 23 17:38:02 2025 IST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Disabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Write SCT (Get) Feature Control Command failed: scsi error aborted command
Wt Cache Reorder: Unknown (SCT Feature Control command failed)

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (  567) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (1367) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   082   065   044    -    148969226
  3 Spin_Up_Time            PO----   094   094   000    -    0
  4 Start_Stop_Count        -O--CK   100   100   020    -    5
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR--   074   061   045    -    25432363
  9 Power_On_Hours          -O--CK   099   099   000    -    1008
 10 Spin_Retry_Count        PO--C-   100   100   097    -    0
 12 Power_Cycle_Count       -O--CK   100   100   020    -    5
 18 Head_Health             PO-R--   100   100   050    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   070   049   040    -    30 (Min/Max 26/41)
192 Power-Off_Retract_Count -O--CK   100   100   000    -    2
193 Load_Cycle_Count        -O--CK   099   099   000    -    2131
194 Temperature_Celsius     -O---K   030   048   000    -    30 (0 18 0 0 0)
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   200   200   000    -    0
200 Pressure_Limit          PO---K   100   100   001    -    0
240 Head_Flying_Hours       ------   100   253   000    -    127h+27m+40.750s
241 Total_LBAs_Written      ------   100   253   000    -    15267664992
242 Total_LBAs_Read         ------   100   253   000    -    31342030502
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      5  Ext. Comprehensive SMART error log
0x04       GPL     R/O    256  Device Statistics log
0x04       SL      R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x0a       GPL     R/W      8  Device Statistics Notification
0x0c       GPL     R/O   2048  Pending Defects log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x13       GPL     R/O      1  SATA NCQ Send and Receive log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    768  Current Device Internal Status Data log
0x2f       GPL     R/O      1  Set Sector Configuration
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa1       GPL,SL  VS     160  Device vendor specific log
0xa2       GPL     VS   16320  Device vendor specific log
0xa4       GPL,SL  VS     160  Device vendor specific log
0xa6       GPL     VS     192  Device vendor specific log
0xa8-0xa9  GPL,SL  VS     136  Device vendor specific log
0xab       GPL     VS       1  Device vendor specific log
0xad       GPL     VS      16  Device vendor specific log
0xbe-0xbf  GPL     VS   65535  Device vendor specific log
0xc1       GPL,SL  VS       8  Device vendor specific log
0xc3       GPL,SL  VS      56  Device vendor specific log
0xc7       GPL,SL  VS       8  Device vendor specific log
0xc9       GPL,SL  VS       8  Device vendor specific log
0xca       GPL,SL  VS      16  Device vendor specific log
0xcd       GPL,SL  VS       1  Device vendor specific log
0xce       GPL     VS       1  Device vendor specific log
0xcf       GPL     VS     512  Device vendor specific log
0xd1       GPL     VS     656  Device vendor specific log
0xd2       GPL     VS   10000  Device vendor specific log
0xd4       GPL     VS    2048  Device vendor specific log
0xda       GPL,SL  VS       1  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer
0xe2       GPL,SL  -      160  Reserved

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%       987         -
# 2  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       522 (0x020a)
Device State:                        Active (0)
Current Temperature:                    30 Celsius
Power Cycle Min/Max Temperature:     26/41 Celsius
Lifetime    Min/Max Temperature:     18/51 Celsius
Under/Over Temperature Limit Count:   0/5
SMART Status:                        0xc24f (PASSED)
Vendor specific:
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         4 minutes
Temperature Logging Interval:        59 minutes
Min/Max recommended Temperature:     10/40 Celsius
Min/Max Temperature Limit:            5/60 Celsius
Temperature History Size (Index):    128 (82)

Index    Estimated Time   Temperature Celsius
  83    2025-04-18 12:29    32  *************
 ...    ..(  5 skipped).    ..  *************
  89    2025-04-18 18:23    32  *************
  90    2025-04-18 19:22    33  **************
  91    2025-04-18 20:21    33  **************
  92    2025-04-18 21:20    34  ***************
  93    2025-04-18 22:19    32  *************
  94    2025-04-18 23:18    33  **************
 ...    ..(  2 skipped).    ..  **************
  97    2025-04-19 02:15    33  **************
  98    2025-04-19 03:14    32  *************
 ...    ..(  4 skipped).    ..  *************
 103    2025-04-19 08:09    32  *************
 104    2025-04-19 09:08    33  **************
 105    2025-04-19 10:07    33  **************
 106    2025-04-19 11:06    32  *************
 107    2025-04-19 12:05    33  **************
 108    2025-04-19 13:04    32  *************
 109    2025-04-19 14:03    32  *************
 110    2025-04-19 15:02    31  ************
 111    2025-04-19 16:01    32  *************
 112    2025-04-19 17:00    32  *************
 113    2025-04-19 17:59    31  ************
 114    2025-04-19 18:58    32  *************
 115    2025-04-19 19:57    32  *************
 116    2025-04-19 20:56    33  **************
 117    2025-04-19 21:55     ?  -
 118    2025-04-19 22:54    18  -
 119    2025-04-19 23:53    27  ********
 120    2025-04-20 00:52     ?  -
 121    2025-04-20 01:51    20  *
 122    2025-04-20 02:50    30  ***********
 123    2025-04-20 03:49    29  **********
 ...    ..(  3 skipped).    ..  **********
 127    2025-04-20 07:45    29  **********
   0    2025-04-20 08:44    28  *********
 ...    ..(  4 skipped).    ..  *********
   5    2025-04-20 13:39    28  *********
   6    2025-04-20 14:38    29  **********
   7    2025-04-20 15:37    32  *************
   8    2025-04-20 16:36    32  *************
   9    2025-04-20 17:35    28  *********
  10    2025-04-20 18:34    27  ********
  11    2025-04-20 19:33    27  ********
  12    2025-04-20 20:32    27  ********
  13    2025-04-20 21:31    26  *******
  14    2025-04-20 22:30    26  *******
  15    2025-04-20 23:29    25  ******
  16    2025-04-21 00:28    25  ******
  17    2025-04-21 01:27    27  ********
 ...    ..(  7 skipped).    ..  ********
  25    2025-04-21 09:19    27  ********
  26    2025-04-21 10:18    26  *******
  27    2025-04-21 11:17    27  ********
  28    2025-04-21 12:16    27  ********
  29    2025-04-21 13:15    28  *********
  30    2025-04-21 14:14    28  *********
  31    2025-04-21 15:13    29  **********
  32    2025-04-21 16:12    29  **********
  33    2025-04-21 17:11     ?  -
  34    2025-04-21 18:10    26  *******
  35    2025-04-21 19:09    32  *************
  36    2025-04-21 20:08    31  ************
  37    2025-04-21 21:07    30  ***********
  38    2025-04-21 22:06    30  ***********
  39    2025-04-21 23:05    29  **********
  40    2025-04-22 00:04    28  *********
  41    2025-04-22 01:03    28  *********
  42    2025-04-22 02:02    27  ********
  43    2025-04-22 03:01    27  ********
  44    2025-04-22 04:00    39  ********************
  45    2025-04-22 04:59    41  **********************
 ...    ..(  3 skipped).    ..  **********************
  49    2025-04-22 08:55    41  **********************
  50    2025-04-22 09:54    40  *********************
 ...    ..(  3 skipped).    ..  *********************
  54    2025-04-22 13:50    40  *********************
  55    2025-04-22 14:49    39  ********************
 ...    ..(  6 skipped).    ..  ********************
  62    2025-04-22 21:42    39  ********************
  63    2025-04-22 22:41    34  ***************
  64    2025-04-22 23:40    30  ***********
  65    2025-04-23 00:39    31  ************
  66    2025-04-23 01:38    30  ***********
  67    2025-04-23 02:37    30  ***********
  68    2025-04-23 03:36    30  ***********
  69    2025-04-23 04:35    29  **********
 ...    ..(  3 skipped).    ..  **********
  73    2025-04-23 08:31    29  **********
  74    2025-04-23 09:30    28  *********
 ...    ..(  5 skipped).    ..  *********
  80    2025-04-23 15:24    28  *********
  81    2025-04-23 16:23    29  **********
  82    2025-04-23 17:22    29  **********

SCT Error Recovery Control:
           Read:    100 (10.0 seconds)
          Write:    100 (10.0 seconds)

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4               5  ---  Lifetime Power-On Resets
0x01  0x010  4            1008  ---  Power-on Hours
0x01  0x018  6     15267664992  ---  Logical Sectors Written
0x01  0x020  6        27398347  ---  Number of Write Commands
0x01  0x028  6     31342013222  ---  Logical Sectors Read
0x01  0x030  6       245080151  ---  Number of Read Commands
0x01  0x038  6               -  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4            1008  ---  Spindle Motor Power-on Hours
0x03  0x010  4             159  ---  Head Flying Hours
0x03  0x018  4            2131  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x03  0x038  4               0  ---  Number of Realloc. Candidate Logical Sectors
0x03  0x040  4               2  ---  Number of High Priority Unload Events
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x04  0x018  4               0  -D-  Physical Element Status Changed
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              30  ---  Current Temperature
0x05  0x010  1              34  ---  Average Short Term Temperature
0x05  0x018  1               -  ---  Average Long Term Temperature
0x05  0x020  1              48  ---  Highest Temperature
0x05  0x028  1              24  ---  Lowest Temperature
0x05  0x030  1              44  ---  Highest Average Short Term Temperature
0x05  0x038  1              27  ---  Lowest Average Short Term Temperature
0x05  0x040  1               -  ---  Highest Average Long Term Temperature
0x05  0x048  1               -  ---  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              60  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               5  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4              16  ---  Number of Hardware Resets
0x06  0x010  4               8  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
0xff  =====  =               =  ===  == Vendor Specific Statistics (rev 1) ==
0xff  0x008  7               0  ---  Vendor Specific
0xff  0x010  7               0  ---  Vendor Specific
0xff  0x018  7               0  ---  Vendor Specific
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x000a  2            8  Device-to-host register FISes sent due to a COMRESET
0x0001  2            0  Command failed due to ICRC error
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS

Seagate FARM log (GP Log 0xa6) supported [try: -l farm]
6

So already seeing a problem here with sdf/g both missing partition information

sde         ST16000NM001G-2KK103         1 gpt    disk          16000900661248                          
└─sde1                                   1 gpt    part     2048 16000898564096 Solaris /usr & Apple ZFS 76bd052b-d1fb-43d4-a3f0-a08b0e96d8a4
sdf         ST16000NM001G-2KK103         1 gpt    disk          16000900661248                          
└─sdf1                                   1 gpt    part     2048 16000898564096                          
sdg         ST16000NM001G-2KK103         1 gpt    disk          16000900661248                          
└─sdg1                                   1 gpt    part     2048 16000898564096    
sdj         ST16000NM001G-2KK103         1 gpt    disk          16000900661248                          
└─sdj1                                   1 gpt    part     2048 16000898564096 Solaris /usr & Apple ZFS 92f5e246-b303-4c06-9fa1-d4ac188577bd

Yet that data is there

root@truenas[~]# sgdisk -i 1 /dev/sdf
Partition GUID code: 6A898CC3-1DD2-11B2-99A6-080020736631 (Solaris /usr & Mac ZFS)
Partition unique GUID: 091AE219-2F40-4697-900B-7B1AC769488B
First sector: 2048 (at 1024.0 KiB)
Last sector: 31251757055 (at 14.6 TiB)
Partition size: 31251755008 sectors (14.6 TiB)
Attribute flags: 0000000000000000
Partition name: 'data'
root@truenas[~]# sgdisk -i 1 /dev/sdg
Partition GUID code: 6A898CC3-1DD2-11B2-99A6-080020736631 (Solaris /usr & Mac ZFS)
Partition unique GUID: BBEE6AC7-DA9C-4C68-8212-B9AFDE9E6E98
First sector: 2048 (at 1024.0 KiB)
Last sector: 31251757055 (at 14.6 TiB)
Partition size: 31251755008 sectors (14.6 TiB)
Attribute flags: 0000000000000000
Partition name: 'data'
root@truenas[~]#

and matches the guid’s from import

image
image853×176 11.2 KB

And no zpool.cache file present @ /etc/zfs/zpool.cache

root@truenas[~]# blkid --probe /dev/sdf
/dev/sdf: PTUUID="18ed44f6-a487-4c44-bfd7-94bf18d77fe9" PTTYPE="gpt"
root@truenas[~]# blkid --probe /dev/sdg
/dev/sdg: PTUUID="bb5184f0-e8db-421f-b2d8-959ef7732a33" PTTYPE="gpt"
root@truenas[~]# blkid --probe /dev/sdf1
blkid: /dev/sdf1: ambivalent result (probably more filesystems on the device, use wipefs(8) to see more details)
root@truenas[~]# blkid --probe /dev/sdg1
blkid: /dev/sdg1: ambivalent result (probably more filesystems on the device, use wipefs(8) to see more details)
root@truenas[~]# 
root@truenas[~]# 

root@truenas[~]# blkid
/dev/nvme0n1p2: LABEL="fastaccess" UUID="9421301376673073326" UUID_SUB="16569089007551506368" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="91de82e0-1bb0-4243-bda9-a8dd7fe767b9"
/dev/sdd2: LABEL="storage1" UUID="11126875392244974942" UUID_SUB="6556572431640363662" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="77292dd5-47c2-4bb9-87bd-7c9d8b276a06"
/dev/sdk3: LABEL="boot-pool" UUID="14786525513723065291" UUID_SUB="18306755730146993496" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="c2c4a03d-3643-42e2-bc1b-354d7c09cf1b"
/dev/sdk4: UUID="a4a295d6-fb99-662d-4240-2f2a8e9cab72" UUID_SUB="e4b05526-febf-f714-28a7-35315d6b52b0" LABEL="truenas:swap3" TYPE="linux_raid_member" PARTUUID="fbe85d2e-cc8c-4f29-b3f5-275a6852f06e"
/dev/sdk2: LABEL_FATBOOT="EFI" LABEL="EFI" UUID="F36B-6816" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="9da7b734-e3f4-4e8e-a05e-0e4cbfb98005"
/dev/sdi3: LABEL="boot-pool" UUID="14786525513723065291" UUID_SUB="9215254773799461772" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="cb0bf26f-a5a5-436a-9875-12e46e880711"
/dev/sdi4: UUID="a4a295d6-fb99-662d-4240-2f2a8e9cab72" UUID_SUB="dc4f24fa-bd08-e871-a17a-c9edcada5187" LABEL="truenas:swap3" TYPE="linux_raid_member" PARTUUID="b18d5629-c191-465b-9e53-8df71365a6a0"
/dev/sdi2: LABEL_FATBOOT="EFI" LABEL="EFI" UUID="F398-4DDA" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="b3106da1-2354-4860-80f3-168058816b61"
/dev/sde1: LABEL="storage2" UUID="8231419487065136889" UUID_SUB="10387615672472982606" BLOCK_SIZE="4096" TYPE="zfs_member" PARTLABEL="data" PARTUUID="76bd052b-d1fb-43d4-a3f0-a08b0e96d8a4"
/dev/sdc2: LABEL="storage1" UUID="11126875392244974942" UUID_SUB="14440468255160956156" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="9cdf928f-7d07-4e26-a103-05f58bfc7171"
/dev/sdl1: LABEL="storage1" UUID="11126875392244974942" UUID_SUB="2446347451913270650" BLOCK_SIZE="4096" TYPE="zfs_member" PARTLABEL="data" PARTUUID="7912023c-7e6b-4bef-8445-3cef8f64fefe"
/dev/nvme1n1p2: LABEL="fastaccess" UUID="9421301376673073326" UUID_SUB="883900868065551174" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="9696510a-3062-4d0c-84cd-9011608ecc36"
/dev/sda2: LABEL="storage1" UUID="11126875392244974942" UUID_SUB="5482751082009820612" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="43133b2a-0ae2-4104-b4dc-686d734cbb42"
/dev/sdj1: LABEL="storage2" UUID="8231419487065136889" UUID_SUB="18339876429200821433" BLOCK_SIZE="4096" TYPE="zfs_member" PARTLABEL="data" PARTUUID="92f5e246-b303-4c06-9fa1-d4ac188577bd"
/dev/nvme0n1p1: PARTUUID="3d1ae95e-5f7f-4b5b-aee8-c066bccece1f"
/dev/sdd1: PARTUUID="612983c6-22d2-41bb-ae68-2730dd37d93a"
/dev/sdk1: PARTUUID="2e7974ac-494a-44d4-9b6c-e235ecdef046"
/dev/sdi1: PARTUUID="89b220c5-4ac7-4956-87f9-42bcebd218b9"
/dev/sdc1: PARTUUID="840940a7-ffbd-45c2-9dfa-8671d02bfe17"
/dev/nvme1n1p1: PARTUUID="14dab88a-d357-440c-b2ca-3eac736c28a0"
/dev/sda1: PARTUUID="de66399f-0974-4692-bf00-99ccf0e6e9fd"
root@truenas[~]#

And one more time for the labels

root@truenas[~]# /sbin/zdb -l /dev/sde1
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'storage2'
    state: 0
    txg: 4
    pool_guid: 8231419487065136889
    errata: 0
    hostid: 365307668
    hostname: 'truenas'
    top_guid: 2782357268061663128
    guid: 10387615672472982606
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 2782357268061663128
        nparity: 1
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 64003575382016
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 5921271173483334054
            path: '/dev/disk/by-partuuid/091ae219-2f40-4697-900b-7b1ac769488b'
            whole_disk: 0
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 10387615672472982606
            path: '/dev/disk/by-partuuid/76bd052b-d1fb-43d4-a3f0-a08b0e96d8a4'
            whole_disk: 0
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 14831864185663549038
            path: '/dev/disk/by-partuuid/bbee6ac7-da9c-4c68-8212-b9afde9e6e98'
            whole_disk: 0
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 18339876429200821433
            path: '/dev/disk/by-partuuid/92f5e246-b303-4c06-9fa1-d4ac188577bd'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3 
root@truenas[~]# /sbin/zdb -l /dev/sdf1
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'storage2'
    state: 0
    txg: 4
    pool_guid: 8231419487065136889
    errata: 0
    hostid: 365307668
    hostname: 'truenas'
    top_guid: 2782357268061663128
    guid: 5921271173483334054
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 2782357268061663128
        nparity: 1
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 64003575382016
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 5921271173483334054
            path: '/dev/disk/by-partuuid/091ae219-2f40-4697-900b-7b1ac769488b'
            whole_disk: 0
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 10387615672472982606
            path: '/dev/disk/by-partuuid/76bd052b-d1fb-43d4-a3f0-a08b0e96d8a4'
            whole_disk: 0
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 14831864185663549038
            path: '/dev/disk/by-partuuid/bbee6ac7-da9c-4c68-8212-b9afde9e6e98'
            whole_disk: 0
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 18339876429200821433
            path: '/dev/disk/by-partuuid/92f5e246-b303-4c06-9fa1-d4ac188577bd'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3 
root@truenas[~]# /sbin/zdb -l /dev/sdg1
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'storage2'
    state: 0
    txg: 4
    pool_guid: 8231419487065136889
    errata: 0
    hostid: 365307668
    hostname: 'truenas'
    top_guid: 2782357268061663128
    guid: 14831864185663549038
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 2782357268061663128
        nparity: 1
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 64003575382016
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 5921271173483334054
            path: '/dev/disk/by-partuuid/091ae219-2f40-4697-900b-7b1ac769488b'
            whole_disk: 0
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 10387615672472982606
            path: '/dev/disk/by-partuuid/76bd052b-d1fb-43d4-a3f0-a08b0e96d8a4'
            whole_disk: 0
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 14831864185663549038
            path: '/dev/disk/by-partuuid/bbee6ac7-da9c-4c68-8212-b9afde9e6e98'
            whole_disk: 0
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 18339876429200821433
            path: '/dev/disk/by-partuuid/92f5e246-b303-4c06-9fa1-d4ac188577bd'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3 
root@truenas[~]# /sbin/zdb -l /dev/sdj1
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'storage2'
    state: 0
    txg: 4
    pool_guid: 8231419487065136889
    errata: 0
    hostid: 365307668
    hostname: 'truenas'
    top_guid: 2782357268061663128
    guid: 18339876429200821433
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 2782357268061663128
        nparity: 1
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 64003575382016
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 5921271173483334054
            path: '/dev/disk/by-partuuid/091ae219-2f40-4697-900b-7b1ac769488b'
            whole_disk: 0
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 10387615672472982606
            path: '/dev/disk/by-partuuid/76bd052b-d1fb-43d4-a3f0-a08b0e96d8a4'
            whole_disk: 0
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 14831864185663549038
            path: '/dev/disk/by-partuuid/bbee6ac7-da9c-4c68-8212-b9afde9e6e98'
            whole_disk: 0
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 18339876429200821433
            path: '/dev/disk/by-partuuid/92f5e246-b303-4c06-9fa1-d4ac188577bd'
            whole_disk: 0
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3

Forgive me I’m not a linux native and am unsure if this is a red herring but it appears that part of the problem is that I’m missing block device attributes for the two disks that zfs thinks is unavailable. I should be seeing information similar to

root@truenas[~]# blkid --probe /dev/sdc1
/dev/sdc1: PART_ENTRY_SCHEME="gpt" PART_ENTRY_UUID="840940a7-ffbd-45c2-9dfa-8671d02bfe17" PART_ENTRY_TYPE="0657fd6d-a4ab-43c4-84e5-0933c84b4f4f" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="128" PART_ENTRY_SIZE="4194177" PART_ENTRY_DISK="8:32"

Where I get this for the offending devices

root@truenas[~]# blkid --probe /dev/sdf1
blkid: /dev/sdf1: ambivalent result (probably more filesystems on the device, use wipefs(8) to see more details)

on the bright side the data does still exist elsewhere as per the output of sgdisk and zdb.

Where I’m unsure is, what the best plan of action to remedy this is?

7

Well - I am impressed by how much further you have taken the diagnosis by doing all the follow-up commands I might have suggested.

At this point my knowledge is exhausted - you need genuinely expert help. @HoneyBadger ???

2 Likes
8

Sound mate, the initial bit of direction was much appreciated. I’m savvy enough, been working in IT for years so no stranger to the command line or this type of problem solving. But I have little experience working with zfs so am a wary of making the situation im in worse :slight_smile:

That said this is a good opportunity to go down the rabbit hole of zfs.

1 Like
9

Now we’re cooking. Thanks for the ping @Protopia

@Squirrely let’s get a wipefs --no-act /dev/sdf1 and repeat it for all of the offending devices.

1 Like
10

Hey HoneyBadger, thanks for looking in. I got mixed results, no output at all for sdf1

root@truenas[~]# wipefs --no-act /dev/sdf1
root@truenas[~]# wipefs --no-act /dev/sdg1
DEVICE OFFSET        TYPE       UUID                LABEL
sdg1   0x3f000       zfs_member 8231419487065136889 storage2
sdg1   0x3e000       zfs_member 8231419487065136889 storage2
sdg1   0x3d000       zfs_member 8231419487065136889 storage2
sdg1   0x3c000       zfs_member 8231419487065136889 storage2
sdg1   0x3b000       zfs_member 8231419487065136889 storage2
sdg1   0x3a000       zfs_member 8231419487065136889 storage2
sdg1   0x39000       zfs_member 8231419487065136889 storage2
sdg1   0x38000       zfs_member 8231419487065136889 storage2
sdg1   0x37000       zfs_member 8231419487065136889 storage2
sdg1   0x36000       zfs_member 8231419487065136889 storage2
sdg1   0x35000       zfs_member 8231419487065136889 storage2
sdg1   0x34000       zfs_member 8231419487065136889 storage2
sdg1   0x33000       zfs_member 8231419487065136889 storage2
sdg1   0x32000       zfs_member 8231419487065136889 storage2
sdg1   0x31000       zfs_member 8231419487065136889 storage2
sdg1   0x30000       zfs_member 8231419487065136889 storage2
sdg1   0x2f000       zfs_member 8231419487065136889 storage2
sdg1   0x2e000       zfs_member 8231419487065136889 storage2
sdg1   0x2d000       zfs_member 8231419487065136889 storage2
sdg1   0x2c000       zfs_member 8231419487065136889 storage2
sdg1   0x2b000       zfs_member 8231419487065136889 storage2
sdg1   0x2a000       zfs_member 8231419487065136889 storage2
sdg1   0x29000       zfs_member 8231419487065136889 storage2
sdg1   0x28000       zfs_member 8231419487065136889 storage2
sdg1   0x27000       zfs_member 8231419487065136889 storage2
sdg1   0x26000       zfs_member 8231419487065136889 storage2
sdg1   0x25000       zfs_member 8231419487065136889 storage2
sdg1   0x24000       zfs_member 8231419487065136889 storage2
sdg1   0x23000       zfs_member 8231419487065136889 storage2
sdg1   0x7f000       zfs_member 8231419487065136889 storage2
sdg1   0x7e000       zfs_member 8231419487065136889 storage2
sdg1   0x7d000       zfs_member 8231419487065136889 storage2
sdg1   0x7c000       zfs_member 8231419487065136889 storage2
sdg1   0x7b000       zfs_member 8231419487065136889 storage2
sdg1   0x7a000       zfs_member 8231419487065136889 storage2
sdg1   0x79000       zfs_member 8231419487065136889 storage2
sdg1   0x78000       zfs_member 8231419487065136889 storage2
sdg1   0x77000       zfs_member 8231419487065136889 storage2
sdg1   0x76000       zfs_member 8231419487065136889 storage2
sdg1   0x75000       zfs_member 8231419487065136889 storage2
sdg1   0x74000       zfs_member 8231419487065136889 storage2
sdg1   0x73000       zfs_member 8231419487065136889 storage2
sdg1   0x72000       zfs_member 8231419487065136889 storage2
sdg1   0x71000       zfs_member 8231419487065136889 storage2
sdg1   0x70000       zfs_member 8231419487065136889 storage2
sdg1   0x6f000       zfs_member 8231419487065136889 storage2
sdg1   0x6e000       zfs_member 8231419487065136889 storage2
sdg1   0x6d000       zfs_member 8231419487065136889 storage2
sdg1   0x6c000       zfs_member 8231419487065136889 storage2
sdg1   0x6b000       zfs_member 8231419487065136889 storage2
sdg1   0x6a000       zfs_member 8231419487065136889 storage2
sdg1   0x69000       zfs_member 8231419487065136889 storage2
sdg1   0x68000       zfs_member 8231419487065136889 storage2
sdg1   0x67000       zfs_member 8231419487065136889 storage2
sdg1   0x66000       zfs_member 8231419487065136889 storage2
sdg1   0x65000       zfs_member 8231419487065136889 storage2
sdg1   0x64000       zfs_member 8231419487065136889 storage2
sdg1   0x63000       zfs_member 8231419487065136889 storage2
sdg1   0x62000       zfs_member 8231419487065136889 storage2
sdg1   0x61000       zfs_member 8231419487065136889 storage2
sdg1   0x60000       zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdbf000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdbe000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdbd000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdbc000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdbb000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdba000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdb9000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdb8000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdb7000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdb6000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdb5000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdb4000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdb3000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdb2000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdb1000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdb0000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdaf000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdae000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdad000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdac000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdab000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdaa000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fda9000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fda8000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fda7000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fda6000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fda5000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fda4000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fda3000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fda2000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fda1000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fda0000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdff000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdfe000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdfd000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdfc000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdfb000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdfa000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdf9000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdf8000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdf7000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdf6000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdf5000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdf4000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdf3000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdf2000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdf1000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdf0000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdef000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdee000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fded000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdec000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdeb000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fdea000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fde9000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fde8000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fde7000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fde6000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fde5000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fde4000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fde3000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fde2000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fde1000 zfs_member 8231419487065136889 storage2
sdg1   0xe8d7fde0000 zfs_member 8231419487065136889 storage2
sdg1   0x3           ntfs       1E887B64887B38FB    
root@truenas[~]#
11

Strange on the zero results from sdf1 as it called it an ambivalent result. Did drives maybe shift letters between reboots?

Let’s bop the offending label on sdg1 though - assuming you haven’t rebooted between then and now:

  • sudo wipefs --all --backup -t ntfs /dev/sdg1

Do this, reboot, see if you have another disk that’s identified as being present again.

1 Like
12

So glad you prompted me to re-check the drive lettering before acting. I had indeed shutdown last night!

image
image941×245 12.4 KB

the results for the new drive lettering is

root@truenas[~]# blkid --probe /dev/sdd1
/dev/sdd1: VERSION="5000" LABEL="storage2" UUID="8231419487065136889" UUID_SUB="10387615672472982606" BLOCK_SIZE="4096" TYPE="zfs_member" USAGE="filesystem" PART_ENTRY_SCHEME="gpt" PART_ENTRY_NAME="data" PART_ENTRY_UUID="76bd052b-d1fb-43d4-a3f0-a08b0e96d8a4" PART_ENTRY_TYPE="6a898cc3-1dd2-11b2-99a6-080020736631" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="2048" PART_ENTRY_SIZE="31251755008" PART_ENTRY_DISK="8:48"
root@truenas[~]# blkid --probe /dev/sde1
blkid: /dev/sde1: ambivalent result (probably more filesystems on the device, use wipefs(8) to see more details)
root@truenas[~]# blkid --probe /dev/sdg1
blkid: /dev/sdg1: ambivalent result (probably more filesystems on the device, use wipefs(8) to see more details)
root@truenas[~]# blkid --probe /dev/sdh1
/dev/sdh1: VERSION="5000" LABEL="storage2" UUID="8231419487065136889" UUID_SUB="18339876429200821433" BLOCK_SIZE="4096" TYPE="zfs_member" USAGE="filesystem" PART_ENTRY_SCHEME="gpt" PART_ENTRY_NAME="data" PART_ENTRY_UUID="92f5e246-b303-4c06-9fa1-d4ac188577bd" PART_ENTRY_TYPE="6a898cc3-1dd2-11b2-99a6-080020736631" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="2048" PART_ENTRY_SIZE="31251755008" PART_ENTRY_DISK="8:112"
root@truenas[~]# wipefs --no-act /dev/sde1
DEVICE OFFSET        TYPE       UUID                LABEL
sde1   0x3f000       zfs_member 8231419487065136889 storage2
sde1   0x3e000       zfs_member 8231419487065136889 storage2
sde1   0x3d000       zfs_member 8231419487065136889 storage2
sde1   0x3c000       zfs_member 8231419487065136889 storage2
sde1   0x3b000       zfs_member 8231419487065136889 storage2
sde1   0x3a000       zfs_member 8231419487065136889 storage2
sde1   0x39000       zfs_member 8231419487065136889 storage2
sde1   0x38000       zfs_member 8231419487065136889 storage2
sde1   0x37000       zfs_member 8231419487065136889 storage2
sde1   0x36000       zfs_member 8231419487065136889 storage2
sde1   0x35000       zfs_member 8231419487065136889 storage2
sde1   0x34000       zfs_member 8231419487065136889 storage2
sde1   0x33000       zfs_member 8231419487065136889 storage2
sde1   0x32000       zfs_member 8231419487065136889 storage2
sde1   0x31000       zfs_member 8231419487065136889 storage2
sde1   0x30000       zfs_member 8231419487065136889 storage2
sde1   0x2f000       zfs_member 8231419487065136889 storage2
sde1   0x2e000       zfs_member 8231419487065136889 storage2
sde1   0x2d000       zfs_member 8231419487065136889 storage2
sde1   0x2c000       zfs_member 8231419487065136889 storage2
sde1   0x2b000       zfs_member 8231419487065136889 storage2
sde1   0x2a000       zfs_member 8231419487065136889 storage2
sde1   0x29000       zfs_member 8231419487065136889 storage2
sde1   0x28000       zfs_member 8231419487065136889 storage2
sde1   0x27000       zfs_member 8231419487065136889 storage2
sde1   0x26000       zfs_member 8231419487065136889 storage2
sde1   0x25000       zfs_member 8231419487065136889 storage2
sde1   0x24000       zfs_member 8231419487065136889 storage2
sde1   0x23000       zfs_member 8231419487065136889 storage2
sde1   0x7f000       zfs_member 8231419487065136889 storage2
sde1   0x7e000       zfs_member 8231419487065136889 storage2
sde1   0x7d000       zfs_member 8231419487065136889 storage2
sde1   0x7c000       zfs_member 8231419487065136889 storage2
sde1   0x7b000       zfs_member 8231419487065136889 storage2
sde1   0x7a000       zfs_member 8231419487065136889 storage2
sde1   0x79000       zfs_member 8231419487065136889 storage2
sde1   0x78000       zfs_member 8231419487065136889 storage2
sde1   0x77000       zfs_member 8231419487065136889 storage2
sde1   0x76000       zfs_member 8231419487065136889 storage2
sde1   0x75000       zfs_member 8231419487065136889 storage2
sde1   0x74000       zfs_member 8231419487065136889 storage2
sde1   0x73000       zfs_member 8231419487065136889 storage2
sde1   0x72000       zfs_member 8231419487065136889 storage2
sde1   0x71000       zfs_member 8231419487065136889 storage2
sde1   0x70000       zfs_member 8231419487065136889 storage2
sde1   0x6f000       zfs_member 8231419487065136889 storage2
sde1   0x6e000       zfs_member 8231419487065136889 storage2
sde1   0x6d000       zfs_member 8231419487065136889 storage2
sde1   0x6c000       zfs_member 8231419487065136889 storage2
sde1   0x6b000       zfs_member 8231419487065136889 storage2
sde1   0x6a000       zfs_member 8231419487065136889 storage2
sde1   0x69000       zfs_member 8231419487065136889 storage2
sde1   0x68000       zfs_member 8231419487065136889 storage2
sde1   0x67000       zfs_member 8231419487065136889 storage2
sde1   0x66000       zfs_member 8231419487065136889 storage2
sde1   0x65000       zfs_member 8231419487065136889 storage2
sde1   0x64000       zfs_member 8231419487065136889 storage2
sde1   0x63000       zfs_member 8231419487065136889 storage2
sde1   0x62000       zfs_member 8231419487065136889 storage2
sde1   0x61000       zfs_member 8231419487065136889 storage2
sde1   0x60000       zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdbf000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdbe000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdbd000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdbc000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdbb000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdba000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdb9000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdb8000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdb7000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdb6000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdb5000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdb4000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdb3000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdb2000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdb1000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdb0000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdaf000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdae000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdad000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdac000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdab000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdaa000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fda9000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fda8000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fda7000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fda6000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fda5000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fda4000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fda3000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fda2000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fda1000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fda0000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdff000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdfe000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdfd000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdfc000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdfb000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdfa000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdf9000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdf8000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdf7000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdf6000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdf5000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdf4000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdf3000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdf2000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdf1000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdf0000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdef000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdee000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fded000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdec000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdeb000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fdea000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fde9000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fde8000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fde7000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fde6000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fde5000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fde4000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fde3000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fde2000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fde1000 zfs_member 8231419487065136889 storage2
sde1   0xe8d7fde0000 zfs_member 8231419487065136889 storage2
sde1   0x3           ntfs       3AB8FFEAB8FFA295    
root@truenas[~]# wipefs --no-act /dev/sdh1
DEVICE OFFSET        TYPE       UUID                LABEL
sdh1   0x3f000       zfs_member 8231419487065136889 storage2
sdh1   0x3e000       zfs_member 8231419487065136889 storage2
sdh1   0x3d000       zfs_member 8231419487065136889 storage2
sdh1   0x3c000       zfs_member 8231419487065136889 storage2
sdh1   0x3b000       zfs_member 8231419487065136889 storage2
sdh1   0x3a000       zfs_member 8231419487065136889 storage2
sdh1   0x39000       zfs_member 8231419487065136889 storage2
sdh1   0x38000       zfs_member 8231419487065136889 storage2
sdh1   0x37000       zfs_member 8231419487065136889 storage2
sdh1   0x36000       zfs_member 8231419487065136889 storage2
sdh1   0x35000       zfs_member 8231419487065136889 storage2
sdh1   0x34000       zfs_member 8231419487065136889 storage2
sdh1   0x33000       zfs_member 8231419487065136889 storage2
sdh1   0x32000       zfs_member 8231419487065136889 storage2
sdh1   0x31000       zfs_member 8231419487065136889 storage2
sdh1   0x30000       zfs_member 8231419487065136889 storage2
sdh1   0x2f000       zfs_member 8231419487065136889 storage2
sdh1   0x2e000       zfs_member 8231419487065136889 storage2
sdh1   0x2d000       zfs_member 8231419487065136889 storage2
sdh1   0x2c000       zfs_member 8231419487065136889 storage2
sdh1   0x2b000       zfs_member 8231419487065136889 storage2
sdh1   0x2a000       zfs_member 8231419487065136889 storage2
sdh1   0x29000       zfs_member 8231419487065136889 storage2
sdh1   0x28000       zfs_member 8231419487065136889 storage2
sdh1   0x27000       zfs_member 8231419487065136889 storage2
sdh1   0x26000       zfs_member 8231419487065136889 storage2
sdh1   0x25000       zfs_member 8231419487065136889 storage2
sdh1   0x24000       zfs_member 8231419487065136889 storage2
sdh1   0x23000       zfs_member 8231419487065136889 storage2
sdh1   0x7f000       zfs_member 8231419487065136889 storage2
sdh1   0x7e000       zfs_member 8231419487065136889 storage2
sdh1   0x7d000       zfs_member 8231419487065136889 storage2
sdh1   0x7c000       zfs_member 8231419487065136889 storage2
sdh1   0x7b000       zfs_member 8231419487065136889 storage2
sdh1   0x7a000       zfs_member 8231419487065136889 storage2
sdh1   0x79000       zfs_member 8231419487065136889 storage2
sdh1   0x78000       zfs_member 8231419487065136889 storage2
sdh1   0x77000       zfs_member 8231419487065136889 storage2
sdh1   0x76000       zfs_member 8231419487065136889 storage2
sdh1   0x75000       zfs_member 8231419487065136889 storage2
sdh1   0x74000       zfs_member 8231419487065136889 storage2
sdh1   0x73000       zfs_member 8231419487065136889 storage2
sdh1   0x72000       zfs_member 8231419487065136889 storage2
sdh1   0x71000       zfs_member 8231419487065136889 storage2
sdh1   0x70000       zfs_member 8231419487065136889 storage2
sdh1   0x6f000       zfs_member 8231419487065136889 storage2
sdh1   0x6e000       zfs_member 8231419487065136889 storage2
sdh1   0x6d000       zfs_member 8231419487065136889 storage2
sdh1   0x6c000       zfs_member 8231419487065136889 storage2
sdh1   0x6b000       zfs_member 8231419487065136889 storage2
sdh1   0x6a000       zfs_member 8231419487065136889 storage2
sdh1   0x69000       zfs_member 8231419487065136889 storage2
sdh1   0x68000       zfs_member 8231419487065136889 storage2
sdh1   0x67000       zfs_member 8231419487065136889 storage2
sdh1   0x66000       zfs_member 8231419487065136889 storage2
sdh1   0x65000       zfs_member 8231419487065136889 storage2
sdh1   0x64000       zfs_member 8231419487065136889 storage2
sdh1   0x63000       zfs_member 8231419487065136889 storage2
sdh1   0x62000       zfs_member 8231419487065136889 storage2
sdh1   0x61000       zfs_member 8231419487065136889 storage2
sdh1   0x60000       zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdbf000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdbe000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdbd000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdbc000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdbb000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdba000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdb9000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdb8000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdb7000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdb6000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdb5000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdb4000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdb3000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdb2000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdb1000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdb0000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdaf000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdae000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdad000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdac000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdab000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdaa000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fda9000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fda8000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fda7000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fda6000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fda5000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fda4000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fda3000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fda2000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fda1000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fda0000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdff000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdfe000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdfd000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdfc000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdfb000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdfa000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdf9000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdf8000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdf7000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdf6000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdf5000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdf4000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdf3000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdf2000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdf1000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdf0000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdef000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdee000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fded000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdec000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdeb000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fdea000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fde9000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fde8000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fde7000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fde6000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fde5000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fde4000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fde3000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fde2000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fde1000 zfs_member 8231419487065136889 storage2
sdh1   0xe8d7fde0000 zfs_member 8231419487065136889 storage2
root@truenas[~]# 

root@truenas[~]# wipefs --all --backup -t ntfs /dev/sde1
/dev/sde1: 8 bytes were erased at offset 0x00000003 (ntfs): 4e 54 46 53 20 20 20 20
root@truenas[~]#

Rebooted and we now have an apparently accessible pool.

image
image1382×790 72.5 KB

vdev showing where it wasn’t before

image
image892×399 18.2 KB

And one disk marked unavailable

image
image439×287 6.35 KB

  pool: storage2
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:

        NAME                                      STATE     READ WRITE CKSUM
        storage2                                  DEGRADED     0     0     0
          raidz1-0                                DEGRADED     0     0     0
            091ae219-2f40-4697-900b-7b1ac769488b  ONLINE       0     0     0
            76bd052b-d1fb-43d4-a3f0-a08b0e96d8a4  ONLINE       0     0     0
            14831864185663549038                  UNAVAIL      0     0     0  was /dev/disk/by-partuuid/bbee6ac7-da9c-4c68-8212-b9afde9e6e98
            92f5e246-b303-4c06-9fa1-d4ac188577bd  ONLINE       0     0     0

errors: No known data errors
root@truenas[~]#

How would you advise I proceed sensei, resilver or scrub first? And what just happened by wiping that single ntfs signature away? How did it get in there at all?

The outcast disk has already passed a long smart test so am running a round of badblocks to get an idea if the surface is ok. If that passes I’ll add it back into the pool. Open to any other suggestions as to best practice. The disks all passed long smart recently so am sorta confident they are sound. After a resilver, i’m thinking it would be wise to run a non destructive badblocks one disk at a time over the next week or so to confirm platter surface integrity. What is the dogma around here when dealing with an event like this? I know I’m living on the edge with no backups but who can afford that much drive capacity x2. Need a tape drive maybe. I’ve just re-assembled this computer into a new case so have made sure of all the power and data connections. PSU is good quality with capacity to spare. No outstanding bios updates for the HBA’s.

Many many thanks for the assistance guys!

13

Re-check the lettering once more, assuming sdh1 is still where you expect it to be with the ambiguous signatures on blkid then do the wipefs --all --backup -t ntfs /dev/sdh1 and reboot once more.

Should then have a healthy pool again with four drives, once that’s the case give it a scrub.

2 Likes
14

I jumped the gun there and ran badblocks destructive on the last drive now /dev/sdg.

Just so I understand what the logic is here mate: wipefs --all iteratively scans the target device, destroying ‘all’ of the partition-table signatures that are detectable by libblkid. These magic strings have somehow become corrupt for two of my drives. We are diagnosing this via the output of blkid aka ambivalent result which also uses libblkid. From this I surmise this same lib is used when a zfs pool is being instantiated and so reads a disk as unavailable.

a few questions then.

  1. Where do bad signatures come from, is it something I can mitigate against?
  2. If bad signatures make it seem as if a disk is unavailable how is a disk with no signatures better?
  3. What’s the relevance of ‘ntfs’ in the wipefs command where --all is also present?

Thanks again!

15

Oh. Well you’ll definitely want to REPLACE the missing/failed disk with that new one then, let it resilver, and then scrub to be sure.

wipefs --all would indeed destroy all of the filesystem labels as implied; however, the presence of the -t ntfs restricts it to wiping out only those that match the filesystem type. --backup causes it to dump a backup file for what we wiped in your $HOME folder, just in case we want it back (but in most cases, we don’t)

Basically libblkid has/had some incorrect detection logic in ZFS as to how it senses the pool - it’s using headers rather than uberblocks, so the ambivalent[1] result is because it’s confused about what filesystem really exists on the disk/partition, so it’s uncertain to report it as a ZFS pool candidate or NTFS filesystem.

So what we do is wipe out the ntfs filesystem label, removing the ambiguity, and letting ZFS know it can try this disk. Presto, it’s back. No wiping of filesystem or ZFS labels happens.

  1. the actual word they want to use is “ambiguous” meaning “unclear or open to multiple interpretations” - “ambivalent” means “mixed or conflicting feelings” and I don’t believe that AI has advanced enough for blkid to have emotions, and yes I am going to submit a PR for this ↩︎

3 Likes
16

Will do! I’ll read into uberblocks cheers. Then that probably wouldn’t have worked for the second drive (at the time) sdh1, it wasn’t showing any ntfs strings.

As usual I have some reading to do!

Thanks again to yourself and Protopia for the assist.