Can't Import zpool; single dead hdd in raidz2 vdev

TrueNAS General
1

HELP!!! I can’t Import zpool; single dead hdd in raidz2 vdev

How does someone go about fixing an issue like this?

root@truenas[/home/admin]# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
boot-pool 14G 6.33G 7.67G - - 9% 45% 1.00x ONLINE -
root@truenas[/home/admin]# zpool import -a
cannot import ‘Nexsan_2x7_raidz2’: I/O error
Destroy and re-create the pool from
a backup source.
root@truenas[/home/admin]# zpool import
pool: Nexsan_2x7_raidz2
id: 8972354718331620302
state: FAULTED
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the ‘-f’ flag.
see: Message ID: ZFS-8000-5E — OpenZFS documentation
config:

Nexsan_2x7_raidz2                         FAULTED  corrupted data
  raidz2-0                                DEGRADED
    d4f45115-d21b-11ea-bc27-002590db5748  ONLINE
    d51021f0-d21b-11ea-bc27-002590db5748  ONLINE
    d52f2438-d21b-11ea-bc27-002590db5748  ONLINE
    3958782095893599654                   UNAVAIL
    d6630266-d21b-11ea-bc27-002590db5748  ONLINE
    d6a69574-d21b-11ea-bc27-002590db5748  ONLINE
    d6b4ee02-d21b-11ea-bc27-002590db5748  ONLINE
  raidz2-1                                ONLINE
    d4fd3610-d21b-11ea-bc27-002590db5748  ONLINE
    d5b1bc18-d21b-11ea-bc27-002590db5748  ONLINE
    d60ccef4-d21b-11ea-bc27-002590db5748  ONLINE
    d68f5c7c-d21b-11ea-bc27-002590db5748  ONLINE
    d648a21a-d21b-11ea-bc27-002590db5748  ONLINE
    d67abf95-d21b-11ea-bc27-002590db5748  ONLINE
    d6ca4a39-d21b-11ea-bc27-002590db5748  ONLINE

root@truenas[/home/admin]# zpool import -a -m -f
cannot import ‘Nexsan_2x7_raidz2’: I/O error
Destroy and re-create the pool from
a backup source.
root@truenas[/home/admin]# h

root@truenas[/home/admin]# zpool import -X -F -f -n -d /dev/disk/by-id Nexsan_2x7_raidz2
root@truenas[/home/admin]# ls /dev/disk/by-partuuid
113bba29-f280-4b1c-b020-6a1ec517082e d4f45115-d21b-11ea-bc27-002590db5748 d62491f6-d21b-11ea-bc27-002590db5748
53b5106a-13c7-43b7-bf25-0d774fb1c7e4 d4fd3610-d21b-11ea-bc27-002590db5748 d63085d9-d21b-11ea-bc27-002590db5748
541a6042-c2d9-4ee6-aec6-3cd0bb6556dc d5040c50-d21b-11ea-bc27-002590db5748 d648a21a-d21b-11ea-bc27-002590db5748
7e70faf9-4981-4da9-93ea-7245b5a5add7 d51021f0-d21b-11ea-bc27-002590db5748 d653ef56-d21b-11ea-bc27-002590db5748
9b390189-fbf1-4acb-af45-d39feb8894ff d52f2438-d21b-11ea-bc27-002590db5748 d6630266-d21b-11ea-bc27-002590db5748
b1eea782-2940-42a3-9e6b-3e634bcdfc31 d5670897-d21b-11ea-bc27-002590db5748 d67abf95-d21b-11ea-bc27-002590db5748
b33c2617-6683-430a-83b8-83caae9f8eb9 d5a7975b-d21b-11ea-bc27-002590db5748 d68f5c7c-d21b-11ea-bc27-002590db5748
cda86b6f-0959-42af-a53d-0df74300d7bb d5b1bc18-d21b-11ea-bc27-002590db5748 d6a69574-d21b-11ea-bc27-002590db5748
d487e76b-d21b-11ea-bc27-002590db5748 d5d4566b-d21b-11ea-bc27-002590db5748 d6b4ee02-d21b-11ea-bc27-002590db5748
d49fc11d-d21b-11ea-bc27-002590db5748 d5dcb8c0-d21b-11ea-bc27-002590db5748 d6ca4a39-d21b-11ea-bc27-002590db5748
d4bd53d3-d21b-11ea-bc27-002590db5748 d60088b2-d21b-11ea-bc27-002590db5748 f02218c4-fb0b-4f14-aacc-19f13b3a3f36
d4e88df8-d21b-11ea-bc27-002590db5748 d60ccef4-d21b-11ea-bc27-002590db5748 f46cfe74-835a-4d15-b70f-06e7781b5f1c
root@truenas[/home/admin]# zpool import -X -F -f -n -d /dev/disk/by-partuuid Nexsan_2x7_raidz2

root@truenas[/home/admin]# zpool import
pool: Nexsan_2x7_raidz2
id: 8972354718331620302
state: FAULTED
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the ‘-f’ flag.
see: Message ID: ZFS-8000-5E — OpenZFS documentation
config:

Nexsan_2x7_raidz2                         FAULTED  corrupted data
  raidz2-0                                DEGRADED
    d4f45115-d21b-11ea-bc27-002590db5748  ONLINE
    d51021f0-d21b-11ea-bc27-002590db5748  ONLINE
    d52f2438-d21b-11ea-bc27-002590db5748  ONLINE
    3958782095893599654                   UNAVAIL
    d6630266-d21b-11ea-bc27-002590db5748  ONLINE
    d6a69574-d21b-11ea-bc27-002590db5748  ONLINE
    d6b4ee02-d21b-11ea-bc27-002590db5748  ONLINE
  raidz2-1                                ONLINE
    d4fd3610-d21b-11ea-bc27-002590db5748  ONLINE
    d5b1bc18-d21b-11ea-bc27-002590db5748  ONLINE
    d60ccef4-d21b-11ea-bc27-002590db5748  ONLINE
    d68f5c7c-d21b-11ea-bc27-002590db5748  ONLINE
    d648a21a-d21b-11ea-bc27-002590db5748  ONLINE
    d67abf95-d21b-11ea-bc27-002590db5748  ONLINE
    d6ca4a39-d21b-11ea-bc27-002590db5748  ONLINE

root@truenas[/home/admin]# zpool import -X -F -f -n -d /dev/disk/by-id Nexsan_2x7_raidz2
root@truenas[/home/admin]#

oot@truenas[/home/admin]# zpool import -FX -f -m -d /dev/disk/by-partuuid Nexsan_2x7_raidz2
cannot import ‘Nexsan_2x7_raidz2’: one or more devices is currently unavailable
root@truenas[/home/admin]# zdb -e Nexsan_2x7_raidz2

Configuration for import:
vdev_children: 2
version: 5000
pool_guid: 8972354718331620302
name: ‘Nexsan_2x7_raidz2’
state: 0
hostid: 1362372599
hostname: ‘truenas’
vdev_tree:
type: ‘root’
id: 0
guid: 8972354718331620302
children[0]:
type: ‘raidz’
id: 0
guid: 3981250722323924461
nparity: 2
metaslab_array: 60
metaslab_shift: 37
ashift: 12
asize: 13987726491648
is_log: 0
create_txg: 4
children[0]:
type: ‘disk’
id: 0
guid: 2558360558814569309
DTL: 289
create_txg: 4
path: ‘/dev/sde2’
children[1]:
type: ‘disk’
id: 1
guid: 10926754174243817966
DTL: 288
create_txg: 4
path: ‘/dev/sdi2’
children[2]:
type: ‘disk’
id: 2
guid: 13733395212906534443
DTL: 287
create_txg: 4
path: ‘/dev/sdf2’
children[3]:
type: ‘disk’
id: 3
guid: 3958782095893599654
path: ‘/dev/gptid/d614a52b-d21b-11ea-bc27-002590db5748’
phys_path: ‘id1,enc@n500304800374a57d/type@0/slot@4/elmdesc@Slot_04/p2’
not_present: 1
DTL: 286
create_txg: 4
children[4]:
type: ‘disk’
id: 4
guid: 17217824180768895110
DTL: 285
create_txg: 4
path: ‘/dev/sdc2’
children[5]:
type: ‘disk’
id: 5
guid: 8051454238781092430
DTL: 284
create_txg: 4
path: ‘/dev/sdh2’
children[6]:
type: ‘disk’
id: 6
guid: 8535382695637425029
DTL: 283
create_txg: 4
path: ‘/dev/sda2’
children[1]:
type: ‘raidz’
id: 1
guid: 6983679603137452165
nparity: 2
metaslab_array: 52
metaslab_shift: 37
ashift: 12
asize: 13987726491648
is_log: 0
create_txg: 4
children[0]:
type: ‘disk’
id: 0
guid: 2335491171700643644
DTL: 282
create_txg: 4
path: ‘/dev/sdl2’
children[1]:
type: ‘disk’
id: 1
guid: 3136592629563023909
DTL: 281
create_txg: 4
path: ‘/dev/sdj2’
children[2]:
type: ‘disk’
id: 2
guid: 11736308515974993500
DTL: 280
create_txg: 4
path: ‘/dev/sdg2’
children[3]:
type: ‘disk’
id: 3
guid: 13479457084483254071
DTL: 279
create_txg: 4
path: ‘/dev/sdo2’
children[4]:
type: ‘disk’
id: 4
guid: 5119149219636539909
DTL: 278
create_txg: 4
path: ‘/dev/sdm2’
children[5]:
type: ‘disk’
id: 5
guid: 14902355098894633373
DTL: 277
create_txg: 4
path: ‘/dev/sdk2’
children[6]:
type: ‘disk’
id: 6
guid: 15750852432423240846
DTL: 276
create_txg: 4
path: ‘/dev/sdn2’
load-policy:
load-request-txg: 18446744073709551615
load-rewind-policy: 2
zdb: can’t open ‘Nexsan_2x7_raidz2’: Input/output error

ZFS_DBGMSG(zdb) START:
spa.c:6623:spa_import(): spa_import: importing Nexsan_2x7_raidz2
spa_misc.c:418:spa_load_note(): spa_load(Nexsan_2x7_raidz2, config trusted): LOADING
vdev.c:161:vdev_dbgmsg(): disk vdev ‘/dev/sde2’: best uberblock found for spa Nexsan_2x7_raidz2. txg 9256120
spa_misc.c:418:spa_load_note(): spa_load(Nexsan_2x7_raidz2, config untrusted): using uberblock with txg=9256120
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 2558360558814569309: vdev_path changed from ‘/dev/disk/by-partuuid/d4f45115-d21b-11ea-bc27-002590db5748’ to ‘/dev/sde2’
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 10926754174243817966: vdev_path changed from ‘/dev/disk/by-partuuid/d51021f0-d21b-11ea-bc27-002590db5748’ to ‘/dev/sdi2’
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 13733395212906534443: vdev_path changed from ‘/dev/disk/by-partuuid/d52f2438-d21b-11ea-bc27-002590db5748’ to ‘/dev/sdf2’
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 17217824180768895110: vdev_path changed from ‘/dev/disk/by-partuuid/d6630266-d21b-11ea-bc27-002590db5748’ to ‘/dev/sdc2’
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 8051454238781092430: vdev_path changed from ‘/dev/disk/by-partuuid/d6a69574-d21b-11ea-bc27-002590db5748’ to ‘/dev/sdh2’
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 8535382695637425029: vdev_path changed from ‘/dev/disk/by-partuuid/d6b4ee02-d21b-11ea-bc27-002590db5748’ to ‘/dev/sda2’
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 2335491171700643644: vdev_path changed from ‘/dev/disk/by-partuuid/d4fd3610-d21b-11ea-bc27-002590db5748’ to ‘/dev/sdl2’
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 3136592629563023909: vdev_path changed from ‘/dev/disk/by-partuuid/d5b1bc18-d21b-11ea-bc27-002590db5748’ to ‘/dev/sdj2’
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 11736308515974993500: vdev_path changed from ‘/dev/disk/by-partuuid/d60ccef4-d21b-11ea-bc27-002590db5748’ to ‘/dev/sdg2’
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 13479457084483254071: vdev_path changed from ‘/dev/disk/by-partuuid/d68f5c7c-d21b-11ea-bc27-002590db5748’ to ‘/dev/sdo2’
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 5119149219636539909: vdev_path changed from ‘/dev/disk/by-partuuid/d648a21a-d21b-11ea-bc27-002590db5748’ to ‘/dev/sdm2’
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 14902355098894633373: vdev_path changed from ‘/dev/disk/by-partuuid/d67abf95-d21b-11ea-bc27-002590db5748’ to ‘/dev/sdk2’
vdev.c:2499:vdev_update_path(): vdev_copy_path: vdev 15750852432423240846: vdev_path changed from ‘/dev/disk/by-partuuid/d6ca4a39-d21b-11ea-bc27-002590db5748’ to ‘/dev/sdn2’
spa_misc.c:2311:spa_import_progress_set_notes_impl(): ‘Nexsan_2x7_raidz2’ Loading checkpoint txg
spa_misc.c:2311:spa_import_progress_set_notes_impl(): ‘Nexsan_2x7_raidz2’ Loading indirect vdev metadata
spa_misc.c:2311:spa_import_progress_set_notes_impl(): ‘Nexsan_2x7_raidz2’ Checking feature flags
spa_misc.c:404:spa_load_failed(): spa_load(Nexsan_2x7_raidz2, config trusted): FAILED: error getting refcount for feature org.illumos:lz4_compress [error=52]
spa_misc.c:418:spa_load_note(): spa_load(Nexsan_2x7_raidz2, config trusted): UNLOADING
ZFS_DBGMSG(zdb) END
root@truenas[/home/admin]#

---------WEBGUI IMPORT Error-----------
Error: concurrent.futures.process.RemoteTraceback:
“”"
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs/pool_actions.py", line 227, in import_pool
zfs.import_pool(found, pool_name, properties, missing_log=missing_log, any_host=any_host)
File “libzfs.pyx”, line 1369, in libzfs.ZFS.import_pool
File “libzfs.pyx”, line 1397, in libzfs.ZFS.__import_pool
libzfs.ZFSException: cannot import ‘Nexsan_2x7_raidz2’ as ‘Nexsan_2x7_raidz2’: I/O error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3.11/concurrent/futures/process.py”, line 256, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 112, in main_worker
res = MIDDLEWARE._run(*call_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 46, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 34, in call
with Client(f’ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock’, py_exceptions=True) as c:
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 40, in call
return methodobj(*params)
^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 191, in nf
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs/pool_actions.py", line 207, in import_pool
with libzfs.ZFS() as zfs:
File “libzfs.pyx”, line 529, in libzfs.ZFS.exit
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs/pool_actions.py", line 231, in import_pool
raise CallError(f’Failed to import {pool_name!r} pool: {e}', e.code)
middlewared.service_exception.CallError: [EZFS_IO] Failed to import ‘Nexsan_2x7_raidz2’ pool: cannot import ‘Nexsan_2x7_raidz2’ as ‘Nexsan_2x7_raidz2’: I/O error
“”"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 469, in run
await self.future
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 511, in _run_body
rv = await self.method(*args)
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 187, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 47, in nf
res = await f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool/import_pool.py", line 113, in import_pool
await self.middleware.call(‘zfs.pool.import_pool’, guid, opts, any_host, use_cachefile, new_name)
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1564, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1425, in _call
return await self._call_worker(name, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1431, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1337, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1321, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EZFS_IO] Failed to import ‘Nexsan_2x7_raidz2’ pool: cannot import ‘Nexsan_2x7_raidz2’ as ‘Nexsan_2x7_raidz2’: I/O error

2

My guess is that you have 1 dead drive - but this is not what is stopping the pool importing. There are a couple of issue reports on OpenZFS from a single user who had this happen twice after a power failure (only user, happened twice - suggesting that he had a hardware issue), but there was never a diagnosis or fix.

I suspect that your pool is toast and you need to rebuild and restore from backup.

But before you do that, we should probably check that your hardware design is suitable. So please post a description of your disk subsystem hardware (storage controllers, disk models and how they are connected).

3

Is this virtualised TrueNAS on Proxmox, by any (mis)chance?

4

root@truenas[/home/admin]# !69
echo 1 >> /sys/module/zfs/parameters/zfs_max_missing_tvds
root@truenas[/home/admin]# !70
echo 0 >> /sys/module/zfs/parameters/spa_load_verify_metadata
root@truenas[/home/admin]# !71
echo 0 >> /sys/module/zfs/parameters/spa_load_verify_data
root@truenas[/home/admin]# !73
zpool import -FX Nexsan_2x7_raidz2
cannot import ‘Nexsan_2x7_raidz2’: one or more devices is currently unavailable
root@truenas[/home/admin]# zpool import
pool: Nexsan_2x7_raidz2
id: 8972354718331620302
state: FAULTED
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the ‘-f’ flag.
see: Message ID: ZFS-8000-5E — OpenZFS documentation
config:

Nexsan_2x7_raidz2                         FAULTED  corrupted data
  raidz2-0                                DEGRADED
    d4f45115-d21b-11ea-bc27-002590db5748  ONLINE
    d51021f0-d21b-11ea-bc27-002590db5748  ONLINE
    d52f2438-d21b-11ea-bc27-002590db5748  ONLINE
    3958782095893599654                   UNAVAIL
    d6630266-d21b-11ea-bc27-002590db5748  ONLINE
    d6a69574-d21b-11ea-bc27-002590db5748  ONLINE
    d6b4ee02-d21b-11ea-bc27-002590db5748  ONLINE
  raidz2-1                                ONLINE
    d4fd3610-d21b-11ea-bc27-002590db5748  ONLINE
    d5b1bc18-d21b-11ea-bc27-002590db5748  ONLINE
    d60ccef4-d21b-11ea-bc27-002590db5748  ONLINE
    d68f5c7c-d21b-11ea-bc27-002590db5748  ONLINE
    d648a21a-d21b-11ea-bc27-002590db5748  ONLINE
    d67abf95-d21b-11ea-bc27-002590db5748  ONLINE
    d6ca4a39-d21b-11ea-bc27-002590db5748  ONLINE

root@truenas[/home/admin]# zpool import -f Nexsan_2x7_raisz2
cannot import ‘Nexsan_2x7_raisz2’: no such pool available
root@truenas[/home/admin]# zpool import -f Nexsan_2x7_raidz2
cannot import ‘Nexsan_2x7_raidz2’: I/O error
Destroy and re-create the pool from
a backup source.
root@truenas[/home/admin]# zpool import -f -o readonly=on Nexsan_2x7_raidz2
cannot import ‘Nexsan_2x7_raidz2’: I/O error
Destroy and re-create the pool from
a backup source.
root@truenas[/home/admin]# zdb -eul Nexsan_2x7_raidz2

LABEL 0

version: 5000
name: 'Nexsan_2x7_raidz2'
state: 0
txg: 9241005
pool_guid: 8972354718331620302
errata: 0
hostid: 1362372599
hostname: 'truenas'
top_guid: 6983679603137452165
guid: 11736308515974993500
vdev_children: 2
vdev_tree:
    type: 'raidz'
    id: 1
    guid: 6983679603137452165
    nparity: 2
    metaslab_array: 52
    metaslab_shift: 37
    ashift: 12
    asize: 13987726491648
    is_log: 0
    create_txg: 4
    children[0]:
        type: 'disk'
        id: 0
        guid: 2335491171700643644
        path: '/dev/disk/by-partuuid/d4fd3610-d21b-11ea-bc27-002590db5748'
        phys_path: 'id1,enc@n500304800374a57d/type@0/slot@c/elmdesc@Slot_12/p2'
        DTL: 282
        create_txg: 4
    children[1]:
        type: 'disk'
        id: 1
        guid: 3136592629563023909
        path: '/dev/disk/by-partuuid/d5b1bc18-d21b-11ea-bc27-002590db5748'
        phys_path: 'id1,enc@n500304800374a57d/type@0/slot@a/elmdesc@Slot_10/p2'
        DTL: 281
        create_txg: 4
    children[2]:
        type: 'disk'
        id: 2
        guid: 11736308515974993500
        path: '/dev/disk/by-partuuid/d60ccef4-d21b-11ea-bc27-002590db5748'
        phys_path: 'id1,enc@n500304800374a57d/type@0/slot@9/elmdesc@Slot_09/p2'
        DTL: 280
        create_txg: 4
    children[3]:
        type: 'disk'
        id: 3
        guid: 13479457084483254071
        path: '/dev/disk/by-partuuid/d68f5c7c-d21b-11ea-bc27-002590db5748'
        phys_path: 'id1,enc@n500304800374a57d/type@0/slot@f/elmdesc@Slot_15/p2'
        DTL: 279
        create_txg: 4
    children[4]:
        type: 'disk'
        id: 4
        guid: 5119149219636539909
        path: '/dev/disk/by-partuuid/d648a21a-d21b-11ea-bc27-002590db5748'
        phys_path: 'id1,enc@n500304800374a57d/type@0/slot@d/elmdesc@Slot_13/p2'
        DTL: 278
        create_txg: 4
    children[5]:
        type: 'disk'
        id: 5
        guid: 14902355098894633373
        path: '/dev/disk/by-partuuid/d67abf95-d21b-11ea-bc27-002590db5748'
        phys_path: 'id1,enc@n500304800374a57d/type@0/slot@b/elmdesc@Slot_11/p2'
        DTL: 277
        create_txg: 4
    children[6]:
        type: 'disk'
        id: 6
        guid: 15750852432423240846
        path: '/dev/disk/by-partuuid/d6ca4a39-d21b-11ea-bc27-002590db5748'
        phys_path: 'id1,enc@n500304800374a57d/type@0/slot@e/elmdesc@Slot_14/p2'
        DTL: 276
        create_txg: 4
features_for_read:
    com.delphix:hole_birth
    com.delphix:embedded_data
labels = 0 1 2 3 
Uberblock[0]
magic = 0000000000bab10c
version = 5000
txg = 9254816
guid_sum = 3805491763838847193
timestamp = 1726245672 UTC = Fri Sep 13 09:41:12 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[1]
magic = 0000000000bab10c
version = 5000
txg = 9255905
guid_sum = 3805491763838847193
timestamp = 1726251247 UTC = Fri Sep 13 11:14:07 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[2]
magic = 0000000000bab10c
version = 5000
txg = 9255906
guid_sum = 3805491763838847193
timestamp = 1726251252 UTC = Fri Sep 13 11:14:12 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[3]
magic = 0000000000bab10c
version = 5000
txg = 9255907
guid_sum = 3805491763838847193
timestamp = 1726251258 UTC = Fri Sep 13 11:14:18 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[4]
magic = 0000000000bab10c
version = 5000
txg = 9255908
guid_sum = 3805491763838847193
timestamp = 1726251263 UTC = Fri Sep 13 11:14:23 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[5]
magic = 0000000000bab10c
version = 5000
txg = 9255365
guid_sum = 3805491763838847193
timestamp = 1726248483 UTC = Fri Sep 13 10:28:03 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[6]
magic = 0000000000bab10c
version = 5000
txg = 9253638
guid_sum = 3805491763838847193
timestamp = 1726239640 UTC = Fri Sep 13 08:00:40 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[7]
magic = 0000000000bab10c
version = 5000
txg = 9253127
guid_sum = 3805491763838847193
timestamp = 1726237024 UTC = Fri Sep 13 07:17:04 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[8]
magic = 0000000000bab10c
version = 5000
txg = 9255944
guid_sum = 3805491763838847193
timestamp = 1726251447 UTC = Fri Sep 13 11:17:27 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[9]
magic = 0000000000bab10c
version = 5000
txg = 9256105
guid_sum = 3805491763838847193
timestamp = 1726252271 UTC = Fri Sep 13 11:31:11 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[10]
magic = 0000000000bab10c
version = 5000
txg = 9256106
guid_sum = 3805491763838847193
timestamp = 1726252276 UTC = Fri Sep 13 11:31:16 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[11]
magic = 0000000000bab10c
version = 5000
txg = 9256107
guid_sum = 3805491763838847193
timestamp = 1726252282 UTC = Fri Sep 13 11:31:22 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[12]
magic = 0000000000bab10c
version = 5000
txg = 9256108
guid_sum = 3805491763838847193
timestamp = 1726252287 UTC = Fri Sep 13 11:31:27 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[13]
magic = 0000000000bab10c
version = 5000
txg = 9255565
guid_sum = 3805491763838847193
timestamp = 1726249507 UTC = Fri Sep 13 10:45:07 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[14]
magic = 0000000000bab10c
version = 5000
txg = 9256046
guid_sum = 3805491763838847193
timestamp = 1726251969 UTC = Fri Sep 13 11:26:09 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[15]
magic = 0000000000bab10c
version = 5000
txg = 9255695
guid_sum = 3805491763838847193
timestamp = 1726250172 UTC = Fri Sep 13 10:56:12 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[16]
magic = 0000000000bab10c
version = 5000
txg = 9254992
guid_sum = 3805491763838847193
timestamp = 1726246573 UTC = Fri Sep 13 09:56:13 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[17]
magic = 0000000000bab10c
version = 5000
txg = 9255505
guid_sum = 3805491763838847193
timestamp = 1726249199 UTC = Fri Sep 13 10:39:59 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[18]
magic = 0000000000bab10c
version = 5000
txg = 9256050
guid_sum = 3805491763838847193
timestamp = 1726251990 UTC = Fri Sep 13 11:26:30 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[19]
magic = 0000000000bab10c
version = 5000
txg = 9255507
guid_sum = 3805491763838847193
timestamp = 1726249210 UTC = Fri Sep 13 10:40:10 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[20]
magic = 0000000000bab10c
version = 5000
txg = 9255956
guid_sum = 3805491763838847193
timestamp = 1726251508 UTC = Fri Sep 13 11:18:28 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[21]
magic = 0000000000bab10c
version = 5000
txg = 9255765
guid_sum = 3805491763838847193
timestamp = 1726250531 UTC = Fri Sep 13 11:02:11 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[22]
magic = 0000000000bab10c
version = 5000
txg = 9252790
guid_sum = 3805491763838847193
timestamp = 1726235299 UTC = Fri Sep 13 06:48:19 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[23]
magic = 0000000000bab10c
version = 5000
txg = 9254359
guid_sum = 3805491763838847193
timestamp = 1726243332 UTC = Fri Sep 13 09:02:12 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[24]
magic = 0000000000bab10c
version = 5000
txg = 9256120
guid_sum = 3805491763838847193
timestamp = 1726252348 UTC = Fri Sep 13 11:32:28 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[25]
magic = 0000000000bab10c
version = 5000
txg = 9255961
guid_sum = 3805491763838847193
timestamp = 1726251534 UTC = Fri Sep 13 11:18:54 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[26]
magic = 0000000000bab10c
version = 5000
txg = 9256058
guid_sum = 3805491763838847193
timestamp = 1726252031 UTC = Fri Sep 13 11:27:11 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[27]
magic = 0000000000bab10c
version = 5000
txg = 9255707
guid_sum = 3805491763838847193
timestamp = 1726250234 UTC = Fri Sep 13 10:57:14 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[28]
magic = 0000000000bab10c
version = 5000
txg = 9255708
guid_sum = 3805491763838847193
timestamp = 1726250239 UTC = Fri Sep 13 10:57:19 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[29]
magic = 0000000000bab10c
version = 5000
txg = 9255965
guid_sum = 3805491763838847193
timestamp = 1726251555 UTC = Fri Sep 13 11:19:15 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[30]
magic = 0000000000bab10c
version = 5000
txg = 9255870
guid_sum = 3805491763838847193
timestamp = 1726251068 UTC = Fri Sep 13 11:11:08 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3 
Uberblock[31]
magic = 0000000000bab10c
version = 5000
txg = 9255615
guid_sum = 3805491763838847193
timestamp = 1726249763 UTC = Fri Sep 13 10:49:23 2024
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
    labels = 0 1 2 3

root@truenas[/home/admin]# zpool import -FXT 9241005 Nexsan_2x7_raidz2
cannot import ‘Nexsan_2x7_raidz2’: one or more devices is currently unavailable
root@truenas[/home/admin]# zpool import -FXT 9255615 Nexsan_2x7_raidz2
cannot import ‘Nexsan_2x7_raidz2’: one or more devices is currently unavailable

5

Truenas scale on bare-metal.

Its a Nexsan NST 2000

1 Like
6

Thanks for the link.
If you’ve reached the point of desperate measures that would be
zpool import -fFX Nexsan_2x7_raidz2

7

If you do manage to get the pool force-imported, then you need to run a scrub on it as a next step.

8

This looks like some serious hardware, and at a first glance it looks like it is well engineered for the NAS space.

However I have not been able to find very much information about the HBA in these boxes, but they seem to have dual path controllers which may need special software support.

I have also not been able to find very much about whether Nexsan support running TrueNAS on their hardware or whether TrueNAS supports Nexsan hardware.

There are a few posts on the old Community forums about Nexsan systems, so I guess that other people have successfully used some of their hardware.

My first thoughts are to wonder how the HBA is configured…

9

Indeed, non-zero risk that it’s using some form of RAID controller.

This site suggests the NST 2000 supports various RAID levels:
https://sunstarco.com/unified-hybrid-storage-smb/

10

The good news is that if it is in RAID mode, at least they have it configured as JBODs.

11

zsh: command not found: screen
root@truenas[/home/admin]# zpool import -fFX Nexsan_2.7_raidz2
cannot import ‘Nexsan_2.7_raidz2’: no such pool available
root@truenas[/home/admin]# zpool import -fFX Nexsan_2x7_raidz2
cannot import ‘Nexsan_2x7_raidz2’: one or more devices is currently unavailable
root@truenas[/home/admin]# zpool import Nexsan_2x7_raidz2
cannot import ‘Nexsan_2x7_raidz2’: I/O error
Destroy and re-create the pool from
a backup source.
root@truenas[/home/admin]#
root@truenas[/home/admin]#

12

should i upload a debug?

13

Is there no way to online the pool in a partial state with the one good vdev? It feels like there should be vdev repair tools (disk swap/exchange/etc) to work with. I.e. fix the vdev then attempt to import/online etc.

Or is there a more manual way to recreate / re-import / online the pool?

14

ZFS does not have repair tools beyond what a scrub could do. I tries hard not to break, but when it does break the official policy is: “Restore from backup”.