Reimporting pool in truenas scale fails

Hello,
yesterday one of my 2 pools was suddenly missing. Today I had 2 unassigned disks instead. (Bare metal, Dragonfish-24.04.2.5)

I tried to import the pool in the GUI:

details
 Error importing pool
[EZFS_BADDEV] Failed to import 'data16' pool: cannot import 'data16' as 'data16': one or more devices is currently unavailable
remove_circle_outline
More info...
 Error: concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/pool_actions.py", line 227, in import_pool
    zfs.import_pool(found, pool_name, properties, missing_log=missing_log, any_host=any_host)
  File "libzfs.pyx", line 1369, in libzfs.ZFS.import_pool
  File "libzfs.pyx", line 1397, in libzfs.ZFS.__import_pool
libzfs.ZFSException: cannot import 'data16' as 'data16': one or more devices is currently unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
    res = MIDDLEWARE._run(*call_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
    with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 191, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/pool_actions.py", line 207, in import_pool
    with libzfs.ZFS() as zfs:
  File "libzfs.pyx", line 529, in libzfs.ZFS.__exit__
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/pool_actions.py", line 231, in import_pool
    raise CallError(f'Failed to import {pool_name!r} pool: {e}', e.code)
middlewared.service_exception.CallError: [EZFS_BADDEV] Failed to import 'data16' pool: cannot import 'data16' as 'data16': one or more devices is currently unavailable
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 469, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 511, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 47, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/import_pool.py", line 113, in import_pool
    await self.middleware.call('zfs.pool.import_pool', guid, opts, any_host, use_cachefile, new_name)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1564, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1425, in _call
    return await self._call_worker(name, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1431, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1337, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EZFS_BADDEV] Failed to import 'data16' pool: cannot import 'data16' as 'data16': one or more devices is currently unavailable

The same problem as a shell command:

root@truenas[~]# zpool import -F -R /mnt data16 
cannot import 'data16': I/O error
        Destroy and re-create the pool from
        a backup source.
root@truenas[~]# zpool import -d /dev/disk/by-id
   pool: data16
     id: 9136863319262047294
  state: ONLINE
status: Some supported features are not enabled on the pool.
        (Note that they may be intentionally disabled if the
        'compatibility' property is set.)
 action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
 config:

        data16                            ONLINE
          mirror-0                        ONLINE
            wwn-0x5000c500c71d3efe-part2  ONLINE
            wwn-0x5000c500e3b82913-part2  ONLINE

Information of the metadata isn’t promising either:

      
root@truenas[~]# zdb -e data16 

Configuration for import:
        vdev_children: 1
        version: 5000
        pool_guid: 9136863319262047294
        name: 'data16'
        state: 0
        hostid: 1018141771
        hostname: 'truenas'
        vdev_tree:
            type: 'root'
            id: 0
            guid: 9136863319262047294
            children[0]:
                type: 'mirror'
                id: 0
                guid: 6318700606881195242
                metaslab_array: 132
                metaslab_shift: 34
                ashift: 12
                asize: 15998748196864
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 8005179225333517566
                    whole_disk: 0
                    DTL: 17579
                    create_txg: 4
                    path: '/dev/sdb2'
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 11974721465407886050
                    whole_disk: 0
                    DTL: 17578
                    create_txg: 4
                    path: '/dev/sdf2'
        load-policy:
            load-request-txg: 18446744073709551615
            load-rewind-policy: 2
MOS Configuration:
        version: 5000
        name: 'data16'
        state: 0
        txg: 5391635
        pool_guid: 9136863319262047294
        errata: 0
        hostid: 1018141771
        hostname: 'truenas'
        com.delphix:has_per_vdev_zaps
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 9136863319262047294
            create_txg: 4
            children[0]:
                type: 'mirror'
                id: 0
                guid: 6318700606881195242
                metaslab_array: 132
                metaslab_shift: 34
                ashift: 12
                asize: 15998748196864
                is_log: 0
                create_txg: 4
                com.delphix:vdev_zap_top: 129
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 8005179225333517566
                    path: '/dev/disk/by-partuuid/dde3f5ff-166c-473a-9c4b-e29c54b6b129'
                    whole_disk: 0
                    DTL: 17579
                    create_txg: 4
                    com.delphix:vdev_zap_leaf: 130
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 11974721465407886050
                    path: '/dev/disk/by-partuuid/513e9377-340b-4b13-b6b2-b25579e118b8'
                   whole_disk: 0
                    DTL: 17578
                    create_txg: 4
                    com.delphix:vdev_zap_leaf: 131
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data

Uberblock:
        magic = 0000000000bab10c
        version = 5000
        txg = 5392324
        guid_sum = 16988720543175094536
        timestamp = 1738455572 UTC = Sun Feb  2 01:19:32 2025
        mmp_magic = 00000000a11cea11
        mmp_delay = 0
        mmp_valid = 0
        checkpoint_txg = 0

All DDTs are empty
BRT: unsupported on this pool

Metaslabs:
        vdev          0      ms_unflushed_phys object 136
        metaslabs   931   offset                spacemap          free        
        ---------------   -------------------   ---------------   ------------
        metaslab      0   offset            0   spacemap    471   free    11.8G
        On-disk histogram:              fragmentation 3
                         12:    542 **********************
                         13:    423 *****************
                         14:    276 ***********
                         15:    933 *************************************
                         16:    758 ******************************
                         17:   1019 ****************************************
                         18:    827 *********************************
                         19:    117 *****
                         20:     65 ***
                         21:    132 ******
                         22:    216 *********
                         23:     45 **
                         24:     34 **
                         25:     33 **
                         26:     17 *
                         27:      3 *
                         28:      1 *
                         29:      0 
                         30:      0 
                         31:      0 
                         32:      1 *
space map object 471:
  smp_length = 0x19750
  smp_alloc = 0x10b394000
        Flush data:
        unflushed txg=5392323

        metaslab      1   offset    400000000   spacemap    144   free    15.9G
        On-disk histogram:              fragmentation 0
                         12:    415 ****************************************
                         13:     37 ****
                         14:     24 ***
                         15:      8 *
                         16:     17 **
                         17:     13 **
                         18:      8 *
                         19:     17 **
                         20:     40 ****
                         21:    239 ************************
                         22:     34 ****
                         23:     45 *****
                         24:     59 ******
                         25:     60 ******
                         26:     37 ****
                         27:     21 ***
                         28:      0 
                         29:      0 
                         30:      0 
                         31:      0 
                         32:      1 *
space map object 144:
  smp_length = 0x23578
  smp_alloc = 0x52ed000
        Flush data:
        unflushed txg=5392321
       metaslab      2   offset    800000000   spacemap    143   free    6.53G
        On-disk histogram:              fragmentation 5
                         12:    970 *************************
                         13:    669 *****************
                         14:   1085 ****************************
                         15:   1369 ***********************************
                         16:   1607 ****************************************
                         17:   1342 **********************************
                         18:    737 *******************
                         19:     36 *
                         20:    171 *****

....


       328    2    16K   128K   168K     512   128K  100.00  SPA space map
       331    1    32K   128K    48K     512   256K  100.00  bpobj
       336    1    32K    512      0     512    512    0.00  DSL directory
       337    1    32K    512      0     512    512  100.00  DSL props
       338    1    32K    512      0     512    512  100.00  DSL directory child map
       339    2    32K    16K    36K     512    32K  100.00  zap
       340    1    32K    512      0     512    512  100.00  DSL props
       341    1    32K    512      0     512    512  100.00  DSL directory child map
       342    1    32K    512      0     512    512  100.00  zap
       343    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
       344    1    32K    512      0     512    512  100.00  zap
       345    1    32K    512      0     512    512  100.00  DSL dataset snap map
       346    1    32K    512      0     512    512  100.00  DSL deadlist map
       347    2    32K    16K    36K     512    32K  100.00  zap
       348    1    32K    512      0     512    512  100.00  DSL props
       349    1    32K    512      0     512    512  100.00  DSL directory child map
       350    1    32K    512      0     512    512  100.00  zap
       351    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
       354    1    32K   128K    12K     512   128K  100.00  bpobj
       382    1    32K    512      0     512    512  100.00  zap
       383    1    32K    512      0     512    512  100.00  DSL deadlist map
       384    1    32K   128K    12K     512   128K  100.00  bpobj
       385    2    16K   128K   936K     512   512K  100.00  SPA space map
       386    2    16K   128K   600K     512   384K  100.00  SPA space map
       388    1    32K    512      0     512    512    0.00  DSL directory
       389    1    32K    16K    24K     512    32K  100.00  DSL props
       390    1    32K    512      0     512    512  100.00  DSL directory child map
       391    1    32K    512      0     512    512  100.00  zap
       392    1    32K    512      0     512    512  100.00  DSL dataset snap map
       393    1    32K    512      0     512    512  100.00  DSL deadlist map
       395    1    32K   128K    12K     512   128K  100.00  bpobj
       399    1    32K    512      0     512    512    0.00  DSL directory
       400    1    32K    16K    24K     512    32K  100.00  DSL props
       401    1    32K    512      0     512    512  100.00  DSL directory child map
       402    1    32K    512      0     512    512  100.00  zap
       403    1    32K    512      0     512    512  100.00  DSL dataset snap map
       404    1    32K    512      0     512    512  100.00  DSL deadlist map
       407    1    32K    512      0     512    512    0.00  DSL directory
       408    1    32K    512      0     512    512  100.00  DSL props
       409    1    32K    512      0     512    512  100.00  DSL directory child map
       410    1    32K    512      0     512    512  100.00  zap
       411    1    32K    512      0     512    512  100.00  DSL dataset snap map
       413    1    32K    512      0     512    512  100.00  DSL deadlist map
       418    1    32K    512      0     512    512  100.00  zap
       420    1    32K    512      0     512    512  100.00  zap
       421    1    32K    512      0     512    512  100.00  DSL dir clones
       422    1    32K   128K    60K     512   256K  100.00  bpobj
       437    2    32K    16K    36K     512    32K  100.00  zap
       438    1    32K    16K    24K     512    32K  100.00  DSL props
       439    1    32K    512      0     512    512  100.00  DSL directory child map
       440    2    32K    16K    36K     512    32K  100.00  zap
       441    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
       471    2    16K   128K   240K     512   128K  100.00  SPA space map
       490    2    32K    16K    36K     512    32K  100.00  zap
       491    1    32K    512      0     512    512  100.00  DSL props
       492    1    32K    512      0     512    512  100.00  DSL directory child map
       493    1    32K    512      0     512    512  100.00  zap
       494    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
       495    1    32K    512      0     512    512  100.00  DSL deadlist map
       512    1    32K    512      0     512    512  100.00  DSL dataset snap map
       513    1    32K    512      0     512    512  100.00  DSL deadlist map
       516    2    16K   128K   828K     512   512K  100.00  SPA space map
       518    1    32K   128K    12K     512   128K  100.00  bpobj
       520    2    16K   128K   456K     512   256K  100.00  SPA space map
       521    2    16K   128K   624K     512   384K  100.00  SPA space map
       527    2    32K    16K    36K     512    32K  100.00  zap
       528    1    32K    512      0     512    512  100.00  DSL props
       530    1    32K    512      0     512    512  100.00  DSL directory child map
       531    1    32K    512      0     512    512  100.00  zap
       532    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
       535    2    32K    16K    36K     512    32K  100.00  zap
       536    1    32K    512      0     512    512  100.00  DSL props
       537    1    32K    512      0     512    512  100.00  DSL directory child map
       538    1    32K    512      0     512    512  100.00  zap
       539    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
       546    1    32K   128K    12K     512   128K  100.00  bpobj
       547    1    32K   128K    12K     512   128K  100.00  bpobj
       552    1    32K   128K    12K     512   128K  100.00  bpobj
       561    1    32K   128K    12K     512   128K  100.00  bpobj
       627    1    32K    512      0     512    512    0.00  DSL directory
       628    1    32K    16K    24K     512    32K  100.00  DSL props
       629    1    32K    512      0     512    512  100.00  DSL directory child map
       630    1    32K    512      0     512    512  100.00  zap
       631    1    32K    512      0     512    512  100.00  DSL dataset snap map
       632    1    32K    512      0     512    512  100.00  DSL deadlist map
       635    2    32K    16K    36K     512    32K  100.00  zap
       636    1    32K    512      0     512    512    0.00  DSL directory
       637    1    32K    16K    24K     512    32K  100.00  DSL props
       638    1    32K    512      0     512    512  100.00  DSL directory child map
       639    1    32K    512      0     512    512  100.00  zap
       640    1    32K    512      0     512    512  100.00  zap
       641    1    32K    512      0     512    512  100.00  DSL deadlist map
       642    1    32K   128K    12K     512   128K  100.00  bpobj
       651    1    32K   128K    12K     512   128K  100.00  bpobj
       653    1    32K    512      0     512    512    0.00  DSL directory
       654    1    32K    512      0     512    512  100.00  DSL props
       655    1    32K    512      0     512    512  100.00  DSL directory child map
       656    1    32K    512      0     512    512  100.00  zap
       657    1    32K    512      0     512    512  100.00  DSL dataset snap map
       658    1    32K   128K    48K     512   128K  100.00  bpobj
       659    1    32K    512      0     512    512  100.00  DSL deadlist map
       662    1    32K    512      0     512    512    0.00  DSL directory
       663    1    32K    512      0     512    512  100.00  DSL props
       664    1    32K    512      0     512    512  100.00  DSL directory child map
       665    1    32K    512      0     512    512  100.00  zap
       666    1    32K    512      0     512    512  100.00  DSL dataset snap map
       667    1    32K    512      0     512    512  100.00  DSL deadlist map
       675    1    32K    512      0     512    512  100.00  zap
       676    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
       677    1    32K    512      0     512    512  100.00  zap
       678    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
       679    1    32K    512      0     512    512  100.00  zap
       680    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
       681    1    32K    512      0     512    512  100.00  zap
       682    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
       708    1    32K    512      0     512    512    0.00  DSL directory
       709    1    32K    16K    24K     512    32K  100.00  DSL props
       710    1    32K    512      0     512    512  100.00  DSL directory child map
...
      6270    1    32K    512      0     512    512  100.00  zap
      6271    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
      6408    1    32K    512      0     512    512  100.00  zap
      6409    1    32K    512      0     512    512  100.00  DSL deadlist map
      6410    1    32K    512      0     512    512  100.00  zap
      6411    1    32K    512      0     512    512  100.00  DSL deadlist map
      6412    1    32K    512      0     512    512  100.00  zap
      6413    1    32K    512      0     512    512  100.00  DSL deadlist map
      6414    1    32K    512      0     512    512  100.00  zap
      6415    1    32K    512      0     512    512  100.00  DSL deadlist map
      6427    1    32K   128K    12K     512   128K  100.00  bpobj
      6444    1    32K    512      0     512    512  100.00  zap
      6445    1    32K    512      0     512    512  100.00  DSL deadlist map
      6446    1    32K    512      0     512    512  100.00  zap
      6447    1    32K    512      0     512    512  100.00  DSL deadlist map
      6448    1    32K    512      0     512    512  100.00  zap
      6449    1    32K    512      0     512    512  100.00  DSL deadlist map
      6450    1    32K    512      0     512    512  100.00  zap
      6451    1    32K    512      0     512    512  100.00  DSL deadlist map
      6452    1    32K    512      0     512    512  100.00  zap
      6453    1    32K    512      0     512    512  100.00  DSL deadlist map
      6454    1    32K    512      0     512    512  100.00  zap
      6455    1    32K    512      0     512    512  100.00  DSL deadlist map
      6456    1    32K    512      0     512    512  100.00  zap
      6457    1    32K    512      0     512    512  100.00  DSL deadlist map
      6458    1    32K    512      0     512    512  100.00  zap
      6459    1    32K    512      0     512    512  100.00  DSL deadlist map
      6460    1    32K    512      0     512    512  100.00  zap
      6461    1    32K    512      0     512    512  100.00  DSL deadlist map
      6462    1    32K    512      0     512    512  100.00  zap
      6463    1    32K    512      0     512    512  100.00  DSL deadlist map
      6464    1    32K    512      0     512    512  100.00  zap
      6465    1    32K    512      0     512    512  100.00  DSL deadlist map
      6466    1    32K    512      0     512    512  100.00  zap
      6467    1    32K    512      0     512    512  100.00  DSL deadlist map
      6468    1    32K    512      0     512    512  100.00  zap
      6469    1    32K    512      0     512    512  100.00  DSL deadlist map
      6534    1    32K   128K    12K     512   128K  100.00  bpobj
      6538    1    32K   128K    12K     512   128K  100.00  bpobj
      6556    1    32K    512      0     512    512  100.00  zap
      6557    1    32K    512      0     512    512  100.00  DSL deadlist map
      6558    1    32K    512      0     512    512  100.00  zap
      6559    1    32K    512      0     512    512  100.00  DSL deadlist map
      6560    1    32K    512      0     512    512  100.00  zap
      6561    1    32K    512      0     512    512  100.00  DSL deadlist map
      6562    1    32K    512      0     512    512  100.00  zap
      6563    1    32K    512      0     512    512  100.00  DSL deadlist map
      6564    1    32K    512      0     512    512  100.00  zap
      6565    1    32K    512      0     512    512  100.00  DSL deadlist map
      6566    1    32K    512      0     512    512  100.00  zap
      6567    1    32K    512      0     512    512  100.00  DSL deadlist map
      6568    1    32K    512      0     512    512  100.00  zap
      6569    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
      6570    1    32K    512      0     512    512  100.00  zap
      6571    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
      6572    1    32K    512      0     512    512  100.00  zap
      6573    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
      6574    1    32K    512      0     512    512  100.00  zap
      6575    1    32K    512      0     512    512  100.00  DSL deadlist map
      6576    1    32K    512      0     512    512  100.00  zap
      6577    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
      6578    1    32K    512      0     512    512  100.00  zap
      6579    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
      6580    1    32K    512      0     512    512  100.00  zap
      6581    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
      6582    1    32K    512      0     512    512  100.00  zap
      6583    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
      6584    1    32K    512      0     512    512  100.00  zap
      6585    1    32K    512      0     512    512  100.00  DSL deadlist map
      6586    2    32K   128K   156K     512   512K  100.00  bpobj
      6587    1    32K   128K    12K     512   128K  100.00  bpobj
      6601    1    32K   128K    12K     512   128K  100.00  bpobj
      6658    1    32K   128K    24K     512   128K  100.00  bpobj
      6670    1    32K   128K    12K     512   128K  100.00  bpobj
      6671    1    32K   128K    12K     512   128K  100.00  bpobj
      6677    1    32K   128K    12K     512   128K  100.00  bpobj
      6678    1    32K   128K    84K     512   256K  100.00  bpobj
      6753    2    16K   128K   504K     512   384K  100.00  SPA space map
      7726    1    32K   128K    12K     512   128K  100.00  bpobj
      8257    1    32K   128K    12K     512   128K  100.00  bpobj
      8537    2    16K   128K   624K     512   384K  100.00  SPA space map
      8538    2    16K   128K   792K     512   512K  100.00  SPA space map
      8814    1    16K   128K   228K     512   128K  100.00  SPA space map
      8815    2    16K   128K   516K     512   256K  100.00  SPA space map
      9277    1    16K   128K   204K     512   128K  100.00  SPA space map
      9278    2    16K   128K   132K     512   128K  100.00  SPA space map
      9298    1    32K   128K    12K     512   128K  100.00  bpobj
      9432    1    32K   128K    12K     512   128K  100.00  bpobj
      9552    1    32K    512      0     512    512  100.00  zap
      9553    1    32K    512      0     512    512  100.00  DSL deadlist map
      9554    1    32K    512      0     512    512  100.00  zap
      9555    1    32K    512      0     512    512  100.00  DSL deadlist map
      9556    1    32K    512      0     512    512  100.00  zap
      9557    1    32K    512      0     512    512  100.00  DSL deadlist map
      9558    1    32K    512      0     512    512  100.00  zap
      9559    1    32K    512      0     512    512  100.00  DSL deadlist map
      9560    1    32K    512      0     512    512  100.00  zap
      9561    1    32K    512      0     512    512  100.00  DSL deadlist map
      9562    1    32K    512      0     512    512  100.00  zap
      9563    1    32K    512      0     512    512  100.00  DSL deadlist map
      9565    1    32K   128K    36K     512   128K  100.00  bpobj
      9641    2    16K   128K   684K     512   640K   60.00  SPA space map
      9642    2    16K   128K   576K     512   384K  100.00  SPA space map
      9889    2    16K   128K   324K     512   640K   40.00  SPA space map
      9890    2    16K   128K   336K     512   256K  100.00  SPA space map
     10551    2    16K   128K   900K     512   512K  100.00  SPA space map
     10552    2    16K   128K   588K     512   384K  100.00  SPA space map
     11038    2    16K   128K   744K     512   512K  100.00  SPA space map
     11562    2    16K   128K   288K     512   256K  100.00  SPA space map
     11563    2    16K   128K   132K     512   128K  100.00  SPA space map
     11616    2    16K   128K   624K     512   384K  100.00  SPA space map
     11617    2    16K   128K   408K     512   256K  100.00  SPA space map
     11838    2    16K   128K   744K     512   640K   80.00  SPA space map
     11839    2    16K   128K   444K     512   256K  100.00  SPA space map
     11851    2    16K   128K   396K     512   256K  100.00  SPA space map
     11852    2    16K   128K   924K     512   512K  100.00  SPA space map
     12129    1    16K   128K   180K     512   128K  100.00  SPA space map
     12130    2    16K   128K   888K     512   512K  100.00  SPA space map
     12268    1    16K   128K   144K     512   128K  100.00  SPA space map
     12269    2    16K   128K   840K     512   640K   80.00  SPA space map
     12290    2    16K   128K   504K     512   256K  100.00  SPA space map
     12825    2    16K   128K   204K     512   128K  100.00  SPA space map
     12826    2    16K   128K   324K     512   256K  100.00  SPA space map
     13310    1    32K   128K    12K     512   128K  100.00  bpobj
     13516    2    16K   128K   792K     512   512K  100.00  SPA space map
     13517    2    16K   128K   528K     512   384K  100.00  SPA space map
     14029    2    16K   128K   984K     512   640K   80.00  SPA space map
     14220    2    16K   128K   576K     512   384K  100.00  SPA space map
     14221    2    16K   128K   348K     512   256K  100.00  SPA space map
     14239    2    16K   128K   696K     512   384K  100.00  SPA space map
     14240    2    16K   128K   504K     512   384K  100.00  SPA space map
     14479    2    16K   128K   420K     512   256K  100.00  SPA space map
     14480    2    16K   128K   600K     512   640K   60.00  SPA space map
     14740    2    16K   128K   864K     512   640K   80.00  SPA space map
     14741    2    16K   128K   972K     512   512K  100.00  SPA space map
     15011    2    16K   128K   324K     512   256K  100.00  SPA space map
     15012    2    16K   128K   324K     512   256K  100.00  SPA space map
     16262    1    32K    512      0     512    512  100.00  DSL deadlist map
     16264    1    32K    512      0     512    512  100.00  DSL deadlist map
     16266    1    32K    512      0     512    512  100.00  DSL deadlist map
     16270    1    32K    512      0     512    512  100.00  DSL deadlist map
     16272    1    32K    512      0     512    512  100.00  DSL deadlist map
     16278    1    32K    512      0     512    512  100.00  DSL deadlist map
     16437    2    16K   128K   504K     512   640K   60.00  SPA space map
     16438    2    16K   128K   660K     512   384K  100.00  SPA space map
     16751    2    16K   128K   780K     512   640K   80.00  SPA space map
     16752    2    16K   128K   744K     512   384K  100.00  SPA space map
     17417    1    32K   128K    12K     512   128K  100.00  bpobj
     17466    2    16K   128K   288K     512   256K  100.00  SPA space map
     17467    2    16K   128K   612K     512   384K  100.00  SPA space map
     17578    1    16K     4K      0     512     4K    0.00  SPA space map
     17579    1    16K     4K      0     512     4K    0.00  SPA space map
     18424    2    16K   128K   852K     512   512K  100.00  SPA space map
     18425    2    16K   128K   684K     512   384K  100.00  SPA space map
     18811    2    16K   128K   708K     512   640K   80.00  SPA space map
     18812    2    16K   128K   372K     512   256K  100.00  SPA space map
     19034    2    16K   128K   924K     512   512K  100.00  SPA space map
     19035    2    16K   128K  1008K     512   512K  100.00  SPA space map
     19069    1    16K   128K    84K     512   128K  100.00  SPA space map
     19070    2    16K   128K   456K     512   256K  100.00  SPA space map
     19437    2    16K   128K   876K     512   512K  100.00  SPA space map
     19438    2    16K   128K   912K     512   512K  100.00  SPA space map
     19745    2    16K   128K   720K     512   384K  100.00  SPA space map
     20093    2    16K   128K   276K     512   256K  100.00  SPA space map
     20094    2    16K   128K   684K     512   640K   60.00  SPA space map
     20226    1    32K    512      0     512    512    0.00  DSL directory
     20227    1    32K    16K    24K     512    32K  100.00  DSL props
     20228    1    32K    512      0     512    512  100.00  DSL directory child map
     20229    1    32K    512      0     512    512  100.00  zap
     20230    1    32K    512      0     512    512  100.00  DSL dataset snap map
     20231    1    32K    512      0     512    512  100.00  DSL deadlist map
     20359    1    32K    512      0     512    512    0.00  DSL directory
     20360    1    32K    16K    24K     512    32K  100.00  DSL props
     20361    1    32K    512      0     512    512  100.00  DSL directory child map
     20362    1    32K    512      0     512    512  100.00  zap
     20363    1    32K    512      0     512    512  100.00  DSL dataset snap map
     20364    1    32K    512      0     512    512  100.00  DSL deadlist map
     20893    1    16K   128K   204K     512   128K  100.00  SPA space map
     20894    2    16K   128K   564K     512   384K  100.00  SPA space map
     20911    1    16K   128K   144K     512   128K  100.00  SPA space map
     20912    2    16K   128K   816K     512   512K  100.00  SPA space map
     21006    2    16K   128K   840K     512   640K   80.00  SPA space map
     21880    2    32K    16K    36K     512    32K  100.00  zap
     21881    1    32K    16K    24K     512    32K  100.00  DSL props
     21882    1    32K    512      0     512    512  100.00  DSL directory child map
     21883    1    32K    512      0     512    512  100.00  zap
     21884    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     21959    2    32K    16K    36K     512    32K  100.00  zap
     21960    1    32K    512      0     512    512  100.00  DSL props
     21961    1    32K    512      0     512    512  100.00  DSL directory child map
     21962    1    32K    512      0     512    512  100.00  zap
     21963    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     21968    2    32K    16K    36K     512    32K  100.00  zap
     21969    1    32K    512      0     512    512  100.00  DSL props
     21970    1    32K    512      0     512    512  100.00  DSL directory child map
     21971    1    32K    512      0     512    512  100.00  zap
     21972    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     22098    2    32K    16K    36K     512    32K  100.00  zap
     22099    1    32K    512      0     512    512  100.00  DSL props
     22100    1    32K    512      0     512    512  100.00  DSL directory child map
     22101    1    32K    512      0     512    512  100.00  zap
     22102    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     22103    1    32K    512      0     512    512  100.00  DSL deadlist map
     22105    2    32K    16K    36K     512    32K  100.00  zap
     22106    1    32K    512      0     512    512  100.00  DSL props
     22107    1    32K    512      0     512    512  100.00  DSL directory child map
     22108    1    32K    512      0     512    512  100.00  zap
     22109    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     22110    1    32K    512      0     512    512  100.00  DSL deadlist map
     22114    1    32K   128K    12K     512   128K  100.00  bpobj
     22724    2    16K   128K   504K     512   384K  100.00  SPA space map
     22725    2    16K   128K   588K     512   384K  100.00  SPA space map
     22812    2    16K   128K   168K     512   640K   20.00  SPA space map
     22813    2    16K   128K   396K     512   256K  100.00  SPA space map
     23273    2    16K   128K   648K     512   384K  100.00  SPA space map
     23274    2    16K   128K   684K     512   384K  100.00  SPA space map
     23345    2    16K   128K   948K     512   512K  100.00  SPA space map
     23346    2    16K   128K   372K     512   256K  100.00  SPA space map
     23435    2    16K   128K   420K     512   256K  100.00  SPA space map
     23436    2    16K   128K   936K     512   512K  100.00  SPA space map
     23445    2    16K   128K   936K     512   512K  100.00  SPA space map
     23446    2    16K   128K   744K     512   512K  100.00  SPA space map
     25526    2    16K   128K   756K     512   384K  100.00  SPA space map
     25527    2    16K   128K   972K     512   640K  100.00  SPA space map
     25535    2    16K   128K    60K     512   128K  100.00  SPA space map
     25536    2    16K   128K   852K     512   512K  100.00  SPA space map
     26229    2    32K    16K    36K     512    32K  100.00  zap
     26230    1    32K    512      0     512    512  100.00  DSL props
     26231    1    32K    512      0     512    512  100.00  DSL directory child map
     26232    1    32K    512      0     512    512  100.00  zap
     26233    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     26234    1    32K    512      0     512    512  100.00  DSL deadlist map
     26236    2    32K    16K    36K     512    32K  100.00  zap
     26237    1    32K    512      0     512    512  100.00  DSL props
     26238    1    32K    512      0     512    512  100.00  DSL directory child map
     26239    1    32K    512      0     512    512  100.00  zap
     26329    2    32K    16K    36K     512    32K  100.00  zap
     26330    1    32K    512      0     512    512  100.00  DSL props
     26331    1    32K    512      0     512    512  100.00  DSL directory child map
     26332    1    32K    512      0     512    512  100.00  zap
     26333    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     26334    1    32K    512      0     512    512  100.00  DSL deadlist map
     26337    2    32K    16K    36K     512    32K  100.00  zap
     26338    1    32K    512      0     512    512  100.00  DSL props
     26339    1    32K    512      0     512    512  100.00  DSL directory child map
     26340    1    32K    512      0     512    512  100.00  zap
     26341    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     26342    1    32K    512      0     512    512  100.00  DSL deadlist map
     26416    2    32K    16K    36K     512    32K  100.00  zap
     26417    1    32K    512      0     512    512  100.00  DSL props
     26418    1    32K    512      0     512    512  100.00  DSL directory child map
     26419    1    32K    512      0     512    512  100.00  zap
     26420    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     26421    1    32K    512      0     512    512  100.00  DSL deadlist map
     26423    2    32K    16K    36K     512    32K  100.00  zap
     26424    1    32K    512      0     512    512  100.00  DSL props
     26425    1    32K    512      0     512    512  100.00  DSL directory child map
     26426    1    32K    512      0     512    512  100.00  zap
     26427    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     26428    1    32K    512      0     512    512  100.00  DSL deadlist map
     26505    2    32K    16K    36K     512    32K  100.00  zap
     26506    1    32K    512      0     512    512  100.00  DSL props
     26507    1    32K    512      0     512    512  100.00  DSL directory child map
...
     52692    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     52693    1    32K    512      0     512    512  100.00  DSL deadlist map
     52695    2    32K    16K    36K     512    32K  100.00  zap
     52696    1    32K    512      0     512    512  100.00  DSL props
     52697    1    32K    512      0     512    512  100.00  DSL directory child map
     52698    1    32K    512      0     512    512  100.00  zap
     52699    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     52700    1    32K    512      0     512    512  100.00  DSL deadlist map
     52718    1    32K   128K    12K     512   128K  100.00  bpobj
     52780    2    32K    16K    36K     512    32K  100.00  zap
     52781    1    32K    512      0     512    512  100.00  DSL props
     52782    1    32K    512      0     512    512  100.00  DSL directory child map
     52783    1    32K    512      0     512    512  100.00  zap
     52784    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     52785    1    32K    512      0     512    512  100.00  DSL deadlist map
     52789    1    32K    512      0     512    512  100.00  zap
     52790    1    32K    512      0     512    512  100.00  DSL deadlist map
     52791    1    32K    512      0     512    512  100.00  zap
     52792    1    32K    512      0     512    512  100.00  DSL deadlist map
     52793    1    32K    512      0     512    512  100.00  zap
     52794    1    32K    512      0     512    512  100.00  DSL deadlist map
     52795    1    32K    512      0     512    512  100.00  zap
     52796    1    32K    512      0     512    512  100.00  DSL deadlist map
     52797    1    32K    512      0     512    512  100.00  zap
     52798    1    32K    512      0     512    512  100.00  DSL deadlist map
     52799    1    32K    512      0     512    512  100.00  zap
     52800    1    32K    512      0     512    512  100.00  DSL deadlist map
     52802    1    32K   128K    12K     512   128K  100.00  bpobj
     52866    2    32K    16K    36K     512    32K  100.00  zap
     52867    1    32K    512      0     512    512  100.00  DSL props
     52868    1    32K    512      0     512    512  100.00  DSL directory child map
     52869    1    32K    512      0     512    512  100.00  zap
     52870    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     52871    1    32K    512      0     512    512  100.00  DSL deadlist map
     52876    1    32K   128K    48K     512   128K  100.00  bpobj
     52879    1    32K   128K    12K     512   128K  100.00  bpobj
     53068    2    16K   128K   936K     512   512K  100.00  SPA space map
     53079    2    16K   128K   720K     512   384K  100.00  SPA space map
     53080    2    16K   128K   648K     512   384K  100.00  SPA space map
     53111    2    16K   128K   564K     512   384K  100.00  SPA space map
     53112    2    16K   128K   240K     512   128K  100.00  SPA space map
     53214    2    16K   128K   840K     512   512K  100.00  SPA space map
     53215    2    16K   128K   780K     512   512K  100.00  SPA space map
     53598    2    16K   128K   780K     512   512K  100.00  SPA space map
     53599    2    16K   128K   360K     512   256K  100.00  SPA space map
     54067    2    16K   128K   492K     512   384K  100.00  SPA space map
     54068    2    16K   128K   468K     512   256K  100.00  SPA space map
     54317    2    16K   128K   480K     512   384K  100.00  SPA space map
     54431    2    16K   128K    36K     512   640K   20.00  SPA space map
     54432    2    16K   128K    84K     512   128K  100.00  SPA space map
     55141    2    16K   128K   504K     512   384K  100.00  SPA space map
     55174    2    16K   128K   324K     512   256K  100.00  SPA space map
     55175    2    16K   128K   960K     512   512K  100.00  SPA space map
     56165    2    16K   128K   360K     512   256K  100.00  SPA space map
     56166    2    16K   128K   840K     512   640K   80.00  SPA space map
     56666    2    16K   128K   204K     512   128K  100.00  SPA space map
     56667    2    16K   128K   924K     512   512K  100.00  SPA space map
     56765    2    16K   128K   768K     512   384K  100.00  SPA space map
     56766    2    16K   128K   276K     512   128K  100.00  SPA space map
     56939    2    16K   128K   408K     512   256K  100.00  SPA space map
     56940    2    16K   128K   564K     512   384K  100.00  SPA space map
     56961    2    16K   128K   480K     512   384K  100.00  SPA space map
     56962    2    16K   128K   504K     512   384K  100.00  SPA space map
     57221    2    16K   128K   360K     512   256K  100.00  SPA space map
     57222    2    16K   128K   360K     512   256K  100.00  SPA space map
     57272    2    16K   128K   996K     512   640K   80.00  SPA space map
     57273    2    16K   128K   552K     512   384K  100.00  SPA space map
     57584    1    32K   128K    12K     512   128K  100.00  bpobj
     57657    1    32K    512      0     512    512  100.00  zap
     57658    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
     57659    1    32K    512      0     512    512  100.00  zap
     57660    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
     57663    1    32K    512      0     512    512  100.00  zap
     57664    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
     57665    1    32K    512      0     512    512  100.00  zap
     57666    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
     57667    1    32K    512      0     512    512  100.00  zap
     57668    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
     57669    1    32K    512      0     512    512  100.00  zap
     57670    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
     57682    1    32K   128K    60K     512   256K  100.00  bpobj
     57770    1    32K   128K    48K     512   128K  100.00  bpobj
     57875    1    32K    512      0     512    512  100.00  zap
     57876    1    32K     1K    12K     512     1K  100.00  DSL deadlist map
     57878    1    32K   128K    12K     512   128K  100.00  bpobj
     57880    1    32K   128K    12K     512   128K  100.00  bpobj
     57898    1    32K   128K    12K     512   128K  100.00  bpobj
     58392    2    16K   128K    36K     512   640K   20.00  SPA space map
     58393    2    16K   128K   468K     512   256K  100.00  SPA space map
     58557    2    16K   128K    72K     512   640K   20.00  SPA space map
     58558    2    16K   128K   468K     512   256K  100.00  SPA space map
     58589    1    32K   128K    24K     512   128K  100.00  bpobj
     58609    2    16K   128K   744K     512   512K  100.00  SPA space map
     58610    2    16K   128K   216K     512   128K  100.00  SPA space map
     59007    2    16K   128K   840K     512   512K  100.00  SPA space map
     59264    2    16K   128K   636K     512   384K  100.00  SPA space map
     59991    1    16K   128K   180K     512   128K  100.00  SPA space map
     60036    2    16K   128K   744K     512   512K  100.00  SPA space map
     60037    2    16K   128K   180K     512   128K  100.00  SPA space map
     60063    2    16K   128K   276K     512   128K  100.00  SPA space map
     60093    2    16K   128K   288K     512   256K  100.00  SPA space map
     60094    2    16K   128K   888K     512   640K   80.00  SPA space map
     60549    2    16K   128K   348K     512   256K  100.00  SPA space map
     60550    2    16K   128K   312K     512   256K  100.00  SPA space map
     60659    2    16K   128K   828K     512   512K  100.00  SPA space map
     60660    2    16K   128K   300K     512   256K  100.00  SPA space map
     60920    2    16K   128K   624K     512   384K  100.00  SPA space map
     60921    2    16K   128K   744K     512   512K  100.00  SPA space map
     61101    2    16K   128K   276K     512   256K  100.00  SPA space map
     61904    2    16K   128K   732K     512   384K  100.00  SPA space map
     61905    2    16K   128K   132K     512   128K  100.00  SPA space map
     62050    2    16K   128K   444K     512   256K  100.00  SPA space map
     62051    2    16K   128K   444K     512   256K  100.00  SPA space map
     62073    2    16K   128K   684K     512   384K  100.00  SPA space map
     62074    2    16K   128K   384K     512   256K  100.00  SPA space map
     62226    1    32K   128K    12K     512   128K  100.00  bpobj
     62311    2    16K   128K   588K     512   384K  100.00  SPA space map
     62312    2    16K   128K   840K     512   512K  100.00  SPA space map
     62343    2    16K   128K   444K     512   256K  100.00  SPA space map
     62344    2    16K   128K   600K     512   640K   60.00  SPA space map
     62951    1    32K    512      0     512    512  100.00  zap
     62952    1    32K    512      0     512    512  100.00  DSL deadlist map
     62953    1    32K    512      0     512    512  100.00  zap
     62954    1    32K    512      0     512    512  100.00  DSL deadlist map
     62955    1    32K    512      0     512    512  100.00  zap
     62956    1    32K    512      0     512    512  100.00  DSL deadlist map
     62983    1    16K   128K    96K     512   128K  100.00  SPA space map
     62984    2    16K   128K   324K     512   256K  100.00  SPA space map
     63108    1    32K    512      0     512    512  100.00  zap
     63109    1    32K    512      0     512    512  100.00  DSL deadlist map
     63110    1    32K    512      0     512    512  100.00  zap
     63111    1    32K    512      0     512    512  100.00  DSL deadlist map
     63112    1    32K    512      0     512    512  100.00  zap
     63113    1    32K    512      0     512    512  100.00  DSL deadlist map
     63114    1    32K    512      0     512    512  100.00  zap
     63115    1    32K    512      0     512    512  100.00  DSL deadlist map
     63129    1    32K   128K    36K     512   128K  100.00  bpobj
     63143    1    32K   128K    12K     512   128K  100.00  bpobj
     63487    1    32K    512      0     512    512    0.00  DSL directory
     63550    2    16K   128K   924K     512   640K   80.00  SPA space map
     63551    2    16K   128K   588K     512   384K  100.00  SPA space map
     63731    2    32K    16K    36K     512    32K  100.00  zap
     63732    1    32K    512      0     512    512  100.00  DSL props
     63733    1    32K    512      0     512    512  100.00  DSL directory child map
     63734    1    32K    512      0     512    512  100.00  zap
     63735    1    32K    16K    24K     512    32K  100.00  DSL dataset snap map
     63736    1    32K    512      0     512    512  100.00  DSL deadlist map
     63762    2    16K   128K   636K     512   640K   60.00  SPA space map
     63763    2    16K   128K    96K     512   128K  100.00  SPA space map

    Dnode slots:
        Total used:          1195
        Max used:           63763
        Percent empty:  98.125872

dmu_object_next() = 52
zsh: IOT instruction (core dumped)  zdb -e data16
root@truenas[~]# 2025 Feb  5 22:36:25 truenas Process 19260 (zdb) of user 0 dumped core.

Module libudev.so.1 from deb systemd-252.19-1~deb12u1.amd64
Stack trace of thread 19260:
#0  0x00007f4694eb7e2c __pthread_kill_implementation (libc.so.6 + 0x8ae2c)
#1  0x00007f4694e68fb2 __GI_raise (libc.so.6 + 0x3bfb2)
#2  0x00007f4694e53472 __GI_abort (libc.so.6 + 0x26472)
#3  0x000055fe4906af3f n/a (zdb + 0x1ef3f)
#4  0x000055fe4906ca2e n/a (zdb + 0x20a2e)
#5  0x000055fe4905684f n/a (zdb + 0xa84f)
#6  0x00007f4694e5424a __libc_start_call_main (libc.so.6 + 0x2724a)
#7  0x00007f4694e54305 __libc_start_main_impl (libc.so.6 + 0x27305)
#8  0x000055fe49056b71 n/a (zdb + 0xab71)

Stack trace of thread 19262:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19261:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19264:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19263:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19268:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19282:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19265:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19270:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19329:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19266:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19334:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19272:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19267:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19338:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19274:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19367:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.so.5 + 0x59359)
#4  0x00007f4695510b38 n/a (libzpool.so.5 + 0x56b38)
#5  0x00007f4694eb6134 start_thread (libc.so.6 + 0x89134)
#6  0x00007f4694f367dc __clone3 (libc.so.6 + 0x1097dc)

Stack trace of thread 19340:
#0  0x00007f4694eb2e96 __futex_abstimed_wait_common64 (libc.so.6 + 0x85e96)
#1  0x00007f4694eb5558 __pthread_cond_wait_common (libc.so.6 + 0x88558)
#2  0x00007f4695511779 cv_wait (libzpool.so.5 + 0x57779)
#3  0x00007f4695513359 n/a (libzpool.
root@truenas[~]# fdisk -l /dev/sdf
Disk /dev/sdf: 14.55 TiB, 16000900661248 bytes, 31251759104 sectors
Disk model: ST16000NM001G-2K
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 34D04AC2-1A97-489E-B817-4B80A6F5A73E

Device       Start         End     Sectors  Size Type
/dev/sdf1      128     4194304     4194177    2G Linux swap
/dev/sdf2  4194432 31251759070 31247564639 14.6T Solaris /usr & Apple ZFS
root@truenas[~]# blkid
/dev/sdd2: LABEL_FATBOOT="EFI" LABEL="EFI" UUID="7CF6-7EBE" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="cd746b28-e052-4a26-9a61-13a34ec284b7"
/dev/sdd3: LABEL="boot-pool" UUID="5046100318348024170" UUID_SUB="14499026116622580121" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="c748d15e-482a-4008-bee7-6982ffc1cab0"
/dev/sdd4: UUID="ab7b8ac9-0259-fff6-d595-584789e54321" UUID_SUB="5af503f8-9b6e-f60a-4a9d-d9a5527973e3" LABEL="truenas:swap0" TYPE="linux_raid_member" PARTUUID="17bf2d50-772b-4da8-9022-0dc51673619b"
/dev/sdb2: LABEL="data16" UUID="9136863319262047294" UUID_SUB="8005179225333517566" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="dde3f5ff-166c-473a-9c4b-e29c54b6b129"
/dev/sde2: LABEL="data" UUID="7291055728607969938" UUID_SUB="2482156194395675769" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="91a5f965-46d2-4502-9f3f-301e850d6ae6"
/dev/sdc2: LABEL="data" UUID="7291055728607969938" UUID_SUB="17604897078112715004" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="62815455-1ae7-4d9a-be90-07792c8bca57"
/dev/sda4: UUID="ab7b8ac9-0259-fff6-d595-584789e54321" UUID_SUB="df798149-6156-a624-d5c7-6ff8c1e9bd1f" LABEL="truenas:swap0" TYPE="linux_raid_member" PARTUUID="ee425f00-eed4-4036-a575-840690143ad9"
/dev/sda2: LABEL_FATBOOT="EFI" LABEL="EFI" UUID="3619-525E" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="90e07bb3-6138-4d4b-80ca-b4b6e21584c7"
/dev/sda3: LABEL="boot-pool" UUID="5046100318348024170" UUID_SUB="15489100258567321204" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="1d6b742d-da70-4085-a756-fb9bd8481adf"
/dev/mapper/md127: UUID="b1d25cdb-b6c0-4eb5-b6b3-89bf3ae82db3" TYPE="swap"
/dev/sdf2: LABEL="data16" UUID="9136863319262047294" UUID_SUB="11974721465407886050" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="513e9377-340b-4b13-b6b2-b25579e118b8"
/dev/sdf1: PARTUUID="d7390a07-b0f8-47b9-ad67-8091c1611615"
/dev/sdd1: PARTUUID="73e34d63-e85d-41e2-a42e-7a2b14fe2504"
/dev/sdb1: PARTUUID="78daf6f8-9b6a-401e-91d8-7b5f82596358"
/dev/sde1: PARTUUID="8e1cba94-facc-4797-ba3b-3c73b15bfb75"
/dev/sdc1: PARTUUID="a5d87da9-ead3-4849-a713-ae8b005b9862"
/dev/sda1: PARTUUID="b9e06b7f-f1ac-4fe6-b73a-5ccca66e8ecd"

Help is much appreciated

Have you tried zpool import -R /mnt -d /dev/disk/by-partuuid data16 perhaps with -f or -F?

Thanks for your feedback:

root@truenas[~]# zpool import -R /mnt -d /dev/disk/by-partuuid data16
cannot import 'data16': one or more devices is currently unavailable
root@truenas[~]# man zpool import
root@truenas[~]# zpool import -F -R /mnt -d /dev/disk/by-partuuid data16
cannot import 'data16': I/O error
        Destroy and re-create the pool from
        a backup source.

@honeybadger Can you help here? My ZFS pool recovery knowledge doesn’t go any further.

@leeds you need to list the hardware you are using and specifiy how you are attaching the disks to the motherboard

Take a look at NugentS or Protopia expandable sections. Looks like a little triangle. Click on those and see their details. This is the sort of info we would like to see. Both of them are good examples.

Curious that it’s calling out an unavailable device. :thinking:

zpool import -FXn -R /mnt -o readonly=on data16

?