I think if I can retrieve the data some how, I am going to start on a fresh build.
This truenas server is actually running on a mini pc with an m2 stata expansion 
Ok I ran the following with only my Barracuda plugged in
truenas_admin@truenas[~]$ sudo zpool import
[sudo] password for truenas_admin:
pool: DataSet
id: 8610289484681839548
state: DEGRADED
status: One or more devices contains corrupted data.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:
DataSet DEGRADED
mirror-0 DEGRADED
c912daae-836f-4228-bde0-631de6bd8ed6 ONLINE
09cc475a-e7ab-4509-8ca4-0cfdd152913f UNAVAIL
truenas_admin@truenas[~]$ sudo zpool import -F -R /mnt -o readonly=on DataSet
cannot iterate filesystems: I/O error
zsh: segmentation fault sudo zpool import -F -R /mnt -o readonly=on DataSet
truenas_admin@truenas[~]$ sudo zpool status
pool: DataSet
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: scrub repaired 0B in 01:24:23 with 0 errors on Sun Jul 6 01:24:25 2025
config:
NAME STATE READ WRITE CKSUM
DataSet DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
c912daae-836f-4228-bde0-631de6bd8ed6 ONLINE 0 0 8
9870796191026097272 UNAVAIL 0 0 0 was /dev/disk/by-partuuid/09cc475a-e7ab-4509-8ca4-0cfdd152913f
errors: 1 data errors, use '-v' for a list
pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:00:11 with 0 errors on Mon Jul 21 03:45:12 2025
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
sdb3 ONLINE 0 0 0
errors: No known data errors
When I look at my DataSet in the GUI, it shows the following error
[EFAULT] Failed retreiving GROUP quotas for DataSet
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 76, in get_quota
with libzfs.ZFS() as zfs:
File "libzfs.pyx", line 534, in libzfs.ZFS.__exit__
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 78, in get_quota
quotas = resource.userspace(quota_props)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "libzfs.pyx", line 3800, in libzfs.ZFSResource.userspace
libzfs.ZFSException: cannot get used/quota for DataSet: dataset is busy
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.11/concurrent/futures/process.py", line 256, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
res = MIDDLEWARE._run(*call_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
return methodobj(*params)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 80, in get_quota
raise CallError(f'Failed retreiving {quota_type} quotas for {ds}')
middlewared.service_exception.CallError: [EFAULT] Failed retreiving GROUP quotas for DataSet
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 208, in call_method
result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1526, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1457, in _call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_quota.py", line 48, in get_quota
quota_list = await self.middleware.call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1626, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1465, in _call
return await self._call_worker(name, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1471, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1377, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1361, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EFAULT] Failed retreiving GROUP quotas for DataSet
[ENOENT] Path /mnt/DataSet not found
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 208, in call_method
result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1526, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1468, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1361, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 55, in nf
res = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/filesystem.py", line 445, in stat
raise CallError(f'Path {_path} not found', errno.ENOENT)
middlewared.service_exception.CallError: [ENOENT] Path /mnt/DataSet not found
If I try with just the old WD plugged in
truenas_admin@truenas[~]$ sudo zpool import
[sudo] password for truenas_admin:
pool: DataSet
id: 8610289484681839548
state: DEGRADED
status: One or more devices were being resilvered.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
config:
DataSet DEGRADED
mirror-0 DEGRADED
14739960081565547348 UNAVAIL
09cc475a-e7ab-4509-8ca4-0cfdd152913f ONLINE
fe2d4d34-1dac-45f5-8320-c12ed3b436aa UNAVAIL
truenas_admin@truenas[~]$ sudo zpool import -F -R /mnt -o readonly=on DataSet
cannot import 'DataSet': insufficient replicas
Destroy and re-create the pool from
a backup source.
truenas_admin@truenas[~]$ sudo zpool import -FX -R /mnt -o readonly=on DataSet
cannot import 'DataSet': one or more devices is currently unavailable
Ah… if this thing uses a SATA port multiplier, it could be the source of your troubles. Even with two drives.
Do you have another computer where you can plug two or three drives to good SATA ports, served by the board’s chipset?
ah… yes I do, great idea.
Will give this a test on another desktop
Plugged in my hdds to a different desktop with a motherboard sata input. Same results
How did you set up the different desktop? Did you do a fresh TrueNAS install to a boot drive and upload your config? You need to be tell us all your steps as you leave out a lot of data or do unexpected things.
straight boot from the existing drives. I can do a fresh install if that would help
Wait and let etorix decide on that.
I don’t think a fresh install would change anything.
We have (hopefully) a better SATA controller.
Maybe adding -f would help?
We can re-check labels, to assess where things are after these attempts at importing a single drive and resilvering. I suspect that things have further diverged and both drives cannot be re-united as a mirror.
After that… it would be a job for @HoneyBadger , but I fear he’s overburdened.
I think i’ve exhausted every import command. Here are my labels with the old WD and barracuda plugged in
truenas_admin@truenas[/]$ sudo zdb -l /dev/sdb1
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'DataSet'
state: 0
txg: 3285195
pool_guid: 8610289484681839548
errata: 0
hostid: 1168512994
hostname: 'truenas'
top_guid: 17597434126019592049
guid: 14739960081565547348
vdev_children: 1
vdev_tree:
type: 'mirror'
id: 0
guid: 17597434126019592049
whole_disk: 0
metaslab_array: 133
metaslab_shift: 34
ashift: 12
asize: 4000780386304
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 14739960081565547348
path: '/dev/disk/by-partuuid/c912daae-836f-4228-bde0-631de6bd8ed6'
whole_disk: 0
DTL: 1790
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 9870796191026097272
path: '/dev/disk/by-partuuid/09cc475a-e7ab-4509-8ca4-0cfdd152913f'
whole_disk: 0
DTL: 965
create_txg: 4
removed: 1
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
com.klarasystems:vdev_zaps_v2
labels = 0 1 2 3
truenas_admin@truenas[/]$ sudo zdb -l /dev/sdc1
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'DataSet'
state: 0
txg: 3070748
pool_guid: 8610289484681839548
errata: 0
hostid: 1168512994
hostname: 'truenas'
top_guid: 17597434126019592049
guid: 9870796191026097272
vdev_children: 1
vdev_tree:
type: 'mirror'
id: 0
guid: 17597434126019592049
whole_disk: 0
metaslab_array: 133
metaslab_shift: 34
ashift: 12
asize: 4000780386304
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 14739960081565547348
path: '/dev/disk/by-partuuid/c912daae-836f-4228-bde0-631de6bd8ed6'
whole_disk: 0
not_present: 1
DTL: 1790
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 9870796191026097272
path: '/dev/disk/by-partuuid/09cc475a-e7ab-4509-8ca4-0cfdd152913f'
whole_disk: 0
DTL: 965
create_txg: 4
degraded: 1
aux_state: 'err_exceeded'
children[2]:
type: 'disk'
id: 2
guid: 1248796505571215142
path: '/dev/disk/by-partuuid/fe2d4d34-1dac-45f5-8320-c12ed3b436aa'
whole_disk: 0
DTL: 192
create_txg: 4
resilver_txg: 3070745
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
com.klarasystems:vdev_zaps_v2
labels = 0 1 2 3
The Barracuda is sdb above, and the old WD is sdc, correct?
They are, in effect, members of different pools by now. I would not attempt any import with both plugged in.
The old WD has aux_state: 'err_exceeded', which I do not like.
The new WD is at resilver_txg: 3070745, only 3 txg behind the old WD; so there might be some hope.
I would first unplug the Barracuda, and let the pool import with the two WD. Force if it doesn’t work out on its own. Hope that the resilver can complete… and then replace the old WD by another new drive.
If this doesn’t work, or if you’ve lost data from the old WD, you may unplug both WD and import the Barracuda with a new disk to copy the data to, or to add as a new member. And then replace the Barracuda…
New drive purchase ahead anyway. But two distinct chances to get your data out.
Whatever you do, do not swap the new WD from one partner to another. Or wipe it first.
For completeness, what’s the label on the new WD?
because I wiped it, there’s nothing there
truenas_admin@truenas[~]$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 447.1G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 512M 0 part
└─sda3 8:3 0 446.6G 0 part
sdb 8:16 0 3.6T 0 disk
sdc 8:32 0 3.6T 0 disk
└─sdc1 8:33 0 3.6T 0 part
truenas_admin@truenas[~]$ sudo zdb -l /dev/sdb
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3
truenas_admin@truenas[~]$ sudo zpool import
pool: DataSet
id: 8610289484681839548
state: DEGRADED
status: One or more devices were being resilvered.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
config:
DataSet DEGRADED
mirror-0 DEGRADED
14739960081565547348 UNAVAIL
09cc475a-e7ab-4509-8ca4-0cfdd152913f ONLINE
fe2d4d34-1dac-45f5-8320-c12ed3b436aa UNAVAIL
With the two WDs plugged in, I’ve tried to import but failed
truenas_admin@truenas[~]$ sudo zpool import -f -o readonly=on -R /mnt DataSet
cannot import 'DataSet': insufficient replicas
Destroy and re-create the pool from
a backup source.
truenas_admin@truenas[~]$ sudo zpool import -fF -o readonly=on -R /mnt DataSet
cannot import 'DataSet': I/O error
Destroy and re-create the pool from
a backup source.
truenas_admin@truenas[~]$ sudo zpool import -fFX -o readonly=on -R /mnt DataSet
cannot import 'DataSet': one or more devices is currently unavailable
I am not sure what to do with the Barracuda? Could I use my current new WD since I wiped it anyways? Whenever I tried to import with the Barracuda itself in read only, it kept giving the segmentation fault error I mentioned
If you cannot import the Barracuda alone, having an extra disk cannot help. 
Maybe @HoneyBadger or @NickF1227 will have an idea?
I unfortunately don’t have any good ideas here.
Whatever happned here has basically left us with only one option I can think of. I’ve not personally used this, but I do know of people who have been successful. Klennet is basically Recuva for ZFS.
This looks like hardware issues stacked with the power event, but I’d have to dig in a bit more to better understand what happened.
If you send me a debug I can try and figure out something. Go to System->Advanced and generate a debug. You can DM me a link to it in whatever storage platform you use.
Thanks, I will look into klennet
I will forward you the debug as well. Really appreciate it.
Jul 24 04:24:43 truenas kernel: ata4.00: status: { DRDY }
Jul 24 04:25:05 truenas kernel: ata4.00: exception Emask 0x10 SAct 0xe0000 SErr 0x4010000 action 0xe frozen
Jul 24 04:25:05 truenas kernel: ata4.00: irq_stat 0x80400040, connection status changed
Jul 24 04:25:05 truenas kernel: ata4: SError: { PHYRdyChg DevExch }
Jul 24 04:25:05 truenas kernel: ata4.00: failed command: READ FPDMA QUEUED
Jul 24 04:25:05 truenas kernel: ata4.00: cmd 60/00:88:18:1f:27/08:00:36:00:00/40 tag 17 ncq dma 1048576 in
Jul 24 12:42:36 truenas kernel: ata3.00: status: { DRDY }
Jul 24 12:42:49 truenas kernel: ata3.00: exception Emask 0x10 SAct 0x80 SErr 0x4010000 action 0xe frozen
Jul 24 12:42:49 truenas kernel: ata3.00: irq_stat 0x80400040, connection status changed
Jul 24 12:42:49 truenas kernel: ata3: SError: { PHYRdyChg DevExch }
Jul 24 12:42:49 truenas kernel: ata3.00: failed command: READ FPDMA QUEUED
Jul 24 12:42:49 truenas kernel: ata3.00: cmd 60/08:38:48:28:00/00:00:76:00:00/40 tag 7 ncq dma 4096 in
res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
Is this the same drive moved to a differant SATA port on the 24th?
I believe so, I tried removing the Barracuda and WD individually that day.
I must have tried another port as well during that time
I am going to rebuild at this point. Bought a ups and wont be using Barracudas ever again. Will also schedule weekly snapshots 
Thank you everyone for your time