Pool i/o suspended

That drive that couldn’t be initialized was never part of my pool, and never had any data written to it. It was purchased to expand my pool but was dead the whole time and I never knew. It’s basically just been sitting in my server for no reason.

I have all of the drives connected via an LSI 9207-8i HBA in IT Mode except for one which is connected via onboard sata on the motherboard.

Motherboard: ASUS PRIME H610I-PLUS itx

Cpu: Intel i3 14100

Nvme ssd: PNY CS1030 M.2 2280 500GB

boot ssd: 128gb ssd 870 EVO

Ram: Timetec 32GB (2x16GB) DDR4 2666MHz

yeah if that dead disk was never in the pool, ignore it. ur real hole is the UNAVAIL member on raidz1

from the config-backup boot that actually works:
-zpool import
grab the UNAVAIL gptid from that output

then a known good 16tb goes in, confirm it in /dev/disk/by-id/, and:
-zpool replace tank THAT-UNAVAIL-GPTID /dev/disk/by-id/NEWDRIVE

cant replace b4 import. dont expand. dont initialize anything. let resilver hit 100% b4 u touch another disk

also that m.2 5gbe nic sharing lanes w/ the 9207 can drop disks. if the same member keeps bouncing UNAVAIL after replace, pull the nic and retest

Yeah I’ve saved the gptid of that unavailable drive. I don’t have an active resilver right now, should I import my tank? Last time I tried that from my actual truenas instance and not from the usb shell it put me in a boot loop. I already yanked the 5gbe nic and went back to the onboard lan a while ago to be safe.

I ended up importing via zpool import -f -o readonly=on tank and all my data is there but I can’t access it. So I’ve tried to copy some files/directories out via command line. I’ve used cp, rsync and install but they wont seem to copy all the files, just some. I’m sure its a permissions error but does anyone know how to fix that so I can transfer some files over? This is the error when I go to datasets

`Error: concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 76, in get_quota
    with libzfs.ZFS() as zfs:
  File "libzfs.pyx", line 534, in libzfs.ZFS.__exit__
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 78, in get_quota
    quotas = resource.userspace(quota_props)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "libzfs.pyx", line 3834, in libzfs.ZFSResource.userspace
libzfs.ZFSException: cannot get used/quota for tank: dataset is busy

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 116, in main_worker
    res = MIDDLEWARE._run(*call_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 47, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 41, in _call
    return methodobj(*params)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py", line 80, in get_quota
    raise CallError(f'Failed retreiving {quota_type} quotas for {ds}')
middlewared.service_exception.CallError: [EFAULT] Failed retreiving GROUP quotas for tank
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 323, in process_method_call
    result = await method.call(app, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py", line 52, in call
    result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 911, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 720, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 174, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_quota.py", line 48, in get_quota
    quota_list = await self.middleware.call(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1005, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 728, in _call
    return await self._call_worker(name, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 734, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 640, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 624, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EFAULT] Failed retreiving GROUP quotas for tank
 `

You need to post details on what you have done and what you are trying to do. I think we are expecting you to follow @nite_route directions and it seems like you are doing something different or have attempted but aren’t letting us know the results. We aren’t there to see your system so a detailed summary of where you are now could help.

Apologies. I attempted to do this:

from the config-backup boot that actually works:
-zpool import
grab the UNAVAIL gptid from that output

then a known good 16tb goes in, confirm it in /dev/disk/by-id/, and:
-zpool replace tank THAT-UNAVAIL-GPTID /dev/disk/by-id/NEWDRIVE

cant replace b4 import. dont expand. dont initialize anything. let resilver hit 100% b4 u touch another disk

I was unable to successfully import from my truenas instance (not usb shell). It says that I have insufficient replicas. So in my truenas instance again I tried zpool import -f -o readonly=on tank which did work. I can see my tank pool, but I don’t have access to any of the data on it. So I’ve just been attempting to copy over the files/directories from the datasets to another pool on my nas. However cp, rclone, rsync etc all aren’t able to copy over the entire contents of the directories.

For example the directory /mnt/tank/configs is around 150gb, but all those commands only copy over the same amount, around 400 megabytes out of the total 150gb. And then when trying to move those files over the network via smb to my macbook, I don’t even have permission to copy/move/delete them over smb. So at this point I’m just trying to save some of the stuff that I can since the pool is back although it’s in read only but I am unable to.

I would just wait to hear from another user. I don’t know what gets broken with mounting the pool read only. It may just be a simple thing like having to use ‘root’ or ‘sudo’ to copy between pools. I would expect file sharing to break also but I’m not sure on how to work around that.

You could also try browsing or searching the forum for users that had to mount RO and recover the data.

Appreciate the reply!

for the copy off a readonly import u need root. plain cp as truenas_admin hits perms.

from that same boot:
-mount | rg tank
-sudo rsync -aH /mnt/tank/SOME/PATH/ /mnt/OTHERPOOL/rescue/

or sudo cp -a. if tank isnt under /mnt check zfs get mountpoint tank

dont flip it rw yet till u got a spare ready for the replace

Thanks for the reply.

mount | rg tank doesn’t work, “command not found: rg”

sudo rsync -aH and sudo cp -a both have the same problem, they dont copy all of the data even though truenas clearly sees it.

zfs get mountpoint tank shows this which looks corrct since I can ls -l /mnt/tank/ and see my directories and copy some files just not all.

NAME  PROPERTY     VALUE     SOURCE
tank  mountpoint   /tank     default

lsattr /mnt didn’t return any immutable flags either.

When using the import pool command before this was the message. “cannot mount: failed to create mountpoint: Read-only file system Import was successful, but unable to mount some datasets”

Yeah no matter what I do I cannot get it to import my pool in rw. It only imports in read only and that doesn’t let me transfer my data out or replace the disk. I think its over for me.

You need to set up the folders for the mount points first, and give it full read-write access.

What’s probably happening is it’s first mounting the root dataset as read-only over the directory /tank. Since /tank is the mount point for a read-only filesystem, it cannot create the needed folders for the child dataset mount points if they don’t already exist, since that requires writable access.

What you can do is include the -N flag in the import command, which will not automatically mount the datasets of the pool.

Now you can create your own directories and mount the datasets to these locations.

zpool import -f -o readonly=on -N -R /recovery tank

mkdir /recovery /recovery/child1 /recovery/child2 /recovery/child2/baby2

Then mount the datasets you wish to recover files from.

zfs mount tank/child1

Remember that if you mount a lower level dataset before a higher level dataset, it won’t likely work because the lower level filesystem is read-only and won’t be able to create the needed directories if they don’t already exist.

I’m so sorry but this is completely over my head lmao. Say I want to copy files from /mnt/tank/photos to another pool called backup at /mnt/backup/photos how would I go about this? I’m truly appreciative of all your help.

I can’t just tell you what to do, since you need to understand some basics before proceeding. You don’t want to run commands blindly for something as critical as this.

You cannot import your pool normally, which would have allowed you to do useful things like changing pool and dataset properties, creating new snapshots, and automatically mounting the datasets without requiring special steps. That’s why you can’t rely on the same procedures when the pool is imported read-only.

It’s very possible to try a zfs send/recv from one pool to another, while your “tank” pool is imported read-only. This is because zfs send/recv (a block-based operation) does not require a dataset to be mounted. The problem with that is it would only be as useful as the latest snapshots that exist on each dataset, but you won’t be able to take a new snapshot. Another caveat is that metadata corruption or I/O errors would abort the entire send anyways, since you cannot use a “partial” ZFS replication on the destination.

Importing a pool as read-only is usually done for emergency operations. This means you need to take extra planning and steps to do what you want to do.

You can’t rely on all datasets being mounted automatically. Every mounted filesystem is read-only, which means the needed directories to be used as mount points for child datasets cannot be created. You might ask why you can’t just mount all child datasets on their own directories outside of a common (root) directory. The reason you can’t do this is because you would need to change the “mountpoint” property of these datasets, but you can’t do that because the pool is read-only.

That leaves you with a tedious option: Create the directory structure ahead of time and only mount grandchildren and children as needed. Copy the files from these mounted read-only filesystems and repeat the process for other children and their parents.


Import the pool as read-only, don’t let it automount anything, and set its “altroot” to /tank:

zpool import -f -o readonly=on -N -R /tank tank

The “altroot” property will default to /tank anyways, but this lets you see how you can use something different like /recovery if you wanted to.

List the dataset hierarchy, so you know what directory structure to create before mounting anything:

zfs list -t fs -o name -r tank

I’m assuming you’re not using zvols or VMs.

Here’s what that might look like:

tank
tank/child1
tank/child1/grandchildA
tank/child1/grandchildB
tank/child2
tank/child3
tank/child3/grandchildC

Use mkdir to create those directories under the root path /tank :

mkdir /tank
mkdir /tank/child1
mkdir /tank/child1/grandchildA
mkdir /tank/child1/grandchildB
mkdir /tank/child2
mkdir /tank/child3
mkdir /tank/child3/grandchildC


I skipped shell shortcuts and other options because it would only make these steps more complex than they need to be.

Now mount the first grandchild that you want to copy files from.

zfs mount tank/child1/grandchildA

The read-only filesystem contents can be found under the associated mount point /tank/child1/grandchildA/ .

You should then be able to use rsync to transfer everything to a backup destination path:

rsync -avxxhH --progress /tank/child1/grandchildA/ /mnt/backup/child1/grandchildA/

The slashes at the end of the paths are not optional.

Repeat the process with the other grandchildren, children, and parents.

You might bump into a lot of issues, especially if you don’t use sudo or you have special permissions and ACLs. The problem with a file-based solution is you do not get a perfect replica of a filesystem as the result. At least with rsync, you can issue the same command and it will skip files that already transferred successfully.

You might bump into serious problems if there is metadata corruption or I/O errors due to faulty or failing hardware.