Datasets not found/reachable after pool issues

[hommsy]

While attending another issue that happened in my TrueNAS system, I managed to create more problems

The pool in question here is called ‘personal’. It is made up of 2x4TB WD drives. They were mirrored so that if one drive fails the other can be used to recover the data.

Everything was running fine, I was dealing with another pool where a hard drive failed. In my hastiness and unfocused state, I disconnected both 2x4TB drives of the personal pool. The TrueNAS GUI was showing errors and warnings etc.

I reconnected the drives, and went into a shell, and did sudo zpool import personal. I TrueNAS GUI looked normal again like this:

At this point some of the self-hosted apps Im hosting were complaining that their directories are no longer available. When I check TrueNAS for example to edit permissions of these directories/shares/datasets I get this:

Here is the block of code/error in that error message:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 201, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1353, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1251, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 181, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 50, in nf
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/filesystem.py", line 409, in stat
    raise CallError(f'Path {_path} not found', errno.ENOENT)
middlewared.service_exception.CallError: [ENOENT] Path /mnt/personal/nfs-phone-media not found

What went wrong? Is it recoverable?
Thanks.

[Johnny_Fartpants]

I’d suggest you zpool export personal from the command line and reboot your TrueNAS. Hopefully it will auto import the pool and you should be good but if not import it via the UI and see if that fixes it.

[hommsy]

can’t seem to export it

admin@truenas[~]$ sudo zpool list
[sudo] password for admin: 
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool    47G  2.46G  44.5G        -         -     3%     5%  1.00x    ONLINE  -
personal   3.62T  1.08T  2.54T        -         -    10%    29%  1.00x    ONLINE  -
admin@truenas[~]$ sudo zpool export personal
cannot unmount '/var/db/system/samba4': pool or dataset is busy

[hommsy]

I think relevant to say is when I disconnected the HDD’s from the personal pool, the apps using that pool were probably using it…(?) I now hear a click from one of these HDDs every 5-6 seconds. Maybe I broke filesystem or something? Should I scrub or something?

[Johnny_Fartpants]

Restart if you haven’t already and let’s see how things look.

[neofusion]

I suspect that your manual import (whyyyyy??) caused the pool to be mounted in the wrong place, causing the apps path to be invalid.

The last column of your zpool list would have shown /mnt as the altroot, if it was mounted in the proper place.

Your initial import should have been sudo zpool import -R /mnt personal, if you needed to do it in the shell, for some reason.

[hommsy]

@Johnny_Fartpants rebooted but no change.

@neofusion as you can tell im a novice. I read around the forum and the web and the manual import was suggested. I could not import it via GUI as it was greyed out.

[neofusion]

I have not tried this myself, but this is my propsed course of action:

1. Try exporting the pool again, if it fails due to being in use, move the system dataset to the boot-pool (if possible) and try again. You do this in System Settings → Advanced → Storage.
2. Try exporting again, if you can’t because it’s still in use, restart and try this step again.
3. If step 2 worked, try importing using the GUI.
4. If it fails try importing using sudo zpool import -R /mnt personal, which has the proper altroot flag set.

[hommsy]

I stopped all services that might rely on the datasets in personal. I restart TrueNAS and tried to export but no luck.

I check in System Settings → Advanced → Storage and it seems I can only edit the swap?!

[neofusion]

Don’t touch the swap setting, only change the top option “Select Pool” to boot-pool instead of personal.

This is just a temporary thing to hopefully stop the personal pool from being in use.

[hommsy]

that seems to work. I dont get the error when I edit permissions of the dataset. However the services that use the dataset in the personal pool still cant find it. Should I redo the System Settings → Advanced → Storage and change it to personal? rather than boot-pool

[neofusion]

Have you done the export/import dance that I described earlier? Until you do your personal pool is mounted in the wrong spot, which will cause different issues now and down the road.

When zpool list shows the proper altroot (/mnt) you can move the system dataset back to personal, if it doesn’t end up there automagically on the next import.

[hommsy]

I danced your dance and can confirm

admin@truenas[~]$ sudo zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool    47G  2.59G  44.4G        -         -     3%     5%  1.00x    ONLINE  -
personal   3.62T  1.08T  2.54T        -         -    10%    29%  1.00x    ONLINE  /mnt

I then did System Settings → Advanced → Storage and changed it to back to personal. The service using the dataset in personal are running again.

Thank you for the support!

[Johnny_Fartpants]

Excellent, well done @neofusion