Need help with corrupted pool/data and bad mountpoints

[sizzam]

my pool keeps crashing with a kernel panic during access of the internal datasets with an error saying “duplicate mountpoint” and not syncing errors get thrown up as a result. things have not been great for this pool one disk is throwing many checksum errors and it even kernel panics during scrub of pool now.

I need help from someone here who can walk me through step by step and lay it out for someone with the technical prowess of a moron when it comes to linux/zfs/truenas. (eg. what commands to run exactly as needed to debug and gather info as well as what to run to fix said issues in absolute laymans terms)

My current pool layout is a 4 wide no redundancy vdev with 2 datasets, one dataset was a smb share and one is a iscsi zvol. both are having issues being seen by the middleware and are unable to be accessed or moved with replication.

here is the result of one command.

I am running dragonfish atm as newer versions would not import my pools

the result of zpool status -v

and here are the commands i ran so far to get to this point.
before these commands I was unable to mount the pool without a kernel panic not syncing 3 error.

sudo zfs set mountpoint=/mnt/NFS NFS
sudo zpool import -N -o readonly=on -f -R /mnt -F -T 9878227061012675051 NFS
sudo zpool scrub NFS
sudo zpool status -v NFS
sudo zfs set share.smb=on NFS/sizzam
sudo zpool status -x
sudo zpool export NFS
sudo zpool import -o readonly=on -f -R /mnt -F -T 9878227061012675051 NFS
sudo zfs set mountpoint=none NFS
sudo zfs set mountpoint=none NFS/nfs
sudo zfs list -r -o NFS,canmount,mountpoint syspool/ROOT
sudo zpool import -o readonly=on -f -R /mnt -F -T 9878227061012675051 NFS

some of these commands failed to run so if there are some here that dont make sense they probably are one of the failed ones.

[sizzam]

one of the errors I get when I open datasets.

Error: concurrent.futures.process.RemoteTraceback:
“”"
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs/dataset_quota.py", line 76, in get_quota
with libzfs.ZFS() as zfs:
File “libzfs.pyx”, line 534, in libzfs.ZFS.exit
File “/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_quota.py”, line 78, in get_quota
quotas = resource.userspace(quota_props)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “libzfs.pyx”, line 3800, in libzfs.ZFSResource.userspace
libzfs.ZFSException: cannot get used/quota for NFS: dataset is busy

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3.11/concurrent/futures/process.py”, line 261, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 112, in main_worker
res = MIDDLEWARE._run(*call_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 46, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 34, in _call
with Client(f’ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock’, py_exceptions=True) as c:
File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 40, in call
return methodobj(*params)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs/dataset_quota.py", line 80, in get_quota
raise CallError(f’Failed retreiving {quota_type} quotas for {ds}')
middlewared.service_exception.CallError: [EFAULT] Failed retreiving USER quotas for NFS
“”"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 211, in call_method
result = await self.middleware.call_with_audit(message[‘method’], serviceobj, methodobj, params, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1529, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1460, in call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 179, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool/dataset_quota.py", line 48, in get_quota
quota_list = await self.middleware.call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1629, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1468, in _call
return await self._call_worker(name, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1474, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1380, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1364, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EFAULT] Failed retreiving USER quotas for NFS