Unable to disconnect pool due to ghost tailscale process

TrueNAS General
1

Truenas version: ElectricEel-24.10.2.4
Hi everyone,
I’m trying to Export/disconnect a pool so that I can rename it, when I try to do this I get the following error:

[EFAULT] Failed to stop docker service: [EFAULT] Failed to umount dataset: cannot unmount '/mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone': unmount failed

This error causes the app services to shut down and become unusable unless the system is restarted.

  1. What started this issue was me trying to Export/disconnect a pool without stopping all the apps.
  2. I have deleted TailScale from the apps list since, however this didn’t fix the issue, the directory still exists in /mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone, I’m going to reinstall all apps anyway so I don’t mind loosing them.
  3. What I tried doing was removing the dir that was causing issues (/mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone) with rmdir, it was removed but after when I tried to disconnect pool I got this error:

[EFAULT] Failed to stop docker service: [EFAULT] Failed to umount dataset: cannot unmount '/mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone': no such pool or dataset

  1. I ran lsof to see if any files from this dir where open, and got the following:
truenas_admin@truenas[~]$ lsof /mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone
lsof: WARNING: can't stat() zfs file system /mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /mnt/.ix-apps/docker/overlay2/f56620b6357deba3b2a7d57c51d54dce5c6c177ee69968748646ae9317f7a442/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /mnt/.ix-apps/docker/overlay2/617cc887933be8799d2c339266d3de490ae9f7937408dcb98584df8e28c40b2f/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /mnt/.ix-apps/docker/overlay2/bdd3ee48521063f1955e8b51e9405a94b3cc736d3d1aeb7cb309272886703b8b/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() nsfs file system /run/docker/netns/55cf075613b6
      Output information may be incomplete.
lsof: status error on /mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone: No such file or directory
  1. I then tried deleting all tailscale snapshots, but there was one that I wasn’t able to remove, I would get the following error:
Warning: 1 of 1 snapshots could not be deleted.
*** [EINVAL] options.defer: Please set this attribute as ‘NAS 16TB 2 Vdevs Mirrored /ix-apps/app_mounts/tailscale@1.2.7’ snapshot has dependent clones: NAS 16TB 2 Vdevs Mirrored /ix-apps/app_mounts/tailscale-1.2.7-clone
  1. I ended up finally being able to promote “/ix-apps/app_mounts/tailscale-1.2.7-clone” with “zfs promote”, and then ran “zfs destroy -r mnt/ix-apps/app_mounts/tailscale-1.2.7-clone”, it destroyed the snapshot, it no longer appears in the snapshots tab, but when I try to disconnect pool, I still get the

[EFAULT] Failed to stop docker service: [EFAULT] Failed to umount dataset: cannot unmount ‘/mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone’: unmount failed

  1. No idea what’s using the service, ran the following command “lsof +D /mnt/.ix-apps/app_mounts” Got the following output:
COMMAND   PID          USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
zsh     27361 truenas_admin  cwd    DIR   0,82       17   34 /mnt/.ix-apps/app_mounts
sudo    35661          root  cwd    DIR   0,82       17   34 /mnt/.ix-apps/app_mounts
zsh     35662          root  cwd    DIR   0,82       17   34 /mnt/.ix-apps/app_mounts
lsof    44551          root  cwd    DIR   0,82       17   34 /mnt/.ix-apps/app_mounts
lsof    44552          root  cwd    DIR   0,82       17   34 /mnt/.ix-apps/app_mounts
  1. When I try to rmdir tailscale, I get
root@truenas[/mnt/.ix-apps/app_mounts]# rmdir tailscale
rmdir: failed to remove 'tailscale': Device or resource busy
  1. Running “ps -ef | grep /mnt/.ix-apps/app_mounts/tailscale”, I get the following output:
    root 67106 56490 0 18:20 pts/4 00:00:00 grep /mnt/.ix-apps/app_mounts/tailscale
    Which I think is the process that’s running

However I can’t for the life of me end it, only thing that’s worked has been sudo kill -9 56490, but that just created a new process now

  1. tried killing the process again, here’s the shell:
root@truenas[/mnt/.ix-apps/app_mounts]# ps ax | grep tailscale
1155464 pts/3 S+ 0:00 grep tailscale
root@truenas[/mnt/.ix-apps/app_mounts]# kill 1155464
kill: kill 1155464 failed: no such process
  1. Now ran a command to see if I could locate the parent process, I think I found the parent process ID that doesn’t change, but can’t kill it for shit:
root@truenas[/mnt/.ix-apps/app_mounts]# ps -o pid,ppid,cmd -U root | grep tailscale
1169788 1126922 grep tailscale
  1. At this point it’s still showing the following error:
[EFAULT] Failed to stop docker service: [EFAULT] Failed to umount dataset: cannot unmount '/mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone': unmount failed

I still have tailscale-1.2.7-clone in my .ix-apps/app_mounts, but I no longer have the regular tailscale folder.

If anyone has similar experience or any tips, I’d greatly appreciate it, this has been extremely frustrating, and I’d do anything to get it resolved without having to reinstall all of Truenas Scale. If any more info is needed please let me know.

This is a repost from the previous thread: [EFAULT] Failed to stop docker service: [EFAULT] Failed to umount dataset

I tried to format it better in this one

[EFAULT] Failed to stop docker service: [EFAULT] Failed to umount dataset
2

Here is the full error log when trying to unmount pool

concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_actions.py", line 77, in umount
    with libzfs.ZFS() as zfs:
  File "libzfs.pyx", line 534, in libzfs.ZFS.__exit__
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_actions.py", line 79, in umount
    dataset.umount(force=options['force'])
  File "libzfs.pyx", line 4287, in libzfs.ZFSDataset.umount
libzfs.ZFSException: cannot unmount '/mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone': unmount failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 112, in main_worker
    res = MIDDLEWARE._run(*call_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 34, in _call
    with Client(f'ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call
    return methodobj(*params)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/dataset_actions.py", line 82, in umount
    raise CallError(f'Failed to umount dataset: {e}')
middlewared.service_exception.CallError: [EFAULT] Failed to umount dataset: cannot unmount '/mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone': unmount failed
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/update.py", line 97, in do_update
    await self.middleware.call('service.stop', 'docker')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1629, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1460, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service.py", line 267, in stop
    await service_object.after_stop()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service_/services/docker.py", line 72, in after_stop
    await self.mount_umount_ix_apps(False)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service_/services/docker.py", line 19, in mount_umount_ix_apps
    await self.middleware.call('zfs.dataset.umount', docker_ds, {'force': True})
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1629, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1468, in _call
    return await self._call_worker(name, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1474, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1380, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
middlewared.service_exception.CallError: [EFAULT] Failed to umount dataset: cannot unmount '/mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone': unmount failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 509, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 554, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/export.py", line 105, in export
    await delegate.delete(attachments)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/attachments.py", line 31, in delete
    await (await self.middleware.call('docker.update', {'pool': None})).wait(raise_error=True)
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 457, in wait
    raise self.exc_info[1]
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 509, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 554, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/update.py", line 111, in do_update
    raise CallError(f'Failed to stop docker service: {e}')
middlewared.service_exception.CallError: [EFAULT] Failed to stop docker service: [EFAULT] Failed to umount dataset: cannot unmount '/mnt/.ix-apps/app_mounts/tailscale-1.2.7-clone': unmount failed