Evening,
Been running Core virtualised for years in VMWare with HBA passed through attached to a 60TB encrypted pool.
I recently explored Scale again, installed the latest 24_04 in a new VM. Restored config and all worked well. Pretty impressed with the progress. Due to unrelated issues though discussed elsewhere, I am having to reboot regularly to restore the sluggish GUI - having to unlock the pool each time.
But today, I installed StorJ as an app on Scale, which created the IX-Applications dataset in the process on my pool. On the subsiquent restart, I had a bit of a panic as I got an error unlocking the pool as it would not accept my passphrase.
directory is not empty (please provide "force" flag to override this error and file/directory will be renamed once the dataset is unlocked)
I reverted back to core, and it unlocked fine, so looks like a GUI realted issue rather than underlying ZFS, and seems identical to this thread from 2 years ago.
https://www.truenas.com/community/threads/can-not-unlock-pool-after-upgrade-to-22-02-0-1-directory-is-not-empty-some-datasets-mount-empty-with-workaround.100103/
Is this a bug, that should be reported again?
Thanks
CC
Are there actual files saved in the “path” of where the root dataset is normally mounted (when it’s not currently mounted)?
Example: /mnt/mypool/
1 Like
I was about say no, as should be just datasets - but when I looked there was a test.img file there from speed tests I did. Is that significant?
Thanks
Mike
Most likely as true as complains about a non empty path. Move the file and try again.
So now had time to review these suggestions. I am still unable to unlock, with teh correct passphrase, in Scale as I was once able to. I am pretty sure teh change that was made that day was to specify teh pool I wanted apps to use, where scale created the ix-applications dataset.
I still get a GUI error trying to unlock - I do not want to force as it suggests, as not sure how safe this is given the causxe is unclear. It unlocks fine in core.
If I list the mount point I get this
Thoughts?
Edit: Looks to be same issue on many threads. Including this one that had no replies or real solutions.
https://www.truenas.com/community/threads/directory-is-not-empty-during-unlocking-pool-once-the-apps-pool-is-set-ix-applications.116238/
Is this ix-applications an empty directory?
Yes, but listed as a mount point.
If it’s empty, and it’s not currently mounted, why not just remove this empty directory?
zfs mount | grep "ix-applications"
Is it currently mounted at this path? If not, then you can safely remove the empty directory.
rmdir /mnt/HouseOfBogarde/ix-applications
1 Like
Thanks winnie. So I dont understand. I started Scale without the pool - there is a mount point to my pool, no mention of ix-applications.
If I attach the pool - its there. If I bring the pool up under Truenas Core, ix-applications is no where to be seen.
Im not following where this mount point is coming from - I know it was added by Scale, but I dont want to make changes to scale with the pool attached - ideally
CC
We’ll start from the top.
Without doing anything, while in SCALE, what are the following outputs? (It’s hard to troubleshoot blind.)
zpool list
zfs mount
ls -la /mnt/HouseOfBogarde
What might be happening is you have a residual “ix-applications” folder inside the top-level root dataset of your pool.
Sorry, I appreciate the total blindness, and appologise for my limited knowledge here. If I am honest, I am trying to gain more confidence in Scale, after years of faithfull Truenas Core service - so I am more interested in what happened to a clean install that left me unable to unlock the pool - and as there is a history of this in other threads, would be good to undertsnad the root cause.
So scale without pool attached, and
ls -la /mnt/HouseOfBogarde
And yes its there - so I can safely remove this?
rmdir /mnt/HouseOfBogarde/ix-applications
On the live pool, mounted on core - none of this exists.
And winnie; bless you for all your help
CC
No need to apologize. No one is really an expert in anything. 
But it’s desired, etiquette-wise, to post the output of all commands, even if you think they are pointless. (It’s also preferable to paste the text in codeblocks, rather than screenshots of text output.)
Perhaps you need to use sudo to remove the empty directory. Or perhaps the directory is not empty?
ls -lR /mnt/HouseOfBogarde/
If it outputs more files/folders beyond “ix-applications”, then it means there are things stored under this path.
No existing folders/files should exist in the mountpoint path of the dataset when attempting to mount the dataset, which includes “unlocking + mounting”. It’s possible that SCALE has extra safeguards which Core ignores.
But let’s try the above first.
I’m basing all of the above under the assumption that the “HouseOfBogarde” pool is not imported.
Im sorry for the etiquet, should know better. Was being very careful to mask contents - lets use putty so I can copy paste properly.
/mnt/HouseOfBogarde/:
total 1
drwxr-xr-x 2 root root 2 May 8 20:36 ix-applications
/mnt/HouseOfBogarde/ix-applications:
You are correct, the folder “ix-applications” appears empty, and pool is not attached or mounted at this point.
Even su to root I cannot remove the directory
# su root
# rmdir /mnt/HouseOfBogarde/ix-applications
rmdir: failed to remove '/mnt/HouseOfBogarde/ix-applications': Operation not permitted
Is this because its pointed to a pool that is not mounted? If so that makes sense, but that folder does not exist on that pool anyway! What am I missing? 
CC
So the directory is 100% empty?
The pool is not imported? No datasets, other than on the boot-pool, are mounted?
What are the permissions on the “ix-applications” folder?
getfacl /mnt/HouseOfBogarde/ix-applications
Corrent - 100% empty, no pool, datasets, just the boot pool present.
# file: mnt/HouseOfBogarde/ix-applications
# owner: root
# group: root
user::rwx
group::r-x
other::r-x
I also checked there were no old snapshots referencing it. There must be a process using it or locking it right? So I tried
lsof | grep "HouseOfBogarde/ix-applications"
But nothing. This all came about from a fresh install too. Setting the Apps pool location. Unsetting it, removing the dataset. Worrying
Found the bug again here Rogue ix-applications folder no solution really. I had tried zfs destroy, which also fails to find the dataset, which I woudl expect as I removed it from the Web UI
Even
chmod -R 777 ix-applications
rmdir ix-applications
rmdir: failed to remove 'ix-applications': Operation not permitted
CC
So had another look this morning, and it occured to me that I wonder if kubernetes is still holding onto this folder, even though its all gone in the GUI.
Sure enough
cli -c 'app kubernetes config'
+--------------------+--------------------------------+
| dataset | HouseOfBogarde/ix-applications |
+--------------------+--------------------------------+
k3s kubectl get namespaces
INFO[0000] Acquiring lock file /mnt/HouseOfBogarde/ix-applications/k3s/data/.loc k
INFO[0000] Preparing data dir /mnt/HouseOfBogarde/ix-applications/k3s/data/1ed3c c852f61a2a236a55f59fc8c26103fa2db11fca1b22730ef07456c4811d7
With no apps installed, and pool unset, I would expect Kubernetes to be disabled or not running, and this is confusing to me, but I am out of my depth.
systemctl status k3s
â—‹ k3s.service - Lightweight Kubernetes
Loaded: loaded (/lib/systemd/system/k3s.service; disabled; preset: disabled)
Active: inactive (dead)
Docs: https://k3s.io
I am unable to work out how to stop or remove kubernetes. So would welcome any further help to learn about these bugs, before I do a fresh install to recover to a working system.
Thanks
CC
This is the first time I’ve ever heard of issues caused by K3s on TrueNAS SCALE. 
Do you have another dataset that you can temporarily “set” as your “Apps” pool, which hopefully will “unstick” it from your HouseOfBogarde pool?
If so, you could possibly set it to another pool, and then unset it, and hopefully that will fix the issue of the protected / in-use folder.
1 Like
Can you point k3s to a different pool in the app GUI?
Maybe that will update the dataset entry in the k3s config to something other than HouseOfBogarde.
There’s a cli command app kubernetes update pool=_____ that I am not sure how to use, but it might be worth investigating.
I dont really have another pool I can use other than the boot pool.
Whats odd is if I attach the pool, I cannot unlock it because of this ix-applications directory. I think this is a bug, there are quite a few threads over two years where this has come up before. I have done a clean install, now have the GUI unresponsive issue, but I will see if the issue is repeatable.
Thanks
CC
1 Like
Might be worth filing an official bug report.
2 Likes
