Mysterious Permissions Issues in Syncthing/Nextcloud App Installs (24.04)

Trying to put something on a fresh 24.04 build that will allow users to autosync data from client devices. Started with Syncthing, went to NextCloud, then back to Syncthing. Same errors in both apps.

After getting the Syncthing app to finally deploy (which took ~1 billion years of troubleshooting), adding a folder to sync results in the following output in the Syncthing UI:

2024-10-18 22:04:18: Failed to create folder root directory: mkdir /mnt/TestPool: permission denied

2024-10-18 22:04:18: Error on folder “test” (xqcgv-gpurp): folder path missing

Logs tell the same story:

2024-10-18 15:04:05.526268-07:00[FHAD7] 2024/10/18 22:04:05 INFO: My name is “truenas”

2024-10-18 15:04:05.526273-07:00[FHAD7] 2024/10/18 22:04:05 INFO: TCP listener ([::]:22000) starting

2024-10-18 15:04:05.526282-07:00[FHAD7] 2024/10/18 22:04:05 INFO: Ready to synchronize “test” (trntu-26mcl) (sendreceive)

2024-10-18 15:04:05.526288-07:00[FHAD7] 2024/10/18 22:04:05 INFO: QUIC listener ([::]:22000) starting

2024-10-18 15:04:18.074829-07:00[FHAD7] 2024/10/18 22:04:18 INFO: Adding folder “test” (xqcgv-gpurp)

2024-10-18 15:04:18.074907-07:00[FHAD7] 2024/10/18 22:04:18 INFO: No stored folder metadata for “xqcgv-gpurp”; recalculating

2024-10-18 15:04:18.074943-07:00[FHAD7] 2024/10/18 22:04:18 WARNING: Failed to create folder root directory: mkdir /mnt/TestPool: permission denied

2024-10-18 15:04:18.074957-07:00[FHAD7] 2024/10/18 22:04:18 INFO: Ready to synchronize “test” (xqcgv-gpurp) (sendreceive)

2024-10-18 15:04:18.074973-07:00[FHAD7] 2024/10/18 22:04:18 INFO: Failed initial scan of sendreceive folder “test” (xqcgv-gpurp)

2024-10-18 15:04:18.074984-07:00[FHAD7] 2024/10/18 22:04:18 WARNING: Error on folder “test” (xqcgv-gpurp): folder path missing

The trouble is, the ACL for the dataset is perfect. Syncthing runs as UID/GID 568, which corresponds to “apps.” “apps” in the ACL has all the necessary perms, no mask or default issues, and I have verified that “apps” can r/w/e in the dataset with sudo su apps and mkdiring and touching stuff. I even tried giving ownership of the dataset to “apps,” since I was feeling generous.

NextCloud throws identical perm errors.

I suspect that this permission issue might have to do with the pool mount itself being restricted to root ownership and access only. This seems kinda weird, since dataset perms allow for SMB and bash access from regular users with the correct ACL settings. Also, why would ixSystem restrict the pool to root only, from installation, if it was gonna break all the cool apps you can put on your TrueNAS?

I have also tried checking all the Docker and Kubernetes configs, but neither Docker nor Kubernetes seems to be controllable from a terminal in TrueNAS Scale 24.04 - command not found.

A solution to this would also solve Syncthing permission denied on custom install

I have read approx. 1.468 trillion posts on this topic, including the ones in this forum, and have even resorted to consulting my disabled colleague ChatGPT, and have yet to solve. Any ideas?

Collection of links for reference, if desired:

Syncthing permission denied | TrueNAS Community
SyncThing - Directory Access Denied | TrueNAS Community
Not sure why Syncthing does not have permission on dataset on truenas scale | TrueNAS Community
Syncthing permission denied on custom install
Syncthing permission denied
Weird permission error on syncthing
Syncthing - Failed to create folder root directory mkdir /Test: permission denied.. | TrueNAS Community

I should add that adding folders in the Syncthing web UI that exist outside of the pool works just fine. That seems important

Update: running the Syncthing app as root makes it work. Because the pool is root-only by default and the datasets within it are permed correctly, I take this to mean that Syncthing needs root access in order to sync folders within a ZFS pool in TrueNAS Scale. Syncthing itself tells me this a bad idea. Seems kinda broken? Maybe this is an issue in 24.04? Would love input.

Update update: removing all access control settings for the dataset and allowing literally anyone to access it still throws “Permission denied” in Syncthing. This, combined with the fact that it works when running as root, is enough for me to conclude that Syncthing needs root permission at the pool level to traverse /mnt/[Pool]/[Dataset]. I assume this is the case with Nextcloud, too.

Of course, this is not how Syncthing is supposed to run, so what gives?

Final Update

I changed the ACL of the pool itself, adding syncthing as a user. I also opened up the ACL of the dataset which I wanted to sync. Those are screenshots #1 and #2. Shockingly, I still got the permission errors in screenshot #3.

Running syncthing as root still works.

I’m probably missing something painfully obvious, but that’s it for me. Seems like many have this problem, too, and have also not found any solutions. I’ll be going with a different OS for my current project, unless something jumps out. Wish everyone the best in the future.