Docker Compose & ACL/Permissins HELL

Apps and Virtualization
, , ,
1

Background:
I originally came from OMV (Open Media Vault), running docker compose based off of Smart Home Beginner (now SimpleHomeLab.com) Ultimate Docker Media Server. I love docker-compose, because it’s an easy way to manage a whole docker stack that I can backup and restore easily (isn’t that suppose to be the beauty of Docker). When TrueNAS SCALE was moving to Linux with scaling, clustering and high availability, I thought it was my chance to move to a more professionally maintained environment (read: Better Financed/Supported). I feared K8’s, but figured the benefits would outway the complexity. I latched on to TrueCharts, as they had better support and more app options, with an established ingress that was more cohesive. Unfortunately, the rude/unfriendly/toxic nature of the TrueCharts main dev gave me second thoughts and caused a riff with iXSystems now TrueNAS, only exacerbated by the (in my opinion poor kubernetes support) and eventually abandoning clustering all together (anyone else find it weird that SCALE name has been removed now). With clustering out, docker was the logical transition. I thought this was GREAT, back to simple compose stack and less abstraction. Unfortunately, the TrueNAS middlewares still maintained a confusing abstraction, creating problems such as inability to keep apps on the same docker network. I considered Portainer as my Docker UI manager, but apparently it doesn’t have a 1:1 pairity with docker compose, should I want to change managers. I tried moving to dockge, but it doesn’t currently support docker secrets. I settled on Arcane, as it can read already created docker compose stacks and add to them.

The Problem:
I fail to really understand how to set proper ACL’s for my datasets and, after selecting my docker dataset and recursively applying ACL’s to all child datasets, my docker compose stack doesn’t deploy properly. I tried adding environmental value PUID & PGID for “apps” to all containers, but multiple containers still have issues. Looks like I got myself into permissions hell and I don’t know how to get out. I appreciate @Stux & @serversathome @technotim & many other YouTubers for providing video Tutorials, but I haven’t quite found a comprehensive guide from start to finish for docker compose in TrueNAS. Maybe I missed the right tutorials? Can anyone point me to a tutorial on how to setup the TrueNAS environment to run an entire docker compose stack? And/or help me straighten out permissions hell?

2

Don’t know if can help, but when i moved on Scale from Core (jails) and Linux VM (NFS) i was a bit puzzled too by ACL.
My dataset structure Is a main dataset docker with many nested dataset, one each app; if the specific app need different permissions (eg postgres), i will use many other nested dataset the app need, and set ACL accordingly.
So for what Is my experience Is to not miss to set the traverse permission for the user in the respective higher dataset, or you can end in the app user can’t reach the desired dataset.

3

Sorry, that isn’t helpful.

4

So @lawrencesystems has a video on how to set permissions, but doesn’t really give best practices on how to setup if you want to do docker compose? https://youtu.be/59NGNZ0kO04

@Stux appears to stick with POSIX

@serversathome seems to use NFSv4 ACL (“Apps” dataset preset)

Again, my goal was to have a dataset I could run a docker compose through, ideally via Arcane, but also just video command line with nested stacks according to the aforementioned UDMS guide, but also to share via SMB, so I could easily add yaml files via my windows machine. I’m guessing there is no simple way to do this, hence my docker stack not working.

1 Like
5

I recommend not using ACLs at all, if you can avoid it. If you just want separation between containers, you don’t need them. For my system, I simply created different users for different containers and data stores (like “videos” for my video archive, “photos” for my photo archive, etc.), then set the different containers to run as specific users/groups with those PUID/PGID env variables, so that the containers are limited in how they can access data.

6

There really isn’t a best practices when it comes to permissions. It all depends on the image you want to run and and which user that image needs to run.
For example postgres wants to run as user 999. If you try to run it as a different it will simply fail to start with a permission error. Other images can be run as any user you want.

I do not run my apps directly on truenas but rather in an lxc managed by portainer, but inside that lxc i use the same permissions the truenas apps would use. 568 as general apps user, 999 for postgres and in rare cases if the container needs it 0 for root.

On the truenas side i gave user 568, 999 and my smb user permissions and have not encountered any problems. I do not like posix so all i use is nfsv4 for acl.

Hope that helps…

7

Without understanding exactly what you are trying to accomplish it seems like you are trying to make something simple very complicated.

TrueNAS docker support is not special in any particular way, unless you use TrueNAS Apps, in which case it is only special by having the docker compose pre-configured for you for those TrueNAS Apps. Otherwise, it works like docker on any Linux platform.

  1. Create users and/or groups for the apps under Credentials->Users / Groups. There are some built-in users and groups that you can use if you wish, for example apps (568) is a generic apps user and group.
  2. Under Datasets, create an apps dataset using the dataset preset “Apps”
  3. Create child datasets underneath your apps dataset as you wish, but I would suggest at least one per app if not more to e.g. enable individual snapshot restores. I do it on the command line with zfs create, but you can do it via the GUI too. Do not set NFSv4 ACLs.
  4. Use chown/chgrp/chmod to set the permissions on the child datasets as you want, matching the user and group that the container will run as.
  5. Deploy apps however you choose, whether that is via TrueNAS Apps or docker compose YAML – but only via the web UI, not the command line.
  6. Configure automatic snapshots for your apps dataset unless you like to live dangerously or your containers are completely disposable.

Only certain containers use PUID/PGID environment variables to set the user – generally those that need to start as root and then drop privileges later. Other containers can be run with “user:” to set the uid/gid.

I suggest to avoid ACLs until you are very comfortable deploying without ACLs. You can likely get everything you need without ACLs and simpler.

2 Likes
8

Just regarding docker networks, for every Docker Compose you create TrueNAS will automatically create a network. You have different options if you want to have containers on the same network, but only when you create the apps using docker compose – if you use TrueNAS Apps you cannot change the app networking.

  1. Create the containers in the same docker compose – they will automatically end up on the network ix__default. This is easiest.
  2. Create the containers in different apps/docker compose and use external networks. For example:

I have a caddy reverse proxy in its own application “caddy”. TrueNAS automatically creates a network “ix-caddy_default” when it starts. I want to have other apps like jellyfin use the caddy reverse proxy. To do this I need them to be in the caddy network, but I want to keep them contained in their own app so I can start and stop them independently. At the top of the jellyfin docker compose I have:

networks:
  default:
    external: False
  ix-caddy_default:
    external: True

And then in the Jellyfin service:

services:
  jellyfin:
    networks:
      - default
      - ix-caddy_default

In this case the container will join both its own default network (ix-jellyfin_default) and the caddy network.

Also, you are not meant to play with TrueNAS on the command line. While it will pick up a lot of things, I would not be deploying docker compose outside of the TrueNAS web UI. If you were not able to find it (because it is a little unobvious) Apps → Discover Apps → … (3 dots on the right) → Install via YAML.

2 Likes
9

My goal is to have a simple docker stack that’s easy to backup & recreate on another server, as needed. That’s suppose to be part of the beauty of docker, right? …TrueNAS implementation of docker lacks too many essentials, so I’ve rolled my own stack & let TrueNAS do what it does best: storage.

I’ve utilized the Ultimate Docker Media Server mentioned above, but permissions have me in a jam. I’ve got my own docker network within my stack. I’ve had trouble finding TrueNAS tutorials that guide through setting up images with appropriate permissions, & why permissions are set the way they are. Essentially I’m unsure how to diagnose & resolve permissions problems.

1 Like
10

Same here. Haven’t done anything because I don’t get any of these :baby:
But I will, and the day I do, that you say, is what I wanna have. As portable and repeatable as possible.

Thanks for the links and ideas.

Note to self: learn Linux stuff :slight_smile:

11

I don’t know where you got the easy to backup part. IMO, it isn’t easy by any means. At least there won’t be images with the single purpose of backing up docker containers if it were easy out of the box.

Regarding the recreation on another server, the easiest most robust way to (re)deploy the container to another server is to use docker swarm or kubernetes. These solutions have shared storages and assume multi-node by their very nature. So migrating to another server (node) is done with a few clicks.
:warning:DISCLAIMER!:warning: I have zero experience with docker swarm and very limited experience with k8s.

Also, you can manage your docker containers via portainer. Moreover, portainer can manage several servers (aka environments) from its UI. You just need to deploy the portainer agent (or edge agent) container – all configs would be stored on the machine with portainer. You can even migrate your stacks from one node to another.
However! It doesn’t support shared storage, so you would need to transfer your data-volumes manually, or your containers should use shared storage (smb/nfs/etc) for the data-volumes.

I use this approach for a few years now – it’s okayish so far. I even manage my VPS containers via local portainer (although, it wasn’t the easiest set up because of the desired security level).

1 Like
12

I interpret that you want to run some containers in a way that is portable, so if you choose to move away from TrueNAS then there is minimum effort. You want to have a docker compose stack that you can just copy portably to another computer. TrueNAS Apps are not really designed for this, but Dockge is. I don’t use Dockge so I deployed it with a simple container stack for jellyfin. You can follow this approach to deploy anything.

Step 0: make a snapshot and a backup of everything

  • Please make sure you have a snapshot and a backup of everything in case you are not happy with how this turns out

Step 1: make an Apps dataset

  • In the TrueNAS web UI create a new dataset for apps, and chose Dataset Preset = Apps.
  • DO NOT choose to setup ACL.
  • I will assume it is tank/apps and mounted as /mnt/tank/apps

Step 2: deploy Dockge datasets

  • If you skip ahead and look at the TrueNAS App for Dockge you will see it needs 2 datasets: stacks and data.
  • Let’s create datasets for Dockge, stacks and data. Under Datasets click on “tank/apps” and add a dataset “dockge” of Dataset Preset = Generic (default). DO NOT choose to setup ACL.
  • Under “dockge” add datasets for stacks and data, also Dataset Preset = Generic (default). DO NOT choose to setup ACL.
  • You will now have these datasets:
  1. tank/apps
  2. tank/apps/dockge
  3. tank/apps/dockge/stacks
  4. tank/apps/dockge/data

Step 3: deploy the Dockge app

  • Go to Apps → Discover Apps, search for Dockge and click Install.
  • Scroll down to Storage Configuration. Change the two volumes to Type = Host Path
  • Enter the host path for Stacks: /mnt/tank/apps/dockge/stacks
  • Enter the host path for Data: /mnt/tank12t/apps/dockge/data
  • DO NOT choose Enable ACL
  • Click install at the bottom and wait for Dockge to deploy

Step 3: open Dockge

  • Back on the Apps page, click Dockge and the on the right click Web UI
  • Set your userid and password

Step 4: Jellyfin pre-setup

  • Let’s deploy Jellyfin. Here is the docker-compose we will use for inspiration: Container | Jellyfin. I give you the docker-compose I deployed below…
  • We want Jellyfin to run as a specified user. In the TrueNAS web UI, make a user jellyfin, user id=4000, group jellyfin, group id = 4000 (or any numbers you like that are not used)
  • The docker-compose example says we need two volumes, config and cache.
  • In the TrueNAS web UI make some more datasets, exactly like you did before. We will create:
  1. tank/apps/jellyfin
  2. tank/apps/jellyfin/cache
  3. tank/apps/jellyfin/config
  • click on tank/apps/jellyfin/cache and edit permissions. Set the permissions user: jellyfin, group: media, select Apply User, select Apply Group, and Save. DO NOT click Set ACL.
  • repeat for tank/apps/jellyfin/config

Step 5: Make a dummy media library

  • the docker-compose example also said we needed a volume for media.
  • I would like you to do this first with a dummy dataset so that you can experiment with permissions safely.
  • It is pretty normal that the media library is on a SMB share, so let’s start with that.
  • Add a dataset tank/media, type SMB share, share name: media
  • Click “Go to ACL”
  • You can set this up in different ways. My datasets have Owner: my account, Owner Group: my account. If you do this you will see below that owner and owner group “my account” received Modify permissions to the dataset.
  • If you don’t do this you need to give yourself access. Click Add Item, Who: User, User: your account, Permissions: Modify
  • You will also need to give jellyfin access. Click Add Item, Who: User: User: jellyfin, Permissions: read
  • Click “Apply Recursively” and then “Save Access Control List”.
  • Connect to the share and put some media files, or copy media files into the dataset to use for testing.
  • Once you have this working with a test you can follow the same principle to share real data with a container.
  • Note that the TrueNAS GUI only lets you change permissions on datasets. If you want more fine-grained control you need to use the command line.

Step 6: Deploy Jellyfin

  • Go back to Dockge. Click “+ Compose”. Give the stack a name: jellyfin
  • Paste the compose YAML you want to use in the top right. This is what I deployed:
services:
  jellyfin:
    image: jellyfin/jellyfin
    container_name: jellyfin
    user: 4000:4000 # matching the jellyfin user and group that you used above
    ports:
      - 8096:8096
    networks:
      - jellyfin_network
    volumes:
      - /mnt/tank/apps/jellyfin/config:/config
      - /mnt/tank/apps/jellyfin/cache:/cache
      - /mnt/tank/media:/media:ro # mounted read-only for safety as Jellyfin doesn't need write permissions to your media library
    restart: 'unless-stopped'
  • Underneath Internal Networks, add a network: jellyfin_network
  • Click Save.
  • You will find your stacks neatly arranged on the server under /tank/apps/dockge/stacks. You can copy those and take anywhere.
root@fileserver[/mnt/tank/apps/dockge/stacks/jellyfin]# ls -la
total 28
drwxr-xr-x 2 root root   4 Oct  3 14:57 .
drwxr-xr-x 3 root root   3 Oct  3 14:57 ..
-rw-r--r-- 1 root root  25 Oct  3 15:01 .env
-rw-r--r-- 1 root root 341 Oct  3 15:01 compose.yaml

Step 6: Start Jellyfin

  • Go to http://your-truenas:8096
  • Login to Jellyfin
  • Try adding your media library. With some luck it will work the first time but if you have trouble please post enough information and screenshots that we can help.

You can follow the above procedure for deploying any container that needs to access any file on your TrueNAS.

Back to your comments:

The original design of TrueNAS (from FreeNAS) was a bootable USB stick that had your config, and then your data disks. If something happened to your server you could move the USB stick and data disks and be up and running in minutes. If something happened to the USB stick you could write a new one and restore a small config file and be up and running in minutes. If something happened to the data… well, that’s why you have ZFS, snapshots, mirrors or raidz, and backups. :slight_smile: This is all still essentially true today: TrueNAS is designed for easy recovery of a complete system and not configuration exports of individual bits like container stacks.

Further, containers are designed with TrueNAS Apps in mind. TrueNAS Apps use a kind of pre-chewed application template and make no sense on any platform other than TrueNAS. So again portability is not a priority.

TrueNAS also has as a priority to “remove sharp edges”. That means it doesn’t necessarily present all the configurability you might have elsewhere. In exchange you are less likely to shoot yourself in the foot. They view that as better for their customers overall.

Whether you like it is up to you, but having gone through the above, you have a perfectly valid TrueNAS config that probably is more like what you wanted. You can also decide to not use TrueNAS and copy the stacks and data elsewhere, or you can decide not to use Dockge and cut-and-paste the compose.yaml into individual TrueNAS Apps to run “natively”. At the very least you should not be in ACL & Permission Hell anymore. :slight_smile:

Best wishes,

Evan

3 Likes
13

I’ll just note for everyone else that Dockge is only there to service GuyFromMars’s requirement to manage containers out of a simple stack of docker compose.yaml files. All the above applies just as well out of TrueNAS directly without Dockge.

1 Like
14

I want to give a big thanks to @Evan123 for this very detailed explanation of how to set up custom compose apps using Dockge and configuring the permissions/ ACLs correctly.
I am currently transforming my Docker in VM + Jailmaker (for containers that need GPU access) setup that originated back from FreeNAS days to compose stacks directly on TrueNAS.
As I wanted to get permissions right this time, your guide was really very helpful!

One thing I changed is that I also created a SMB share for the dockge/stacks directory, so that I can manage the YAML files and, more importantly, any auxiliary configuration directly using the file manager.

Also, the context about differences between Dockge (and Portainer) and the TrueNAS apps with regard to portability is very helpful.
IMO, the container management experience with YAML files is currently really basic.

1 Like
15

This thread helped me a lot, thanks so much, sorry to revive it! Question for @Evan123 : the person in this video https://www.youtube.com/watch?v=AzGq2lJSKpo seems to be using just normal (generic) POSIX based datasets for config files (you only have to watch the first couple of minutes to see what I mean) whereas you use the “apps” type dataset… either way would be OK?

Also, in your Jellyfin compose example you just use normal volumes instead of the bind type that is talked about in the TrueNAS documentation, not a real problem I guess? I’m still learning… thanks!

1 Like
16

Hi @HerrRay— The apps dataset type has small differences only, For example granting access to the “apps” user that is used by default by TrueNAS apps. If you are concerned you can just try creating one of each type and use zfs get all to compare the differences.

I’m not exactly sure what you are referring to for the volumes in my docker-compose example as those are bind mounts in the container, but for production use never use IX Volumes and always use bind mounts (called Host Paths in TrueNAS Apps).

17

Thanks for your quick reply! Maybe I’m confused, but what I thought is the following: when I convert an app to a “custom app” (clicking on the … menu next to Application Info in the upper right corner when you click on the application in the list of installed apps, if you know what I mean) TrueNAS will generate a yml file with something like this for a volume:

volumes:
  - bind:
      create_host_path: False
      propagation: rprivate
    read_only: False
    source: /mnt/tank/apps/jellyfin/config
    target: /config
    type: bind

I thought just specifying volume as in your example with just a path:path is not a bind volume? Maybe I’m totally wrong, I’m just starting out with Docker-like things more thoroughly… (I’ve read the TrueNAS docs and the linked Docker documentation on this but maybe I misunderstood…)

Edit: last sentence

18

When you convert a TrueNAS App to docker-compose syntax it includes a lot of unnecessary syntax. You can leave it or remove it — it’s not wrong it’s just very verbose and may be confusing when you compare against other sources like the Jellyfin reference docker-compose.

19

Right! Learning a lot, thanks!

So, when I just put in:

volumes:
      - /mnt/tank/apps/jellyfin/config:/config

it will be a bind volume…

As a matter of fact, I’ve put in some other apps like this as custom yml files with no problems, but I wanted to do things more correctly so I started to read a little bit…

Could you recommend some reading material to further my starting-out understanding?

In any case thanks @Evan123 for the already supplied help in this thread!

20

Yes, that’s right.

I wouldn’t overthink it. When you make a new compose go on the container’s official repository’s page on dockerhub or GitHub and use their docker-compose.yaml as a reference. If you want to do something unusual it will be fastest to ask your favourite AI to make the config and then explain it to you with references to the documentation so you can double-check.