Video: Setting up Sandboxes with Jailmaker for Docker, Dockge and Jellyfin

Stux · April 27, 2024, 11:55am

G’day Guys,

I decided to make a video describing TrueNAS Scale Dragonfish’s new Sandbox feature.

In the video I demonstrate Jailmaker, and describe how to set up networking, Docker, and Dockge the Docker Compose stacks manager, as well as a Jellyfin Server.

Although its possible to script the installation, I go through a fairly complicated install, and demonstate both MAC-VLAN networking as well as Bridge Networking.

Enjoy

YouTube: Setting up Sandboxes with Jailmaker

IMPORTANT: Jailmaker 2.0 removed the install function used in the above video, where I use install, you should instead follow the instructions for creating an alias, otherwise you will see the following error: jlmkr.py: error: argument : invalid choice: 'install'

Other common issues:

How do I install jailmaker your way?

make a jailmaker dataset on the root of your pool using generic presets (in the examples below my pool is called “tank”)
in a shell session, cd into the jailmaker dataset and clone the repo

cd /mnt/tank/jailmaker
git clone https://github.com/Jip-Hop/jailmaker.git .

follow the instructions for creating an alias
set a post-init startup command to run the jailmaker startup command, eg:
/mnt/tank/jailmaker/jlmkr.py startup

How do I upgrade the version of jailmaker I have installed?

If you followed my instructions to clone the jailmaker repo, then just cd into your jailmaker dataset and pull the new version

cd /mnt/tank/jailmaker
git pull

That will update your jailmaker install. If you need to adjust your jail’s config you can compare to the current version of the template you used to create the jail .

Otherwise, re-install the jailmaker script to update it, alternatively, delete it and clone jailmaker’s repo as per my install instructions above.

Does the sandbox persist within each scale update?

Yes.

This is a quote from the TrueNAS docs: “These containers persist across upgrades in 24.04 (Dragonfish) and later SCALE major versions”

When I run jlmkr I get `zsh: permission denied: ./jlmkr.py`

check the permissions on jlmkr.py script. If you cloned the repo, they should be correct.
if you are not logged in as root, then you need to use sudo, if you setup the alias, the alias should do this for you
check the “Exec” property on the jailmaker dataset. It needs to be “ON” or “Inherited (ON)”. You may not have Exec defaulting to ON on your root dataset.

kris · April 27, 2024, 12:04pm

I ran through a bit of this, fantastic in-depth tutorial @Stux !!

Thats worth a tweet on my side

giftcardreloaded · May 20, 2024, 2:35pm

Thanks for this resource!

at the end of the Jellyfin setup( ~17:37), you choose /data as a media folder and are presented subfolders from your /tank/media directory. See attached pic with recording subfolder visible:

Following along to this point, I did not have an option to select sub-folders in my setup.

I wonder if this could be due to any of the following:

My “media” folder (/pool1/library) has child datasets for each media type: music, shows, and movies. Could nested datasets be a problem?
Could it be a permissions issue? I have made sure that the jellyfin user & group have access to these datasets.
Jellyfin had a major release since this video was recorded. I doubt this would change things, but who knows.
I got the bindings wrong. (most likely)

Jellyfin’s Dockge config:

volumes:
      - /mnt/data/Jellyfin/config:/config
      - /mnt/data/Jellyfin/transcodes:/config/data/transcodes
      - /mnt/media:/data

Jailmaker config:

systemd_nspawn_user_args=--network-macvlan=eno1
        --resolv-conf=bind-host
        --system-call-filter='add_key keyctl bpf'
        --bind='/mnt/pool1/docker/data:/mnt/data'
        --bind='/mnt/pool1/docker/stacks:/opt/stacks'
        --bind='/mnt/pool1/temp:/home/downloads'
        --bind='/mnt/pool1/library:/mnt/media'

If I’m understanding it correctly /data should map to /mnt/media which should map to /mnt/pool1/library

Anything jump out to you here? Hopefully this is a quick tweak.

Thanks again for all that you do @Stux!

Stux · May 20, 2024, 9:27pm

I suspect the issue is #2. Permissions issue.

Your config looks correct.

Jellyfin needs to be able to traverse and read the media dataset. It shouldn’t matter that you have child datasets.

BUT, Jellyfin is a Linux docker container. It’s going to use the Unix permissions, not ACLs.

I use world-readable on my media dataset, as I’m already using the group for “media writers”

To make sure it’s correct you can shell into the jellyfin container and check that the content of the /data directory is correct.

winnielinnie · May 20, 2024, 11:37pm

I believe this might contribute to the “unable to read/traverse” issue as well?

Because if systemd-nspawn is similar to FreeBSD jails, then you must bind/mount each real filesystem (dataset) that you wish to have access from within the “jail”.

Simply giving access to a parent dataset does not necessarily give access to all children datasets underneath. (Imagine non-nested independent filesystems.)

giftcardreloaded · May 20, 2024, 11:39pm

Can you explain what this means in the context of your earlier comment that “[Jellyfin is] going to use the Unix permissions, not ACLs.”?

To make something “world-readable” in my case, should I be changing the necessary directory permission(s) in docker (chmod 444?) or the ACL for my library dataset in TrueNAS?

both dockge (/data) and docker (/mnt/media) show the right subdirectories

winnielinnie · May 20, 2024, 11:40pm

What if you try to look further down? (Not simply checking if the subdirectories exist.)

giftcardreloaded · May 20, 2024, 11:44pm

Yep, everything looks in order. Movies and files, etc. are there.

giftcardreloaded · May 20, 2024, 11:57pm

Would I need to map each of my library sub-directories in the jellyfin docker compose file and the docker config?

volumes:
      - /mnt/data/Jellyfin/config:/config
      - /mnt/data/Jellyfin/transcodes:/config/data/transcodes
      - /mnt/media:/data
      - /mnt/media/shows:/data/shows
      - /mnt/media/movies:/data/movies
      - /mnt/media/music:/data/music

and

systemd_nspawn_user_args=--network-macvlan=eno1
        --resolv-conf=bind-host
        --system-call-filter='add_key keyctl bpf'
        --bind='/mnt/pool1/docker/data:/mnt/data'
        --bind='/mnt/pool1/docker/stacks:/opt/stacks'
        --bind='/mnt/pool1/temp:/home/downloads'
        --bind='/mnt/pool1/library/shows:/mnt/media/shows'
        --bind='/mnt/pool1/library/movies:/mnt/media/movies'
        --bind='/mnt/pool1/library/music:/mnt/media/music'

winnielinnie · May 21, 2024, 12:39am

For both? I’m not sure, since I do not use SCALE. However, for Core (FreeBSD jails), you need to configure a “mountpoint” for every dataset that you wish to access within the jail; even if they are child datasets perfectly nested underneath the parent.

So in your case, if “shows”, “movies”, and “music” are each their own dataset (not simply subfolders), then I would assume the same principle applies to SCALE (systemd-nspawn + docker). Perhaps it’s not enough to simply do it once in the “jail”, and you need to do it again in docker (the next “onion layer” in a sense)?

So I assume this will work:

As long as such binds are already defined for the jail itself.

Otherwise, only having /mnt/media:/data will not suffice to overcome the “independent datasets” hurdle.

(If it’s anything like a FreeBSD jail, you’ll have to restart the “Linux jail” for such changes to take effect.)

Why haven’t I played around with SCALE in a VM yet? That’s a good question.

Stux · May 21, 2024, 3:25am

I did test this

some actual nested datasets…

bind mounted into a jail like so:

systemd_nspawn_user_args=--network-bridge=br0
        --resolv-conf=bind-host
        --system-call-filter='add_key keyctl bpf'
        --bind='/mnt/tank/docker/data:/mnt/data'
        --bind='/mnt/tank/docker/compose:/compose'
        --bind='/mnt/tank/docker/mirror:/mnt/docker-mirror'
        --bind='/mnt/tank/docker/jellyfin:/mnt/jellyfin'
        --bind='/mnt/tank/media:/mnt/media'

NOTE: the above setup is an older jail, and not how I recommend in my video

when I shell into the docker and check the jellyfin mount… .the children are visible

root@chronus[/mnt/tank/jailmaker]# jlmkr shell docker
Connected to machine docker. Press ^] three times within 1s to exit session.
root@docker-jail:~$ cd /mnt/jellyfin/
root@docker-jail:/mnt/jellyfin$ ll
total 578
drwxr-xr-x  2 1005 1007  2 May  4 18:05 transcodes
drwxr-xr-x  6 1005 1007  6 Apr 12 08:59 cache
drwxr-xr-x 10 root root 10 Mar 27 19:15 ..
drwxr-xr-x  5 root root  5 Mar 25 15:43 .
drwxr-xr-x  7 1005 1007 14 Mar 25 15:14 config
root@docker-jail:/mnt/jellyfin$ ll config/
total 191
drwxr-xr-x 2 1005 1007    5 May 21 10:01 log
-rw-r--r-- 1 1005 1007 2174 Apr 24 18:28 encoding.xml
drwxr-xr-x 5 root root    5 Mar 25 15:43 ..
drwxr-xr-x 7 1005 1007   14 Mar 25 15:14 .
drwxr-xr-x 2 root 1007    2 Mar 25 15:13 cache
drwxr-xr-x 3 1005 1007    3 Mar 17 08:36 users
-rw-r--r-- 1 1005 1007  767 Mar  5 17:28 dlna.xml
-rw-r--r-- 1 1005 1007  263 Mar  5 17:22 branding.xml
-rw-r--r-- 1 1005 1007 5762 Mar  5 17:22 system.xml
-rw-r--r-- 1 1005 1007 1453 Mar  4 13:42 network.xml
-rw-r--r-- 1 1005 1007 2181 Mar  4 13:21 migrations.xml
drwxr-xr-x 7 1005 1007    7 Mar  4 13:19 data
drwxr-xr-x 4 1005 1007    4 Mar  4 13:19 dlna
-rw-r--r-- 1 1005 1007 1362 Mar  4 13:19 logging.default.json
root@docker-jail:/mnt/jellyfin$ ll transcodes/
total 545
drwxr-xr-x 2 1005 1007 2 May  4 18:05 .
drwxr-xr-x 5 root root 5 Mar 25 15:43 ..
root@docker-jail:/mnt/jellyfin$ ll cache/
total 51
drwxr-xr-x 2 1005 1007  33 May 21 10:36 temp
drwxr-xr-x 2 1005 1007 354 May 21 00:37 omdb
drwxr-xr-x 2 1005 1007   3 Apr 12 08:59 imagesbyname
drwxr-xr-x 6 1005 1007   6 Apr 12 08:59 .
drwxr-xr-x 5 root root   5 Mar 25 15:43 ..
drwxr-xr-x 3 1005 1007   3 Mar  4 14:17 images
root@docker-jail:/mnt/jellyfin$

Now, for this jellyfin instance, each “dataset” is separately mounted… but that’s out of necessity… so I can move the cache and transcodes out of the config directory… so that I don’t have to backup their changes

    volumes:
      - /mnt/jellyfin/config:/config
      - /mnt/jellyfin/cache:/config/cache
      - /mnt/jellyfin/transcodes:/config/data/transcodes
      - /mnt/media:/data

In the past I did have child datasets in media. I believe that worked, but I eliminated the child datasets so that mv between directories in the media dataset were moves… not copies.

And I just tested again by mounting the jellyfin dataset into the jellyfin container at “/configroot” by adding another mount to the jellyfin compose file, ie - /mnt/jellyfin:/configroot

root@docker-jail:/compose/jellyfin$ docker compose up -d
[+] Running 1/1
 ✔ Container jellyfin  Started                                                                                                                                                    11.9s 
root@docker-jail:/compose/jellyfin$ docker compose exec jellyfin bash
root@docker-jail:/# cd /configroot/
root@docker-jail:/configroot# ll
total 578
drwxr-xr-x 5 root root  5 Mar 25 04:43 ./
drwxr-xr-x 1 root root 11 May 21 03:14 ../
drwxr-xr-x 6 abc  abc   6 Apr 11 22:59 cache/
drwxr-xr-x 7 abc  abc  14 Mar 25 04:14 config/
drwxr-xr-x 2 abc  abc   2 May  4 08:05 transcodes/
root@docker-jail:/configroot# cd config/
root@docker-jail:/configroot/config# ll
total 191
drwxr-xr-x 7 abc  abc    14 Mar 25 04:14 ./
drwxr-xr-x 5 root root    5 Mar 25 04:43 ../
-rw-r--r-- 1 abc  abc   263 Mar  5 06:22 branding.xml
drwxr-xr-x 2 root abc     2 Mar 25 04:13 cache/
drwxr-xr-x 7 abc  abc     7 Mar  4 02:19 data/
drwxr-xr-x 4 abc  abc     4 Mar  4 02:19 dlna/
-rw-r--r-- 1 abc  abc   767 Mar  5 06:28 dlna.xml
-rw-r--r-- 1 abc  abc  2174 May 21 03:15 encoding.xml
drwxr-xr-x 2 abc  abc     5 May 21 00:01 log/
-rw-r--r-- 1 abc  abc  1362 Mar  4 02:19 logging.default.json
-rw-r--r-- 1 abc  abc  2181 Mar  4 02:21 migrations.xml
-rw-r--r-- 1 abc  abc  1453 Mar  4 02:42 network.xml
-rw-r--r-- 1 abc  abc  5762 Mar  5 06:22 system.xml
drwxr-xr-x 3 abc  abc     3 Mar 16 21:36 users/
root@docker-jail:/configroot/config#

yep, child datasets work in jails and nested docker containers.

giftcardreloaded · May 21, 2024, 4:20am

Correct, “shows”, “movies”, and “music” are datasets in my case.

winnielinnie:

So I assume this will work:
giftcardreloaded:
      - /mnt/media:/data
      - /mnt/media/shows:/data/shows
      - /mnt/media/movies:/data/movies
      - /mnt/media/music:/data/music
As long as such binds are already defined for the jail itself.

I tried this and was not successful. Unless I’m misunderstanding what you mean by binds being “already defined for the jail itself”, I assume this means I would need to define the correct mappings in Dockge & the docker config script.

Is it enough to declare folders in a config for them to get created? For example, when mapping /mnt/pool1/library to /mnt/data (and then to /data), do /mnt/data and /data exist as mountpoints in their respective environments? Or are they created because those folders are specified?

Reason I ask is because I’m not sure if

      - /mnt/media:/data
      - /mnt/media/shows:/data/shows
      - /mnt/media/movies:/data/movies
      - /mnt/media/music:/data/music

with create the “shows”, “movies” and “music” sub-folders in the mount points just because I write them out.

Thanks for your help and advice so far.

Stux · May 21, 2024, 4:40am

both sides of a docker mount point get created if they don’t exist.

but this means the permissions are a bit of a crapshoot.

When I tested, both sides end up as drwxr-xr-x 2 root root

For jellyfin, the jellyfin user needs to be able to traverse your /data mount IN the container.

Which means they need to be able to traverse your media/library mount in the jail.

So, shell into the jail, and check the mount

root@docker-jail:/mnt/media$ ll
total 4344143
drwxrwxr-x 194 root 1008        199 May 21 14:25  recordings
drwxrwxr-x   6 root 1008          9 May 14 10:38  .
drwxrwxr-x   3 1004 1008          4 May 13 17:59  music
drwxr-xr-x  10 root root         10 Mar 27 19:15  ..
drwxrwxr-x   6 root 1008          7 Mar  9 13:48  movies
drwxrwxr-x   8 root 1008         10 Mar  7 14:02  tvseries

1008 is my 'media writers" group, for things that can write to the media directory.
BUT all the directories are world-readable, ie o=r-x

I don’t want jellyfin able to modify media

ANYWAY, you need to verify that the jail has at least read access to the sub directories in your media mount.

And then you need to verify one level deeper.

root@docker-jail:/mnt/media/recordings$ ll
total 10468
-rwxrwxr-x   1 root 1008 357282 May 21 14:32  fetchtv_save_list.json
drwxrwxr-x   2 root 1008    117 May 21 14:32  Horrible_Histories
drwxrwxr-x 194 root 1008    199 May 21 14:30  .
drwxrwxr-x   2 root 1008    341 May 21 14:11  The_Big_Bang_Theory
drwxrwxr-x   2 root 1008    140 May 21 14:01  The_Next_Step
drwxrwxr-x   2 root 1008     88 May 21 13:40  Octonauts
...

Again world readable.

Then you can try checking the permissions from inside your Jellyfin container…

You can do this with dockge, by entering the containers shell, or on the cmd line by entering the jail, then into the /opt/stacks/jellyfin directory and docker compose exec jellyfin bash

root@docker-jail:/compose/jellyfin$ docker compose exec jellyfin bash
root@docker-jail:/# cd /data
root@docker-jail:/data# ll
total 4344143
drwxrwxr-x   6 root 1008          9 May 14 00:38  ./
drwxr-xr-x   1 root root         10 May 21 04:29  ../
drwxrwxr-x   6 root 1008          7 Mar  9 02:48  movies/
drwxrwxr-x   3 1004 1008          4 May 13 07:59  music/
drwxrwxr-x 194 root 1008        199 May 21 04:35  recordings/
drwxrwxr-x   8 root 1008         10 Mar  7 03:02  tvseries/

Notice, container doesn’t know about the groups… but world readable

giftcardreloaded · May 21, 2024, 3:13pm

Very helpful, thanks @Stux . Hopefully my last question and I can get this working:

root@docker:/mnt# ls -l
total 50
drwxr-xr-x  3 root root  3 May 19 15:03 data
drwxrwx--- 15 root root 16 Apr 14 10:16 media
drwxr-xr-x  5 root root  5 May  5 09:39 pool1
root@docker:/mnt# chmod a=rX media
chmod: changing permissions of 'media': Operation not permitted

Why can’t I change the directory permissions as root in docker? I can chgrp just fine.

Stux · May 21, 2024, 5:40pm

That’s a bind mounted dataset right? Try changing it in the TrueNAS dataset screen.

(You could also try in the TrueNAS shell)

giftcardreloaded · May 21, 2024, 6:54pm

That message was in the TrueNAS shell after jlmkr shell docker :

root@docker:/mnt# ls -l
total 50
drwxr-xr-x  3 root root  3 May 19 15:03 data
drwxrwx--- 15 root root 16 Apr 14 10:16 media
drwxr-xr-x  5 root root  5 May  5 09:39 pool1
root@docker:/mnt# chmod a=rX media
chmod: changing permissions of 'media': Operation not permitted

I tried changing permissions on my library dataset in Truenas

root@truenas[/mnt/pool1]# chmod a=rX library  
chmod: changing permissions of 'library': Operation not permitted

I wonder if I’m not confusing myself now. Which folder’s permissions get edits in truenas?

mooglestiltzkin · May 21, 2024, 7:44pm

another reason to use jellyfin is it’s free. no paywalls for the features unlike the alternatives.

functionally it also just works.

if you expect responsive live transcoding for jellyfin, it’s recommended you setup graphics card acceleration for it. If however you use the jellyfin media player and feed it to an external player with ffmpeg codec, you can avoid live transcoding, as it will simply direct playback (for best quality and responsiveness).

dockge is great, and what i am using right now. But it’s not the only option. There is portainer which is another popular option which i started with first. And there are probably others as well. But if you are starting out with docker, or even an advance user, dockge is simply great.

thx for the youtube stux

Stux · May 21, 2024, 9:35pm

If this command did work you’d be removing write privs from all. Ie user,group and other

You’d probably meant chmod -R o+rX library

BUT in the dataset view. The screenshot. Click library then edit permissions (I think) then in the ACL make sure there is an @everyone permission with read. Apply that recursively if necessary.

@owner, @group & @everyone maps to the Unix user, group and other perms.

giftcardreloaded · May 22, 2024, 8:06pm

ahhhh this fixed it!! Thank you! I was definitely over complicating things… glad it came down to a simple fix.

BitByteBit · June 5, 2024, 11:26am

Thanks for posting this great video (and the one on bridges) @Stux - extremely useful!
I used them as a basis for setting up Syncthing in Jailmaker, which so far is working perfectly.
A lightweight Syncthing “jail” was the last thing stopping me migrating from CORE to SCALE, and now with Sandboxes, Jailmaker and these resources it’s finally sorted and I’ve made the move.
Cheers