Linux Jails (sandboxes / containers) with Jailmaker

FUSE works for me: jailmaker/templates/lxd/config at main · Jip-Hop/jailmaker · GitHub

But I haven’t tried in combination with user namespacing.

Off topic: I “bookmarked” this thread, which adds a little gray icon next to the title. The gray bookmark icon looks similar to the gray “lock” icon.

More than once I wondered “Why did the mods lock this thread?!”

:laughing:

2 Likes

Just posting to say that adding the bind for /dev/fuse to the config helped fix the Borg Backup feature for Nextcloud AIO for myself.

Also wanted to thank @Jip-Hop for making jailmaker, glad to be finally off Truecharts (also migrated all apps from Truecharts to jailmaker and couldn’t be happier)

4 Likes

Ah brilliant! I didn’t even think of simply passing the bind mount for /dev/fuse. I’ll give it a try.

Once I get that going, I would say jailmaker is the most hassle-free method for running Nextcloud AIO. Even with the upcoming native Docker Compose support, I’m staying with jailmaker for the added peace of mind that user namespacing gives me from a security point of view. While not as good as full isolation (separate VM + running rootless), it’s good enough for home use.

The next step for me is learning how to use Podman.

2 Likes

In the context of Jailmaker running docker with Dockge. Can anyone think of a way to prevent different compose to connect to each other?

Right now they can talk together via the IP used in the jail.

To make this slightly more complicated, I am running nginx in one of these compose and would like it to talk to other containers, but only nginx and not other containers between each other across “compose”.

Fun fact: I found I can prevent all traffic between compose by putting the jail IP as -p IPAddressDeny=jail-ip in jailmaker’s config… However my nginx can’t connect to anything either then :face_with_diagonal_mouth:

Any ideas? Thanks!

By default Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by the service’s name.

You should create different networks if you don’t want all your containers to be able to connect. In your nginx compose you’d have something like this:

    networks:
      - nginx
      - proxy_service1
      - proxy_service2
      - proxy_service3

And each service would be on their dedicated network which connects only the service and your nginx reverse proxy. But you’d be better off discussing this in another topic as this is generic docker compose stuff and not directly related to jailmaker. Good luck!

2 Likes

I have been able to move my PlexServer from TrueChart to Jail, in dockage.
In the jail I see the nvidia-card, but in the Plex docker the GPU is not found. Wondering what I did wrong. In the jail config the gpu_passthrough_nvidia=1
From with a shell in the jail I see the nvidia-card:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3060        Off | 00000000:08:00.0 Off |                  N/A |
|  0%   62C    P0              35W / 170W |      1MiB / 12288MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

here is the response from docker info command:

Client: Docker Engine - Community
 Version:    26.1.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 7
  Running: 7
  Paused: 0
  Stopped: 0
 Images: 9
 Server Version: 26.1.4
 Storage Driver: overlay2
  Backing Filesystem: zfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d2d58213f83a351ca8f528a95fbd145f5654e957
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.29-production+truenas
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 125.7GiB
 Name: apps
 ID: 4e592b63-d8d6-4e53-859f-d77694b218ab
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Here is the PlexServer compose.yaml file:

version: "2"
services:
  plex:
    container_name: plex
    image: plexinc/pms-docker:plexpass
    restart: unless-stopped
    ports:
      - 32400:32400/tcp
      - 3005:3005/tcp
      - 8324:8324/tcp
      - 32469:32469/tcp
      - 1900:1900/udp
      - 32410:32410/udp
      - 32412:32412/udp
      - 32413:32413/udp
      - 32414:32414/udp
    environment:
      - PLEX_UID=568
      - PLEX_GID=568
      - PUID=568
      - PGID=568
      - TZ=America/Toronto
      - PLEX_CLAIM=******************
      - ADVERTISE_IP=http://192.168.x.x:32400/
      - HOSTNAME=New-Mikey
      - VERSION=docker
    hostname: New-Mikey
    volumes:
      - /mnt/appconfig/plex/PlexConfig:/config
      - /mnt/appconfig/plex/PlexConfig/Transcode:/transcode
      - /mnt/share:/data
networks: {}

Any help would be appreciated :slight_smile:

Thanks, the issue was that the restart button in dockge doesn’t actually restart the compose. (at least the network config don’t apply then). I had to do docker compose down. (perhaps the stop and inative button would have worked.

1 Like

According to

You need to set the container runtime to “nvidia”

runtime: nvidia in your compose file should do it.

Added the:
runtime: nvidia

No difference :frowning:
really don’t know how they did it in Truechart. Plex seems to be a beast on its own.

See my thread when we got this originally working. You need the NVIDIA_* vars.

You can also see my working Plex config as well.

1 Like

Went through all the step adding the NVIDA_* environemnt
with runtime: nvidia … without … everything should be working … Plex does not want to behave :frowning:
By the way I am running the docker template on Jailmaker 2.0.0
Truenas verions 24.04.1.1
Everithing is showing fine, from the Plex Container terminal:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3060        Off | 00000000:08:00.0 Off |                  N/A |
|  0%   61C    P0              42W / 170W |      1MiB / 12288MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

But Plex itself won’t recongnize the gpu

I got It working !!! PlexServer is now recognizing my NVIDIA card.
Here is the secret sauce. in the config for the jail add the following mounted point:

--bind='/dev/dri:/dev/dri'

This is to be able to show the renderD128 driver to Plex.

Then in your Plex Compose.yaml something that look like:

services:
  plex:
    container_name: plex
    image: plexinc/pms-docker:plexpass
    restart: unless-stopped
    runtime: nvidia # <- Trick one
    ports:
      - 32400:32400/tcp
    environment:
      - PLEX_UID=568
      - PLEX_GID=568
      - PUID=568
      - PGID=568
      - TZ=America/Toronto
      - PLEX_CLAIM=claim-***************
      - ADVERTISE_IP=http://192.168.xxx.xxx:32400/
      - HOSTNAME=New-PlexServer
      - VERSION=docker
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,video,utility
    hostname: New-PlexServer
    volumes:
      - /mnt/appconfig/plex/PlexConfig:/config
      - /mnt/appconfig/plex/PlexConfig/Transcode:/transcode
      - /mnt/share:/data
      - /dev/dri:/dev/dri #### for hardware transcoding <-- Trick 2
networks: {}
2 Likes

SMI won’t show working in the jail. You have to run it from truenas to see if it’s working.

Anyone running rootless docker (not podman) in an unprivileged jail?

Oh, (strictly self-censored) - the questions discussed are really too complex.
IMHO: Needed a small tutorial for launching self-made python scripts (or other linux apps) in jails. My task, I was thinking to create VM, but it’s not the best from resources viewpoint. Docker container with contains Ubuntu, AFAIK. Jail looks the best, but afraid about apps, 5 ones are working now.
So, needed step-by-step tiutorial of

  1. Creating jail - formally exists in Documentation, but needed more - which network ports are safe 2 use with apps, etc, mine is listening in 3000-ths. Adding more experience from this thread needed.
  2. Update python\linux\other packages in jail
  3. Daemonize app script template ( at least start\stop) and entering it in autostart.

Will be greatly useful indeed.

Hey all, if anyone could help me that’d be great. I’m new to dockge and jails so I’m hitting a bit of a wall when it comes to errors. I have Jellyfin running fine and as far as I can tell the Automatted Ripping Machine (I can at least get to the web GUI with no errors I have not actually tested the system yet for reasons that will be clear in a second)
But when I try to set up MakeMKV and Handbreak neither will create a container. I had them working for a time but after making what seemed like a minor change to where they put their finished files I could no longer access them. I undid the changes but now I’m getting this error message on Dockge:

makemkv-makemkv-1  | [cont-init   ] 10-xdg-runtime-dir.sh: executing...
makemkv-makemkv-1  | [cont-init   ] 10-xdg-runtime-dir.sh: mkdir: can't create directory '': No such file or directory
makemkv-makemkv-1  | [cont-init   ] 10-xdg-runtime-dir.sh: terminated with error 1.
makemkv-makemkv-1 exited with code 1

I’ve tried everything short of redoing my entire docker jail and I’m totally lost. I’ve tried entirely new compose files even deleting the old data from my docker storage pool in TrueNas. If anyone knows what this issue is and how I might fix it that’d be great.

I’m new to jailmaker and followed the youtube and web instructions to get it stood up. Able to get dockge and Plex up and running, but having trouble with hardware encoding with an Intel processor and Quick Sync. I’ve set in the jail:

gpu_passthrough_intel=1

and I see that the start of the docker jail shows:

--bind=/dev/dri

But I can’t get Plex to do any hardware transcoding. I was finally able to get the dropdown in Plex to at least show the hardware transcoding device if i added the following in the compose.yaml in dockge:

volumes:
    - /dev/dri:/dev/dri

But it still won’t actually use the device. What am I doing wrong??

Make sure the above is in your Jail config.

In addition of the the following in your docker compose:

volumes:
   - /dev/dri:/dev/dri

some people I found have sucess in adding instead this:

devices:
   - /dev/:/dev/

You might have to add to your docker compose this section

    deploy:
      resources:
        reservations:
          devices:
            - capabilities:
                - gpu

Also not sure about the additional environment variable that Plex might need for the coresponding NVIDIA ones. (could not find anything in my search.)

I was trying to do some search on intel passthrough, could not find something that are a bit up to date, everything I found was about 2 to 3 years old and not really helpful.

Since I don’t have an intel GPU in my system, can’t really help you further in testing the configuration.

Also did you test the video trancoding in you Plex by playing something not in the native playback ?

Found this post that might be a start:
How to Setup Hardware-Accelerated Streaming in Plex with Intel Quick Sync on Unraid | Chris Sandvik (csandvik.com)

Thanks for the advice. I tried your devices line, but that threw an error and wouldn’t start the plex instance. BUT, I put in:

devices:
   - /dev/dri:/dev/dri

and that did work! So I have that same line in both volumes and devices (not sure if I need both, but it’s working).

EDIT:
Just for completeness, putting in my entire compose.yaml here:

services:
  plex:
    image: lscr.io/linuxserver/plex:latest
    container_name: plex
    network_mode: host
    environment:
      - PUID=3000
      - PGID=3000
      - TZ=America/Denver
      - VERSION=docker
    volumes:
      - /mnt/data/plex/config:/config
      - /mnt/data/plex/transcodes:/transcodes
      - /mnt/Media:/Media
      - /dev/dri:/dev/dri
    restart: unless-stopped
    devices:
      - /dev/dri:/dev/dri
networks: {}
1 Like