NOTICE: Modified post and steps below for the fully automated config-ee file. All you need to do now is
jlmkr create --start -c config-ee <jail-name>and it will be started and ready to go with docker and nvidia gpu support.
ALERT: Keep in mind, this is ONLY for Nvidia GPU based jails. This doesn’t apply for standard jails. For standard jails, you can continue to use the template config. This is ONLY for Electric Eel!
DISCLAIMER: With the GPU you will lose the ability to install packages inside the jail… to patch or update the jail, you’ll have to disable the GPU and comment out all the extra mounts in the config file to do so at this point. I also have not tested this with TrueNAS’s Apps running, so I have no idea at this point if they will conflict, it shouldn’t, but you never know.
REQUIRED: Make sure you are on the latest version of the
jlmkr.pyscript, otherwise this likely won’t work.
- Install the Nvidia drivers in EE. It can be done by going to:
Apps > Configuration > Settings > Tick Install NVIDIA Drivers. Click Save and reboot your TrueNAS server when it’s finished installing the drivers. - Create and start jail with custom config. Copy, paste, edit, and save the config code below, then run the following command:
jlmkr create --start -c config-ee <jail-name>
Custom config: config-ee:
NOTE: Make sure to set your network interface and data mounts to the datasets on your hosts that you use for your applications… I left in some example ones you can modify. If you don’t modify these to match your system, the jail will fail to start. TAKE HEED!
startup=1
gpu_passthrough_intel=0
gpu_passthrough_nvidia=1
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1
# Use macvlan networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-macvlan=eno1 instead of --network-bridge
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
#
# When using different network adapters, update the line above the following:
# --system-call-filter='add_key keyctl bpf'
# otherwise there will be wierd docker, dns, interface, and storage issues.
#
systemd_nspawn_user_args=--network-bridge=br0
--system-call-filter='add_key keyctl bpf'
--bind-ro='/dev/dri:/dev/dri'
--bind-ro='/etc/ssl:/etc/ssl'
--bind-ro='/usr/bin/containerd:/usr/bin/containerd'
--bind-ro='/usr/bin/containerd-shim:/usr/bin/containerd-shim'
--bind-ro='/usr/bin/containerd-shim-runc-v1:/usr/bin/containerd-shim-runc-v1'
--bind-ro='/usr/bin/containerd-shim-runc-v2:/usr/bin/containerd-shim-runc-v2'
--bind-ro='/usr/bin/curl:/usr/bin/curl'
--bind-ro='/usr/bin/docker:/usr/bin/docker'
--bind-ro='/usr/bin/docker-proxy:/usr/bin/docker-proxy'
--bind-ro='/usr/bin/dockerd:/usr/bin/dockerd'
--bind-ro='/usr/bin/dockerd-rootless-setuptool.sh:/usr/bin/dockerd-rootless-setuptool.sh'
--bind-ro='/usr/bin/dockerd-rootless.sh:/usr/bin/dockerd-rootless.sh'
--bind-ro='/usr/bin/nvidia-container-cli:/usr/bin/nvidia-container-cli'
--bind-ro='/usr/bin/nvidia-container-runtime:/usr/bin/nvidia-container-runtime'
--bind-ro='/usr/bin/nvidia-container-runtime-hook:/usr/bin/nvidia-container-runtime-hook'
--bind-ro='/usr/bin/nvidia-container-toolkit:/usr/bin/nvidia-container-toolkit'
--bind-ro='/usr/bin/nvidia-ctk:/usr/bin/nvidia-ctk'
--bind-ro='/usr/bin/runc:/usr/bin/runc'
--bind-ro='/usr/sbin/iptables:/usr/sbin/iptables'
--bind-ro='/usr/libexec/docker:/usr/libexec/docker'
--bind='/mnt/tank/media/:/mnt/media'
--bind='/mnt/tank/stacks:/opt/stacks'
# Script to run on the HOST before starting the jail
# Load kernel module and config kernel settings required for docker
pre_start_hook=#!/usr/bin/bash
set -euo pipefail
echo 'PRE_START_HOOK'
echo 1 > /proc/sys/net/ipv4/ip_forward
modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
# Only used while creating the jail
distro=debian
release=bookworm
# Configure docker inside the jail:
# https://docs.docker.com/engine/install/debian/#install-using-the-repository
# Will also configure the NVIDIA Container Toolkit if gpu_passthrough_nvidia=1 during initial setup
# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
initial_setup=#!/usr/bin/bash
set -euo pipefail
# The /usr/bin/nvidia-smi will be present when gpu_passthrough_nvidia=1
if [ -f /usr/bin/nvidia-smi ]; then
# Create docker group
groupadd docker
# Create docker.service unit file
cat <<EOF >> /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service containerd.service time-set.target
Wants=network-online.target containerd.service
Requires=docker.socket
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP \$MAINPID
TimeoutStartSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
EOF
# Create docker.socket unit file
cat <<EOF >> /lib/systemd/system/docker.socket
[Unit]
Description=Docker Socket for the API
[Socket]
# If /var/run is not implemented as a symlink to /run, you may need to
# specify ListenStream=/var/run/docker.sock instead.
ListenStream=/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
EOF
# Create containerd.service unit
cat <<EOF >> /lib/systemd/system/containerd.service
# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target
[Service]
#uncomment to enable the experimental sbservice (sandboxed) version of containerd/cri integration
#Environment="ENABLE_CRI_SANDBOXES=sandboxed"
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
EOF
# Enable Docker
systemctl enable docker.service
systemctl enable docker.socket
systemctl enable containerd.service
# Configure nvidia runtime
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker
fi
docker info
# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed
--property=Type=notify
--property=RestartForceExitStatus=133
--property=SuccessExitStatus=133
--property=Delegate=yes
--property=TasksMax=infinity
--collect
--setenv=SYSTEMD_NSPAWN_LOCK=0
systemd_nspawn_default_args=--keep-unit
--quiet
--boot
--bind-ro=/sys/module
--inaccessible=/sys/module/apparmor
- Profit? You should be able to install
Dockage,Plex,Jellyfin, etc at this point.
Output of Plex working in new jail:
nvidia-smi
Thu Oct 31 16:04:22 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.05 Driver Version: 550.127.05 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce GTX 1080 Ti Off | 00000000:2B:00.0 Off | N/A |
| 0% 56C P2 61W / 280W | 191MiB / 11264MiB | 1% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 603809 C ...lib/plexmediaserver/Plex Transcoder 188MiB |
+-----------------------------------------------------------------------------------------+