i could probably nuke it and redo from scratch to get it working again,
but i’m trying to tinker to first figure out how to recover jailmaker after an upgrade of truenas. best find out now before actually deploying for real 
here’s mine. my jail is also called “docker”
This is from a slightly older release of Jailmaker. I’m working on upgrading to current today (involves switching to datasets etc)
startup=1
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Use macvlan networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-bridge=br1 instead of --network-macvlan
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=--network-bridge=br0
--resolv-conf=bind-host
--system-call-filter='add_key keyctl bpf'
--bind-ro='/mnt/tank/WebRoot/Sites/XXXX.com:/mnt/XXXweb'
--bind='/mnt/tank/docker/data:/mnt/data'
--bind='/mnt/tank/docker/compose:/compose'
--bind='/mnt/tank/docker/mirror:/mnt/docker-mirror'
--bind='/mnt/tank/docker/jellyfin:/mnt/jellyfin'
--bind='/mnt/tank/media:/mnt/media'
# Script to run on the HOST before starting the jail
# Load kernel module and config kernel settings required for docker
pre_start_hook=#!/usr/bin/bash
set -euo pipefail
echo 'PRE_START_HOOK'
echo 1 > /proc/sys/net/ipv4/ip_forward
modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
# Only used while creating the jail
distro=debian
release=bookworm
# Install docker inside the jail:
# https://docs.docker.com/engine/install/debian/#install-using-the-repository
# NOTE: this script will run in the host networking namespace and ignores
# all systemd_nspawn_user_args such as bind mounts
initial_setup=#!/usr/bin/bash
set -euo pipefail
apt-get update && apt-get -y install ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
apt-get install -y nano openssh-server bash-completion
curl https://raw.githubusercontent.com/docker/docker-ce/master/components/cli/contrib/completion/bash/docker -o /etc/bash_completion.d/docker.sh
# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed
--property=Type=notify
--property=RestartForceExitStatus=133
--property=SuccessExitStatus=133
--property=Delegate=yes
--property=TasksMax=infinity
--collect
--setenv=SYSTEMD_NSPAWN_LOCK=0
systemd_nspawn_default_args=--keep-unit
--quiet
--boot
--bind-ro=/sys/module
--inaccessible=/sys/module/apparmor
I install a few extra things into the jail… openssh, nano, compose plugin etc
I keep most of my docker data in tank/docker/data, ie I have directories in that dataset for each application. I then keep a directory in the compose dataset for each application… and in the directory I write a compose file etc for each applicaation. Just how I do it.
having the single mount for data means I don’t need to keep restarting the jail to add a new dataset.
Jellyfin is big enough that I have a separate dataset for it and all its caches etc…
and then my media is outside that docker hierarchy.
1 Like
Just an overall recap of the current status of moving from QNAP QTS to truenas.
- switch QTS to truenas - completed
- Fan fix. have to go bios to set smartfans to manual - completed.
- lcd fix. post init and shutdown script not working - incomplete
- restore shares to truenas datasets. did this using rsync using qnap hybrid backup sync to truenas dataset - complete
- set scheduled smart tests for SSD & hdds. weekly for short, monthly for long - complete
- run raid scrub and also set a monthly schedule run - complete
- setup ACLS - complete
- enable smb and nfs. Can access the datasets via windows 11 file explorer using smb - complete
- setup docker using jailmaker - incomplete
- test kubernetes app deploy. tested for unifi controller it worked. but i removed pool for app because don’t plan on using it since i am trying to use jailmaker for docker instead - complete
- configured bridging for networking - complete
- setup sfp+ 10g fiber - complete
- change hostname, time zone and miscelanous settings. Don’t think i missed anything - complete
- checked reporting, yes it works great. Even has netdata which is cool. It even saved me by notifying i had a temperature issue, which i then noticed my fans weren’t working so i managed to fix that thx to truenas alerts and checking the monitoring tools like netdata to troubleshoot - complete
- setup users - complete
- setup acls - complete
- snapshots - incomplete
I think that more or less covers most of what you need for a NAS. So basically more or less a success. Only major thing to solve is the docker situation. Others with better technical skills have already done so. So i just have to keep at it
If you are not planning on running any docker containers, then your move to truenas will be much easier.
truenas uses kubernetes k3’s listed under the apps section in the UI. You just select the app from the catalog, do the minor adjustments in settings and deploy.
But even if you wanted docker, seems the VM route is the easiest to do. Jailmaker is a bit more complicated but is probably a bit doable once more complete guides that account for everything come around. hopefully my documentation as i go along will help somewhat with that.
A picture is worth a thousand words - confucius
1 Like
And that is your headings for that future guide 
1 Like
why is data and compose separate?
i assume data has the docker deployed configs. but compose has the compose file? is that correct?
but if you run the compose won’t it deploy where you ran it?
Example mine is like
docker/portainer/docker compose.yaml
so i would browse to that portainer directory, then do a docker compose up. It detects that yaml and runs it, then deploys within that same location.
That’s my only understanding of that 
and usually i have both a docker compose.yaml, and also i kept a copy within portainer saved as a custom deploy script for quick re-deploy.
portainer doesn’t like you deploying docker compose yamls. instead they want you to deploy docker compose within the portainer UI so it can manage it. So i keep the docker compose yaml as a backup, then copy the content to portainer, then deploy from portainer. So that’s how i manage that.
portainer also supports .env so you don’t need a .env file. just copy the values into portainer and it will deploy them fine.
I don’t use portainer. I find it too fragile.
Instead I create a compose file for each service I want to run.
my compose datasets contains a directory for each service, and each directory has a “docker-compose.yml” file in it, and anything else.
I bring up a service by cd-ing into a directory and typing “docker compose up -d”, etc.
(or using dockge)
As such, the compose directories I want to be able to snapshot/rollback/backup etc on separate scehdule, and to different locatations, vs the data directory, which contains all the data sub-directories that I host mount into containers/dockers etc.
I used to have a dataset per docker container… but then I had to keep on restarting the jail everytime I wanted to add a new service (ie a docker composition)
1 Like
Maybe this will make a bit more sense
root@chronus[/mnt/tank/docker/compose]# cd jellyfin
root@chronus[/mnt/tank/docker/compose/jellyfin]# ll
total 26
drwxr-xr-x 2 root 4 Mar 25 15:44 ./
drwxr-xr-x 10 root 11 Mar 25 15:59 ../
-rw-r--r-- 1 root 17 Mar 25 15:13 .env
-rw-r--r-- 1 root 707 Mar 25 15:44 docker-compose.yml
root@chronus[/mnt/tank/docker/compose/jellyfin]# cat docker-compose.yml
services:
jellyfin:
image: linuxserver/jellyfin:latest
container_name: jellyfin
network_mode: host
environment:
- PUID=1005
- PGID=1007
- TZ=Etc/UTC
- JELLYFIN_PublishedServerUrl=https://jellyfin.XXXX.com
volumes:
- /mnt/jellyfin/config:/config
- /mnt/jellyfin/cache:/config/cache
- /mnt/jellyfin/transcodes:/config/data/transcodes
- /mnt/media:/data
# - /path/to/tvseries:/data/tvseries
# - /path/to/movies:/data/movies
# ports:
# - 8096:8096
# - 8920:8920 #optional
# - 7359:7359/udp #optional
# - 1900:1900/udp #optional
restart: unless-stopped
networks: {}
root@chronus[/mnt/tank/docker/compose/jellyfin]#
1 Like
Note, the PUID/PGID refer to the actual jellyfin user on my NAS.
root@chronus[/mnt/tank/docker/compose/jellyfin]# cat /etc/passwd | g 1005
jellyfin:x:1005:1007:jellyfin:/nonexistent:/usr/sbin/nologin
1 Like
sorry back. was busy making memes 
1 Like
yeah your compose dataset is pretty much what i am doing as well.
but when the docker deploys it dumps its config, data everything into that same location. But i see you are pointing yours to a different directory under data.
so compose/ directory for each service.
then for data/ also directory for each service?
is that how it is?
I just assumed that the compose had to be where the docker data stuff get generated into, guess not. Maybe i should do that too. Then i can keep things clean to separate out the docker compose yamls for deploying, and the data configs the apps generate out and use.
yeah i think i need to do this also.
setup a user in the jailmaker, maybe also give root using the sudo command
useradd USERNAME
passwd USERNAME
If you want the ability to run commands as root, add the user to the sudo group usermod -aG sudo USERNAME
then to find puid pgid
not sure if we also have to go truenas later and create that user there using the same puid and pgid or not as well
and the other thing is whether to change owner from root to this new user or not? but if it has root powers, then no need right?
i thought you said the docker needs to be a slightly older one that works because the latest docker has issues with jailmaker. but your config says stable, did you forget to change there?
i’m not sure how to downgrade, so i simply intend to nuke/delete the jailmaker and start from scratch. Basically just run the jailmaker docker script again (this script has already been modified by me)
though i’m still not too sure about the pathing
correct?
/mnt/xxxxx/jailmaker/jails/docker/rootfs/docker
possible new pathing to improve it to?
/mnt/xxxxx/jailmaker/jails/docker/rootfs/mnt/docker
this not needed?
/mnt/xxxxx/jailmaker/jails/docker/
Yes. I keep the data and compose separate per composition
Makes it easy to move a service, say into a vm or whatever
1 Like
I just haven’t updated to the broken version yet
1 Like
No. That doesn’t look right
Rootfs is your jails file system on your pool. The jail will have its / directory (ie root) set to point to that directory.
You should try to keep the jail as thin and disposable as possible… which is why I mount the docker config and data into the jail.
The config and data should be stored on your pool outside of the jail, and the mounted in using a bind argument in the jail config
1 Like
I’ll try redo it and report back my results
how do you set this to 26.0.1?
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
https://download.docker.com/linux/debian/dists/bookworm/
was trying to browse here
hm
so
/mnt/xxxxx/jailmaker/jails/docker/rootfs/
/mnt/xxxxx/docker/
*data and compose directory go here
When creating a jail, an entire Linux filesystem is created in the ‘rootfs’ folder within the jail’s folder of the jailmaker directory E.g
/mnt/tank/vault/jailmaker/jails/jailname/rootfs .No files from the TrueNAS host will be available.
this is what i came up with
--bind-ro='/mnt/xxxxx2/Storage:/mnt/Storage'
--bind-ro='/mnt/xxxxx2/Storage2:/mnt/Storage2'
--bind='/mnt/xxxxx/jailmaker/jails/rootfs/docker/:/mnt/docker'
--bind='/mnt/xxxxx/docker/data/:/mnt/data'
--bind='/mnt/xxxxx/docker/data/:/mnt/compose'
then the docker jailmaker script is in /mnt/xxxxx/docker/jailmaker/docker
jlmkr create --start --config /mnt/xxxxx/jailmaker/config docker
What’s this bit though.
that means you’re mounting the “/docker” directory in the jail to “/mnt/docker” in the jail.
Which is a bit weird.
What’s in this docker directory?
And if its in your jailmaker/jails directory, wouldn’t it be inside a jailname directory… ie docker?
‘/mnt/xxxxx/jailmaker/jails/docker/rootfs/docker/’
What’s in that directory?
I would suggest moving the contents somewhere else 
, using either mv, or rsync…
Then
--bind='/mnt/xxxxx/docker:/mnt/docker'
In my jail I decided to mount most things in the /mnt directory, mainly to keep the root clean, and was similar to how truenas does it anyway
in the vm pool i nuked the ixsystem, app, jailmaker and docker datasets. it said a service running, but still allowed me to delete. i don’t know how to stop the service first
anyway recreated dataset and added the acl presets
oo.ooo
Job for jlmkr-docker.service failed.
See "systemctl status jlmkr-docker.service" and "journalctl -xeu jlmkr-docker.service" for details.
Failed to start jail docker...
In case of a config error, you may fix it with:
jlmkr edit docker
root@xxxxx[/mnt/xxxxx/jailmaker]#
Apr 24 xxxxxsystemd[1]: Starting jlmkr-docker.service - My nspawn jail docker [created with jailmaker]...
░░ Subject: A start job for unit jlmkr-docker.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit jlmkr-docker.service has begun execution.
░░
░░ The job identifier is 2775.
Apr 24 xxxxx .ExecStartPre[386283]: PRE_START_HOOK
Apr 24 xxxxx systemd-nspawn[386286]: Failed to stat /mnt/xxxxx/docker/compose/: No such file or directory
Apr 24 xxxxx systemd[1]: jlmkr-docker.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit jlmkr-docker.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 24 xxxxx systemd[1]: jlmkr-docker.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit jlmkr-docker.service has entered the 'failed' state with result 'exit-code'.
Apr 24 xxxxx xxxxx systemd[1]: Failed to start jlmkr-docker.service - My nspawn jail docker [created with jailmaker].
░░ Subject: A start job for unit jlmkr-docker.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit jlmkr-docker.service has finished with a failure.
░░
░░ The job identifier is 2775 and the job result is failed.
root@xxxxx[/mnt/xxxxx/jailmaker]#