QNAP TS-877 Truenas Journal

TrueNAS General
301

restarted truenas

jlmkr list
NAME   RUNNING STARTUP GPU_INTEL GPU_NVIDIA OS     VERSION ADDRESSES
docker False   True    False     True       debian 12      -        
root@xxxxx[/mnt/xxxxx/jailmaker]# jlmkr status docker
Unit jlmkr-docker.service could not be found.
root@xxxxx[/mnt/xxxxx/jailmaker]#

jlmkr restart docker
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
Hit:4 https://download.docker.com/linux/debian bookworm InRelease
Reading package lists... Done                        
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
ca-certificates is already the newest version (20230311).
curl is already the newest version (7.88.1-10+deb12u5).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
Hit:3 https://download.docker.com/linux/debian bookworm InRelease
Hit:4 http://deb.debian.org/debian-security bookworm-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
docker-ce is already the newest version (5:26.1.0-1~debian.12~bookworm).
docker-ce-cli is already the newest version (5:26.1.0-1~debian.12~bookworm).
containerd.io is already the newest version (1.6.31-1).
docker-buildx-plugin is already the newest version (0.14.0-1~debian.12~bookworm).
docker-compose-plugin is already the newest version (2.26.1-1~debian.12~bookworm).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Skipped mounting /dev/nvidia-modeset, it doesn't exist on the host...

Starting jail docker with the following command:

systemd-run --property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Delegate=yes --property=TasksMax=infinity --collect --setenv=SYSTEMD_NSPAWN_LOCK=0 --unit=jlmkr-docker --working-directory=./jails/docker '--description=My nspawn jail docker [created with jailmaker]' --property=ExecStartPre=/mnt/xxxxx/jailmaker/jails/docker/.ExecStartPre -- systemd-nspawn --keep-unit --quiet --boot --bind-ro=/sys/module --inaccessible=/sys/module/apparmor --machine=docker --directory=rootfs --bind=/dev/nvidia-uvm --bind=/dev/nvidiactl --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-ptxjitcompiler.so.545.23.08 --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libnvcuvid.so.545.23.08 --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libcuda.so.545.23.08 --bind-ro=/usr/bin/nvidia-smi --bind-ro=/usr/lib/nvidia/current/nvidia-smi --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-encode.so.545.23.08 --bind=/dev/nvidia0 --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-ml.so.545.23.08 --bind-ro=/usr/bin/nvidia-persistenced --bind=/dev/nvidia-uvm-tools --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-cfg.so.545.23.08 --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-nvvm.so.545.23.08 --network-bridge=br1 --resolv-conf=bind-host '--system-call-filter=add_key keyctl bpf' --bind-ro=/mnt/xxxxx/Storage:/mnt/Storage --bind-ro=/mnt/xxxxx/Storage2:/mnt/Storage2 --bind=/mnt/xxxxx/jailmaker/docker/:/mnt/docker --bind=/mnt/xxxxx/docker/data/:/mnt/data --bind=/mnt/xxxxx/docker/compose/:/mnt/compose

Job for jlmkr-docker.service failed.
See "systemctl status jlmkr-docker.service" and "journalctl -xeu jlmkr-docker.service" for details.

Failed to start jail docker...
In case of a config error, you may fix it with:
jlmkr edit docker

root@xxxxx[/mnt/xxxxx/jailmaker]#

root@xxxx[/mnt/xxxx/jailmaker]# jlmkr remove 
usage: jlmkr.py remove [-h] jail_name
jlmkr.py remove: error: the following arguments are required: jail_name
root@xxxx[/mnt/xxxx/jailmaker]# jlmkr remove docker

CAUTION: Type "docker" to confirm jail deletion!

docker


Cleaning up: jails/docker.
Removing ZFS Dataset xxxx/jailmaker/jails/docker

hm nope still the same i tried to reinstall the docker jail

jlmkr create --start --config /mnt/xxxx/docker/config docker

Job for jlmkr-docker.service failed.
See "systemctl status jlmkr-docker.service" and "journalctl -xeu jlmkr-docker.service" for details.

Failed to start jail docker...
In case of a config error, you may fix it with:
jlmkr edit docker

jail is made in

/mnt/xxxxx/jailmaker

docker install script config in /mnt/xxxxx/docker

this new issue started when i tried to nuke the dataset. last i saw was a warning about an active service which i didnā€™t shutdown before i deleted.

rebooted, redid reinstall from scratch, but now iā€™m stuck x-x;

302

do you have ACLs set on the jailmaker and children datasets? I think you mentioned doing thatā€¦ you probably donā€™t want that

303

it was behaving weirdly.

so i deleted the datasets, remade them and added the presets for smb permissions that just worked.

anyway ill try strip acl.

I was trying to reach my docker folder still on the jailmaker location to copy it to the new dataset outside the jailmaker like you suggested, but i couldnā€™t copy over. so i thought it was permission issue.

anyway iā€™ll just grab them from my backup later, no problem.

i also deleted the ixsystem dataset. to my understanding truena creates that and apps whenever you try to use the truenas apps to configure something.

does jailmaker need the stuff in ixsystem dataset? :thinking:

304

also the other thing i did differently, i placed the docker config in the docker dataset that resides outside the jailmaker.

305

was stuck so tried something different.

was trying to setup a user called docker

useradd docker

usermod -aG sudo docker

su -l docker

deluser docker

the above all worked, but this didnā€™t work
passwd USERNAME

so because i couldnā€™t set a password because it kept spitting out

Authentication token manipulation error when trying to ..

so gave up on that. will figure it out later

I was following jailmakerā€™s guide

306

setting up snapshots.

this video is probably the best at explaining it

the gist is, set an auto (scheduled snapshot task) snapshot daily retain 1 week, with recursive enabled.

If you did manually, you may run into the issue of snapshots using space more and more over time and you didnā€™t realize.

Of course your requirements may be different and need to change accordingly. But for most people this is the recommendation.

This however is merely snapshots within the same NAS.

But there is snapshot replication to a remote nas. Explained in the video below

From what iā€™m told, this is a good backup solution especially when paired with another truenas acting as the backup storing the remote replication snapshots. Looks promising and worth looking into.

307

Managed to fix docker in jailmaker

apparently i had to change graphics card to 0 for nvidia, and i had to create the dataset in my mount bind was was not yet created

Now trying to figure out how to get to the bind mounts via jailmaker shell docker

ls
initial_startup
root@docker:~# cd /mnt/docker
root@docker:/mnt/docker#
1 Like
308

Ok so successfully

  1. figured out how to navigate to the dataset located OUTSIDE the jailmaker dataset

  2. deploy docker compose


root@docker:~# ls
initial_startup
root@docker:~# cd /mnt/docker
root@docker:/mnt/docker# dir
compose  config  data
root@docker:/mnt/docker# cd compose/whoami
root@docker:/mnt/docker/compose/whoami# docker compose up d
no configuration file provided: not found
root@docker:/mnt/docker/compose/whoami# docker compose up
no configuration file provided: not found
root@docker:/mnt/docker/compose/whoami# docker compose up clear
WARN[0000] /mnt/docker/compose/whoami/docker-compose.yaml: `version` is obsolete 
no such service: clear
root@docker:/mnt/docker/compose/whoami# clr
bash: clr: command not found
root@docker:/mnt/docker/compose/whoami# docker compose up
[+] Running 4/4
 āœ” whoami Pulled                                                                                                                                                                                         13.0s 
   āœ” 9a3f489abe7a Pull complete                                                                                                                                                                           1.6s 
   āœ” 442cc7f997f8 Pull complete                                                                                                                                                                           1.5s 
   āœ” 52416ed98bd1 Pull complete                                                                                                                                                                           1.3s 
[+] Running 1/2
 āœ” Network whoami_default     Created                                                                                                                                                                     0.2s 
 ā ™ Container whoami-whoami-1  Created                                                                                                                                                                     0.1s 
Attaching to whoami-1
whoami-1  | 2024/04/24 xxxxxx Starting up on port 2001

i thought ls would work but it didnā€™t.

what i needed to do was cd to change directory to the /mnt/docker which i recall assigning in the jailmaker docker script config beforehand.

once done, then i can type dir to list directories

then i browsed to the whoami directory where i placed a docker-compose.yaml

version: '3.9'

services:
  whoami:
    image: traefik/whoami
    command:
       # It tells whoami to start listening on 2001 instead of 80
       - --port=2001
       - --name=iamfoo

then i tried

docker-compose up

this doesnā€™t work. it has been changed to docker compose up

so i tried again, still didnā€™t work. why?

Because now you had to remove the version line inside the yaml file since that has been made obsolete.

so went into the yaml and made it like this

services:
  whoami:
    image: traefik/whoami
    command:
       # It tells whoami to start listening on 2001 instead of 80
       - --port=2001
       - --name=iamfoo

docker compose up

now it worked.

So now i need to do that for portainer and itā€™s lift off :smiling_face_with_three_hearts:

309

This is how networking is setup for the docker

785Ɨ568 35.4 KB

is set in bridge mode to br1. The first one is the physical link itā€™s bound io.

The vb-docker no idea how that appeared xd. assume itā€™s jailmaker related.

so after compose up for whoami, it says itā€™s active on portā€¦

so what is the docker ip?

192.168.0.210 ?

no. because when i used android app ā€œnet analyzerā€ to scan network and i saw a 192.168.0.24 named docker.

i assume jailmaker did this? or something

so i tried 192.168.0.24:2001

but didnā€™t work. scanner says no website.

so iā€™ll try another app to test.

310

Ok i deployed openspeedtest

2287Ɨ1176 93.7 KB

https://hub.docker.com/r/openspeedtest/latest

services:
    speedtest:
        restart: unless-stopped
        container_name: openspeedtest
        ports:
            - '3000:3000'
            - '3001:3001'
        image: openspeedtest/latest
1 Like
311

In conclussion, when moving from QNAP QTS to truenas, and you want to setup docker containers.

Either, do so via VM (which is the officially supported method), OR, do what i did. Use jailmaker GitHub - Jip-Hop/jailmaker: Persistent Linux 'jails' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts thanks to systemd-nspawn!

Iā€™ve detailed all the steps i did and what issues i ran into, and finally how i managed to deploy a working docker container.

Later i will deploy portainer, then deploy all my docker containers through that.

So that concludes my docker container walkthrough (if you can call it that).

The next big project is getting traefik to work. For truenas, they suggested setting a static ip on truenas, thus freeing ports 80 and 443 to work for traefik to work with docker containers.

will have to see how that goes.

Why do i want traefik?

Itā€™s a reverse proxy.

Typicall you use it if you want to expose your nas online, so to limit exposure you slot in a reverse proxy such as nginx proxy manager or traefik between your NAS and the router, and the internet.

So you are exposing FEW ports, rather than many. Then you can even add authentication layer thus further locking down access through few gaps.

I however am using it for a local homelab setup. meaning, i donā€™t expose my nas online at all since i donā€™t remote access. instead, reverse proxy is restricted to lan only network. So why bother?

Because traefik can handle all the ports. So i can setup a local domain url, without having to append any ports at the back each time to access app service web urls. Just the main domain url is sufficient to access the service.

Also makes accessing pages snappier.

313

500Ɨ624 54.2 KB

HOW TO INSTALL JAILMAKER FOR DOCKER CONTAINERS

if i were to try summarize

setup the networking first. this video teaches how to setup bridges

then follow this guide (donā€™t deviate)

then this guide (learn the jlmkr commands)

Quick crash course for jlmkr

  1. do not bind mount to the jailmaker dataset or any child residing in it. bind mount is only meant for external datasets residing outside the jailmaker dataset.

e.g.

ā€“bind=ā€˜/mnt/tank/docker/:/mnt/dockerā€™
ā€“bind=ā€˜/mnt/tank/docker/data/:/mnt/dataā€™
ā€“bind=ā€˜/mnt/tank//docker/compose/:/mnt/composeā€™
ā€“bind-ro=ā€˜/mnt/tank2/Storage:/mnt/Storageā€™
ā€“bind-ro=ā€˜/mnt/tank2/Storage2:/mnt/Storage2ā€™

  1. after installing jailmaker, the next step is to install the docker using the docker script by jailmaker. modify the config before deploying it.

  2. you have to downgrade docker to a working version

Apparently, docker 26.0.2 broke things. Downgrading to 26.0.1 fixes things.

To downgrade do this
jlmkr shell docker

apt-get install docker-ce=5:26.0.1-1~debian.12~bookworm docker-ce-cli=5:26.0.1-1~debian.12~bookworm containerd.io docker-buildx-plugin docker-compose-plugin

  1. once docker is deployed, you are wondering why you canā€™t access. that is because you have to start the docker first e.g. jlmkr start docker after which you can then do jlmkr shell docker at this point you can then run docker commands e.g. docker version

  2. to change directory depends on the bind mounts you set earlier. using my example from before

go to the docker shell through jlmkr, then do cd /mnt/compose/portainer this is where i store my docker-compose.yaml files to deploy the docker container. to deploy, type docker compose up -d The old method was docker-compose up -d but this has been deprecated so do it the new way. Another thing that has changed, within the docker compose yaml, the version line, remove it because that has been deprecated to in the newer docker versions.

  1. after deploying my docker container, i used my android app ā€œNet Analyzerā€ did a network scan and found my docker ip. Just append at the back :port for the app

notes: when upgrading truenas, you may need to run the jailmkaker script to make it work again. Donā€™t worry itā€™s easy

./jlmkr.py install

Setting up the startup for jailmaker is important. donā€™t forget to set that up.
GitHub - Jip-Hop/jailmaker: Persistent Linux 'jails' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts thanks to systemd-nspawn!

then finally my guide if you get stuck. Itā€™s more of a journal than a proper guide but u can see what i did, what issues i encountered, how i resolved it

i also have a issue ticket where i covered the issues i faced and got some help there as well

took me a few days to go through the guides and test to learn what it does and how to use it. itā€™s doable and iā€™m not even that technical a user but i could figure it out. you just need to be able to google and ask questions.

note: more clarification on docker bind mounts, how that works for jail and docker.

QNAP TS-877 Truenas Journal - #335 by Stux

314

some additional explanations about how these bind mounts are used based on my example.

i already explained the /mnt/compose is for deploying dockers.

/mnt/data is where we will store our permanent configs. example, in say jellyfin docker compose yaml, i can set it to save itā€™s permanent config in data e.g. /mnt/data/jellyfinā€¦ (the structure within will be dependent on the container setup, refer to the container project website for that app. usually they have docker compose examples)

then the /mnt/tank/ these are where my media is stored. so for docker containers like jellyfin, i can simply specify that the media dataset is located at /mnt/tank/ *at this point it will let you browse the sub folders within that location

Notice you can either set bind mounts to read only, or read & write. Change it to what you require. Some containers require ro access, so be mindful of your requirements for this.

well this is just an example of what i use these bind mounts for. You milleage may vary.

315

docker compose up -d

d option is to demonize (ie run in background)

1 Like
316

jlmkr list will list your jails and IPs.

1 Like
317

yeah i thought that was the case, but when i did it like that it still ran. tired xd

318

good tip. but i the ip was cut off at the right so i couldnā€™t fully see it

319


Lockszmith-GH2 hours ago

It looks like debian already pushed 26.1.0 fixing the issue with 26.0.2

root@docker:~# apt-cache policy docker-ce | head
docker-ce:
Installed: 5:26.1.0-1~debian.12~bookworm
Candidate: 5:26.1.0-1~debian.12~bookworm
Version table:
*** 5:26.1.0-1~debian.12~bookworm 500
500 Index of linux/debian/ bookworm/stable amd64 Packages
100 /var/lib/dpkg/status
5:26.0.2-1~debian.12~bookworm 500
500 Index of linux/debian/ bookworm/stable amd64 Packages
5:26.0.1-1~debian.12~bookworm 500

320

see my screenshot, this is what i see. the ip gets cut off x-x; how to get around that xd

763Ɨ131 14.2 KB

321

portainer works

1217Ɨ924 43.9 KB

docker compose up -d

docker-compose.yaml for portainer

services:
  portainer:
    image: portainer/portainer-ce:latest
    container_name: portainer

    environment:
      PUID: 0
      PGID: 0
      TZ: xxxxxxxxxxx

    ports:
      - 8000:8000
      - 9443:9443
      - 9000:9000 # for http
    volumes:
      - /mnt/xxxxxxxxx/docker/data/portainer/data:/data
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped

any tips for the user? can set the container to setup using a non root user? how to go about this?

when i tried to create a docker user through shell i couldnā€™t set a password so i got stuck there x-x;