QNAP TS-877 Truenas Journal

mooglestiltzkin · April 24, 2024, 2:40am

yes i did. but it didn’t work. maybe i need to reboot?

Stux · April 24, 2024, 2:45am

FWIW, my “docker” jail came up perfectly after the update to 24.04 release.

I’m guessing you haven’t quite got the install right. For example. Did you get the patch to the jlmkr script right in the post-init command?

Also, you need to set startup=1 for jail to start at startup when jlmkr starts

mooglestiltzkin · April 24, 2024, 2:57am

well before upgrading truenas, i did get the jailmaker to work kind of.

like, i could access the docker commands like docker version

note: In truenas shell via UI, you had to jlmkr shell docker in order to then begin using docker commands. the docker is simply the jailname i called docker fyi.

but what i couldn’t do at the time was ls (which is list directories). So i wasn’t able to cd (to change directory) to my docker compose.yaml file to docker compose up. Still figuring this out.

So i tried adding the bind mount to the jailmaker docker script config (you basically run this to install the docker for jailmaker). But did not see any improvement.

Now i’m sus it is a permission issue. One thing i didn’t yet do is create a user in jailmaker, which might be what i need to do to move forward on this. But i’m not sure what is the correct approach here.

doc says you create a user, then if they need root, you can do a sudo command to give them such permissions.

can i create a user called docker then assign that root permissions?

and do i then have to go truenas beforehand to create that user first before i do that? these things aren’t quite explained. They assume the person would know these things, but i don’t since i’m a newbie

Anyway this is what my docker script looks like

startup=0
gpu_passthrough_intel=1
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1

# Use macvlan networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-bridge=br1 instead of --network-macvlan
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=--network-bridge=br1
	--resolv-conf=bind-host
	--system-call-filter='add_key keyctl bpf'
	--bind='/mnt/xxxxxx/jailmaker/docker/:/docker'

# Script to run on the HOST before starting the jail
# Load kernel module and config kernel settings required for docker
pre_start_hook=#!/usr/bin/bash
	set -euo pipefail
	echo 'PRE_START_HOOK'
	echo 1 > /proc/sys/net/ipv4/ip_forward
	modprobe br_netfilter
	echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
	echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

# Only used while creating the jail
distro=debian
release=bookworm

# Install docker inside the jail:
# https://docs.docker.com/engine/install/debian/#install-using-the-repository
# NOTE: this script will run in the host networking namespace and ignores
# all systemd_nspawn_user_args such as bind mounts
initial_setup=#!/usr/bin/bash
	set -euo pipefail
	
	apt-get update && apt-get -y install ca-certificates curl
	install -m 0755 -d /etc/apt/keyrings
	curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
	chmod a+r /etc/apt/keyrings/docker.asc
	
	echo \
	"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
	$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
	tee /etc/apt/sources.list.d/docker.list > /dev/null
	apt-get update
	apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed
	--property=Type=notify
	--property=RestartForceExitStatus=133
	--property=SuccessExitStatus=133
	--property=Delegate=yes
	--property=TasksMax=infinity
	--collect
	--setenv=SYSTEMD_NSPAWN_LOCK=0

systemd_nspawn_default_args=--keep-unit
	--quiet
	--boot
	--bind-ro=/sys/module
	--inaccessible=/sys/module/apparmor

Yes i see what you mean. The config i need to change from 0 to 1.

I may also have to change the docker version to the other you mention which works, since the latest docker broke networking or something for jailmaker?

mooglestiltzkin · April 24, 2024, 3:02am

i could probably nuke it and redo from scratch to get it working again,

but i’m trying to tinker to first figure out how to recover jailmaker after an upgrade of truenas. best find out now before actually deploying for real

Stux · April 24, 2024, 3:13am

here’s mine. my jail is also called “docker”

This is from a slightly older release of Jailmaker. I’m working on upgrading to current today (involves switching to datasets etc)

startup=1
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0

# Use macvlan networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-bridge=br1 instead of --network-macvlan
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=--network-bridge=br0
	--resolv-conf=bind-host
	--system-call-filter='add_key keyctl bpf'
	--bind-ro='/mnt/tank/WebRoot/Sites/XXXX.com:/mnt/XXXweb'
	--bind='/mnt/tank/docker/data:/mnt/data'
	--bind='/mnt/tank/docker/compose:/compose'
	--bind='/mnt/tank/docker/mirror:/mnt/docker-mirror'
	--bind='/mnt/tank/docker/jellyfin:/mnt/jellyfin'
	--bind='/mnt/tank/media:/mnt/media'

# Script to run on the HOST before starting the jail
# Load kernel module and config kernel settings required for docker
pre_start_hook=#!/usr/bin/bash
	set -euo pipefail
	echo 'PRE_START_HOOK'
	echo 1 > /proc/sys/net/ipv4/ip_forward
	modprobe br_netfilter
	echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
	echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

# Only used while creating the jail
distro=debian
release=bookworm

# Install docker inside the jail:
# https://docs.docker.com/engine/install/debian/#install-using-the-repository
# NOTE: this script will run in the host networking namespace and ignores
# all systemd_nspawn_user_args such as bind mounts
initial_setup=#!/usr/bin/bash
	set -euo pipefail
	
	apt-get update && apt-get -y install ca-certificates curl
	install -m 0755 -d /etc/apt/keyrings
	curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
	chmod a+r /etc/apt/keyrings/docker.asc
	
	echo \
	"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
	$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
	tee /etc/apt/sources.list.d/docker.list > /dev/null
	apt-get update
	apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin 
	apt-get install -y nano openssh-server bash-completion
        curl https://raw.githubusercontent.com/docker/docker-ce/master/components/cli/contrib/completion/bash/docker -o /etc/bash_completion.d/docker.sh

# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed
	--property=Type=notify
	--property=RestartForceExitStatus=133
	--property=SuccessExitStatus=133
	--property=Delegate=yes
	--property=TasksMax=infinity
	--collect
	--setenv=SYSTEMD_NSPAWN_LOCK=0

systemd_nspawn_default_args=--keep-unit
	--quiet
	--boot
	--bind-ro=/sys/module
	--inaccessible=/sys/module/apparmor

I install a few extra things into the jail… openssh, nano, compose plugin etc

I keep most of my docker data in tank/docker/data, ie I have directories in that dataset for each application. I then keep a directory in the compose dataset for each application… and in the directory I write a compose file etc for each applicaation. Just how I do it.

having the single mount for data means I don’t need to keep restarting the jail to add a new dataset.
Jellyfin is big enough that I have a separate dataset for it and all its caches etc…
and then my media is outside that docker hierarchy.

mooglestiltzkin · April 24, 2024, 3:13am

Just an overall recap of the current status of moving from QNAP QTS to truenas.

switch QTS to truenas - completed
Fan fix. have to go bios to set smartfans to manual - completed.
lcd fix. post init and shutdown script not working - incomplete
restore shares to truenas datasets. did this using rsync using qnap hybrid backup sync to truenas dataset - complete
set scheduled smart tests for SSD & hdds. weekly for short, monthly for long - complete
run raid scrub and also set a monthly schedule run - complete
setup ACLS - complete
enable smb and nfs. Can access the datasets via windows 11 file explorer using smb - complete
setup docker using jailmaker - incomplete
test kubernetes app deploy. tested for unifi controller it worked. but i removed pool for app because don’t plan on using it since i am trying to use jailmaker for docker instead - complete
configured bridging for networking - complete
setup sfp+ 10g fiber - complete
change hostname, time zone and miscelanous settings. Don’t think i missed anything - complete
checked reporting, yes it works great. Even has netdata which is cool. It even saved me by notifying i had a temperature issue, which i then noticed my fans weren’t working so i managed to fix that thx to truenas alerts and checking the monitoring tools like netdata to troubleshoot - complete
setup users - complete
setup acls - complete
snapshots - incomplete

I think that more or less covers most of what you need for a NAS. So basically more or less a success. Only major thing to solve is the docker situation. Others with better technical skills have already done so. So i just have to keep at it

If you are not planning on running any docker containers, then your move to truenas will be much easier.

truenas uses kubernetes k3’s listed under the apps section in the UI. You just select the app from the catalog, do the minor adjustments in settings and deploy.

But even if you wanted docker, seems the VM route is the easiest to do. Jailmaker is a bit more complicated but is probably a bit doable once more complete guides that account for everything come around. hopefully my documentation as i go along will help somewhat with that.

A picture is worth a thousand words - confucius

Stux · April 24, 2024, 3:17am

And that is your headings for that future guide

mooglestiltzkin · April 24, 2024, 3:25am

why is data and compose separate?

i assume data has the docker deployed configs. but compose has the compose file? is that correct?

but if you run the compose won’t it deploy where you ran it?

Example mine is like

docker/portainer/docker compose.yaml

so i would browse to that portainer directory, then do a docker compose up. It detects that yaml and runs it, then deploys within that same location.

That’s my only understanding of that

and usually i have both a docker compose.yaml, and also i kept a copy within portainer saved as a custom deploy script for quick re-deploy.

portainer doesn’t like you deploying docker compose yamls. instead they want you to deploy docker compose within the portainer UI so it can manage it. So i keep the docker compose yaml as a backup, then copy the content to portainer, then deploy from portainer. So that’s how i manage that.

portainer also supports .env so you don’t need a .env file. just copy the values into portainer and it will deploy them fine.

Stux · April 24, 2024, 3:40am

I don’t use portainer. I find it too fragile.

Instead I create a compose file for each service I want to run.

my compose datasets contains a directory for each service, and each directory has a “docker-compose.yml” file in it, and anything else.

I bring up a service by cd-ing into a directory and typing “docker compose up -d”, etc.

(or using dockge)

As such, the compose directories I want to be able to snapshot/rollback/backup etc on separate scehdule, and to different locatations, vs the data directory, which contains all the data sub-directories that I host mount into containers/dockers etc.

I used to have a dataset per docker container… but then I had to keep on restarting the jail everytime I wanted to add a new service (ie a docker composition)

Stux · April 24, 2024, 3:44am

Maybe this will make a bit more sense

root@chronus[/mnt/tank/docker/compose]# cd jellyfin
root@chronus[/mnt/tank/docker/compose/jellyfin]# ll
total 26
drwxr-xr-x  2 root   4 Mar 25 15:44 ./
drwxr-xr-x 10 root  11 Mar 25 15:59 ../
-rw-r--r--  1 root  17 Mar 25 15:13 .env
-rw-r--r--  1 root 707 Mar 25 15:44 docker-compose.yml
root@chronus[/mnt/tank/docker/compose/jellyfin]# cat docker-compose.yml 
services:
  jellyfin:
    image: linuxserver/jellyfin:latest
    container_name: jellyfin
    network_mode: host
    environment:
      - PUID=1005
      - PGID=1007
      - TZ=Etc/UTC
      - JELLYFIN_PublishedServerUrl=https://jellyfin.XXXX.com
    volumes:
      - /mnt/jellyfin/config:/config
      - /mnt/jellyfin/cache:/config/cache
      - /mnt/jellyfin/transcodes:/config/data/transcodes
      - /mnt/media:/data
    #      - /path/to/tvseries:/data/tvseries
    #      - /path/to/movies:/data/movies
    #    ports:
    #      - 8096:8096
    #      - 8920:8920 #optional
    #      - 7359:7359/udp #optional
    #      - 1900:1900/udp #optional
    restart: unless-stopped
networks: {}
root@chronus[/mnt/tank/docker/compose/jellyfin]#

Stux · April 24, 2024, 3:45am

Note, the PUID/PGID refer to the actual jellyfin user on my NAS.

root@chronus[/mnt/tank/docker/compose/jellyfin]# cat /etc/passwd | g 1005
jellyfin:x:1005:1007:jellyfin:/nonexistent:/usr/sbin/nologin

mooglestiltzkin · April 24, 2024, 4:17am

sorry back. was busy making memes

mooglestiltzkin · April 24, 2024, 4:28am

yeah your compose dataset is pretty much what i am doing as well.

but when the docker deploys it dumps its config, data everything into that same location. But i see you are pointing yours to a different directory under data.

so compose/ directory for each service.

then for data/ also directory for each service?

is that how it is?

I just assumed that the compose had to be where the docker data stuff get generated into, guess not. Maybe i should do that too. Then i can keep things clean to separate out the docker compose yamls for deploying, and the data configs the apps generate out and use.

mooglestiltzkin · April 24, 2024, 4:34am

yeah i think i need to do this also.

setup a user in the jailmaker, maybe also give root using the sudo command

useradd USERNAME

passwd USERNAME

If you want the ability to run commands as root, add the user to the sudo group usermod -aG sudo USERNAME

then to find puid pgid

not sure if we also have to go truenas later and create that user there using the same puid and pgid or not as well

github.com

Jip-Hop/jailmaker/blob/main/docs/basicconfig.md

# User Management
The root user (also known as the superuser or su) can access any file, make system changes, and lots of room for security vulnerabilities.
For this reason you should aspire to run services as a non-root user.

### Create a non-root user
`useradd USERNAME`

Where username can be anything, but should reflect the service/jail's name for diagnostic.

Then a password should be created as some commands require a non-blank password to be inserted:
`passwd USERNAME`

If you want the ability to run commands as root, add the user to the sudo group
`usermod -aG sudo USERNAME`

This WILL require a non-blank password, and any command run with sudo will be run as root not as the user. But it saves time compared to switching users to root to install/change things then switching back.

### Switch to user
`su -l USERNAME`

This file has been truncated. show original

and the other thing is whether to change owner from root to this new user or not? but if it has root powers, then no need right?

mooglestiltzkin · April 24, 2024, 4:39am

i thought you said the docker needs to be a slightly older one that works because the latest docker has issues with jailmaker. but your config says stable, did you forget to change there?

i’m not sure how to downgrade, so i simply intend to nuke/delete the jailmaker and start from scratch. Basically just run the jailmaker docker script again (this script has already been modified by me)

though i’m still not too sure about the pathing

correct?
/mnt/xxxxx/jailmaker/jails/docker/rootfs/docker

possible new pathing to improve it to?

/mnt/xxxxx/jailmaker/jails/docker/rootfs/mnt/docker

this not needed?
/mnt/xxxxx/jailmaker/jails/docker/

Stux · April 24, 2024, 4:41am

Yes. I keep the data and compose separate per composition

Makes it easy to move a service, say into a vm or whatever

Stux · April 24, 2024, 4:43am

I just haven’t updated to the broken version yet

Stux · April 24, 2024, 4:45am

No. That doesn’t look right

Rootfs is your jails file system on your pool. The jail will have its / directory (ie root) set to point to that directory.

You should try to keep the jail as thin and disposable as possible… which is why I mount the docker config and data into the jail.

The config and data should be stored on your pool outside of the jail, and the mounted in using a bind argument in the jail config

mooglestiltzkin · April 24, 2024, 5:16am

I’ll try redo it and report back my results

how do you set this to 26.0.1?

	echo \
	"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
	$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
	tee /etc/apt/sources.list.d/docker.list > /dev/null
	apt-get update
	apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

https://download.docker.com/linux/debian/dists/bookworm/

was trying to browse here

mooglestiltzkin · April 24, 2024, 5:26am

hm

so

/mnt/xxxxx/jailmaker/jails/docker/rootfs/

/mnt/xxxxx/docker/

*data and compose directory go here

github.com

Jip-Hop/jailmaker/blob/main/docs/storage.md

# Default storage system
When creating a jail, an entire Linux filesystem is created in the 'rootfs' folder within the jail's folder of the jailmaker directory E.g `/mnt/tank/vault/jailmaker/jails/jailname/rootfs`. No files from the TrueNAS host will be available. 

Common locations for services are:
`/home` for user accessible files
`/var/www/` for webpages
`/tmp` for temporary application data such as build files

# Linking folders to TrueNAS folders
To allow file access by either the jail, another jail, or TrueNAS a bind can be made. A bind creates a link between two locations. Think of this as a portal, anything that goes in one side is visible from the other side and vice versa.

Note that creating a file in the jail or TrueNAS will reflect in both binded locations, so be careful of overwrites and corruption.

### Setup
Add the following to your user arguments during setup or into the jail's config file, with your two linked locations separated by a colon:
```
--bind='/host/path/to/:/jail/path/to'
```
Where `/host/path/to/` is the folder on the TrueNAS filesystem you want shared.
And where `/jail/path/to/` is the folder you want those shared files accessible by the jail.

This file has been truncated. show original

When creating a jail, an entire Linux filesystem is created in the ‘rootfs’ folder within the jail’s folder of the jailmaker directory E.g /mnt/tank/vault/jailmaker/jails/jailname/rootfs . No files from the TrueNAS host will be available.

this is what i came up with

	--bind-ro='/mnt/xxxxx2/Storage:/mnt/Storage'
	--bind-ro='/mnt/xxxxx2/Storage2:/mnt/Storage2'
	--bind='/mnt/xxxxx/jailmaker/jails/rootfs/docker/:/mnt/docker'
	--bind='/mnt/xxxxx/docker/data/:/mnt/data'
	--bind='/mnt/xxxxx/docker/data/:/mnt/compose'

then the docker jailmaker script is in /mnt/xxxxx/docker/jailmaker/docker

jlmkr create --start --config /mnt/xxxxx/jailmaker/config docker