Linux Jails (sandboxes / containers) with Jailmaker

Thank you so much for your help!
Below is the config file under “jailmaker/templates/docker/”

startup=0
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1

# Use macvlan networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-macvlan=eno1 instead of --network-bridge
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=--network-bridge=br1
    --resolv-conf=bind-host
    --system-call-filter='add_key keyctl bpf'

# Script to run on the HOST before starting the jail
# Load kernel module and config kernel settings required for docker
pre_start_hook=#!/usr/bin/bash
    set -euo pipefail
    echo 'PRE_START_HOOK'
    echo 1 > /proc/sys/net/ipv4/ip_forward
    modprobe br_netfilter
    echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
    echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

# Only used while creating the jail
distro=debian
release=bookworm

# Install docker inside the jail:
# https://docs.docker.com/engine/install/debian/#install-using-the-repository
# Will also install the NVIDIA Container Toolkit if gpu_passthrough_nvidia=1 during initial setup
# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
initial_setup=#!/usr/bin/bash
    set -euo pipefail

    apt-get update && apt-get -y install ca-certificates curl
    install -m 0755 -d /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
    chmod a+r /etc/apt/keyrings/docker.asc

    echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
    $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
    tee /etc/apt/sources.list.d/docker.list > /dev/null
    
    apt-get update
    apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    
    # The /usr/bin/nvidia-smi will be present when gpu_passthrough_nvidia=1
    if [ -f /usr/bin/nvidia-smi ]; then
        curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey -o /etc/apt/keyrings/nvidia.asc
        chmod a+r /etc/apt/keyrings/nvidia.asc
        curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
        sed 's#deb https://#deb [signed-by=/etc/apt/keyrings/nvidia.asc] https://#g' | \
        tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

        apt-get update
        apt-get install -y nvidia-container-toolkit

        nvidia-ctk runtime configure --runtime=docker
        systemctl restart docker
    fi

    docker info

# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed
    --property=Type=notify
    --property=RestartForceExitStatus=133
    --property=SuccessExitStatus=133
    --property=Delegate=yes
    --property=TasksMax=infinity
    --collect
    --setenv=SYSTEMD_NSPAWN_LOCK=0

systemd_nspawn_default_args=--keep-unit
    --quiet
    --boot
    --bind-ro=/sys/module
    --inaccessible=/sys/module/apparmor

Yeah, but what is the config used for your jail that has the problem.

There should be a directory called “jails” in your jailmaker dataset.

My guess is probably I haven’t went that far. The error happened when I input:
jlmkr create
Then input “1” => went to Nano => pasted the config => “^W” write to file => “^X” exit Nano. Right after exit, I got this error. : ( I haven’t got chance to enter docker name.

I don’t have jails folder under jailmaker:

image

You’re right

I tested it, and if i duplicate the intel line, I get the same error, but on line 4, and the jail doesn’t get created.

But I can reproduce your exact error by pasting the config in twice.

Traceback (most recent call last):
  File "/mnt/tank/jailmaker/jlmkr.py", line 2144, in <module>
    main()
  File "/mnt/tank/jailmaker/jlmkr.py", line 2139, in main
    sys.exit(func(**args))
             ^^^^^^^^^^^^
  File "/mnt/tank/jailmaker/jlmkr.py", line 1302, in create_jail
    jail_name, config, start_now = interactive_config()
                                   ^^^^^^^^^^^^^^^^^^^^
  File "/mnt/tank/jailmaker/jlmkr.py", line 1051, in interactive_config
    config.read_file(f)
  File "/usr/lib/python3.11/configparser.py", line 734, in read_file
    self._read(f, source)
  File "/mnt/tank/jailmaker/jlmkr.py", line 208, in _read
    return super()._read(lines, fpname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/configparser.py", line 1112, in _read
    raise DuplicateOptionError(sectname, optname,
configparser.DuplicateOptionError: While reading from '/tmp/tmpgr5t9x_1' [line 83]: option 'gpu_passthrough_intel' in section 'a' already exists

And this error is on Line 83… the last line…

configparser.DuplicateOptionError: While reading from '/tmp/tmpgr5t9x_1' [line 83]: option 'gpu_passthrough_intel' in section 'a' already exists

you see, the first line is startup=0 which gets appended to the last line of the previous paste… thus the gpu_passthrough_intel=0 is the first repeated line at line 83.

If you’re having trouble with the config pasting you can instead use this command run from your jailmaker dataset

jlmkr create -c templates/docker/config jail-name

and then edit it using jlmkr edit jail-name

and then start it with jlmkr start jail-name

where jail-name is you jail name…

1 Like

Thanks Stux! Will give it a try tonight : )

Wasn’t sure if this should go here or in its own thread, but I’m getting a crash on mongo in my docker jail (have installed in both dockge and portainer, trying multiple versions of mongo from 4 to 7).

Can pull and install no problem, but when I try to use mogosh to set up a new database or user, the whole container crashes with a “terminal no longer available” message. I read somewhere that can be an issue with the mongo container not having certain CPU virtualisation features passed through.

Thoughts & experiences?

Sorry I can’t post more details just now, as I’m on the road.

I didn’t see anything that requires enhanced privileges, but I did see this:

Which I think could interact badly with Dragonfish ARC

  1. ensure you’ve applied the lru_gen fix if using dragonfish 24.04.0
  2. perhaps limit the mongo cache size.

https://hub.docker.com/_/mongo

I think I have found a way to limit the memory usage per container by adding these to systemd_run_default_args in the jail config

        --property=MemoryAccounting=true
        --property=MemoryHigh=16G
        --property=MemoryMax=20G
        --property=MemorySwapMax=0

It is not possible to add them to systemd_nspawn_user_args because jailmaker uses --keep-unit which seems to make them useless (I am not a systemd expert) but that is what is mentioned in systemd-nspawn man page.

Also note that swap properties are important otherwise swap will be used when MemoryMax is exceeded.

It should be possible to limit CPU usage too but I am unable to test it at the moment.

More stuff are here: systemd.resource-control

I tested it by writing some code that keeps leaking memory on purpose.
In my case the allocator gets throttled significantly after it hits the MemoryHigh limit. I ran out of patience to test the MemoryMax but maybe someone else can check it.

This is the code for the allocator. Build it via:
gcc -o alloc alloc.c
copy it to the truenas server and then to the container. It takes the number of pages as an argument (1 page is 4K)


#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>

#define PAGE_SIZE  4096   // bytes per page
#define MB         256    // pages per MB
#define GB         262144 // pages per GB

int main(int argc, char *argv[]) {
  if (argc != 2) {
    fprintf(stderr, "Usage: %s <num of pages>\n", argv[0]);
    return 1;
  }

  int n_pages = atoi(argv[1]);
  if (!n_pages) {
    perror("atoi");
    return 1;
  }

  /*
   * Trying to allocate a guge chunk of n_pages * PAGE_SIZE fails.
   * Let's allocate in small pieces
   */
  for (int x = 0; x < n_pages; x++) {
    void *mem = malloc(PAGE_SIZE);
    if (!mem) {
      perror("malloc");
      return 1;
    }

    memset(mem, 0x0, PAGE_SIZE);
    if (x + 1 >= GB) {
      fprintf(stdout, "Allocated %ld pages (%f GB)\n", x + 1, (x + 1) / (float)GB);
    } else if (x + 1 >= MB) {
      fprintf(stdout, "Allocated %ld pages (%f MB)\n", x + 1, (x + 1) / (float)MB);
    } else {
      fprintf(stdout, "Allocated %ld pages (%lld bytes)\n", x + 1, (x + 1) * PAGE_SIZE);
    }
  }

  return 0;
}

I apologize if I could not fully test and/or the code has any bugs but I had a long exhausting day.

1 Like

Here’s the link suggesting a need for AVX support to run Mongo >5.0:docker - MongoDB 5.0+ requires a CPU with AVX support. Container failed to start - Stack Overflow

Steps to reproduce crash:

  1. I’ve installed mongo using the instructions here:
    How to Install and Run MongoDB using Docker Compose - Pi My Life Up
  2. Using Dockge, open a bash shell in the mongo container, then in the shell, enter the mongo shell (‘mongosh’)
  3. Make and switch to a new database (‘use admin’)
  4. Try to create a new user for the admin database:
    db.createUser({ user:“admin”, pwd:“password”, roles:[{role:“root”, db:“admin”}] });

The terminal hangs, and no user is created.

Using the above example I can’t create any new databases or users in mongosh as “not authorized to execute commands”, despite security not being enabled in /etc/mongo.conf

Anyway - this is probably more an error with my lack of mongo skills, rather than a jail or dockge problem, so I’ll leave it there. My intention had been to run UniFi controller in a docker - the new linuxserver docker method requires a separate mongodb container. As is is, I’ve resorted to running UniFi and mongo in the same VM which is also running my nextcloud instance.

Ryzen5 3600 supports AVX and AVX2. This capability should be available to the sandbox.

I don’t think that is your issue.

AMD Ryzen 5 3600 Specs | TechPowerUp CPU Database.

Could this be a network problem that docker-compose is not visible in Dockge?

I encountered, see picture a difference while unzipping the installation file of dockge.
I refer to the tutorial TrueNAS Scale: Setting up Sandboxes with Jailmaker by @Stux
, which I have followed step by step many times.

Shell Dockge Running Stux

Docker compose (1)

Docker compose (2)

Check this out @skittlebrau regarding your multiple bridges question on the old forum.

Brilliant! I will give this a go in a test container.

Thanks for the excellent guide on getting Jailmaker up and running with Dockge. It was easy to follow and trivial to get running. I’m looking forward to leaving TrueCharts and Kubernetes behind.

1 Like

Yep, This really revolutionized the APP world for Scale. Thanks to Jip-Hop for developing jailmaker and to Stux for showing us how to implement it and Dockage with a great video. I left Truecharts far behind. I have hopes that the Truenas folks will work this capability into their product.

3 Likes

Dockage is right… Dockage · GitHub

1 Like

So there’s a fourth one? Great. Now I’m feeling even more dyslexic. :face_with_spiral_eyes:

1 Like

Well, Dockage is not right if you’re referring to Dockge which is what was in my video

Your right…My bad…Dockge…It’s great whatever its called.

1 Like