Linux Jails (sandboxes / containers) with Jailmaker

Apps and Virtualization
,
482

TODO

  1. Add GPU attachment to containers.
  2. Integrate ZFS datasets or zvols into instance creation.

NOTE: If you’re running this in a VM, you’re may run into Nesting issues. You need to configure your VM to nest from the host then it “should” work fine… The config options I added will load the needed kernel modules, sysctl values, and Incus instance configs when the containers attached to the docker profile are powered on.

NOTE: This guide is intended for TrueNAS SCALE 25.04, BUT in theory the configs should work on any host that has incus installed and running. Give a test and let me know how it goes.

This guide assumes you already have your bridges and networking configured and ready to go. I’m assuming you are using br0 for the bridge. If not, go ahead and adjust the config below with your specific interfaces.

It’s very straightforward process with this config. No need for any scripts at this point as cloud-init handles all the dirty work. The only thing missing at this point would be ZFS datasets or zvols for the instances and GPU support. I’ll likely be able to add this functionality to the docker cloud-init profile.

Getting started

  1. Download the following yaml config to your TrueNAS 25.04 host. Please note, when you modify this config, it will be the same on all hosts created from the profile moving forward. Don’t add anything to the profile that you don’t want on EVERY instance created from the profile.
    a. Configure your appropriate mount points on your TrueNAS host where you will be hosting your app data.
    b. Set your timezone.
    c. Feel free to modify anything else you might need, like adding additional packages you would like in your base image.

docker-init.yaml:

description: Docker Profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: default
    type: disk
  data:
    path: /mnt/data
    source: /mnt/pool/data/apps
    shift: true
    type: disk
  stacks:
    path: /opt/stacks
    source: /mnt/pool/data/stacks
    shift: true
    type: disk
config:
  # Start instances on boot
  boot.autostart: "true"
 
  # Load needed kernel modules
  linux.kernel_modules: br_netfilter

  # Enable nesting
  security.nesting: "true"

  cloud-init.user-data: |
    #cloud-config

    # Enable docker sysctl values
    write_files:
      - path: /etc/sysctl.d/20-docker.conf
        content: |
          net.ipv4.conf.all.forwarding=1
          net.bridge.bridge-nf-call-iptables=1
          net.bridge.bridge-nf-call-ip6tables=1

    # Set timezone
    timezone: US/Eastern

    # apt update and apt upgrade
    package_update: true
    package_upgrade: true

    # Install apt repos and packages needed for docker
    apt:
      preserve_sources_list: true
      sources:
        docker.list:
          source: deb [arch=amd64] https://download.docker.com/linux/debian $RELEASE stable
          keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
          filename: docker.list
    packages:
      - apt-transport-https
      - ca-certificates
      - curl
      - gpg
      - docker-ce
      - docker-ce-cli
      - containerd.io
      - docker-buildx-plugin
      - docker-compose-plugin

    # create the docker group
    groups:
      - docker

    # Add default auto created user to docker group
    system_info:
      default_user:
        groups: [docker]

    # Install dockge
    runcmd:
      - [ mkdir, -p, /opt/dockge ]
      - [ cd, /opt/dockge ]
      - [ curl, https://raw.githubusercontent.com/louislam/dockge/master/compose.yaml, --output, compose.yaml ]
      - [ docker, compose, up, -d ]
  1. Import the profile into incus. Any future docker instances can now use this profile to create new instances moving forward.
incus profile create docker < docker-init.yaml
  1. Build a docker instance. You can build as many as you like. docker1 is the instance name which will show up when you list the Incus instances by running incus ls.
incus launch images:debian/bookworm/cloud -p docker docker1
  1. Access the new Incus instance shell.
incus exec docker1 -- bash
  1. Configure static IP and DNS resolver. Once the host is built, you should configure a static IP address and point to your DNS server. Edit the following files with your favorite editor.
/etc/systemd/network/10-cloud-init-eth0.network

Output below. Modify your Address and Gateway.

[Match]
Name=eth0

[Network]
#DHCP=ipv4
DHCP=false
Address=192.168.0.30/24
Gateway=192.168.0.1
LinkLocalAddressing=no
LLDP=yes
EmitLLDP=customer-bridge

/etc/systemd/resolved.conf

Output below. Modify DNS to point to your DNS server and Domains to your search domain if needed. If you don’t need the search domains, just comment out the Domains line.

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the resolved.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
#
# See resolved.conf(5) for details.

[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
DNS=192.168.0.8
#FallbackDNS=
Domains=lan.domain.co
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=yes
#LLMNR=yes
#Cache=yes
#CacheFromLocalhost=no
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
  1. Restart container
reboot
  1. Verify everything looks good. You should see output similar to below when everything is up and running. Notice that you will have eth0 which is the containers bridge to the TrueNAS hosts’ br0 interface. You will also have docker0 for Docker’s interface to eth0. Finally you’ll have a br-* interface that Dockge is using.
incus ls                      
+-------------+---------+------------------------------+------+-----------+-----------+
|    NAME     |  STATE  |             IPV4             | IPV6 |   TYPE    | SNAPSHOTS |
+-------------+---------+------------------------------+------+-----------+-----------+
| docker1     | RUNNING | 192.168.0.30 (eth0)          |      | CONTAINER | 0         |
|             |         | 172.18.0.1 (br-7e7ee82b01bf) |      |           |           |
|             |         | 172.17.0.1 (docker0)         |      |           |           |
+-------------+---------+------------------------------+------+-----------+-----------+

Seriously, once you have the profile in place, it will take you less than a couple of minutes to have a brand new docker instance with Dockge running. This config can be modified to do whatever you want and you can create different profiles to spin up different instances instantly.

2 Likes