Do we know how they are planning on implementing Incus? Are they going to phase out Apps, then incorporate Incus as Apps and use it for VM/Custom Apps? I was playing with their demo yesterday and it’s pretty good actually.
I’ll download the nightly and create a VM and test it out.
My understanding is there is absolutely no intention of phasing out the docker apps.
Incus will be for Linux container lightweight virtualization, conceptually Linux Jails.
I need to look into Incus more. I’m assuming it’s leveraging the kvm module in the kernel for VM’s and LXC for containers.
Incus supports heavy and lightweight virtualization.
Ie separate kernel vs shared kernel.
The full virtualization uses QEMU under the hood, the lightweight uses LXC.
LXC is like systemd-nspawn and uses kernel services like cgroups and network namespaces to share a kernel with Linux containers. Yes. Docker does this too, but in a different way, more ephemeral.
QEMU uses KVM.
I’m still pulling on this idea I had. Oh and sorry, @dasunsrule32. It will be student teaches master (a new trick) if this works. So, I hope you have some of those old-timey-sock-keeper-uper-things because I’m about to try and knock 'em off (later, when I get back to my server…in a few days).
*anticipation: built!*
.
.
.
.
.
.
.
EDIT: it’s not really that “big” of an idea, but every once and a while you have to pick a fight with the biggest kid on the playground to really feel alive.
Yeah, makes sense. If I get some time over the next week, I’ll setup the nightly and give it a whirl.
Looks like they do have containers and vm’s enabled in the nightlies now. Incus is there. Time to play.
Have some more work to do… bbbuutt, I have a cloud-init “jail” running on Incus with a new docker profile and docker installed and running in LXC. I’ll automate it further soon.
I have the following to do still:
Load modules/configure on TrueNAS host:
modprobe br_netfilter
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
Build docker host:
incus profile create docker < docker-init.yaml
incus launch images:debian/bookworm/cloud -p docker docker-test
Newly created profile:
incus profile ls
+---------+-------------------------+---------+
| NAME | DESCRIPTION | USED BY |
+---------+-------------------------+---------+
| default | Default TrueNAS profile | 1 |
+---------+-------------------------+---------+
| docker | Docker Profile | 1 |
+---------+-------------------------+---------+
Incus images:
incus list
+-------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| docker-test | RUNNING | 172.17.0.1 (docker0) | fd42:dd50:e5ae:bf19:216:3eff:feb8:5612 (eth0) | CONTAINER | 0 |
| | | 10.105.180.223 (eth0) | | | |
+-------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
Docker information:
docker info
Client: Docker Engine - Community
Version: 27.3.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.17.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.7
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 27.3.1
Storage Driver: overlay2
Backing Filesystem: zfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: true
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
runc version: v1.1.14-0-g2c9f560
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.6.44-production+truenas
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.828GiB
Name: docker-test
ID: 4707c753-6312-4cb7-a8b6-38125878e065
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Sample “working” (very alpha) cloud-init config… This will get you a working instance running with all updates applied, packages installed, and docker running on boot.
docker-init.yaml
description: Docker Profile
devices:
eth0:
name: eth0
network: incusbr0
type: nic
root:
path: /
pool: default
type: disk
config:
cloud-init.user-data: |
#cloud-config
package_update: true
package_upgrade: true
apt:
preserve_sources_list: true
sources:
docker.list:
source: deb [arch=amd64] https://download.docker.com/linux/debian $RELEASE stable
keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
filename: docker.list
packages:
- apt-transport-https
- ca-certificates
- curl
- gpg
- host
- netcat-openbsd
- docker-ce
- docker-ce-cli
- containerd.io
- docker-buildx-plugin
- docker-compose-plugin
# Enable ipv4 forwarding
write_files:
- path: /etc/sysctl.d/enabled_ipv4_forwarding.conf
content: |
net.ipv4.conf.all.forwarding=1
# create the docker group
groups:
- docker
# Add default auto created user to docker group
system_info:
default_user:
groups: [docker]
Log output: /var/log/cloud-init-output.log
Cloud-init v. 22.4.2 running 'init-local' at Tue, 05 Nov 2024 21:22:38 +0000. Up 0.92 seconds.
Cloud-init v. 22.4.2 running 'init' at Tue, 05 Nov 2024 21:22:38 +0000. Up 1.58 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: | eth0 | True | 10.105.180.223 | 255.255.255.0 | global | 00:16:3e:b8:56:12 |
ci-info: | eth0 | True | fe80::216:3eff:feb8:5612/64 | . | link | 00:16:3e:b8:56:12 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: +++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++
ci-info: +-------+--------------+--------------+-----------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+--------------+--------------+-----------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 10.105.180.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 10.105.180.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: | 2 | 10.105.180.1 | 0.0.0.0 | 255.255.255.255 | eth0 | UH |
ci-info: +-------+--------------+--------------+-----------------+-----------+-------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | 0 | fe80::/64 | :: | eth0 | U |
ci-info: | 2 | local | :: | eth0 | U |
ci-info: | 3 | multicast | :: | eth0 | U |
ci-info: +-------+-------------+---------+-----------+-------+
2024-11-05 21:22:39,034 - util.py[WARNING]: Unable to get zpool status of pool: Unexpected error while running command.
Command: ['zpool', 'status', 'pool']
Exit code: -
Reason: [Errno 2] No such file or directory: b'zpool'
Stdout: -
Stderr: -
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub
The key fingerprint is:
SHA256:wopaJK/UjXTYyPqpIs1De73rATVw5iLUDSuDYHaRMes root@docker-test
The key's randomart image is:
+---[RSA 3072]----+
|.o.**oo |
|+o..+*. |
|. +.o + |
| o+++ . |
|. .E.oo S |
| +=.+o . |
| *++.o. |
|++* o .. |
|=..= .+o |
+----[SHA256]-----+
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_dsa_key
Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub
The key fingerprint is:
SHA256:zp6R6CEaGy0KYz+vIyDcOrI29aqPrP1m79Dg06LzaS8 root@docker-test
The key's randomart image is:
+---[DSA 1024]----+
| |
| |
| |
| |
|. . . S |
|o. = + + . |
|=.* O = = |
|=X+XEO o o |
|B=@%X*= o |
+----[SHA256]-----+
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key
Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub
The key fingerprint is:
SHA256:XAJoT6CPdcdgyz9gnE0hhI6Ut0zHAZx/g79oeW9dmDw root@docker-test
The key's randomart image is:
+---[ECDSA 256]---+
| ooOB.o. |
| o.O++X |
| ..B.*O.= . |
| .++o+=oo |
| . . oS. . o |
| .. E . |
| o . . o |
| + o . . |
| . . o. |
+----[SHA256]-----+
Generating public/private ed25519 key pair.
Your identification has been saved in /etc/ssh/ssh_host_ed25519_key
Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub
The key fingerprint is:
SHA256:OCQ5yaAU74eVdFQ4n7TxNLSQwEDZTYlQ2yvToN+nkCw root@docker-test
The key's randomart image is:
+--[ED25519 256]--+
| oo .=O=B++. |
|...o.+o*o*.o. |
|. .*o.o+.*.. |
| . o+..o+.. |
| o oooSo |
| . o.= |
| E = . . |
| . . o |
| . |
+----[SHA256]-----+
Cloud-init v. 22.4.2 running 'modules:config' at Tue, 05 Nov 2024 21:22:40 +0000. Up 3.00 seconds.
2024-11-05 21:22:40,392 - cc_set_passwords.py[WARNING]: Ignoring config 'ssh_pwauth: None'. SSH service 'ssh' is not installed.
Hit:1 http://deb.debian.org/debian bookworm InRelease
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 https://download.docker.com/linux/debian bookworm InRelease [43.3 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [190 kB]
Get:6 http://deb.debian.org/debian-security bookworm-security/main Translation-en [116 kB]
Get:7 https://download.docker.com/linux/debian bookworm/stable amd64 Packages [31.3 kB]
Fetched 484 kB in 1s (410 kB/s)
Reading package lists...
Cloud-init v. 22.4.2 running 'modules:final' at Tue, 05 Nov 2024 21:22:43 +0000. Up 6.14 seconds.
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version (20230311).
ca-certificates set to manually installed.
gpg is already the newest version (2.2.40-1.1).
The following additional packages will be installed:
apparmor bind9-libs docker-ce-rootless-extras git git-man iptables less
libbrotli1 libcurl3-gnutls libcurl4 liberror-perl libfstrm0 libgdbm-compat4
libgdbm6 libglib2.0-0 libglib2.0-data libicu72 libip6tc2 libjansson4
libjemalloc2 liblmdb0 libltdl7 libmaxminddb0 libnetfilter-conntrack3
libnfnetlink0 libnftables1 libnftnl11 libnghttp2-14 libperl5.36
libprotobuf-c1 libpsl5 librtmp1 libslirp0 libssh2-1 libuv1 libxml2 nftables
patch perl perl-modules-5.36 pigz publicsuffix shared-mime-info slirp4netns
xdg-user-dirs xz-utils
Suggested packages:
apparmor-profiles-extra apparmor-utils aufs-tools cgroupfs-mount
| cgroup-lite gettext-base git-daemon-run | git-daemon-sysvinit git-doc
git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn firewalld kmod
gdbm-l10n low-memory-monitor mmdb-bin ed diffutils-doc perl-doc
libterm-readline-gnu-perl | libterm-readline-perl-perl make
libtap-harness-archive-perl
The following NEW packages will be installed:
apparmor apt-transport-https bind9-host bind9-libs containerd.io curl
docker-buildx-plugin docker-ce docker-ce-cli docker-ce-rootless-extras
docker-compose-plugin git git-man iptables less libbrotli1 libcurl3-gnutls
libcurl4 liberror-perl libfstrm0 libgdbm-compat4 libgdbm6 libglib2.0-0
libglib2.0-data libicu72 libip6tc2 libjansson4 libjemalloc2 liblmdb0
libltdl7 libmaxminddb0 libnetfilter-conntrack3 libnfnetlink0 libnftables1
libnftnl11 libnghttp2-14 libperl5.36 libprotobuf-c1 libpsl5 librtmp1
libslirp0 libssh2-1 libuv1 libxml2 netcat-openbsd nftables patch perl
perl-modules-5.36 pigz publicsuffix shared-mime-info slirp4netns
xdg-user-dirs xz-utils
0 upgraded, 55 newly installed, 0 to remove and 0 not upgraded.
Need to get 160 MB of archives.
After this operation, 616 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 perl-modules-5.36 all 5.36.0-7+deb12u1 [2,815 kB]
Get:2 https://download.docker.com/linux/debian bookworm/stable amd64 containerd.io amd64 1.7.22-1 [29.5 MB]
Get:3 http://deb.debian.org/debian bookworm/main amd64 libgdbm6 amd64 1.23-3 [72.2 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 libgdbm-compat4 amd64 1.23-3 [48.2 kB]
Get:5 http://deb.debian.org/debian bookworm/main amd64 libperl5.36 amd64 5.36.0-7+deb12u1 [4,218 kB]
Get:6 http://deb.debian.org/debian bookworm/main amd64 perl amd64 5.36.0-7+deb12u1 [239 kB]
Get:7 http://deb.debian.org/debian bookworm/main amd64 pigz amd64 2.6-1 [64.0 kB]
Get:8 http://deb.debian.org/debian bookworm/main amd64 less amd64 590-2.1~deb12u2 [132 kB]
Get:9 http://deb.debian.org/debian bookworm/main amd64 libjansson4 amd64 2.14-2 [40.8 kB]
Get:10 http://deb.debian.org/debian bookworm/main amd64 libnftnl11 amd64 1.2.4-2 [61.6 kB]
Get:11 http://deb.debian.org/debian bookworm/main amd64 libnftables1 amd64 1.0.6-2+deb12u2 [299 kB]
Get:12 http://deb.debian.org/debian bookworm/main amd64 nftables amd64 1.0.6-2+deb12u2 [70.3 kB]
Get:13 http://deb.debian.org/debian bookworm/main amd64 libuv1 amd64 1.44.2-1+deb12u1 [136 kB]
Get:14 http://deb.debian.org/debian bookworm/main amd64 libfstrm0 amd64 0.6.1-1 [21.6 kB]
Get:15 http://deb.debian.org/debian bookworm/main amd64 libjemalloc2 amd64 5.3.0-1 [275 kB]
Get:16 http://deb.debian.org/debian bookworm/main amd64 liblmdb0 amd64 0.9.24-1 [45.0 kB]
Get:17 http://deb.debian.org/debian bookworm/main amd64 libmaxminddb0 amd64 1.7.1-1 [29.8 kB]
Get:18 http://deb.debian.org/debian bookworm/main amd64 libnghttp2-14 amd64 1.52.0-1+deb12u1 [72.4 kB]
Get:19 http://deb.debian.org/debian bookworm/main amd64 libprotobuf-c1 amd64 1.4.1-1+b1 [27.5 kB]
Get:20 http://deb.debian.org/debian bookworm/main amd64 libicu72 amd64 72.1-3 [9,376 kB]
Get:21 http://deb.debian.org/debian bookworm/main amd64 libxml2 amd64 2.9.14+dfsg-1.3~deb12u1 [687 kB]
Get:22 http://deb.debian.org/debian bookworm/main amd64 bind9-libs amd64 1:9.18.28-1~deb12u2 [1,427 kB]
Get:23 https://download.docker.com/linux/debian bookworm/stable amd64 docker-buildx-plugin amd64 0.17.1-1~debian.12~bookworm [30.3 MB]
Get:24 http://deb.debian.org/debian bookworm/main amd64 bind9-host amd64 1:9.18.28-1~deb12u2 [307 kB]
Get:25 http://deb.debian.org/debian bookworm/main amd64 xz-utils amd64 5.4.1-0.2 [471 kB]
Get:26 http://deb.debian.org/debian bookworm/main amd64 apparmor amd64 3.0.8-3 [616 kB]
Get:27 http://deb.debian.org/debian bookworm/main amd64 apt-transport-https all 2.6.1 [25.2 kB]
Get:28 http://deb.debian.org/debian bookworm/main amd64 libbrotli1 amd64 1.0.9-2+b6 [275 kB]
Get:29 http://deb.debian.org/debian bookworm/main amd64 libpsl5 amd64 0.21.2-1 [58.7 kB]
Get:30 http://deb.debian.org/debian bookworm/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2+b2 [60.8 kB]
Get:31 http://deb.debian.org/debian bookworm/main amd64 libssh2-1 amd64 1.10.0-3+b1 [179 kB]
Get:32 http://deb.debian.org/debian bookworm/main amd64 libcurl4 amd64 7.88.1-10+deb12u7 [390 kB]
Get:33 http://deb.debian.org/debian bookworm/main amd64 curl amd64 7.88.1-10+deb12u7 [315 kB]
Get:34 http://deb.debian.org/debian bookworm/main amd64 libip6tc2 amd64 1.8.9-2 [19.4 kB]
Get:35 http://deb.debian.org/debian bookworm/main amd64 libnfnetlink0 amd64 1.0.2-2 [15.1 kB]
Get:36 http://deb.debian.org/debian bookworm/main amd64 libnetfilter-conntrack3 amd64 1.0.9-3 [40.7 kB]
Get:37 http://deb.debian.org/debian bookworm/main amd64 iptables amd64 1.8.9-2 [360 kB]
Get:38 http://deb.debian.org/debian bookworm/main amd64 libcurl3-gnutls amd64 7.88.1-10+deb12u7 [385 kB]
Get:39 http://deb.debian.org/debian bookworm/main amd64 liberror-perl all 0.17029-2 [29.0 kB]
Get:40 http://deb.debian.org/debian-security bookworm-security/main amd64 git-man all 1:2.39.5-0+deb12u1 [2,054 kB]
Get:41 http://deb.debian.org/debian-security bookworm-security/main amd64 git amd64 1:2.39.5-0+deb12u1 [7,256 kB]
Get:42 http://deb.debian.org/debian bookworm/main amd64 libglib2.0-0 amd64 2.74.6-2+deb12u3 [1,400 kB]
Get:43 http://deb.debian.org/debian bookworm/main amd64 libglib2.0-data all 2.74.6-2+deb12u3 [1,209 kB]
Get:44 https://download.docker.com/linux/debian bookworm/stable amd64 docker-ce-cli amd64 5:27.3.1-1~debian.12~bookworm [15.0 MB]
Get:45 http://deb.debian.org/debian bookworm/main amd64 libltdl7 amd64 2.4.7-7~deb12u1 [393 kB]
Get:46 http://deb.debian.org/debian bookworm/main amd64 libslirp0 amd64 4.7.0-1 [63.0 kB]
Get:47 http://deb.debian.org/debian bookworm/main amd64 netcat-openbsd amd64 1.219-1 [41.5 kB]
Get:48 http://deb.debian.org/debian bookworm/main amd64 patch amd64 2.7.6-7 [128 kB]
Get:49 http://deb.debian.org/debian bookworm/main amd64 publicsuffix all 20230209.2326-1 [126 kB]
Get:50 http://deb.debian.org/debian bookworm/main amd64 shared-mime-info amd64 2.2-1 [729 kB]
Get:51 http://deb.debian.org/debian bookworm/main amd64 slirp4netns amd64 1.2.0-1 [37.5 kB]
Get:52 http://deb.debian.org/debian bookworm/main amd64 xdg-user-dirs amd64 0.18-1 [54.4 kB]
Get:53 https://download.docker.com/linux/debian bookworm/stable amd64 docker-ce amd64 5:27.3.1-1~debian.12~bookworm [25.6 MB]
Get:54 https://download.docker.com/linux/debian bookworm/stable amd64 docker-ce-rootless-extras amd64 5:27.3.1-1~debian.12~bookworm [9,587 kB]
Get:55 https://download.docker.com/linux/debian bookworm/stable amd64 docker-compose-plugin amd64 2.29.7-1~debian.12~bookworm [12.7 MB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 160 MB in 2s (65.8 MB/s)
Selecting previously unselected package perl-modules-5.36.
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 15590 files and directories currently installed.)
Preparing to unpack .../00-perl-modules-5.36_5.36.0-7+deb12u1_all.deb ...
Unpacking perl-modules-5.36 (5.36.0-7+deb12u1) ...
Selecting previously unselected package libgdbm6:amd64.
Preparing to unpack .../01-libgdbm6_1.23-3_amd64.deb ...
Unpacking libgdbm6:amd64 (1.23-3) ...
Selecting previously unselected package libgdbm-compat4:amd64.
Preparing to unpack .../02-libgdbm-compat4_1.23-3_amd64.deb ...
Unpacking libgdbm-compat4:amd64 (1.23-3) ...
Selecting previously unselected package libperl5.36:amd64.
Preparing to unpack .../03-libperl5.36_5.36.0-7+deb12u1_amd64.deb ...
Unpacking libperl5.36:amd64 (5.36.0-7+deb12u1) ...
Selecting previously unselected package perl.
Preparing to unpack .../04-perl_5.36.0-7+deb12u1_amd64.deb ...
Unpacking perl (5.36.0-7+deb12u1) ...
Selecting previously unselected package pigz.
Preparing to unpack .../05-pigz_2.6-1_amd64.deb ...
Unpacking pigz (2.6-1) ...
Selecting previously unselected package less.
Preparing to unpack .../06-less_590-2.1~deb12u2_amd64.deb ...
Unpacking less (590-2.1~deb12u2) ...
Selecting previously unselected package libjansson4:amd64.
Preparing to unpack .../07-libjansson4_2.14-2_amd64.deb ...
Unpacking libjansson4:amd64 (2.14-2) ...
Selecting previously unselected package libnftnl11:amd64.
Preparing to unpack .../08-libnftnl11_1.2.4-2_amd64.deb ...
Unpacking libnftnl11:amd64 (1.2.4-2) ...
Selecting previously unselected package libnftables1:amd64.
Preparing to unpack .../09-libnftables1_1.0.6-2+deb12u2_amd64.deb ...
Unpacking libnftables1:amd64 (1.0.6-2+deb12u2) ...
Selecting previously unselected package nftables.
Preparing to unpack .../10-nftables_1.0.6-2+deb12u2_amd64.deb ...
Unpacking nftables (1.0.6-2+deb12u2) ...
Selecting previously unselected package libuv1:amd64.
Preparing to unpack .../11-libuv1_1.44.2-1+deb12u1_amd64.deb ...
Unpacking libuv1:amd64 (1.44.2-1+deb12u1) ...
Selecting previously unselected package libfstrm0:amd64.
Preparing to unpack .../12-libfstrm0_0.6.1-1_amd64.deb ...
Unpacking libfstrm0:amd64 (0.6.1-1) ...
Selecting previously unselected package libjemalloc2:amd64.
Preparing to unpack .../13-libjemalloc2_5.3.0-1_amd64.deb ...
Unpacking libjemalloc2:amd64 (5.3.0-1) ...
Selecting previously unselected package liblmdb0:amd64.
Preparing to unpack .../14-liblmdb0_0.9.24-1_amd64.deb ...
Unpacking liblmdb0:amd64 (0.9.24-1) ...
Selecting previously unselected package libmaxminddb0:amd64.
Preparing to unpack .../15-libmaxminddb0_1.7.1-1_amd64.deb ...
Unpacking libmaxminddb0:amd64 (1.7.1-1) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../16-libnghttp2-14_1.52.0-1+deb12u1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.52.0-1+deb12u1) ...
Selecting previously unselected package libprotobuf-c1:amd64.
Preparing to unpack .../17-libprotobuf-c1_1.4.1-1+b1_amd64.deb ...
Unpacking libprotobuf-c1:amd64 (1.4.1-1+b1) ...
Selecting previously unselected package libicu72:amd64.
Preparing to unpack .../18-libicu72_72.1-3_amd64.deb ...
Unpacking libicu72:amd64 (72.1-3) ...
Selecting previously unselected package libxml2:amd64.
Preparing to unpack .../19-libxml2_2.9.14+dfsg-1.3~deb12u1_amd64.deb ...
Unpacking libxml2:amd64 (2.9.14+dfsg-1.3~deb12u1) ...
Selecting previously unselected package bind9-libs:amd64.
Preparing to unpack .../20-bind9-libs_1%3a9.18.28-1~deb12u2_amd64.deb ...
Unpacking bind9-host (1:9.18.28-1~deb12u2) ...
Selecting previously unselected package xz-utils.
Preparing to unpack .../22-xz-utils_5.4.1-0.2_amd64.deb ...
Unpacking xz-utils (5.4.1-0.2) ...
Selecting previously unselected package apparmor.
Preparing to unpack .../23-apparmor_3.0.8-3_amd64.deb ...
Unpacking apparmor (3.0.8-3) ...
Selecting previously unselected package apt-transport-https.
Preparing to unpack .../24-apt-transport-https_2.6.1_all.deb ...
Unpacking apt-transport-https (2.6.1) ...
Selecting previously unselected package containerd.io.
Preparing to unpack .../25-containerd.io_1.7.22-1_amd64.deb ...
Unpacking containerd.io (1.7.22-1) ...
Selecting previously unselected package libbrotli1:amd64.
Preparing to unpack .../26-libbrotli1_1.0.9-2+b6_amd64.deb ...
Unpacking libbrotli1:amd64 (1.0.9-2+b6) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../27-libpsl5_0.21.2-1_amd64.deb ...
Unpacking libpsl5:amd64 (0.21.2-1) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../28-librtmp1_2.4+20151223.gitfa8646d.1-2+b2_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2+b2) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../29-libssh2-1_1.10.0-3+b1_amd64.deb ...
Unpacking libssh2-1:amd64 (1.10.0-3+b1) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../30-libcurl4_7.88.1-10+deb12u7_amd64.deb ...
Unpacking libcurl4:amd64 (7.88.1-10+deb12u7) ...
Selecting previously unselected package curl.
Preparing to unpack .../31-curl_7.88.1-10+deb12u7_amd64.deb ...
Unpacking curl (7.88.1-10+deb12u7) ...
Selecting previously unselected package docker-buildx-plugin.
Preparing to unpack .../32-docker-buildx-plugin_0.17.1-1~debian.12~bookworm_amd64.deb ...
Unpacking docker-buildx-plugin (0.17.1-1~debian.12~bookworm) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../33-docker-ce-cli_5%3a27.3.1-1~debian.12~bookworm_amd64.deb ...
Unpacking docker-ce-cli (5:27.3.1-1~debian.12~bookworm) ...
Selecting previously unselected package libip6tc2:amd64.
Preparing to unpack .../34-libip6tc2_1.8.9-2_amd64.deb ...
Unpacking libip6tc2:amd64 (1.8.9-2) ...
Selecting previously unselected package libnfnetlink0:amd64.
Preparing to unpack .../35-libnfnetlink0_1.0.2-2_amd64.deb ...
Unpacking libnfnetlink0:amd64 (1.0.2-2) ...
Selecting previously unselected package libnetfilter-conntrack3:amd64.
Preparing to unpack .../36-libnetfilter-conntrack3_1.0.9-3_amd64.deb ...
Unpacking libnetfilter-conntrack3:amd64 (1.0.9-3) ...
Selecting previously unselected package iptables.
Preparing to unpack .../37-iptables_1.8.9-2_amd64.deb ...
Unpacking iptables (1.8.9-2) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../38-docker-ce_5%3a27.3.1-1~debian.12~bookworm_amd64.deb ...
Unpacking docker-ce (5:27.3.1-1~debian.12~bookworm) ...
Selecting previously unselected package docker-ce-rootless-extras.
Preparing to unpack .../39-docker-ce-rootless-extras_5%3a27.3.1-1~debian.12~bookworm_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:27.3.1-1~debian.12~bookworm) ...
Selecting previously unselected package docker-compose-plugin.
Preparing to unpack .../40-docker-compose-plugin_2.29.7-1~debian.12~bookworm_amd64.deb ...
Unpacking docker-compose-plugin (2.29.7-1~debian.12~bookworm) ...
Selecting previously unselected package libcurl3-gnutls:amd64.
Preparing to unpack .../41-libcurl3-gnutls_7.88.1-10+deb12u7_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.88.1-10+deb12u7) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../42-liberror-perl_0.17029-2_all.deb ...
Unpacking liberror-perl (0.17029-2) ...
Selecting previously unselected package git-man.
Preparing to unpack .../43-git-man_1%3a2.39.5-0+deb12u1_all.deb ...
Unpacking git-man (1:2.39.5-0+deb12u1) ...
Selecting previously unselected package git.
Preparing to unpack .../44-git_1%3a2.39.5-0+deb12u1_amd64.deb ...
Unpacking git (1:2.39.5-0+deb12u1) ...
Selecting previously unselected package libglib2.0-0:amd64.
Preparing to unpack .../45-libglib2.0-0_2.74.6-2+deb12u3_amd64.deb ...
Unpacking libglib2.0-0:amd64 (2.74.6-2+deb12u3) ...
Selecting previously unselected package libglib2.0-data.
Preparing to unpack .../46-libglib2.0-data_2.74.6-2+deb12u3_all.deb ...
Unpacking libglib2.0-data (2.74.6-2+deb12u3) ...
Selecting previously unselected package libltdl7:amd64.
Preparing to unpack .../47-libltdl7_2.4.7-7~deb12u1_amd64.deb ...
Unpacking libltdl7:amd64 (2.4.7-7~deb12u1) ...
Selecting previously unselected package libslirp0:amd64.
Preparing to unpack .../48-libslirp0_4.7.0-1_amd64.deb ...
Unpacking libslirp0:amd64 (4.7.0-1) ...
Selecting previously unselected package netcat-openbsd.
Preparing to unpack .../49-netcat-openbsd_1.219-1_amd64.deb ...
Unpacking netcat-openbsd (1.219-1) ...
Selecting previously unselected package patch.
Preparing to unpack .../50-patch_2.7.6-7_amd64.deb ...
Unpacking patch (2.7.6-7) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../51-publicsuffix_20230209.2326-1_all.deb ...
Unpacking publicsuffix (20230209.2326-1) ...
Selecting previously unselected package shared-mime-info.
Preparing to unpack .../52-shared-mime-info_2.2-1_amd64.deb ...
Unpacking shared-mime-info (2.2-1) ...
Selecting previously unselected package slirp4netns.
Preparing to unpack .../53-slirp4netns_1.2.0-1_amd64.deb ...
Unpacking slirp4netns (1.2.0-1) ...
Selecting previously unselected package xdg-user-dirs.
Preparing to unpack .../54-xdg-user-dirs_0.18-1_amd64.deb ...
Unpacking xdg-user-dirs (0.18-1) ...
Setting up liblmdb0:amd64 (0.9.24-1) ...
Setting up libpsl5:amd64 (0.21.2-1) ...
Setting up libicu72:amd64 (72.1-3) ...
Setting up apt-transport-https (2.6.1) ...
Setting up xdg-user-dirs (0.18-1) ...
Setting up libip6tc2:amd64 (1.8.9-2) ...
Setting up libglib2.0-0:amd64 (2.74.6-2+deb12u3) ...
No schema files found: doing nothing.
Setting up libmaxminddb0:amd64 (1.7.1-1) ...
Setting up libbrotli1:amd64 (1.0.9-2+b6) ...
Setting up libfstrm0:amd64 (0.6.1-1) ...
Setting up netcat-openbsd (1.219-1) ...
update-alternatives: using /bin/nc.openbsd to provide /bin/nc (nc) in auto mode
Setting up libnghttp2-14:amd64 (1.52.0-1+deb12u1) ...
Setting up less (590-2.1~deb12u2) ...
Setting up libnftnl11:amd64 (1.2.4-2) ...
Setting up libjemalloc2:amd64 (5.3.0-1) ...
Setting up libprotobuf-c1:amd64 (1.4.1-1+b1) ...
Setting up libjansson4:amd64 (2.14-2) ...
Setting up apparmor (3.0.8-3) ...
Created symlink /etc/systemd/system/sysinit.target.wants/apparmor.service → /lib/systemd/system/apparmor.service.
Reloading AppArmor profiles
Setting up docker-buildx-plugin (0.17.1-1~debian.12~bookworm) ...
Setting up perl-modules-5.36 (5.36.0-7+deb12u1) ...
Setting up libglib2.0-data (2.74.6-2+deb12u3) ...
Setting up libuv1:amd64 (1.44.2-1+deb12u1) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2+b2) ...
Setting up xz-utils (5.4.1-0.2) ...
update-alternatives: using /usr/bin/xz to provide /usr/bin/lzma (lzma) in auto mode
Setting up containerd.io (1.7.22-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up patch (2.7.6-7) ...
Setting up docker-compose-plugin (2.29.7-1~debian.12~bookworm) ...
Setting up libltdl7:amd64 (2.4.7-7~deb12u1) ...
Setting up docker-ce-cli (5:27.3.1-1~debian.12~bookworm) ...
Setting up libslirp0:amd64 (4.7.0-1) ...
Setting up pigz (2.6-1) ...
Setting up libnfnetlink0:amd64 (1.0.2-2) ...
Setting up git-man (1:2.39.5-0+deb12u1) ...
Setting up libssh2-1:amd64 (1.10.0-3+b1) ...
Setting up docker-ce-rootless-extras (5:27.3.1-1~debian.12~bookworm) ...
Setting up publicsuffix (20230209.2326-1) ...
Setting up libxml2:amd64 (2.9.14+dfsg-1.3~deb12u1) ...
Setting up libgdbm6:amd64 (1.23-3) ...
Setting up libnftables1:amd64 (1.0.6-2+deb12u2) ...
Setting up nftables (1.0.6-2+deb12u2) ...
Setting up bind9-libs:amd64 (1:9.18.28-1~deb12u2) ...
Setting up slirp4netns (1.2.0-1) ...
Setting up libcurl3-gnutls:amd64 (7.88.1-10+deb12u7) ...
Setting up shared-mime-info (2.2-1) ...
Setting up libgdbm-compat4:amd64 (1.23-3) ...
Setting up libnetfilter-conntrack3:amd64 (1.0.9-3) ...
Setting up libcurl4:amd64 (7.88.1-10+deb12u7) ...
Setting up curl (7.88.1-10+deb12u7) ...
Setting up bind9-host (1:9.18.28-1~deb12u2) ...
Setting up libperl5.36:amd64 (5.36.0-7+deb12u1) ...
Setting up iptables (1.8.9-2) ...
update-alternatives: using /usr/sbin/iptables-legacy to provide /usr/sbin/iptables (iptables) in auto mode
update-alternatives: using /usr/sbin/ip6tables-legacy to provide /usr/sbin/ip6tables (ip6tables) in auto mode
update-alternatives: using /usr/sbin/iptables-nft to provide /usr/sbin/iptables (iptables) in auto mode
update-alternatives: using /usr/sbin/ip6tables-nft to provide /usr/sbin/ip6tables (ip6tables) in auto mode
update-alternatives: using /usr/sbin/arptables-nft to provide /usr/sbin/arptables (arptables) in auto mode
update-alternatives: using /usr/sbin/ebtables-nft to provide /usr/sbin/ebtables (ebtables) in auto mode
Setting up perl (5.36.0-7+deb12u1) ...
Setting up docker-ce (5:27.3.1-1~debian.12~bookworm) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Setting up liberror-perl (0.17029-2) ...
Setting up git (1:2.39.5-0+deb12u1) ...
Processing triggers for libc-bin (2.36-9+deb12u8) ...
Cloud-init v. 22.4.2 finished at Tue, 05 Nov 2024 21:23:13 +0000. Datasource DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net]. Up 35.90 seconds
All in all, not bad. ~35s to having a running docker host. This is in a VM too and not on bare metal. I anticipate that be faster when we get to 25.04.
So i have 3 Jails and one with GPU passed through. Id like to keep using jails for the extra flexibility. I am guessing the best approach right now is to just stay on dragonfish and wait for Incus for a migration?
Jail work on EE, at least my one jail with GPU passthrough works fine
If you already have a jail built and working, you’ll be fine. See further up where I discuss the upgrade path. If you need to create new one’s, see the post after on how to do that.
There are some gotcha’s moving to EE, so read those as well. If you’re using nvidia, you need to install those drivers manually as well.
Sweet, I have bridged networking working now… 
More updated and slimmed down profile.
description: Docker Profile
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: default
type: disk
config:
cloud-init.user-data: |
#cloud-config
package_update: true
package_upgrade: true
apt:
preserve_sources_list: true
sources:
docker.list:
source: deb [arch=amd64] https://download.docker.com/linux/debian $RELEASE stable
keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
filename: docker.list
packages:
- apt-transport-https
- ca-certificates
- curl
- gpg
- host
- netcat-openbsd
- docker-ce
- docker-ce-cli
- containerd.io
- docker-buildx-plugin
- docker-compose-plugin
# create the docker group
groups:
- docker
# Add default auto created user to docker group
system_info:
default_user:
groups: [docker]
Mounts are working now, going to fix it so we can write to the directories…
root@docker-test:~# df -h
Filesystem Size Used Avail Use% Mounted on
pool/.ix-virt/containers/docker-test 47G 286M 47G 1% /
none 492K 4.0K 488K 1% /dev
udev 1.8G 0 1.8G 0% /dev/zfs
efivarfs 56K 29K 23K 57% /sys/firmware/efi/efivars
tmpfs 100K 0 100K 0% /dev/incus
pool/data/apps 47G 128K 47G 1% /mnt/data
pool/data/stacks 47G 128K 47G 1% /opt/stacks
tmpfs 100K 0 100K 0% /dev/.incus-mounts
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 784M 112K 784M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
TODO
NOTE: If you’re running this in a VM, you’re may run into Nesting issues. You need to configure your VM to nest from the host then it “should” work fine… The config options I added will load the needed kernel modules, sysctl values, and Incus instance configs when the containers attached to the
dockerprofile are powered on.
NOTE: This guide is intended for TrueNAS SCALE 25.04, BUT in theory the configs should work on any host that has
incusinstalled and running. Give a test and let me know how it goes.
This guide assumes you already have your bridges and networking configured and ready to go. I’m assuming you are using
br0for the bridge. If not, go ahead and adjust the config below with your specific interfaces.
It’s very straightforward process with this config. No need for any scripts at this point as cloud-init handles all the dirty work. The only thing missing at this point would be ZFS datasets or zvols for the instances and GPU support. I’ll likely be able to add this functionality to the docker cloud-init profile.
Getting started
timezone.docker-init.yaml:
description: Docker Profile
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: default
type: disk
data:
path: /mnt/data
source: /mnt/pool/data/apps
shift: true
type: disk
stacks:
path: /opt/stacks
source: /mnt/pool/data/stacks
shift: true
type: disk
config:
# Start instances on boot
boot.autostart: "true"
# Load needed kernel modules
linux.kernel_modules: br_netfilter
# Enable nesting
security.nesting: "true"
cloud-init.user-data: |
#cloud-config
# Enable docker sysctl values
write_files:
- path: /etc/sysctl.d/20-docker.conf
content: |
net.ipv4.conf.all.forwarding=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
# Set timezone
timezone: US/Eastern
# apt update and apt upgrade
package_update: true
package_upgrade: true
# Install apt repos and packages needed for docker
apt:
preserve_sources_list: true
sources:
docker.list:
source: deb [arch=amd64] https://download.docker.com/linux/debian $RELEASE stable
keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
filename: docker.list
packages:
- apt-transport-https
- ca-certificates
- curl
- gpg
- docker-ce
- docker-ce-cli
- containerd.io
- docker-buildx-plugin
- docker-compose-plugin
# create the docker group
groups:
- docker
# Add default auto created user to docker group
system_info:
default_user:
groups: [docker]
# Install dockge
runcmd:
- [ mkdir, -p, /opt/dockge ]
- [ cd, /opt/dockge ]
- [ curl, https://raw.githubusercontent.com/louislam/dockge/master/compose.yaml, --output, compose.yaml ]
- [ docker, compose, up, -d ]
profile into incus. Any future docker instances can now use this profile to create new instances moving forward.incus profile create docker < docker-init.yaml
docker1 is the instance name which will show up when you list the Incus instances by running incus ls.incus launch images:debian/bookworm/cloud -p docker docker1
incus exec docker1 -- bash
/etc/systemd/network/10-cloud-init-eth0.network
Output below. Modify your Address and Gateway.
[Match]
Name=eth0
[Network]
#DHCP=ipv4
DHCP=false
Address=192.168.0.30/24
Gateway=192.168.0.1
LinkLocalAddressing=no
LLDP=yes
EmitLLDP=customer-bridge
/etc/systemd/resolved.conf
Output below. Modify DNS to point to your DNS server and Domains to your search domain if needed. If you don’t need the search domains, just comment out the Domains line.
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it under the
# terms of the GNU Lesser General Public License as published by the Free
# Software Foundation; either version 2.1 of the License, or (at your option)
# any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the resolved.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
#
# See resolved.conf(5) for details.
[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
# Google: 8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
# Quad9: 9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
DNS=192.168.0.8
#FallbackDNS=
Domains=lan.domain.co
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=yes
#LLMNR=yes
#Cache=yes
#CacheFromLocalhost=no
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
reboot
eth0 which is the containers bridge to the TrueNAS hosts’ br0 interface. You will also have docker0 for Docker’s interface to eth0. Finally you’ll have a br-* interface that Dockge is using.incus ls
+-------------+---------+------------------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------------+---------+------------------------------+------+-----------+-----------+
| docker1 | RUNNING | 192.168.0.30 (eth0) | | CONTAINER | 0 |
| | | 172.18.0.1 (br-7e7ee82b01bf) | | | |
| | | 172.17.0.1 (docker0) | | | |
+-------------+---------+------------------------------+------+-----------+-----------+
Seriously, once you have the profile in place, it will take you less than a couple of minutes to have a brand new docker instance with Dockge running. This config can be modified to do whatever you want and you can create different profiles to spin up different instances instantly.
I’m going to break this out to it’s own thread since this is for the new jails moving forward.
EDIT: Created a new thread here. Hopefully we’ll get some community involvement and testing as we move forward.
explanation about incus
i didnt know what it was, had to look it up
can explain how u got namespacing working? i would like to know