Is it possible to compile and install software in a “Custom App” container? That is, without using any repository?
Depends on what you mean… You need to bootstrap a build environment from somewhere, but once you have a suitable userland with the right tools, you can do whatever you want.
I want to build ffmpeg (media transcoding software). In Core, I could just create an empty jail and go to town. FreeBSD was all set up in there.
I don’t really know anything about these containers and such. Am I understanding correctly that there is no OS installed by default in a container? Can you recommend a build environment I could bootstrap? I would need some kind of package manager to install dependencies.
That works well with FreeBSD, which is an OS - kernel, userland, all of it. With Linux, it’s quite a bit different.
In Docker parlance, you’d start with a base image, which is broadly similar to the image used to bootstrap a jail - but you have many to choose from (think along the lines of the various distros as a starting point).
I’ll leave the details of how that translates into TrueNAS Scale to those more familiar with the matter.
Short answer: no, this really isn’t something that fits with the “Custom App.” However, it sounds like exactly what the new “sandbox” feature of Dragonfish was made for.
2 Likes
Not a problem. I haven’t used them, but I understand they’re supposed to be pretty much equivalent to jails.
iX has a great little startup guide:
After installing jailmaker in a dataset and setting up the post-init startup, it refers you to the jailmaker project on github GitHub - Jip-Hop/jailmaker: Persistent Linux 'jails' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts thanks to systemd-nspawn!
Things get a bit rocky at that point, but I’m muddling through.
Here are the available images. Can anyone suggest which might be best? They suggest sticking “with common systemd based distros (Debian, Ubuntu, Arch Linux…)” I suppose if any would serve as well, I might as well use what’s on SCALE, but I don’t know which that is.
sudo ./jlmkr.py images
Downloading the image index
---
DIST RELEASE ARCH VARIANT BUILD
---
almalinux 8 amd64 default 20240425_23:08
almalinux 9 amd64 default 20240425_23:08
alt Sisyphus amd64 default 20240422_01:17
alt p10 amd64 default 20240422_01:17
alt p9 amd64 default 20240422_01:17
archlinux current amd64 default 20240426_04:18
centos 7 amd64 default 20240426_07:08
centos 8-Stream amd64 default 20240426_07:08
centos 9-Stream amd64 default 20240426_07:08
debian bookworm amd64 default 20240424_05:24
debian bullseye amd64 default 20240424_05:24
debian buster amd64 default 20240424_05:24
debian trixie amd64 default 20240424_05:24
fedora 38 amd64 default 20240421_20:33
fedora 39 amd64 default 20240421_20:33
kali current amd64 default 20240426_17:14
mint ulyana amd64 default 20240426_08:51
mint ulyssa amd64 default 20240426_08:51
mint uma amd64 default 20240426_08:51
mint una amd64 default 20240426_08:51
mint vanessa amd64 default 20240426_08:51
mint vera amd64 default 20240426_08:51
mint victoria amd64 default 20240426_08:51
nixos 23.11 amd64 default 20240426_01:12
nixos unstable amd64 default 20240426_01:00
openeuler 20.03 amd64 default 20240425_15:48
openeuler 22.03 amd64 default 20240425_15:48
openeuler 23.09 amd64 default 20240425_15:48
opensuse 15.4 amd64 default 20240426_04:20
opensuse 15.5 amd64 default 20240426_04:20
opensuse tumbleweed amd64 default 20240426_04:20
oracle 7 amd64 default 20240426_07:46
oracle 8 amd64 default 20240426_07:46
oracle 9 amd64 default 20240426_07:46
rockylinux 8 amd64 default 20240426_02:06
rockylinux 9 amd64 default 20240426_02:06
slackware 15.0 amd64 default 20240425_23:08
slackware current amd64 default 20240425_23:08
springdalelinux 7 amd64 default 20240426_06:38
springdalelinux 8 amd64 default 20240426_06:38
springdalelinux 9 amd64 default 20240426_06:38
ubuntu bionic amd64 default 20240426_07:42
ubuntu focal amd64 default 20240426_07:42
ubuntu jammy amd64 default 20240426_07:42
ubuntu lunar amd64 default 20240426_07:42
ubuntu mantic amd64 default 20240426_07:42
ubuntu noble amd64 default 20240426_07:42
ubuntu xenial amd64 default 20240426_07:42
I’d recommend Debian 12 “Bookworm” to match what is running on the host. But Ubuntu can work just as well if that is your cup of tea.
Additionally there are lots of good guides on how to build docker images from scratch. I took a few hours and figured it out a few years back. Its quite straightforward, and is how I originally replaced all my Jails with my own recipe for the containers and the software I wanted inside them.
2 Likes
Thanks kris. Just for my edification, when you say build docker image from scratch, would that be starting with a custom app, or sandbox/jailmaker, or something else altogether?
But for now, I’m lined up with the jailmaker runway and on the glide path, so I guess I’ll continue on and see how it goes.
To build docker images, you’d probably end up doing it in a jail anyway.
Jailmaker makes it so you can install another Linux distribution on top of the active Linux kernel used by Scale.
After that, the jail is just Linux.
Is there any rule of thumb for matching the host’s kernel (or being “close enough” to it)? In the same way we had to avoid using a FreeBSD stable version (for the jail) that is newer than the host’s kernel?
A systemd-nspawn container (sandboxes/jails) uses the host’s kernel.
TrueNAS Scale is based on Debian user land.
That’s the same with FreeBSD jails.
My question is in regards to avoiding a mismatch between a jail (sandbox) “release”, and the host’s kernel.
For FreeBSD, it’s advised against running a jail with a release that is newer than the host’s major version release, since there could be major changes to the ABI.
For example: Packages and libraries built for FreeBSD 14.0 may not run correctly with a host’s 13.x kernel.
I’m curious if this same consideration applies to Linux and sandboxes (“systemd-nspawn”)?
It’s kinda funny, most things seem to work fine as long as you’re not going to some extreme like running cutting-edge packages with a 3.x kernel. The kernel doesn’t break userland stuff very often (to a fault, even), and most software seems pretty conservative when it comes to adopting new syscalls.
1 Like
In theory the same does apply.
But remember docker is a thing. A big thing. And it is essentially the same, run a user land distro on whatever the hosts kernel is.
So, in practise it works well.
Debian bookworm is a good choice I believe. @kris would have a better understanding and he rexcomended that too
1 Like
It’s not clear how one would “upgrade” the sandbox’s release, in the way one could upgrade a jail’s base release.
iocage / jails did a lot under-the-hood, even leveraging ZFS snapshots.
I’ve actually made my jail disposable.
I just re-run the config.
Meanwhile, inside the jail I can just run “apt upgrade”
There is no cloning magic implemented
Debian bookworm is a good choice in general, since you are running Debian on top of Debian. I prefer consistency here because of my own BSD roots. 
But in reality any other systemd-based distro will work just fine with nspawn.
Just as a general primer. Those of us coming from a Jails world on BSD usually followed some rules which are similar but not exactly the same on Linux. On BSD you typically want to always run a jail version which is matching the host version. (I.E. 13.3 jails on 13.3 hosts). You can often safely run older jails on a new host, (I.E. 13.1 Jail on 13.3 Hosts), but depending on what packages you are running you can run into periodic issues with ABI compatibility in that scenario if you are unlucky. FreeBSD does its best to maintain ABI compat in a major version, but issues still sometimes crop up there, so generally you want to always keep like versions running together.
On the Linux you can get away with a bit more. The exponentially larger container ecosystem on Linux would collapse if any ABI issues cropped up in between kernel versions. So they got religion about maintaining ABI compat very early, which is why you can pull and run just about any docker/container image on just about any version of a Linux host these days. Somebody can check my memory on this, but I think going all the way back to the 2.2 kernel days even (It’s been a while). Linus has famously rejected patches which don’t maintain bug-for-bug compatibility between syscalls across versions of Linux, to give you an idea of how hard they follow these rules.
Systemd-nspawn containers (I.E. Sandboxes/Jails in our context) are based on this. They can run just about any Linux inside the container from an ABI compatibility standpoint. The only real limitation there is that they leverage some newer systemd calls for doing fancier tricks and magic related to networking and other service management inside the container from the host directly. This of course requires that you run containers which also leverage systemd internally, which is why the recommendation for any systemd-based distro from that list posted previously.
For those curious what goes on behind the scenes, here’s some good reference materials and sites to browse:
https://wiki.debian.org/nspawn
https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html
2 Likes
In this context, I think this would be a useful thing:
2 Likes