Compile and install software in "Custom App" container?

Is it possible to compile and install software in a “Custom App” container? That is, without using any repository?

Depends on what you mean… You need to bootstrap a build environment from somewhere, but once you have a suitable userland with the right tools, you can do whatever you want.

I want to build ffmpeg (media transcoding software). In Core, I could just create an empty jail and go to town. FreeBSD was all set up in there.

I don’t really know anything about these containers and such. Am I understanding correctly that there is no OS installed by default in a container? Can you recommend a build environment I could bootstrap? I would need some kind of package manager to install dependencies.

That works well with FreeBSD, which is an OS - kernel, userland, all of it. With Linux, it’s quite a bit different.

In Docker parlance, you’d start with a base image, which is broadly similar to the image used to bootstrap a jail - but you have many to choose from (think along the lines of the various distros as a starting point).

I’ll leave the details of how that translates into TrueNAS Scale to those more familiar with the matter.

Short answer: no, this really isn’t something that fits with the “Custom App.” However, it sounds like exactly what the new “sandbox” feature of Dragonfish was made for.


Thanks for the tip @dan . I’m getting dizzy going around all these learning curves :dizzy_face:

Not a problem. I haven’t used them, but I understand they’re supposed to be pretty much equivalent to jails.

iX has a great little startup guide:

After installing jailmaker in a dataset and setting up the post-init startup, it refers you to the jailmaker project on github GitHub - Jip-Hop/jailmaker: Persistent Linux 'jails' on TrueNAS SCALE to install software (docker-compose, portainer, podman, etc.) with full access to all files via bind mounts thanks to systemd-nspawn!
Things get a bit rocky at that point, but I’m muddling through.

Here are the available images. Can anyone suggest which might be best? They suggest sticking “with common systemd based distros (Debian, Ubuntu, Arch Linux…)” I suppose if any would serve as well, I might as well use what’s on SCALE, but I don’t know which that is.

sudo ./ images
Downloading the image index

almalinux        8           amd64  default  20240425_23:08
almalinux        9           amd64  default  20240425_23:08
alt              Sisyphus    amd64  default  20240422_01:17
alt              p10         amd64  default  20240422_01:17
alt              p9          amd64  default  20240422_01:17
archlinux        current     amd64  default  20240426_04:18
centos           7           amd64  default  20240426_07:08
centos           8-Stream    amd64  default  20240426_07:08
centos           9-Stream    amd64  default  20240426_07:08
debian           bookworm    amd64  default  20240424_05:24
debian           bullseye    amd64  default  20240424_05:24
debian           buster      amd64  default  20240424_05:24
debian           trixie      amd64  default  20240424_05:24
fedora           38          amd64  default  20240421_20:33
fedora           39          amd64  default  20240421_20:33
kali             current     amd64  default  20240426_17:14
mint             ulyana      amd64  default  20240426_08:51
mint             ulyssa      amd64  default  20240426_08:51
mint             uma         amd64  default  20240426_08:51
mint             una         amd64  default  20240426_08:51
mint             vanessa     amd64  default  20240426_08:51
mint             vera        amd64  default  20240426_08:51
mint             victoria    amd64  default  20240426_08:51
nixos            23.11       amd64  default  20240426_01:12
nixos            unstable    amd64  default  20240426_01:00
openeuler        20.03       amd64  default  20240425_15:48
openeuler        22.03       amd64  default  20240425_15:48
openeuler        23.09       amd64  default  20240425_15:48
opensuse         15.4        amd64  default  20240426_04:20
opensuse         15.5        amd64  default  20240426_04:20
opensuse         tumbleweed  amd64  default  20240426_04:20
oracle           7           amd64  default  20240426_07:46
oracle           8           amd64  default  20240426_07:46
oracle           9           amd64  default  20240426_07:46
rockylinux       8           amd64  default  20240426_02:06
rockylinux       9           amd64  default  20240426_02:06
slackware        15.0        amd64  default  20240425_23:08
slackware        current     amd64  default  20240425_23:08
springdalelinux  7           amd64  default  20240426_06:38
springdalelinux  8           amd64  default  20240426_06:38
springdalelinux  9           amd64  default  20240426_06:38
ubuntu           bionic      amd64  default  20240426_07:42
ubuntu           focal       amd64  default  20240426_07:42
ubuntu           jammy       amd64  default  20240426_07:42
ubuntu           lunar       amd64  default  20240426_07:42
ubuntu           mantic      amd64  default  20240426_07:42
ubuntu           noble       amd64  default  20240426_07:42
ubuntu           xenial      amd64  default  20240426_07:42

I’d recommend Debian 12 “Bookworm” to match what is running on the host. But Ubuntu can work just as well if that is your cup of tea.

Additionally there are lots of good guides on how to build docker images from scratch. I took a few hours and figured it out a few years back. Its quite straightforward, and is how I originally replaced all my Jails with my own recipe for the containers and the software I wanted inside them.


Thanks kris. Just for my edification, when you say build docker image from scratch, would that be starting with a custom app, or sandbox/jailmaker, or something else altogether?

But for now, I’m lined up with the jailmaker runway and on the glide path, so I guess I’ll continue on and see how it goes.

To build docker images, you’d probably end up doing it in a jail anyway.

Jailmaker makes it so you can install another Linux distribution on top of the active Linux kernel used by Scale.

After that, the jail is just Linux.

Is there any rule of thumb for matching the host’s kernel (or being “close enough” to it)? In the same way we had to avoid using a FreeBSD stable version (for the jail) that is newer than the host’s kernel?

A systemd-nspawn container (sandboxes/jails) uses the host’s kernel.

TrueNAS Scale is based on Debian user land.

That’s the same with FreeBSD jails.

My question is in regards to avoiding a mismatch between a jail (sandbox) “release”, and the host’s kernel.

For FreeBSD, it’s advised against running a jail with a release that is newer than the host’s major version release, since there could be major changes to the ABI.

For example: Packages and libraries built for FreeBSD 14.0 may not run correctly with a host’s 13.x kernel.

I’m curious if this same consideration applies to Linux and sandboxes (“systemd-nspawn”)?

It’s kinda funny, most things seem to work fine as long as you’re not going to some extreme like running cutting-edge packages with a 3.x kernel. The kernel doesn’t break userland stuff very often (to a fault, even), and most software seems pretty conservative when it comes to adopting new syscalls.

1 Like

In theory the same does apply.

But remember docker is a thing. A big thing. And it is essentially the same, run a user land distro on whatever the hosts kernel is.

So, in practise it works well.

Debian bookworm is a good choice I believe. @kris would have a better understanding and he rexcomended that too

1 Like

It’s not clear how one would “upgrade” the sandbox’s release, in the way one could upgrade a jail’s base release.

iocage / jails did a lot under-the-hood, even leveraging ZFS snapshots.

I’ve actually made my jail disposable.

I just re-run the config.

Meanwhile, inside the jail I can just run “apt upgrade”

There is no cloning magic implemented

Debian bookworm is a good choice in general, since you are running Debian on top of Debian. I prefer consistency here because of my own BSD roots. :slight_smile: But in reality any other systemd-based distro will work just fine with nspawn.

Just as a general primer. Those of us coming from a Jails world on BSD usually followed some rules which are similar but not exactly the same on Linux. On BSD you typically want to always run a jail version which is matching the host version. (I.E. 13.3 jails on 13.3 hosts). You can often safely run older jails on a new host, (I.E. 13.1 Jail on 13.3 Hosts), but depending on what packages you are running you can run into periodic issues with ABI compatibility in that scenario if you are unlucky. FreeBSD does its best to maintain ABI compat in a major version, but issues still sometimes crop up there, so generally you want to always keep like versions running together.

On the Linux you can get away with a bit more. The exponentially larger container ecosystem on Linux would collapse if any ABI issues cropped up in between kernel versions. So they got religion about maintaining ABI compat very early, which is why you can pull and run just about any docker/container image on just about any version of a Linux host these days. Somebody can check my memory on this, but I think going all the way back to the 2.2 kernel days even (It’s been a while). Linus has famously rejected patches which don’t maintain bug-for-bug compatibility between syscalls across versions of Linux, to give you an idea of how hard they follow these rules.

Systemd-nspawn containers (I.E. Sandboxes/Jails in our context) are based on this. They can run just about any Linux inside the container from an ABI compatibility standpoint. The only real limitation there is that they leverage some newer systemd calls for doing fancier tricks and magic related to networking and other service management inside the container from the host directly. This of course requires that you run containers which also leverage systemd internally, which is why the recommendation for any systemd-based distro from that list posted previously.

For those curious what goes on behind the scenes, here’s some good reference materials and sites to browse:


In this context, I think this would be a useful thing: