There is popular opinion that there should be some capability to create and use system containers in TrueNAS Scale.
I propose that LXC/Incus would be good solution. LXC is the original lower-level library. Incus is higher level manager that uses LXC. Incus is fork of LXD.
Advantages:
LXC released in 2008, Incus in 2014. So both tested through time.
Actively developed. For orientation, LXC has 11 765 commits, Incus has 34 243 commits.
For comparison systemd-nspawn has only 1504 commits.
Currently systemd-nspawn needs Jailmaker wrapper to be more user-friendly. LXC/Incus are more user friendly already, so there would be no need to maintain wrapper.
Jailmaker already depends on LXC infrastructure for downloading images from images.linuxcontainers.org. So why not use LXC directly.
Incus specifically is able to manage both system containers (through LXC) and VMs (through QEMU). So it can be potentially used as manager for both. Explained here.
iX developers can choose either to use directly lower-level LXC runtime similar to Proxmox. Or they can choose Incus which is higher-level container manager with many useful features and itself uses LXC runtime.
Personally, I think using Incus would be the better choice. Containers can be complicated as they are, so using higher-level manager that simplifies operations seems like a good idea. It also has more features and easier management.
In summary, LXC/Incus would offer popular and tested solution for system containers compatible with ZFS without the need to maintain own wrapper like Jailmaker for nspawn.
“Wayback Machine Downloader” is a great command-line tool for archivists. (I personally use it myself to save archive.org snapshots to my own NAS server for posterity.)
However, because the main GitHub repo is essentially abandoned, in order to keep using it, one must pull it from a forked version and also apply a patch from yet another branch.[1]
With a “jail” that serves as its own operating system with a complete package management system, toolset, and root filesystem, this allows someone to use the “fixed” version of wayback_machine_downloader with the same level of freedom and flexibility as if they were running it on a FreeBSD/Linux distro on their PC.
Even if SCALE has first-class support for Docker, I would not be able to use wayback_machine_downloader to save archived snapshots. (Using VMs for lightweight command-line tools is utter overkill.)
Sometimes you just need a “jail” or “sandbox” with its own network stack on your NAS.
If I should write why are system containers good idea by themselves…
Some software just cant be run in Docker. Good example was mentioned above. But maybe author of the software just didnt package it into Docker. Some people can do it themselves but others cant.
With system containers it’s no problem. People can just install the software as if it was native.
You could say that VM can be used for this. But VM is often unnecessarily heavy. You also have to reserve RAM for VM which is not efficient use. On the other hand system containers are lean and use memory as any other linux process.
Of course, if you want to do something like run Windows you need VM.
I have read about two different types of containers. Cattle and pet containers.
Cattle containers are mainly those running on Docker. They are configured (often by author) to do one thing. You can easily just delete them and recreate. You dont really change them. You just use configuration option that author prepared for you. If something goes wrong you just save one config file and recreate whole container.
Pet containers are suited more for system containers like LXC. These types of containers you develop and change over time. Because they are system containers you can install more software, use them for multiple things and set them uniquely just how you need. They act basically like native OS. But because they are so flexible you cant really just delete them and recreate them like with Docker. They dont have just one config file you need to save. They are unique as a whole.
So I would say system containers cover important area that exists between Docker containers and full-blown VMs.
I’m not sure that LXC support should necessarily be tied to Incus.
Proxmox, which is referenced here, does not use Incus, they wrote their own tool called the Proxmox Container Toolkit. Proxmox Container Toolkit
The reason I bring this up is that Incus touts both container and virtual machine management. In terms of Virtual machines, TrueNAS already has it’s own virtual machine management system. Incus implementation would probably require a large amount of re-work on that front.
Just don’t want to throw the baby out with the bathwater.
Of course both options are possible. Either use LXC directly or use Incus. Whatever iX thinks is best for them.
Incus can manage both containers and VMs, yes. I thought Truenas could use it just for containers and don’t use it for VMs. If they decided to use it for VMs in the future they could but they don’t have to.
In this topic I mentioned both LXC and Incus because they are very close and developed by the same people. But I don’t mean that Incus needs to be used. As you say, if it’s better they can use just LXC.
I don’t know enough about this topic to comment on what’s right or what’s a better path. I just wanted to make sure that other lay-people who read the thread understand that the two are not necessarily tied together.
As S.Graber says incus is not production ready, using it in a sandbox/jail seems the best idea for now.
The SCALE UI for VM’s does not make the best use of libvritd/kvm/qemu as a hypervisor. For example, no option to choose machine type, no option to add a TPM or RNG device, no choice of video type, only qxl. Nor is it designed to be useable via a remote connection to the libvirtd instance running on SCALE.
These, and other short comings, can be side stepped if you are prepared to run a separate instance of libvirtd via jailmaker giving full access to virsh, virtinstall and remote conncetions.
I checked and on the contrary I see it’s regarded as production ready:
A big milestone for Incus was its 6.0 LTS release as that made it suitable for production users. Today we’re seeing around 40% of our users running the LTS release while the rest run the monthly releases.
The only thing iX would need to “support” is the GUI / middleware of creating, configuring, and managing system containers themselves. (A la “jails” under FreeBSD.)
There is zero expectation or support needed for individual applications, libraries, and “software updates”. Everything is managed by the user, and they have the onus to figure it out on their own.
It’s a win-win.
Just give us a fully-supported (+ middleware / GUI) “system container” feature, and leave the responsibility to the user to provide their own “software management and maintenance”.
Of course, I’d argue that a “rolling release distro” should be allowed, such as Arch Linux, since they are not limited by the “version freeze” that we see under Ubuntu, Debian, etc.
Such a feature would not conflict with whatever plans there are for Docker in SCALE.
By sandboxes I presume you mean systemd-nspawn containers.
As I mentioned above there are two levels of software we generally talk about:
runtime: LXC or systemd-nspawn
manager: Incus or Jailmaker
Container runtime is more low level and generally less user-friendly. But even then LXC seems more user friendly than nspawn. But this is subjective.
Container manager is basically wrapper of the runtime. Can do more things and more user friendly. For example Incus is manager for LXC containers but also for VMs and recently added early support for Docker containers. Jailmaker is manager for nspawn.
I understand that there would need to be good reason for going the LXC/Incus direction rather than just stay with systemd-nspawn. I dont mean that systemd-nspawn is bad, nor that there is some big reason why LXC/Incus should be chosen. To me it’s basically multiple of smaller reasons why I think LXC/Incus would be better.
LXC has bigger active community. When dealing with any bug or problem it would be easier to ask in LXC forum or IRC. Systemd only has mailing list.
Systemd-nspawn needs systemd running inside container. If you want to run distro without systemd like Alpine Linux you have a problem. With LXC you can run Alpine Linux no problem.
These two reasons seem to me like the most important ones. But I wrote more in my first post in this thread.
Only running Alpine Linux and other non-systemd distros is really a “use-case”. But I think it’s important one.
We need to go deeper
I heard about running Docker inside Incus but other way around is new for me.
Tbh, I thought if maybe we could just use Docker even for system containers.
I know that it’s made for application containers but I never tried how or if it works for system containers. But I must also admit that bending software so it does something it wasnt designed to do is often not good idea.
Either way first Docker needs to be done and tested before iX does anything with system containers. If they do anything at all I mean.
So we have a lot of time to discuss
And then if they choose nspawn or LXC… Both is good I guess. Even thou I like Incus for reasons mentioned above.
Btw, I tried asking Incus devs if they would consider adding systemd-nspawn runtime support. But as expected they would rather focus on LXC.
In one of my posts I said that Incus can manage even VMs so it could work for both containers and VMs.
But then I thought what about the other way around? So I looked what Truenas Scale use for managing VMs. It uses libvirt (which itself uses QEMU in backend).
So maybe iX devs could just reuse all the infrastructure they already made for managing VMs and just use it also for LXC containers.
I didnt look deep into this but on the first glance it looks like the most straight forward solution.
It definitely sounds easier than implementing Incus.
