Linux Jails (containers/vms) with Incus

Right.

Updating the license would allow Incus to fully integrate any code from truenas_incus_ctl in the future

The Storage Driver and the Incus support in Fangtooth are really separate things.

The potential was amazing once the Incus version in TrueNAS had the Storage Driver…

Meanwhile, if you haven’t already, checkout this cool little tool:

1 Like

I updated the OP Known Issues in regards to the Nvidia GPU issue with the nvidia-container-toolkit as resolved in 25.04.2.

Just spent a few hours reading about the fate of incus in 25.10.

I’ve been on Scale for around 4-5 years, since before the official release of Angelfish.

It has been a roller-coast for the last 4-5 years, having to rework my homelab apps/VMs/containers many times over for each breaking release. I finally found a sense of stability with Jailmaker, and a sense of optimism with Incus. I really thought incus would bring an end to breakages and give me control of my apps/containers.

I’m ready to alight from the roller-coaster and treat TN purely as a storage solution. I’ve been running proxmox on lightweight hardware for a few years with zero issues. I’ll see if I can migrate to proxmox+TN on my nas for my storage+gpu workloads.

Huge shout-out and thanks to @dasunsrule32 for your work on incus, you have clearly put in a lot of work, and it would be heartbreaking to hear about the TNCE direction change.

In the same fashion, @Stux your work on Jailmaker and providing docker solutions has been incredible.

5 Likes

:point_up:
Vox populi, vox dei.

Yep, team Jailmaker here, still running and working with no issue. I wonder if it’s possible to just keep it this way. No need for UI work!

Wait a minute. Are you saying that you can install, manage, and update an operating system with its own package management, user tools, software, filesystem, and network stack, giving you full control without affecting the host NAS and without resorting to VMs, while keeping overhead at nearly zero? :exploding_head:

Finally! I can already think of so many benefits of doing that.

1 Like

Been doing it for over a year with jailmaker :sweat_smile: i run on jail with all my whole docker stack running inside it. survived 3 truenas updates without problems…

1 Like

It looks like I’m going to have to rollback to jailmaker at some point or move on with my own build with Incus/Proxmox. I haven’t decided yet, but I’m leaving heavily towards my own build.

Yeah, that’s what Incus brings, just like jailmaker. I don’t need a GUI, it’s nice to have, but not needed. I just want to know that I’ll have support for what I need. I do like the extra orchestration that Incus brings though vs one off instances in jailmaker.

I can’t afford separate storage and compute boxes, that’s one of the reasons I loved SCALE, but also I could scale out over time if I could afford more boxes.

1 Like

This ā€œjailsā€ thing… I’ll need to look into it. I could really use it for my NAS!

What a simple yet great concept.

1 Like

cough cough stux video incomming

2 Likes

I’ve failed at keeping up with all the truenas direction changes lately.

If I understand, incus is out. What’s going to replace it for lxc like containers/jails? I’ve been sticking with jailmaker while things shake out but it has certainly had it’s issues (such as how it likes to mount the entire /usr/lib/x86_64-linux-gnu folder as read only if you enable nvidia passthrough - which in turn breaks any ability to do apt-get update/installs).

With extensive trial and error, I did figure out I could change one line in the most recent jlmkr.py script and stop it from mounting the entire /usr/lib/x86_64-linux-gnu folder as read only … and that ā€˜seems’ to work for me right now. No idea if I’m causing myself a future problem or not.

For reference, the line I changed is
line 437: library_folders = set(str(Path(x).parent) for x in nvidia_libraries)
changed to: library_folders = set(str(x) for x in nvidia_libraries)

Do I just stick with this solution for now? Is there a better option?

Stick with whatever you’re on for now. From what I understand, is they are looking to migrate to straight LXC for containers and remove Incus.

Also, look at the jailmaker thread. I posted about the library stuff on nvidia gpu passthrough over there.

I believe you: I can also say that wasn’t the case for me.

K3s, apps worked

Docker migration was seamless, apps worked. Including my custom one.

VMs are libvirt and remain libvirt - no need to take a detour into ā€œexperimentalā€. Also I don’t run Home Assistant so I don’t actually need any VMs.

Incus LXC I played with for the giggles, but, ā€œexperimentalā€ - I may try again with 25.10.

My experience with SCALE and then CE has been that apps continue working with each upgrade.

2 Likes

Totally agree with @yorick. I just now updated from 24.10.3 to 25.04.2.1 and I dont have any problems with VMs or anything else.
I believe anyone who did it the same would also be ok.

1 Like

FWIW, I suspect that Incus could be installed in a dataset and launched fairly easily, similar to how jailmaker is installed.

Its only requirement is that the idmap range be created.

3 Likes

Yeah for sure, anything is possible. I’m not sure I have the bandwidth to do that right now. Work is insanely busy right now.

I just came to say goodbye. It’s been a fun ride. I started with FreeNAS 8.3 and my server was updated and upgraded to 25.04.2.1.

I have finished my migration to Proxmox, just a few more details to figure out, but overall it was a pretty smooth transition based on the migration plan I wrote up.

Maybe see some of you on the other side. Good look out there!

3 Likes