Beyond that, even without exposing the “hidden” .ix-virt dataset and its children in the GUI, it would make sense to allow configuring the dataset properties for each new container. This will apply to its specific dataset, of course.
Here’s an example of creating a new jail: This will never happen for iocage on Core. Maybe for Incus on SCALE…?
A jail/container and its dataset (for the jail’s root filesystem) is an important relationship.
This is grounds for libel.
I have always been consistent in making it clear that I use ZSTD-9 for my jails and other general purpose datasets.
I would really want the UI to support this, otherwise I would find it quite confusing to backup or just manage the volumes when they are hidden from the user and managed by TrueNAS without really telling you what’s going on.
Similar to how apps are split into the configs in ix-apps and the volumes in an arbitrary dataset defined and managed by the user.
I dont think this will be done in Fangtooth.
But someone would just need to post feature request and it shouldnt be much work to add this functionality in next release.
How is it currently planned to manage / backup the disks? Backing it up with the entire pool? I think the lack of transparency in the UI is hurting here.
I dont know how iX plans to solve this.
But yes, you can snapshot the entire pool that includes .ix-virt which contains all instances.
But I understand that backup instances individually would be much better option.
Incus of course can do this, its just question of exposing it in middleware and UI. How to back up instances - Incus documentation
But since Truenas itself manages ZFS I think it would take some more work to make sure it plays nicely with Incus which also can manage its ZFS volumes.
Did you test clustering with a separate incus node?
This would be a really awesome feature to merge my Proxmox cluster and the two truenas nodes (die to very different hardware but some failover scenarios)
I sometimes see people confused about what components deal with virtualization and what exactly was changed in Fangtooth. So this is simple hierarchy from top level to bottom.
System containers/Jails: TrueNAS → Incus → LXC
VM before: TrueNAS → libvirt → QEMU → KVM
VM after: TrueNAS → Incus → QEMU → KVM
For VMs, is the changeover from libvirt to incus mostly transparent to users? Is the web interface mostly staying the same?
As much as I love TrueNAS, I’ve just never liked the way VMs are configured through the web UI compared to other solutions like Proxmox or XCP-NG. Seeing the posts here whereby we can work directly with Incus in an easy manner on the command line looks promising.
I’m guessing though that if you create a VM with Incus via CLI, then the TrueNAS middleware doesn’t know about it, right?
Similar UI. It does know about containers and VM’s created via this method. Only issue right now is persistence. We have to recover after updates to TNS. I’m hoping this gets fixed. It’s not a major issue, but definitely annoying.
Config managed by middleware in boot volume can be easily recreated and fixed if something is wrong. Importing Truenas config will always result in the same machine configuration.
If you have something in separate persistent volume and it gets broken its more work to fix it. And middleware doesnt work nicely with arbitrary user-made changes.
How to improve it…
One thing could be custom instance configs in UI. This would be basically the same like direct YAML config in Docker apps.
But I am not sure about putting whole Incus dir on persistent volume. Especially the database.
Why not? They do it with the System Dataset. Move it to the pool of choice. I personally have that dataset running on my main drive SSD, but I could also run that out of my rust pool as well.
Oh, I didnt know that. I thought it needs to be on separate pool.
Ok now I see.
Well, I dont know then. I would need to look into it more. But without knowing the design purpose and if configs need to be on boot-pooI cant really tell much.
One thing I can think of. Maybe middleware expect that upon update it can just create new dataset and do configuration from scratch in new boot environment. If you put some config outside of it then it stays the same across updates and maybe that can be problem? Just guessing.
If all on Incus would be in separate dataset unaffected by updates I guess middleware would need to be designed to deal with it appropriately.
Not saying its impossible, but maybe more work than just moving it.
I think only way for this is feature request and let iX decide if its doable and explain potential problems.
Really, if we just had an option where to store the container/virtualization datasets and it would look at that dataset everytime via the INCUS_DIR env var, we’d be golden because middlewared would know about it after each update.
I just created a dataset, then made an Incus container with that dataset as a Disk, set the owner of the dataset on the TrueNAS side so it maps to root:root in Incus lxc, and finally set Docker data-root with daemon.json to my dataset on /mnt/docker, and installed Docker CE.
And it all just works - with minimal setup complexity. Didn’t have to change Incus security parameters, either.
What flexibility are you gaining with your setup, over mounting a dataset as an Incus Disk?
I would need to look at this configuration, I didn’t look into it.
If Incus is mounting a vdisk on a dataset, which may be what is happening in this configuration, there is an inherent risk of file corruption of the vdisk. This is fine, many virtualization solutions do this like Xen, VMware, KVM, etc, but I prefer to stick with host mounts because of easier snapshots and restores. Adding an vdisk adds an additional layer of complexity as well, especially when related to data recovery and access to such data.
Again, I would need to look into how you have it configured and how this particular configuration works in Incus.
Nope no vdisk, the Disk for incus is like a bind mount plus uid/gid mapping, that’s why on TrueNAS cli chown 2147000001:2147000001 /mnt/<pool>/<dataset> … 1000 more would get you 1000 inside lxc / Incus container.
If I understand what you mean by vdisk … a single file that holds a file system, right? That’s not what this is. All created files are visible as files on the TrueNAS side.
The OS volume is on .ix-virt, and the mounted dataset - “Disk” in TrueNAS UI parlance - is on /mnt/docker inside the container and gets used as the Docker data-root, via /etc/docker/daemon.json inside the container.
Docker sees that its data-root is zfs, and by default uses the overlay2 driver for it. I didn’t muck with the storage driver.
On my particular hardware that happens to be a SATA SSD mirror for .ix-virt, and an Intel P3600 NVMe for the mount that becomes the Docker data-root.
Pretty basic. I’m still wondering what the trade offs if any are though. I am happy you are questioning the idea.
This wouldn’t be required when using the shift=true option when mounting the drive right? Would love for the UI to give us an option for this.
Edit:
Tried it in my nightly VM and adding disks to containers via the command above even shows in the UI. The permissions seem to somewhat work but I can’t for example touch a. I get an permission denied error even though the user owns the directory according to the container. Probably messed up somewhere.
Yes they see shift=true as a security risk. Fair enough. So yes, right now the dataset needs to have the right owner. I think you could also set that from UI, and, yes, it’s not extremely intuitive what the offset is.
As Incus is an experimental feature, I have some patience here for the docs to get better.
An amazing feature would be for TrueNAS middleware to just apply the owner offset when adding a dataset to Incus as a disk … look at current owner and group and add the offset, then chown accordingly. As that’s clearly the user intent.
I am not sure how all of this supposed to work.
2147000001 UID is out of Truenas range in GUI. So, you can’t create a user and give him rights to a dataset from GUI. So, we need to create a dataset with open persmissions or set permissions to a dataset from shell. Correct me if I am wrong.
