Meanwhile, in TrueNAS Land, go for your life. And if it breaks, rollback.
1 Like
Yeah, the OP has the directions in setting up Docker installed in LXC containers. Right now there are quirks, so make sure to read the main post and the rest of thread of you havenāt already.
Iāll modify the OP to include the issues with custom settings not persisting and how to recover between updates.
You can also just run it with the default TrueNAS settings and get persistence, but you lose a lot of flexibility.
I tried setting the INCUS_DIR env var in /etc/environment. This successfully moves Incus dir to specified location but it also breaks Truenas middleware.
Truenas would need to officially support custom Incus location by setting this env var and make middleware compatible with custom location.
Right now it seems the middleware is hardcoded for the default Incus location.
So while this is the case I think moving Incus location isnt good idea.
I think only way to persist custom Incus changes is to backup and restore /var/lib/incus/ after every update.
1 Like
Yeah, added it as a known issue in the main OP on this thread.
Based on what Iāve read in the Incus documentation that containers are unprivileged by default, am I right to assume the PoC example in the OP is created as an unprivileged container? I didnāt see any args in the init that indicated otherwise, so just thought Iād check 
Correct, itās unprivileged.
So I tried creating and installing Windows 10 VM in last Truenas nightly and it works.
Its still quite bumpy but its doable.
The Windows ISO needs to be repacked by distrobuilder for virtio drivers.
You need to create and add zvol so you have disk to install it on, but thats doable in UI.
You need to use VNC client to connect to graphical output.
And dont forget to āpress any keyā during boot to boot from cd-rom.
Only thing you need to do in shell is adding boot.priority of the disk on which is Windows installed in Incus config after the install.
By default it doesnt have boot.priority set and Incus has in its default profile disk called root. This causes that it tries to boot from the root disk and it fails. (At least thats what I think happens)
I was worried it will not work because I was doing nested virtualization. Physical Truenas ā Truenas Nightly VM ā Windows VM
1 Like
Bunch of rough edges there still, some work incoming to make it a bit more refined.
2 Likes
Thanks for that! Itās actually pretty good in itās current state.
I think the main things for me are:
- Persistence of custom settings
- OCI support in incus
Itās been fun playing with it as itās maturing.
Just donāt upgrade your pool 
2 Likes
Where are the containers disks stored?
I have selected a pool and when I create a new container it asks me to assign disks to it. When I donāt assign any disks it still has some kind of persistent storage. Where does that get stored?
In the webui it tells me that something now takes up storage in the pool but when going through the cli into the dataset I canāt find any directories / files not even hidden ones. I would like to limit how much storage one container can take like how it works in proxmox. My guess is also that the āstorageā tab in the container creation menu is like the mount points in proxmox but Iām not sure about that. Thatās probably something future docs should clarify.
Also some docs on how the incus stuff gets stored would be nice for the final release. Similar how the docs for the apps tell you that all docker relevant stuff getās put into a hidden ix-apps dataset inside the selected apps pool.
If you run incus storage ls it should dump the default path that is used to store the containers/vmās. If you follow the guide in the OP, they will be stored in /mnt/pool/jails.
Try using zfs list on your host. You will see all zfs volumes.
.ix-virt is the one where Incus volumes are stored.
To see how its set up inside Incus use these commands on host (you need sudo):
incus storage list you will see there is default pool
incus storage show default will show you that this default pool is mapped to .ix-virt on host
incus storage volume list default will show you what Incus volumes are inside the default pool
What causes that there is some persistent storage by default?
incus config show instance-name you see that instances use default profile, which means in addition to instance-specific config they also use profile config.
incus profile show default you see that this profile defines some things, including root device which is disk on default pool.
If you take all this info together it basically means:
All instances have access to default pool which is their default storage space.
This default pool is mapped to .ix-virt pool on host.
Each instance creates its own volume in this pool and stored its data inside it.
That should be all, hope I didnt say anything wrong.
Incus is very powerful and can do a lot of things. I think to understand it better its good to read its docs.
2 Likes
Thanks for the infos!
Iām not sure however how you would go to change where the default volume gets stored. In proxmox the creation wizard asks me in which location the volume should get stored and how big the containers volume can have as maximum.
Iām asking this because I would like to backup my container disks in a similar way I have done my VM disks so far.
I think I should have called .ix-virt a dataset, not pool. For Incus its location of default pool and for Truenas its just ZFS dataset.
But you can choose where .ix-virt gets stored in UI under Global Settings. There you can choose on which ZFS pool it gets stored.
If you mean to specify directly where each instance gets stored, Incus itself can do it: How to create an instance in a specific storage pool - Incus documentation
You need to first create storage pool somewhere and then set root disk device so it points to that.
But its not currently supported by Truenas UI and will break on update so I think you shouldnt do that.
Iām curious about point #2 on this post.
What would happen if you manually create an .ix-virt dataset in advance, then set its compression and recordsize policy?
Will it be overridden when you set a ādefault poolā for Incus for the first time?
Do child datasets (i.e, each āLinux containerā) inherit the .ix-virt parent dataset properties?
EDIT: I see @Stux is replying. I donāt often like to gamble with my money, but whatās the over-under of the next reply being a link to a YouTube video? 
EDIT 2: I just lost $500 USD by placing a bet that the next reply would be a YouTube video. 
They do. Incus sets a handful of properties when it creates the sub data sets.
TrueNAS wonāt let you manipulate the dataset in the GUI, but you should be able to in the shell.
Additionally, incus stores the various types of ācontainersā that it allocates in a set of sub datasets that it creates in the ix-virt dataset, thus providing a good place to set inheritable defaults if you would like.
1 Like
Not a fan of this. I know that āCore is not SCALEā, but to be able to manage the iocage dataset in Coreās GUI was very useful and convenient.
I really donāt like going around the middlewareās back when it can be avoided.
So after the first-time āinitiationā of setting a ādefaultā pool for Incus/containers, I can change the properties of .ix-virt before creating even a single Linux container?
This will then remain? TrueNAS will not override the properties the next time I create a new Linux container?
I know that sometimes doing certain actions in the GUI could trigger a āsanity checkā by the middleware to ācorrectā unexpected settings. Not sure if it applies to SCALEās implementation of Incus.
What about .ix-virt specifically? Does it inherit encryption of the poolās root dataset?
Unlike the other properties (compression, recordsize), you cannot go back and change a datasetās encryption.
Fairly certain it wonāt on the sub datasets in ix-virt. Iāve read the Incus source code in regards to this.
I havenāt checked the TrueNAS middleware source code to see what it does with the ix-virt props.
1 Like
Yea, TrueNAS Middleware isnāt changing those properties. So if you are uber picky (Looking at you: @winnielinnie) and want ZSTD-10 or something, you can set that before you start creating LXC containers and it should inherit
If there is enough general need to for general audiences to tweak those in the UI, we can consider it eventually. But I suspect most folks are well suited by standard LZ4 compression on that dataset and I donāt want to over-complicate it right out of the gate without good reason.
2 Likes