Linux Jails (containers/vms) with Incus

dasunsrule32 · July 8, 2025, 4:08pm

Yeah, I can do that.

dasunsrule32 · July 8, 2025, 4:12pm

Done.

scyto · July 8, 2025, 8:53pm

I have GPUs working fine on incus LXCs, using it for a frigate server

i found i could only add and remove GPUs using the incus command line - and after doing that it was essential to not poke them with the UI (not ideal. i know)

dasunsrule32 · July 8, 2025, 9:05pm

GPU’s should work fine, just not Nvidia GPU’s in 25.04.1. nvidia-container-toolkit is busted from what I can tell in the version bundled.

dasunsrule32 · July 30, 2025, 7:05pm

Well, apparently Incus isn’t the new hotness. It got introduced and apparently is getting ripped out in 25.04.2 or 25.10.

This is really a horrible way to run things. Why even introduce this in the first place? Who does this in point releases?

Foxtrot314 · July 30, 2025, 7:08pm

I didnt mention 25.04.2. From what I understand this will happen in 25.10

dasunsrule32 · July 30, 2025, 7:09pm

Doesn’t change the fact this is a terrible way to do things. They now have a full track record of this.

winnielinnie · July 30, 2025, 7:11pm

I’m still on Core. What’d I miss?

neofusion · July 30, 2025, 7:12pm

One can still run it like you describe in this thread, right?

I think it’s clear this wasn’t their original intention.

Perhaps they underestimated the work required to implement incus properly? Either way my take is that the devs are also disappointed that it ended up like this.

awasb · July 30, 2025, 7:14pm

The constant “development”? The fresh air that the perfume of those (green) apples brings with every minor update? The Sisyphean feeling of being “involved in the process”? The constant emergence of new tasks and routines just to get things back to normal?

dasunsrule32 · July 30, 2025, 7:15pm

You should be able to on your own build. These configs are straight cloud-init, so it should work with Incus containers without much change. I do have a PoC box with BTRFS volumes and Incus running and they work with these configs for the containers without issue.

You could also drop into dev mode and install the incus packages again I suppose. I’m not willing to do that, I don’t feel like jumping through hoops for this. Going to add in bcache as well and see how it performs.

I wonder if it has anything to do with incus-os?

dasunsrule32 · July 31, 2025, 7:32pm

So containers are still there in the new update, but I haven’t updated to see if it’s Incus still or not. I ended up creating 3 new VM’s for testing some PoC stuff and it looks like those might get nixed on the upgrade. I will test and see.

EDIT: Containers is incus for now.

Side note, nvidia is working again:

nvidia-container-cli -V
cli-version: 1.17.8
lib-version: 1.17.8

mattsteg · July 31, 2025, 8:18pm

It’s barely 3 weeks since Stephane was posting on his blog about TrueNAS integrating Incus and exciting things to come! This was only 24 days ago!

dasunsrule32 · July 31, 2025, 8:29pm

They had them on one of their YouTube videos as well…

dasunsrule32 · July 31, 2025, 8:30pm

Oops…

github.com/lxc/incus

Add TrueNAS Storage Driver

main ← truenas:truenas-storage-driver

opened 09:57AM - 29 May 25 UTC

mrstux

+4099 -33

The TrueNAS Storage driver allows an Incus node to utilise a remote TrueNAS stor…age server to host one or more incus storage pools. When the node is in a cluster, then all members of the cluster can use the storage pool simultaneously. This works well , for example, when live migrating a VM from one Incus node to another. The driver is block-based, ie all Incus volumes are created as zvols on the remote server, and the zvol block devices are accessed on the local incus node via iscsi The driver is modelled after the existing ZFS driver, and supports most existing ZFS functionality, but on a remote TrueNAS server. For example, a local VM can be snapshotted and then cloned. The snapshot and cloning will be performed on the remote server after synchronizing the local filesystem, and the clone will then be activated via iSCSI as necessary. The driver uses the `truenas_incus_ctl` tool, which provides a simple CLI interface to the TrueNAS API to effect changes on the remote server. Additionally, the tool can also activate/deactivate remote zvols using open-iscsi. _If the tool is not found, then the driver is disabled._ The latest version of the truenas tool can be obtained from : https://github.com/truenas/truenas_incus_ctl, and must be installed in the PATH supplied to incusd (v0.7.2+ is required) `open-iscsi` can be installed with `apt install open-iscsi` To login to the remote TrueNAS host first run the command: `sudo truenas_incus_ctl config login` Example: ``` $ sudo truenas_incus_ctl config login [sudo] password for user: Enter a name for this connection: example Enter the TrueNAS hostname or IP address: server.example.com Setting up connection to TrueNAS host: server.example.com Choose authentication method (1 for API Key, 2 for Username/Password): 2 Testing connection to wss://server.example.com/api/current... Enter your TrueNAS username: truenas_admin Enter your TrueNAS password: Generating API key... API key successfully generated Successfully connected to wss://server.example.com/api/current Configuration for 'example' (connecting to server.example.com) saved to /root/.truenas_incus_ctl/config.json ``` After logging in, ensure iSCSI is configured and working with the following command: `sudo truenas_incus_ctl share iscsi setup --test` As an example of the functionality of the TrueNAS tool, this command can be used to list all datasets on the remote server ``` sudo truenas_incus_ctl dataset ls ``` Once logged in and iSCSI has been configured, the following command can be used to create a remote storage pool `incus storage create <poolname> truenas source=[host:]<pool>/<dataset>/[remote-poolname]` `source` is used to describe the location where the storage pool will be created on the remote TrueNAS host `host` is optional, and `remote-poolname` will be set to `poolname` if not supplied. `truenas.config` can be used to specify a specific login config `truenas.host` can be used to specify a specific host, which the tool will lookup in the config.json **Running Tests** There are a number of `INCUS_TRUENAS_` environment variables that can be used to configure the remote TrueNAS that will be used by the driver during the test. The only environment variable that is required is `INCUS_TRUENAS_DATASET` I use the following commands in a script called `start-tests.sh` to run the suite using the default host configured in `truenas_incus_ctl` config file `sudo -E env PATH=$PATH INCUS_BACKEND=truenas INCUS_TRUENAS_DATASET=tank/se/incus-tests ./main-tn.sh $1` Alternatively, I specify a `config` for a different host. Hosts and API keys can also be specified manually to avoid the use of a config file. Alternate config files can be specified. `sudo -E env PATH=$PATH INCUS_VERBOSE=0 INCUS_TMPFS=0 INCUS_BACKEND=truenas INCUS_TRUENAS_DATASET=puddle/se/incus-tests INCUS_TRUENAS_CONFIG=goldeye2 ./main-tn.sh $1` I run the standalone tests using `./start-tests standalone`, as the `cluster` tests require net ns support from `truenas_incus_ctl` which is not implemented. `main-tn.sh` is the same as `main.sh`, except some tests which are problematic on my testing hosts are disabled. `main-tn.sh` should probably be deleted once the `./main.sh standalone` tests are verified When running the test `truenas_incus_ctl` will be launched in daemon mode. It can be useful to prelaunch the daemon to view its console as the test suite runs. for example: `sudo /home/<user>/go/bin/truenas_incus_ctl daemon /home/<user>/tncdaemon.sock` **Incus CI Integration** I would suggest that a VM with a "golden" image of GoldEye with the middleware defer patch is used as a target. The `truenas_incus_ctl` tool should be built from master and the standalone tests used. A preconfigured config.json for the tool could be used to provide the host/api-key to use for the tests. A container dataset should be created and removed after the test suite is run. **Known Issues** - *Creating and Deleting remote Volumes is relatively slow* due to current TrueNAS bottlenecks. Work is ongoing on resolving and hopefully will be incorporated in a future builds of Goldeye. https://github.com/truenas/middleware/pull/16614 - Requires TrueNAS Goldeye for `snapshot rename` capability (currently in Nightlies), although there is a patch that can be easily installed in Fangtooth (25.04) https://github.com/truenas/truenas_incus_ctl#zfssnapshotrename-backport - Cluster tests are not currently passing (requires net ns work-arounds), but cluster usage is supported. Use `standalone` tests - TrueNAS hosts don't currently support zvol-shrinking via API. The driver attempts to zvol-shrink via the tool, and then ignores the inevitable "shrink failure", but warns with the correct `zfs set` command required. If the API is ever changed, the tool will be updated to add support. - Optimized migrations between storage pools are not currently implemented.

dasunsrule32 · July 31, 2025, 8:31pm

I dunno, based on that PR, it seems like they are moving forward? ~~Sounds like containers only. I’d be fine with that.~~ Storage Driver integration only.

mattsteg · July 31, 2025, 8:31pm

Yeah but that way way back in the before-times of…April 2025

mattsteg · July 31, 2025, 8:35pm

There’s not a lot of ambiguity here:

kris

2025-07-29 16:59:09 UTC

General |

I'm a bit shocked that wasn't crystal clear. Incus is being removed from the base system. 100% and entirely. Libvirt is the thing that will handle the backend for both VMS and LXC.

But that said, the way that your message across was bewildering, since I don't know how much clearer that could have been. We also won't be talking as much about backend technology in the future, since folks tend to get all wrapped up around the axel anytime we do

[quote=“kris”]
I’m a bit shocked that wasn’t crystal clear. Incus is being removed from the base system. 100% and entirely. Libvirt is the thing that will handle the backend for both VMS and LXC.
[/quote

Even if kris was on github 8 hours ago talking about licensing on an incus driver

dasunsrule32 · July 31, 2025, 8:37pm

Yeah, all kinds of mixed signaling here…

But it looks like just integration into Incus and TN storage.

Foxtrot314 · July 31, 2025, 9:09pm

This project seems like purely Truenas driver for Incus. Basically, this allows Incus to use Truenas storage but doesnt have anything to do with using Incus as virtualization manager in Truenas.

And because this project was developed in Github Truenas repo it has GPLv3 license, but Incus needs Apache license so Kris as Truenas representative needs to agree to this relicensing.

So basically. Looks like Incus will support Truenas as storage source, but Truenas will not support Incus as virtualization manager.