Linux Jails (containers/vms) with Incus

@cfouche Thanks, I’ll try this evening.
French ?

How did you know ? :sweat_smile:

De quel endroit, j’habite près de Nice.

Do note that configuration changes through the incus UI will be treated as unsupported just as changes from the incus CLI are. c.f MOTD on TrueNAS:

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI, CLI, and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

This means that bug reports for issues that are caused by changing options not exposed via our UI / API will most likely be closed.

We are currently working on adding import and export options for instances and storage volumes.

Feedback via feature requests would be appreciated for gaps in the exposed feature set that are important / critical for proper functionality for how people are using the feature.

I.e. a feature request like “your UI is too limited, just expose the incus UI” is not terribly helpful, whereas “I need the capability to create incus snapshot tasks and be able to replicate them through the normal TrueNAS ZFS replication manager” is more helpful.

Don’t worry, I only use the web ui as a backup, incus in truenas is new and will grow overtime.

Off topic, will the official release of truenas will be on incus 6.0.3 LTS or the just released 6.0.4 ?

I leave Incus web UI :unamused:

Can you do the following and post the output?

incus network ls

incus config show <instance-name>

Hi, my apologies, I could fix it already. It did not work initially, because I picked “br0 / Macvlan”, but I should have picked “br0 / bridged”. It’s working now very well. Am quite glad, because I can finally migrate from jail to incus! yay

1 Like

Ok, I could try this out. I have integrated intel graphics.

I tried two things
a) adding the device (as per lspci)

sudo incus config device add idefix intel-gpu gpu pci=0000:00:02.0 gid=44

b) adding the path

sudo incus config device add idefix dri disk source=/dev/dri path=/dev/dri

Unfortunately, both don’t work. Within the debian container, I get

$ intel_gpu_top
Failed to initialize PMU! (Permission denied)

Just making sure, but you don’t have isolate GPU on in TrueNAS?

Can you dump this output?

incus info --resources|grep GPU -A 20

and

incus config show idefix --expanded

Also, try removing the gid bit and see it can access it as root in the container.

Thank you for taking a look. I get the same result when I omit “gid=44”.

Below is the respective output of the two commands.

$sudo incus info --resources | grep GPU -A20
[sudo] password for admin:
GPU:
NUMA node: 0
Vendor: Intel Corporation (8086)
Product: Alder Lake-UP3 GT2 [Iris Xe Graphics] (46a8)
PCI address: 0000:00:02.0
Driver: i915 (6.12.15-production+truenas)
DRM:
ID: 0
Card: card0 (226:0)
Control: controlD64 (226:0)
Render: renderD128 (226:128)
SR-IOV information:
Current number of VFs: 0
Maximum number of VFs: 7

NICs:

$ sudo incus config show idefix --expanded
architecture: x86_64
config:
boot.autostart: “false”
image.architecture: amd64
image.description: Debian trixie amd64 (20250405_05:24)
image.os: Debian
image.release: trixie
image.serial: “20250405_05:24”
image.type: squashfs
image.variant: default
limits.cpu: 1-8
limits.memory: 8192MiB
raw.idmap: |-
uid 568 568
uid 3000 3000
uid 3003 3003
uid 3004 3004
uid 3005 3005
gid 568 568
gid 3000 3000
gid 3001 3001
gid 3004 3004
gid 3005 3005
gid 3006 3006
user.autostart: “false”
volatile.base_image: 50fdb98bf073f9a17c94e75acfb06b11a8d16485c566d51d0727f7b5c176405d
volatile.cloud-init.instance-id: 8b832df2-13e4-4e3a-9ec2-9ac063305316
volatile.eth0.host_name: vethc26b07c7
volatile.eth0.hwaddr: 00:16:3e:6d:23:41
volatile.eth0.name: eth0
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:2147000001,“Nsid”:0,“Maprange”:568},{“Isuid”:true,“Isgid”:false,“Hostid”:568,“Nsid”:568,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:2147000570,“Nsid”:569,“Maprange”:2431},{“Isuid”:true,“Isgid”:false,“Hostid”:3000,“Nsid”:3000,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:2147003002,“Nsid”:3001,“Maprange”:2},{“Isuid”:true,“Isgid”:false,“Hostid”:3003,“Nsid”:3003,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:3004,“Nsid”:3004,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:3005,“Nsid”:3005,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:2147003007,“Nsid”:3006,“Maprange”:455746},{“Isuid”:false,“Isgid”:true,“Hostid”:2147000001,“Nsid”:0,“Maprange”:568},{“Isuid”:false,“Isgid”:true,“Hostid”:568,“Nsid”:568,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:2147000570,“Nsid”:569,“Maprange”:2431},{“Isuid”:false,“Isgid”:true,“Hostid”:3000,“Nsid”:3000,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:3001,“Nsid”:3001,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:2147003003,“Nsid”:3002,“Maprange”:2},{“Isuid”:false,“Isgid”:true,“Hostid”:3004,“Nsid”:3004,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:3005,“Nsid”:3005,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:3006,“Nsid”:3006,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:2147003008,“Nsid”:3007,“Maprange”:455745}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:2147000001,“Nsid”:0,“Maprange”:568},{“Isuid”:true,“Isgid”:false,“Hostid”:568,“Nsid”:568,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:2147000570,“Nsid”:569,“Maprange”:2431},{“Isuid”:true,“Isgid”:false,“Hostid”:3000,“Nsid”:3000,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:2147003002,“Nsid”:3001,“Maprange”:2},{“Isuid”:true,“Isgid”:false,“Hostid”:3003,“Nsid”:3003,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:3004,“Nsid”:3004,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:3005,“Nsid”:3005,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:2147003007,“Nsid”:3006,“Maprange”:455746},{“Isuid”:false,“Isgid”:true,“Hostid”:2147000001,“Nsid”:0,“Maprange”:568},{“Isuid”:false,“Isgid”:true,“Hostid”:568,“Nsid”:568,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:2147000570,“Nsid”:569,“Maprange”:2431},{“Isuid”:false,“Isgid”:true,“Hostid”:3000,“Nsid”:3000,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:3001,“Nsid”:3001,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:2147003003,“Nsid”:3002,“Maprange”:2},{“Isuid”:false,“Isgid”:true,“Hostid”:3004,“Nsid”:3004,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:3005,“Nsid”:3005,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:3006,“Nsid”:3006,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:2147003008,“Nsid”:3007,“Maprange”:455745}]’
volatile.last_state.idmap: ‘
volatile.last_state.power: RUNNING
volatile.last_state.ready: “false”
volatile.uuid: a08be109-2daf-4903-876c-d5788dbc43d7
volatile.uuid.generation: a08be109-2daf-4903-876c-d5788dbc43d7
devices:
disk0:
path: /share/homes/myuser
source: /mnt/tank-RAIDZ1/homes/myuser
type: disk
eth0:
nictype: bridged
parent: br0
type: nic
intel-gpu:
gid: “44”
pci: “0000:00:02.0”
type: gpu
root:
path: /
pool: default
type: disk
ephemeral: false
profiles:

  • default
    stateful: false
    description: “”

I don’t know if it would be this simple, as I didn’t have an Intel system, but is sr-iov enabled in your BIOS?

Don’t think that matters as you’re not doing passthrough here.

I don’t see that you’re mounting /dev/dri in your config.

Can you ls -la /dev/dri from inside the container, as well as the host?

See OP for example. It would be a path mount.

Without running “incus config device add…” there is no /dev/dri.

After running this command on the host
sudo incus config device add idefix intel-gpu gpu pci=0000:00:02.0 gid=44

this is the output within the container:

root@idefix:~# ls -la /dev/dri
total 1
drwxr-xr-x 2 root root        80 Apr  6 12:03 .
drwxr-xr-x 9 root root       540 Apr  6 12:03 ..
crw-rw---- 1 root video 226,   0 Apr  6 12:03 card0
crw-rw---- 1 root video 226, 128 Apr  6 12:03 renderD128
root@idefix:~# intel_gpu_top 
Failed to initialize PMU! (Permission denied)

Side note: I am running jailmaker on the same host. Using option gpu_passthrough_intel=1 leads effectively to a bind mount of /dev/dri. And then I can execute intel_gpu_top fine. It’s probably not comparable, because jailmaker uses lxc and it’s a privileged container.

Can you try using gpu: type: physical rather than gpuand see if that works?

I was also reading in a few places that may be cosmetic and the hw acceleration may still work. Have you tested it to see? Try running something with hw acceleration enabled then on the host run the intel_gpu_top command.

You can also try setting the following on your host as well:

sysctl kernel.perf_event_paranoid=0

Yes! That worked!

So, to summarize, these commands need to be run on TrueNAS host to enable “intel_gpu_top” running within a debian container to access GPU performance measures:

  1. identify the device

$ lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-UP3 GT2 [Iris Xe Graphics] (rev 0c)

  1. allow any user to access performance measures

sysctl kernel.perf_event_paranoid=0

  1. add GPU to container

sudo incus config device add idefix intel-gpu gpu pci=0000:00:02.0 gid=44 gputype=physical

I could test this by playing a movie in plex. GPU stats are nicely displayed.

Now the 2 mio dollar question - Did this enable only performance monitoring or will the container be able to use the GPU?

I’ll install a desktop environment and find out…

EDIT to share 2 updates:

  • above “sysctl” command must be run as a post-init script. This change will not survive a reboot
  • GPU works for me in a debian trixie container. I can connect using xrdp and something like “glxheads” will use the GPU.
1 Like

I created a container instance on the pool “pool1”.
I defined a disk on a “Bookworm” dataset from an external pool “pool2” (/mnt/pool2/Bookworm). This dataset is mapped to the “/home” directory.

When I try to access “/home” (cd /home), I get the error :

image

“Bookworm” permissions :

It’s an idmapped mount. Root is not an idmapped account.

Thanks for your reply.
I’m not familiar with idmapped mounts. I’ll look into that.

Look at the OP, Step 7