Linux Jails (containers/vms) with Incus

Apps and Virtualization
207

Everything works great and is now no longer running with shift or privileged.

I need to get a feature request in on read-only passthrough mounts, cloud-init support and passing through the gpu via webui but otherwise those are easy enough to do manually through the cli for now.

For now, I can still push the instance creation with incus create instead of launch with my custom yaml with cloud init and then trigger the start with the middleware and everything works as expected with the idmapping correctly working in the instance.

Great update overall.

208

midclt call user.query '[["builtin", "=", false], ["userns_idmap", "=", null], ["local", "=", true]]' '{"select": ["username", "id"]}' | jq will return array of JSON objects with only the username and id keys, but only for local ones that arenā€™t builtin and donā€™t already have the idmap set.

3 Likes
209

All-in-one post with the previous info summarized with examples for working with userns_idmapā€™s.

Modified user query:

midclt call user.query '[["builtin", "=", false], ["userns_idmap", "=", null], ["local", "=", true]]' '{"select": ["username", "id", "uid", "userns_idmap"]}' | jq

Example output:

[
  {
    "id": 78,
    "uid": 373,
    "username": "etesync",
    "userns_idmap": null
  },
...
]

Update user userns_idmap:

midclt call user.update 78 '{"userns_idmap": "DIRECT"}'

Updated user:

[
  {
    "id": 78,
    "uid": 373,
    "username": "etesync",
    "userns_idmap": "DIRECT"
  }
]

Groups query:

midclt call group.query '[["builtin", "=", false], ["userns_idmap", "=", null], ["local", "=", true]]' '{"select": ["name", "id", "gid", "userns_idmap"]}' | jq

Example output:

[
  {
    "id": 117,
    "gid": 373,
    "name": "etesync",
    "userns_idmap": null
  },
...
]

Update group userns_idmap:

midclt call group.update 117 '{"userns_idmap": "DIRECT"}'

Updated group:

[
  {
    "id": 117,
    "gid": 373,
    "name": "etesync",
    "userns_idmap": "DIRECT"
  }
]

Pull in updated userns_idmap.

Restart container:

midclt call virt.instance.restart docker1 -j
Status: (none)
Total Progress: [########################################] 100.00%

Container config output for the raw.idmap:

  incus config show docker1|grep raw.idmap -A 4
  raw.idmap: |-
    uid 568 568
    uid 373 373
    gid 568 568
    gid 373 373

Unfortunately, I donā€™t think this one can be done via cloud-init since restarting via the Web UI will wipe out custom raw.idmapā€™s. Technically, if you didnā€™t interacte with the Web UI and managed it all from cloud-init and the local incus commands, it could work, but that will likely get messy quickly.

Iā€™ll research and see what needs to happen or if a script needs to be made to call the midclt to perform these tasks.

210

I added the apps user (Kept the name the same to avoid confusion) to the OP in the docker-init.yaml and docker-init-nvidia.yaml configs. I removed shift: true from all examples as well. Itā€™s no longer needed.

This means you can create a new dataset with the owner of apps and spin up a new container with the new dataset mounted and be able to write to it by default.

211

You can still use cloud-init if you wanted to for now and understood the risks. I redeployed all of my containers last night to test again other than unifi as its nightmare to deploy automatically right now. I manually changed the config there to remove the shifting and running as privileged and it worked fine after multiple reboots from mdiclt and a full server reboot it started automatically fine.

incus create images:ubuntu/noble/cloud nginx < nginx.yml
midclt call virt.instance.start nginx
incus console nginx --show-log

You can see the whole cloud-init process go through properly in the console from start to finish. (Ctrl+A,Q to leave). Alternatively, you can also live watch the cloud-init process with below instead of the console if you prefer.

incus exec nginx -- tail -f /var/log/cloud-init-output.log

The only real process change with custom config is you canā€™t start/restart the instance with incus directly as it wonā€™t apply the idmap but doing with midclt or when midclt starts during server boot still applies it correctly.

212

Yeah, it could work, but I want to make sure containers/vms made by the scripts in the OP survive and donā€™t need anything weird for them be used in the Web UI as well.

Currently, if you add the userns_idmap and do an incus restart <container>, it will not pick up the those idmapā€™s from the middleware unless you use midclt to reboot, which causes funky permissions issues because itā€™s not inheriting the raw.idmap.

This is fine if youā€™re adding that via cloud-init and donā€™t ever intend to use the Web UI. Plus it would likely break in the future. Thatā€™s why Iā€™m trying to keep it within the ecosystem.

213

Is it safe to clear the DIRECT mapping property state from the automatically apps user if not needed to minimize the amount of mapping pushed since it applies to all instances?

214

I donā€™t think thereā€™s much risk from this direct mapping. What is the concern?

215

The only thing I do via CLI is the initial setup via yaml with cloud-init. All of my config options are WebUI friendly right now other than the Intel GPU passed to plex, but that currently doesnā€™t error in the WebUI like read-only mount did. I can edit the containers there fine without breaking anything. I killed my read-only mounts for now until I see if it ends up being accepted as feature request once I can get time to make it. Everything can be started / stopped / editted from the WebUI without issue. Iā€™m happy to share some of my yaml files directly however Iā€™m not putting them publicly like I did with my past Jail deployment guides in old CORE forums resources as they have some deployment stuff related to my use cases/deployment methods for some apps and I donā€™t have time to troubleshoot for others like I did back then.

1 Like
216

I may just leave it but more just general preference, Iā€™d rather not have extra uid/gid mapped that Iā€™m not explicitly using if though it doesnā€™t harm anything.

217

Sure, I always like to see how others handle similar tasks. :slight_smile:

218

Updated the configs to add the local default and root users to the apps group. Modified the OP as well with information about setting your permissions on your host folders as well. If you set your host data to apps:apps, then your container shouldnā€™t have any issues reading/writing to the datasets passed through as long as the container users are members of the apps group and your host permissions for directories are 775 and files are 664.

219

I have successfully cut over 2 of my jailmaker instances using the scripts (Of course, modified for my environment :wink: ) in OP. Should be pretty straightforward for most with experience. Working great. :slight_smile:

1 Like
220

Have to figure out what is going on with sockets in Dockge. When switching over to Incus even over a bridge, getting slow connections and disconnects between Dockge agents, which I didnā€™t have an issue with in jailmaker. My jails were on Docker 27.5.1, Incus is on 28.0.1. Iā€™ll see if that plays into anything here.

2025-03-13T19:01:53Z [AGENT-MANAGER] ERROR: Error from the socket server: apps1:5001
2025-03-13T19:01:53Z [AGENT-MANAGER] ERROR: Error from the socket server: apps2:5001
2025-03-13T19:01:53Z [AGENT-MANAGER] ERROR: Error from the socket server: dns:5001
2025-03-13T19:01:53Z [AGENT-MANAGER] ERROR: Error from the socket server: proxy:5001
2025-03-13T19:02:14Z [AGENT-MANAGER] ERROR: Error from the socket server: apps2:5001
2025-03-13T19:02:14Z [AGENT-MANAGER] ERROR: Error from the socket server: dns:5001
2025-03-13T19:02:14Z [AGENT-MANAGER] ERROR: Error from the socket server: proxy:5001
2025-03-13T19:02:15Z [AGENT-MANAGER] ERROR: Error from the socket server: apps1:5001

It is working, just slow and annoying right now.

221

:thinking:

Not working in RC1ā€¦ Lookingā€¦

223

Playing with Fangtooth RC1 on a box and created an incus instance and enabled the GPU in devices, but I canā€™t run nvidia-smi in the instance (I did install the container toolkit). Comparing it to my jlmkr instance, there are a lot of missing libnvidia* files from the /usr/lib/x86_64-linux-gnu folder that I would expect to see.

Using incus config show , I noticed the nvidia.runtime: true line is missing from the config.

Is there some way to take this webui built instance and ā€˜addā€™ this to the config? Will this likely resolve my issue and get the GPU visible and usable in the incus instance or is this just not something expected to work?

224

Just something I found out about Incus.
You need to set config image.os: Windows for it to apply some Windows specific things to QEMU.

1 Like
225

Can you run the following command and paste the results formatted as a code block for readability? Thanks.

incus config show <instance-name> --expanded

But getting Nvidia working can be configured with (off the top of my head):

incus config add <instance-name> -c nvidia.runtime=true

PS - I have a config in the OP that will configure Nvidia from the ground up.

226

Going to go through these, I know the default TNS values used to be set too low by defaultā€¦ Not sure if itā€™s been tweaked since apps moved to Docker.

227

So something DNS related since moving to Incusā€¦ looking. Going to the IP connects immediately.