Because the Incus Web UI only installs with a local instance of Incus and it has to be run out of the same installed host, it’s not a feasible option and it can’t be spun up into it’s own container as a result. It’s simple javascript files that have to be installed in the incus/ui path and then set the https connection port via the incus command line application.
Added to the OP to look into adding idmap configs to the cloud-init scripts when it’s out to help move away from disks: shift: true.
I haven’t looked at what your cloud-init scripts are doing (and honestly, probably won’t look at it since it’s beyond scope of what I do here), but TrueNAS will overwrite anything written in raw.idmap on instance restart.
If you want to make sure that the instance catches latest changes to users / groups you’ll want to midclt call virt.instance.restart <instance name> -j.
1 Like
Basically want to set what idmap a container/vm should be using via the cloud-init script. Not wanting to create anything, just use TNS existing idmap’s that either exist or have been created by the user:
For example:
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":2147000001,"Nsid":0,"Maprange":458752},{"Isuid":false,"Isgid":true,"Hostid":2147000001,"Nsid":0,"Maprange":458752}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":2147000001,"Nsid":0,"Maprange":458752},{"Isuid":false,"Isgid":true,"Hostid":2147000001,"Nsid":0,"Maprange":458752}]'
I’m basically wanting to script and create machines that will survive reboots and upgrades within the TNS ecosystem, but give us more flexibility than what the UI will allow us to do. Similar to what Jailmaker gave us.
I may end up scripting and having a template cloud-init config that will be modified based on user input in the future, but it just depends on how things shake out in the UI, etc.
Schedules can be configured via the cloud-init config as well.
snapshots.expiry "When snapshots are to be deleted"
snapshots.pattern "Template for the snapshot name"
snapshots.schedule "Schedule for automatic instance snapshots"
snapshots.schedule.stopped "Whether to automatically snapshot stopped instances"
Key: snapshots.schedule
Type: string
Default: empty
Live update: no
Specify either a cron expression (<minute> <hour> <dom> <month> <dow>), a comma-and-space-separated list of schedule aliases (@startup, @hourly, @daily, @midnight, @weekly, @monthly, @annually, @yearly), or leave empty to disable automatic snapshots.
Note that unlike most other configuration keys, this one must be comma-and-space-separated and not just comma-separated as cron expression can themselves contain commas.
Added cloud-init.network-config to the cloud-init scripts in the OP. Example below. They are commented out, but can be uncommented if you would like to set that manually there during build time of the container/vm.
cloud-init.network-config: |
#cloud-config
version: 1
config:
- type: physical
name: eth0
subnets:
- type: static
address: 192.168.1.100/24
gateway: 192.168.1.1
control: auto
- type: nameserver
interface: eth0
address: 192.168.1.1
search:
- domain.lan
Example output:
+---------+---------+------------------------------+------+-----------+-----------+
| docker3 | RUNNING | 192.168.1.100 (eth0) | | CONTAINER | 0 |
| | | 172.18.0.1 (br-47ff509fa588) | | | |
| | | 172.17.0.1 (docker0) | | | |
+---------+---------+------------------------------+------+-----------+-----------+
This fixes it on bridge interfaces, it will likely fix it in general once the stub listener is stopped.
After reading the thread and also the ticket, I am still missing something. Is it a true statement that the incus implementation is intentionally blocking an inside-the-container mount to an existing dataset on TrueNAS and disabling write because it’s a security risk?
If you use the cloud-init scripts in the OP, it will do host passthrough fine. We will probably update how that is done over time, but for now, it works well enough.
No. That is not true.
2 Likes
Its unfortunate Incus doesnt work similarly like Docker in this regard, where each app knows which manager it belongs to.
Without this kind of separation I am afraid even if Web UI worked in Truenas it would conflict with Truenas UI and middleware because they wouldnt expect the setting changes custom Web UI would make.
Canonical made a weird design choice here. The UI has to be deployed down theincus/ui path to work, then be registered with the local instance. It’s not even a knowing where it’s at issue, it’s that you can only install it in a local incus instance and control that instance. I didn’t go further than that, but I “think” you can cluster and control the instance from other instances and deploy the UI on each instance for HA of the Web UI. You “might” be able to put an LB in front of the separate instances to HA the web interface. It’s a catch-22 with the UI. Normally I would’ve thought it would’ve been a standalone installer that could be configured to point to the individual instances or cluster VIP on the network and maybe it can. But on TNS, it won’t be as easy to deploy the UI without some sort of systemd-sysext image to deploy it to the incus/ui directory… I personally don’t know if that is worth the time or investment…
I think this is something what could be feature request for Incus on their GitHub.
If the problem is explained maybe they will decide to do something about it.
1 Like
From what I can tell in my conversations in the forum and looking at the Incus GH repo, they are not interested in doing much more than clones and builds of the default packages. So unless they decide to fork the UI entirely into their project, I don’t know if we’ll see that.
That is correct. Changes should be made through the truenas webui / API.
Other methods of altering incus configuration are “do at your own risk” and may break under a variety of scenarios with little recourse for improving the situation because at the end of the day it’s unsupported.
IMO, the best way to get access to new features for incus in TrueNAS (if they’re not exposed by the UI / API) is to create feature requests related to it.
As a long time hold out of swapping from Core due to jails, incus support made me make the swap. Given the progress and improvements it seems like based on this thread and jira tickets excited for future improvements.
I’ve done everything with a pushed config yaml file since cloud-init was a hard requirement. I think the only thing I’ve seen the web-ui angry about on what I did via the pushed yaml is you can’t mount read-only host paths and host-paths outside pools (namely to mount a tmpfs for plex transcoding). Otherwise everything else I could replicate in the WebUI other than shift and privileged for now until the direct idmap support is added in.
I’ll have to put in a feature request at some point soon but at least as of this weekend everything I had running on Core jails is now running on Fangtooth beta in containers.
It’s fixed in the nightlies, but it will need chown for the idmap on the datasets. They are working on the docs still.
I was working on some issues with the host passthrough today where I’m able to write fine initially, but the files can’t be deleted from within the container. I only had 15 minutes to look at that today in my testing.
OP has some examples for cloud-init that I put together. I’m trying to keep it within the realm of the default TNS Incus profile so things will continue to work between updates which is currently an issue if you go outside any if the default profile, datasets, etc.
I like it so far, it’s been pretty good. I’ll start migrating of jailmaker once we get to the initial release as I’m constantly updating those cloud-init scripts in the OP.
Welcome aboard!
For 25.04 RC we’ll have the apps user / group (uid/gid 568) mapped and available for containers. I don’t know why you’d need to chown though.
For example see the CI test: middleware/tests/api2/test_virt_instance_acl.py at 9f93527d15914848a9dc86b785b3e33d7c6c66ce · truenas/middleware · GitHub
quite clearly, ACL entries that contained idmapped ids function correctly
1 Like
I’ll take a look when the RC hits. It’s not functioning correctly in the beta, correct? I’m troubleshooting some permissions issues and not sure if I should just wait until then.
ACLs work correctly with idmapped mounts. I fixed before beta NAS-130522 / 25.04 / Fix namespace handling in zpl_permission by anodos325 · Pull Request #244 · truenas/zfs · GitHub . If you’re playing with shifting layer I haven’t tested and so I’d say undefined. You can test with fangtooth nightlies if you’re curious about newer code.
1 Like