Environment: TrueNAS SCALE 25.10.6, VMs managed through the vm.* API
(libvirt/QEMU, machine type pc-i440fx-10.0), DISK devices of type VIRTIO
backed by zvols.
Problem
Storage cannot be added to a running VM. A DISK device created with
vm.device.create on a running VM is stored in the configuration and listed by
vm.device.query, but it is not attached to the running QEMU process. The
guest sees it only after a full VM stop and start. Every storage change
therefore means downtime for everything running inside that VM, and the API
reports success although nothing has changed in the running guest yet.
Business use case
This is a production use case, not a home-lab one. We use TrueNAS as the
storage platform for an internal database development environment: one VM
hosts several isolated database instances, and each developer receives a ZFS
clone of an anonymised production snapshot, attached to that VM as its own
disk. ZFS makes creating the clone instant; attaching it is the bottleneck,
because every new clone needs a VM restart that interrupts all other
developers’ instances.
The same need applies to other common enterprise workloads on TrueNAS VMs, for
example restore tests from snapshots, test and CI environments that get fresh
data per run, or adding capacity to a running application VM.
Current workaround and its cost
We attach a stock of spare clone disks in advance and refill it in scheduled
maintenance windows. That works, but every refill still costs a restart of the
whole VM, the stock ties up pool space, and requests for data that is not in
the stock have to wait for the next window.
Proposed solution
- Live attach (and ideally live detach) for DISK devices on running VMs, the
same way PCI devices are already handled: libvirtattachDeviceFlagswith
VIR_DOMAIN_AFFECT_LIVE | VIR_DOMAIN_AFFECT_CONFIG(QEMUdevice_add). - If that is not planned: return or expose whether a device change is active
in the running domain or only persisted (for example a field in the
vm.device.createresult or invm.device.query), so automation does not
rely on a success that has not taken effect yet.
Technical details
In the vm plugin, live attach/detach logic only exists for PCI devices
(plugins/vm/devices/pci.py). Storage devices have no live path, so the device
is written to the configuration only and takes effect on the next domain start.
The underlying stack already supports this: virtio-blk hot-plug works in QEMU
and libvirt, and the machine type has free PCI slots. vm.device.delete
already refuses correctly on a running VM; a live detach would be the natural
counterpart.
Steps to reproduce:
- Start a VM with one boot disk.
- Create a zvol and call:
vm.device.create {"vm": <id>, "attributes": {"dtype": "DISK", "path": "/dev/zvol/<pool>/<zvol>", "type": "VIRTIO"}} - The call returns a device id.
vm.statusstaysRUNNINGwith the same PID;
vm.device.querylists the new device. - Check the running QEMU process:
tr '\0' '\n' < /proc/<qemu-pid>/cmdline | grep -c <zvol>returns0.
Inside the guest,lsblkdoes not show the disk. vm.stop+vm.start(a reboot inside the guest is not enough, the QEMU
process stays the same): the disk now appears on the QEMU command line and
in the guest.