VM: hot-attach DISK devices to a running VM (vm.device.create currently only persists them)

Environment: TrueNAS SCALE 25.10.6, VMs managed through the vm.* API
(libvirt/QEMU, machine type pc-i440fx-10.0), DISK devices of type VIRTIO
backed by zvols.

Problem

Storage cannot be added to a running VM. A DISK device created with
vm.device.create on a running VM is stored in the configuration and listed by
vm.device.query, but it is not attached to the running QEMU process. The
guest sees it only after a full VM stop and start. Every storage change
therefore means downtime for everything running inside that VM, and the API
reports success although nothing has changed in the running guest yet.

Business use case

This is a production use case, not a home-lab one. We use TrueNAS as the
storage platform for an internal database development environment: one VM
hosts several isolated database instances, and each developer receives a ZFS
clone of an anonymised production snapshot, attached to that VM as its own
disk. ZFS makes creating the clone instant; attaching it is the bottleneck,
because every new clone needs a VM restart that interrupts all other
developers’ instances.

The same need applies to other common enterprise workloads on TrueNAS VMs, for
example restore tests from snapshots, test and CI environments that get fresh
data per run, or adding capacity to a running application VM.

Current workaround and its cost

We attach a stock of spare clone disks in advance and refill it in scheduled
maintenance windows. That works, but every refill still costs a restart of the
whole VM, the stock ties up pool space, and requests for data that is not in
the stock have to wait for the next window.

Proposed solution

  1. Live attach (and ideally live detach) for DISK devices on running VMs, the
    same way PCI devices are already handled: libvirt attachDeviceFlags with
    VIR_DOMAIN_AFFECT_LIVE | VIR_DOMAIN_AFFECT_CONFIG (QEMU device_add).
  2. If that is not planned: return or expose whether a device change is active
    in the running domain or only persisted (for example a field in the
    vm.device.create result or in vm.device.query), so automation does not
    rely on a success that has not taken effect yet.

Technical details

In the vm plugin, live attach/detach logic only exists for PCI devices
(plugins/vm/devices/pci.py). Storage devices have no live path, so the device
is written to the configuration only and takes effect on the next domain start.
The underlying stack already supports this: virtio-blk hot-plug works in QEMU
and libvirt, and the machine type has free PCI slots. vm.device.delete
already refuses correctly on a running VM; a live detach would be the natural
counterpart.

Steps to reproduce:

  1. Start a VM with one boot disk.
  2. Create a zvol and call:
    vm.device.create {"vm": <id>, "attributes": {"dtype": "DISK", "path": "/dev/zvol/<pool>/<zvol>", "type": "VIRTIO"}}
  3. The call returns a device id. vm.status stays RUNNING with the same PID;
    vm.device.query lists the new device.
  4. Check the running QEMU process:
    tr '\0' '\n' < /proc/<qemu-pid>/cmdline | grep -c <zvol> returns 0.
    Inside the guest, lsblk does not show the disk.
  5. vm.stop + vm.start (a reboot inside the guest is not enough, the QEMU
    process stays the same): the disk now appears on the QEMU command line and
    in the guest.

This is a hack, so will only work so long as the host has free loop device slots. You could loop-mount the zvol clones to a common host mount-point, then inside the VM, mount the host paths for the loop mounts via 9PFS.

Thanks for the suggestion – I hadn’t considered 9P, so it’s worth spelling out why it doesn’t fit our case:

  1. The clones are NTFS volumes and need checking inside the guest before they are mounted. Before mounting a clone we read the raw block device in the guest and refuse to mount if it fails (MFT/MFT-mirror consistency, dirty bit). With loop mounts on the host, the NTFS volume would be mounted by TrueNAS itself, with the host’s NTFS driver and outside our checks, before the guest sees anything.

  2. The workload is SQL Server data and log files. We attach databases directly from the clone. Moving block-level I/O to a file-sharing protocol (9P) changes both throughput and locking behaviour in ways we’d rather not test against SQL Server. Our current virtio-blk setup gives us about 3.5× the write throughput of the iSCSI path it replaced. That gain is the reason we moved to this setup.

  3. It doesn’t remove the restart. A 9P share is a new device in the VM definition as well, so it has to be added with a full stop/start once. After that it would avoid the restarts, but it puts in a host-side mount layer we would have to manage and roll back separately.

The pieces underneath SCALE already support this. On a plain Debian/libvirt host, virsh attach-device --live with a virtio-blk disk is routine. SCALE is Debian with the same libvirt/QEMU underneath. I checked our running VM read-only: i440fx machine type, ACPI PCI hotplug not disabled, 12 free slots on PCI bus 0, and virsh and the libvirt socket are present. So nothing technical prevents a live attach. The only gap is that vm.device.create persists a DISK device but never calls libvirt’s live attach for it. We deliberately don’t run virsh behind the middleware’s back, because the VM’s runtime state and the middleware’s view would then disagree.

Our target is about 20 concurrent clones on one VM, each attached to its own SQL Server instance. Today every new clone means a stop/start that restarts all other running instances. Our workaround is a pool of pre-attached, unassigned clone devices, refilled during a nightly window. That works, but it doesn’t scale well to 20 developers. So the request is:

  • Live attach/detach for DISK devices, using the same libvirt call the PCI path already uses.
  • Or, as a minimum, vm.device.create should say explicitly when a change is “persisted only, effective at next start”. Today it returns success while the guest never sees the disk.
  • A related question: with i440fx, PCI bus 0 runs out at around 24 disks. Is there a supported way to go beyond that (a PCI bridge, or virtio-scsi with many LUNs on one controller) through the middleware?