How I restored lost Incus VMs after upgrading to TrueNAS SCALE 26.0 BETA

Apps and Virtualization
1

After upgrading to TrueNAS SCALE 26.0 BETA, my old Incus-based VMs disappeared from the UI. The good news: the VM disks were not deleted. The original .block zvols were still under .ix-virt.

With the help of AI, I restored and documented the recovery process using only the shell: inspect the old Incus DB, create new VM definitions with midclt, reattach the existing zvols, add NIC and VNC devices, and boot the original systems without reinstalling anything.

Sharing it here in case it helps anyone else who lost legacy Incus VMs during the upgrade. Sorry Iā€™m not allowed to upload the txt file:
Recovering Legacy Incus Virtual Machines after Upgrading to TrueNAS SCALE 26

Summary

After upgrading from TrueNAS SCALE 25.x to TrueNAS SCALE 26.x, some legacy Incus-managed virtual machines may disappear from the TrueNAS UI.

In this case, the VM definitions were no longer visible, but the underlying virtual disks were still present as ZFS zvols under .ix-virt.

The recovery was done entirely from the shell using midclt, by:

  1. Finding the original VM zvols.
  2. Verifying that the guest operating systems were still intact.
  3. Reading the old Incus configuration from the Incus database.
  4. Creating new TrueNAS Virtual Machine definitions from the CLI.
  5. Reattaching the original zvols as VM disks.
  6. Adding NIC and VNC display devices from the CLI.

No guest OS reinstall was required.

Important warning

This procedure worked for legacy Incus virtual machines backed by .block zvols.

It is not a generic LXC container recovery procedure.

Before making changes, make sure you understand what each command does and have backups where possible.

1. Locate the old VM zvols

List the old .ix-virt datasets:

zfs list -r <POOL>/.ix-virt

Look for zvols similar to:

<POOL>/.ix-virt/virtual-machines/<OLD_VM_NAME>.block

Example pattern:

<POOL>/.ix-virt/virtual-machines/example-vm.block

If the VM has a .block zvol, it was likely an Incus VM disk rather than a normal LXC container filesystem.

2. Expose the zvol as a block device

By default, some recovered zvols may not appear under /dev/zvol.

Expose the zvol:

sudo zfs set volmode=dev <POOL>/.ix-virt/virtual-machines/<OLD_VM_NAME>.block

Validate:

ls -l /dev/zvol/<POOL>/.ix-virt/virtual-machines/

You should see a symlink for the VM disk.

3. Inspect the disk layout

Check the partition table:

sudo fdisk -l /dev/zvol/<POOL>/.ix-virt/virtual-machines/<OLD_VM_NAME>.block

A typical Linux VM may show:

Disklabel type: gpt
EFI System partition
Linux filesystem partition

4. Verify the guest OS is intact

If the kernel does not automatically expose partition devices for the zvol, use losetup:

sudo losetup -fP --show /dev/zvol/<POOL>/.ix-virt/virtual-machines/<OLD_VM_NAME>.block

Example output:

/dev/loop0

Inspect the partitions:

lsblk -f /dev/loop0

If the root filesystem is Btrfs with subvolumes, mount the root subvolume read-only:

sudo mkdir -p /mnt/testvm
sudo mount -o ro,subvol=@ /dev/loop0p2 /mnt/testvm
cat /mnt/testvm/etc/os-release

If /etc/os-release is readable, the guest OS is still present and the VM disk is intact.

Clean up afterwards:

sudo umount /mnt/testvm 2>/dev/null || true
sudo losetup -d /dev/loop0 2>/dev/null || true

5. Read the old Incus VM configuration

The old Incus database may still contain the original VM metadata.

Query the old VM configuration:

sudo sqlite3 /var/lib/incus/database/global/db.bin '
SELECT i.name, c.key, c.value
FROM instances i
LEFT JOIN instances_config c ON c.instance_id = i.id
ORDER BY i.name, c.key;
'

Useful fields include:

limits.cpu
limits.memory
boot.autostart
security.secureboot
user.autostart
user.ix_vnc_config
volatile.uuid
volatile.eth0.hwaddr

Query the old VM devices:

sudo sqlite3 /var/lib/incus/database/global/db.bin '
SELECT i.name, d.id, d.name, d.type, dc.key, dc.value
FROM instances i
LEFT JOIN instances_devices d ON d.instance_id = i.id
LEFT JOIN instances_devices_config dc ON dc.instance_device_id = d.id
ORDER BY i.name, d.id, dc.key;
'

You can filter a single VM:

sudo sqlite3 /var/lib/incus/database/global/db.bin '
SELECT d.name, d.type, dc.key, dc.value
FROM instances i
JOIN instances_devices d ON d.instance_id = i.id
LEFT JOIN instances_devices_config dc ON dc.instance_device_id = d.id
WHERE i.name="<OLD_VM_NAME>"
ORDER BY d.id, dc.key;
'

6. Confirm TrueNAS can see the old zvol as a VM disk choice

midclt call vm.device.disk_choices | grep '<OLD_VM_NAME>'

If the zvol does not appear, ensure volmode=dev is set:

sudo zfs set volmode=dev <POOL>/.ix-virt/virtual-machines/<OLD_VM_NAME>.block

7. Create a new VM definition from the shell

TrueNAS SCALE 26 VM names may only allow alphanumeric characters. Avoid hyphens and underscores in the new VM name.

Create a new VM definition without any disk attached yet:

midclt call vm.create '{
  "name": "NEWVMNAME",
  "description": "Recovered legacy Incus VM",
  "vcpus": 2,
  "cores": 1,
  "threads": 1,
  "memory": 8192,
  "cpu_mode": "HOST-MODEL",
  "bootloader": "UEFI",
  "bootloader_ovmf": "OVMF_CODE_4M.fd",
  "autostart": true,
  "time": "LOCAL",
  "shutdown_timeout": 90,
  "ensure_display_device": false
}'

The command returns a JSON object. Save the returned VM id.

Example:

"id": 7

In the following commands, replace <VM_ID> with that returned ID.

8. Attach the original zvol as the VM disk

This is the critical recovery step.

Attach the existing zvol without creating a new disk:

midclt call vm.device.create '{
  "vm": <VM_ID>,
  "attributes": {
    "dtype": "DISK",
    "path": "/dev/zvol/<POOL>/.ix-virt/virtual-machines/<OLD_VM_NAME>.block",
    "type": "VIRTIO",
    "create_zvol": false,
    "iotype": "THREADS"
  },
  "order": 1001
}'

Important points:

  • path points to the existing zvol.
  • create_zvol must be false.
  • No data is copied.
  • The original VM disk is reused directly.

9. Add a network interface

Use the appropriate bridge for your system. In this example, <BRIDGE_NAME> is a placeholder.

midclt call vm.device.create '{
  "vm": <VM_ID>,
  "attributes": {
    "dtype": "NIC",
    "type": "VIRTIO",
    "nic_attach": "<BRIDGE_NAME>",
    "trust_guest_rx_filters": false
  },
  "order": 1002
}'

If you need to preserve a previous MAC address, add:

"mac": "<OLD_MAC_ADDRESS>"

Do not include real MAC addresses in public documentation.

10. Add a VNC display

midclt call vm.device.create '{
  "vm": <VM_ID>,
  "attributes": {
    "dtype": "DISPLAY",
    "type": "VNC",
    "resolution": "1920x1080",
    "bind": "0.0.0.0",
    "wait": false,
    "password": "<VNC_PASSWORD>",
    "web": false
  },
  "order": 1003
}'

Do not publish real passwords.

11. Validate the recovered VM definition

midclt call vm.get_instance <VM_ID>

Check that the VM has:

  • One DISK device pointing to the old .block zvol.
  • One NIC device.
  • One DISPLAY device.
  • UEFI bootloader.
  • Expected RAM and CPU values.

12. Start the VM

From the shell:

midclt call vm.start <VM_ID>

Check status:

midclt call vm.status <VM_ID>

Or start it from the TrueNAS UI after confirming the VM appears under Virtual Machines.

13. Optional: update VM resources

If you initially created the VM with conservative settings, adjust RAM or vCPU later:

midclt call vm.update <VM_ID> '{
  "vcpus": 3,
  "cores": 1,
  "threads": 1,
  "memory": 16384,
  "autostart": true,
  "cpu_mode": "HOST-MODEL"
}'

Some fields visible in vm.get_instance may not be accepted by vm.update. For example, enable_secure_boot may be read-only or not accepted as an update field depending on the TrueNAS version.

14. Optional: change the VNC password

Find the display device ID:

midclt call vm.get_instance <VM_ID>

Then update the display device:

midclt call vm.device.update <DISPLAY_DEVICE_ID> '{
  "attributes": {
    "dtype": "DISPLAY",
    "type": "VNC",
    "resolution": "1920x1080",
    "bind": "0.0.0.0",
    "wait": false,
    "password": "<NEW_VNC_PASSWORD>",
    "web": false
  }
}'

If the password is not applied immediately, stop and start the VM.

15. Generic reusable command sequence

Replace the placeholders before running.

OLD_VM_NAME="<OLD_VM_NAME>"
NEW_VM_NAME="<NEWVMNAME>"
POOL="<POOL>"
BRIDGE="<BRIDGE_NAME>"
VCPUS="2"
MEMORY="8192"
VNC_PASSWORD="<VNC_PASSWORD>"

sudo zfs set volmode=dev "${POOL}/.ix-virt/virtual-machines/${OLD_VM_NAME}.block"

midclt call vm.create "{
  \"name\": \"${NEW_VM_NAME}\",
  \"description\": \"Recovered legacy Incus VM\",
  \"vcpus\": ${VCPUS},
  \"cores\": 1,
  \"threads\": 1,
  \"memory\": ${MEMORY},
  \"cpu_mode\": \"HOST-MODEL\",
  \"bootloader\": \"UEFI\",
  \"bootloader_ovmf\": \"OVMF_CODE_4M.fd\",
  \"autostart\": true,
  \"time\": \"LOCAL\",
  \"shutdown_timeout\": 90,
  \"ensure_display_device\": false
}"

Save the returned VM ID, then run:

VM_ID="<VM_ID>"

midclt call vm.device.create "{
  \"vm\": ${VM_ID},
  \"attributes\": {
    \"dtype\": \"DISK\",
    \"path\": \"/dev/zvol/${POOL}/.ix-virt/virtual-machines/${OLD_VM_NAME}.block\",
    \"type\": \"VIRTIO\",
    \"create_zvol\": false,
    \"iotype\": \"THREADS\"
  },
  \"order\": 1001
}"

midclt call vm.device.create "{
  \"vm\": ${VM_ID},
  \"attributes\": {
    \"dtype\": \"NIC\",
    \"type\": \"VIRTIO\",
    \"nic_attach\": \"${BRIDGE}\",
    \"trust_guest_rx_filters\": false
  },
  \"order\": 1002
}"

midclt call vm.device.create "{
  \"vm\": ${VM_ID},
  \"attributes\": {
    \"dtype\": \"DISPLAY\",
    \"type\": \"VNC\",
    \"resolution\": \"1920x1080\",
    \"bind\": \"0.0.0.0\",
    \"wait\": false,
    \"password\": \"${VNC_PASSWORD}\",
    \"web\": false
  },
  \"order\": 1003
}"

Result

Using this procedure, the VM can be recovered without reinstalling the guest OS.

The recovered VM keeps:

  • The original virtual disk.
  • The original EFI partition.
  • The original Linux installation.
  • The original data.

Only the TrueNAS VM definition is recreated.

Suggested places to share this

Good places to share a cleaned-up version of this guide:

  1. TrueNAS Community Forums
    Recommended category: Apps and Virtualization or SCALE.

  2. TrueNAS documentation GitHub
    Useful as a migration note or troubleshooting article suggestion.

  3. Reddit
    r/truenas, preferably linking back to the forum post.

Notes

This recovery method is useful when:

  • The old VM no longer appears in the TrueNAS UI.
  • The .block zvol still exists.
  • The guest OS can be mounted or inspected.
  • TrueNAS 26 can create a new VM via midclt.
  • The old zvol appears in vm.device.disk_choices.

This method does not guarantee recovery if the zvol was deleted, corrupted, or overwritten.

1 Like