Cannot pass NVMe to VM in TrueNAS 26: User error or bug?

TrueNAS General
1

I am trying to create a VM with an NVMe passed through to it. The NVMe is not in use in any pool.

I am getting an error “device is not available”. Traceback and lspci below.

Is this user error, or likely to be a bug in TN 26? If the latter, I’ll open a Jira for it.

It might be user error, if CE needs additional Tunables the way CORE did, to allow PCIe passthrough.

Cannot start domain 'docker':
device.pci_0000_05_00_0: Device pci_0000_05_00_0 is not available

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 387, in process_method_call
    result = await method.call(app, id_, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py", line 57, in call
    result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                                   message_id=id_)
                                                   ^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1118, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                              audit_callback=audit_callback_messages.append, **kwargs)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 945, in _call
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 798, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 217, in wrapped
    result = func(*args)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 55, in start
    self.middleware.libvirt_domains_manager.vms.start(self.pylibvirt_vm(vm, options))
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/truenas_pylibvirt/domain/manager.py", line 66, in start
    raise Error(f"Cannot start domain {domain.configuration.name!r}:\n{error_msg}")
truenas_pylibvirt.error.Error: Cannot start domain 'docker':
device.pci_0000_05_00_0: Device pci_0000_05_00_0 is not available

05:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Intel Corporation Device 00e9
        Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 2
        Memory at df710000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at df700000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
        Capabilities: [60] Express Endpoint, IntMsgNum 0
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Virtual Channel
        Capabilities: [180] Power Budgeting <?>
        Capabilities: [190] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [270] Device Serial Number 55-cd-2e-40-4b-ff-92-df
        Capabilities: [2a0] Secondary PCI Express
        Kernel driver in use: nvme
        Kernel modules: nvme
2

My guess is the below. The Linux kernel has attached the nvme driver to the device, thus, the device is “is not available” for pass through.

I would guess you have to black list the device in the TrueNAS CE kernel first. Not sure how to do that, unless you have no other NVMe devices. In which case, you can black list the nvme kernel module.

3

Hmm good guess. I am testing a little before thinking on how to do anything automatically via udev

With vfio driver bound to the device:

internal error: QEMU unexpectedly closed the monitor (vm='b3889abf-8de6-47b2-b333-1ea4a8c05530'): 2026-04-19T09:41:58.897647Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev0","bus":"pci.0","addr":"0x8"}: vfio 0000:05:00.0: group 2 is not viable Please ensure all devices within the iommu_group are bound to their vfio bus driver.

With no driver bound to the device, same issue.

This is user error still. I have everything (like all my critical host/board devices) in Group 2. I’ll check BIOS to see whether I have ACS/IOMMU options. It’s possible I won’t be able to move this thing into its own group.

4

To close this out. Definitely PEBCAK.

  • Identify IOMMU groups and see whether there’s a slot that’s in its own group
for g in /sys/kernel/iommu_groups/*; do
  echo "Group ${g##*/}:"
  for d in "$g"/devices/*; do
    echo -e "\t$(lspci -nn -s ${d##*/})"
  done
done
  • As needed, move the NVMe physically so it’s in its own IOMMU group. If your motherboard manual has a system block diagram, look for PCIe slots and M.2 that are connected to the PCH, not to the CPU.
  • If the BIOS has ACS in addition to Vt-d, that’s a way to get it into its own group, as well, without needing to move it physically.
  • Verify it’s in its own group, run that command above again. The hard part is done.
  • lspci -D and get the slot of the drive including PCI domain, in my case 0000:0b:00.0
  • Create a script nvme-passthrough.sh somewhere on one of your pools in /mnt or if you like in /root/scripts, make it executable with chmod +x nvme-passthrough.sh
#!/bin/sh
slot=0000:0b:00.0  # Adjust this to be your slot
modprobe vfio-pci
echo $slot > /sys/bus/pci/devices/$slot/driver/unbind 2>/dev/null
echo vfio-pci > /sys/bus/pci/devices/$slot/driver_override
echo $slot > /sys/bus/pci/drivers/vfio-pci/bind
  • Run the script once to see that it works, verify with lspci -v that the drive is bound to the vfio-pci driver
  • In System->Advanced, call this script as a POSTINIT script
  • Reboot, verify with lspci -v that the drive is bound to the vfio-pci driver after reboot

And pass through to a VM

2 Likes
Cannot pass NVME through to a VM