Cannot pass NVMe to VM in TrueNAS 26: User error or bug?

TrueNAS General
4

To close this out. Definitely PEBCAK.

  • Identify IOMMU groups and see whether there’s a slot that’s in its own group
for g in /sys/kernel/iommu_groups/*; do
  echo "Group ${g##*/}:"
  for d in "$g"/devices/*; do
    echo -e "\t$(lspci -nn -s ${d##*/})"
  done
done
  • As needed, move the NVMe physically so it’s in its own IOMMU group. If your motherboard manual has a system block diagram, look for PCIe slots and M.2 that are connected to the PCH, not to the CPU.
  • If the BIOS has ACS in addition to Vt-d, that’s a way to get it into its own group, as well, without needing to move it physically.
  • Verify it’s in its own group, run that command above again. The hard part is done.
  • lspci -D and get the slot of the drive including PCI domain, in my case 0000:0b:00.0
  • Create a script nvme-passthrough.sh somewhere on one of your pools in /mnt or if you like in /root/scripts, make it executable with chmod +x nvme-passthrough.sh
#!/bin/sh
slot=0000:0b:00.0  # Adjust this to be your slot
modprobe vfio-pci
echo $slot > /sys/bus/pci/devices/$slot/driver/unbind 2>/dev/null
echo vfio-pci > /sys/bus/pci/devices/$slot/driver_override
echo $slot > /sys/bus/pci/drivers/vfio-pci/bind
  • Run the script once to see that it works, verify with lspci -v that the drive is bound to the vfio-pci driver
  • In System->Advanced, call this script as a POSTINIT script
  • Reboot, verify with lspci -v that the drive is bound to the vfio-pci driver after reboot

And pass through to a VM

2 Likes
Can't pcie passthrough my NIC (iommu group)
Cannot pass NVME through to a VM