Passing GPU through VM - running out of physical options?

Apps and Virtualization
1

Hi All,

Running SCALE Dragonfish-24.04.0

I want to pass through a dedicated GPU to my Ubuntu VM but running into issues. When creating a VM and assigning the GPU, system reports:

"At least 1 GPU is required by the host for its functions.

With your selection, no GPU is available for the host to consume."

So I tried passing through the GPU as a PCI device, but the VM does not start at all.

I then tried to isolate the GPU but getting the same error in quotes above. My CPU does not have iGPU, so currently there is only the dedicated GPU in the system.

Is my only option to get an additional GPU to set as the main display device and then be able to isolate the secondary GPU? I don’t understand how it’s “required for functions” by the host if it runs totally fine without an iGPU and dedicated GPU?

Thanks for any feedback in advance.

2

Basically yes.

3

I wish GPU SRIOV was here already.

In the future we may be able to split GPUs into Virtual Functions and pass slices of GPUs into VMs etc, but that future is not here.

The good thing with server boards is that they have an IPMI BMC which includes bad onboard video. Which is sufficient for TrueNAS console

4

So I went ahead and installed a GT710, and now I isolated my RTX 3070 which I want to pass through. I spent about 4 hours last night trying to pass it through to the VM and I am kind of giving up.

My problem is when adding the VGA and audio device to the VM, I get an infinite “Please wait” box prompt as I start the VM. And I have no clue what to do next. Somehow I managed to once get the GT710 passed through, but think it was sheer luck. i’ve tried swapping around physical PCIe slots and tried booting from each GPU independently. Perhaps my hardware setup is not optimal for TrueNAS SCALE.

5

Too bad I am just a scrub running his TrueNAS from consumer ATX components XD Would like to have a proper setup one day though

6

For it to work, the GPU and the audio device needs to be in a discreet IOMMU group.

You can check by running this script:

#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU Group %s ' "$n"
  lspci -nns "${d##*/}"
done

Save this script on your truenas as IOMMU.sh
Then run it with

bash IOMMU.sh
7

Thanks for this.

IOMMU Group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2 08:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3070 Lite Hash Rate] [10de:2488] (rev a1)
IOMMU Group 2 08:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)

Seems like they are in group 2 together with those dummy bridges. Do they need to be completely separate in their own IOMMU group?

8

Depending on what is associated with those bridges - they will be passed through aswell - it will not work.

I only ever read about cases working when ONLY the GPU/audio is in that group.

But I am not a expert in PCIe design at all…

9

Thanks for the advice. I will try isolate them into their own groups and hope for the best.