Can't Passthrough PCI Device to SCALE VM

TrueNAS General
, ,
1

Is there any reason I would be getting this error when trying to pass through an Intel Audio device to a VM running on TrueNAS SCALE 24 Dragonfish?

 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 182, in start
    if self.domain.create() < 0:
       ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1373, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2024-04-25T19:01:41.670757Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:00:1b.0","id":"hostdev0","bus":"pci.0","addr":"0x8"}: vfio 0000:00:1b.0: failed to setup container for group 11: Failed to set iommu for container: Operation not permitted

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 198, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1466, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1417, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 47, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 58, in start
    await self.middleware.run_in_thread(self._start, vm['name'])
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1324, in run_in_thread
    return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 68, in _start
    self.vms[vm_name].start(vm_data=self._vm_from_name(vm_name))
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 191, in start
    raise CallError('\n'.join(errors))
middlewared.service_exception.CallError: [EFAULT] internal error: qemu unexpectedly closed the monitor: 2024-04-25T19:01:41.670757Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:00:1b.0","id":"hostdev0","bus":"pci.0","addr":"0x8"}: vfio 0000:00:1b.0: failed to setup container for group 11: Failed to set iommu for container: Operation not permitted

The device is
0000:00:1b.0 ‘Audio device’: 82801JI (ICH10 Family) HD Audio Controller by ‘Intel Corporation’

2

Similar problem reported here

Did you file a bug report ? Try rolling back to Cobia. I for one will wait with the update…

3

Apparently the issue on the linked topic fixed itself after a few reboots.

4

No I did not. I’m not sure if it was working on Cobia. I’ll have to install it and see.

5

Is it the fault of FreeBSD that I can’t even detect my two (NVIDIA and RADEON) GPU devices in order to pass them to a VM?

SCALE sees them.

6

Reverting to Cobia did not help.

7

Sounds like a problem with IOMMU groups, and you probably can’t just isolate the audio device with your motherboard.

1 Like
8

Understandable. See GPU devices not recognized by CORE + can’t be passed through on SCALE

I tried passing the entire controller through to no avail.

9

Its probably the same thing. You can’t only isolate/passthru part of an IOMMU group. You have to pass through all of it

eg d8:00.0 and d8:00.1

d8:00.0 VGA compatible controller: NVIDIA Corporation TU117GL [T1000 8GB] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Lenovo TU117GL [T1000 8GB]
        Flags: bus master, fast devsel, latency 0, IRQ 564, NUMA node 1, IOMMU group 10
        Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 39ffe0000000 (64-bit, prefetchable) [size=256M]
        Memory at 39fff0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at f000 [size=128]
        Expansion ROM at fb000000 [virtual] [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express
        Capabilities: [bb0] Physical Resizable BAR
        Kernel driver in use: nvidia
        Kernel modules: nouveau, nvidia_current_drm, nvidia_current

d8:00.1 Audio device: NVIDIA Corporation Device 10fa (rev a1)
        Subsystem: Lenovo Device 1612
        Flags: bus master, fast devsel, latency 0, IRQ 522, NUMA node 1, IOMMU group 10
        Memory at fb080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
10

Just tried. Found all the 0000:03:00:0 series devices (there were 2) but no go. Same error.

11

I got my VMs with PCI passthrough working for a short period of time, but after a SCALE reboot, it’s broken again for me and I can’t get it working in Dragonfish. Booting into Cobia and the VMs work just fine again. So it’s not an IOMMU issue on my end. Anybody any idea?