Can't Passthrough PCI Device to SCALE VM

TrueNAS General
, ,
1

Is there any reason I would be getting this error when trying to pass through an Intel Audio device to a VM running on TrueNAS SCALE 24 Dragonfish?

 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 182, in start
    if self.domain.create() < 0:
       ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1373, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2024-04-25T19:01:41.670757Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:00:1b.0","id":"hostdev0","bus":"pci.0","addr":"0x8"}: vfio 0000:00:1b.0: failed to setup container for group 11: Failed to set iommu for container: Operation not permitted

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 198, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1466, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1417, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 47, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 58, in start
    await self.middleware.run_in_thread(self._start, vm['name'])
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1324, in run_in_thread
    return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 68, in _start
    self.vms[vm_name].start(vm_data=self._vm_from_name(vm_name))
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 191, in start
    raise CallError('\n'.join(errors))
middlewared.service_exception.CallError: [EFAULT] internal error: qemu unexpectedly closed the monitor: 2024-04-25T19:01:41.670757Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:00:1b.0","id":"hostdev0","bus":"pci.0","addr":"0x8"}: vfio 0000:00:1b.0: failed to setup container for group 11: Failed to set iommu for container: Operation not permitted

The device is
0000:00:1b.0 ‘Audio device’: 82801JI (ICH10 Family) HD Audio Controller by ‘Intel Corporation’

2

Similar problem reported here

Did you file a bug report ? Try rolling back to Cobia. I for one will wait with the update…

3

Apparently the issue on the linked topic fixed itself after a few reboots.

4

No I did not. I’m not sure if it was working on Cobia. I’ll have to install it and see.

5

Is it the fault of FreeBSD that I can’t even detect my two (NVIDIA and RADEON) GPU devices in order to pass them to a VM?

SCALE sees them.

6

Reverting to Cobia did not help.

7

Sounds like a problem with IOMMU groups, and you probably can’t just isolate the audio device with your motherboard.

1 Like
8

Understandable. See GPU devices not recognized by CORE + can’t be passed through on SCALE

I tried passing the entire controller through to no avail.

9

Its probably the same thing. You can’t only isolate/passthru part of an IOMMU group. You have to pass through all of it

eg d8:00.0 and d8:00.1

d8:00.0 VGA compatible controller: NVIDIA Corporation TU117GL [T1000 8GB] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Lenovo TU117GL [T1000 8GB]
        Flags: bus master, fast devsel, latency 0, IRQ 564, NUMA node 1, IOMMU group 10
        Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 39ffe0000000 (64-bit, prefetchable) [size=256M]
        Memory at 39fff0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at f000 [size=128]
        Expansion ROM at fb000000 [virtual] [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express
        Capabilities: [bb0] Physical Resizable BAR
        Kernel driver in use: nvidia
        Kernel modules: nouveau, nvidia_current_drm, nvidia_current

d8:00.1 Audio device: NVIDIA Corporation Device 10fa (rev a1)
        Subsystem: Lenovo Device 1612
        Flags: bus master, fast devsel, latency 0, IRQ 522, NUMA node 1, IOMMU group 10
        Memory at fb080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
10

Just tried. Found all the 0000:03:00:0 series devices (there were 2) but no go. Same error.

11

I got my VMs with PCI passthrough working for a short period of time, but after a SCALE reboot, it’s broken again for me and I can’t get it working in Dragonfish. Booting into Cobia and the VMs work just fine again. So it’s not an IOMMU issue on my end. Anybody any idea?

12

I’m passing PCI USB 3.0 controller to my VM. I upgraded to Dragonfish yesterday and everything worked fine until today’s reboot. Then I had the same error. After another reboot it did fix itself. From what I have read in the other thread there is a chance it will break again, but I don’t really have any ideas what could have changed between the reboots.