PCI device listed as Not Available for passthrough after trying to start VM

Hello! I have an old ATSC Tuner card. It has special functionality I have not found on any other card, but drivers were discontinued after Windows ME. I had an old machine running '98 just to access the card’s features. Now I’m setting up a NAS with virtual machines, I’m trying to see if I can put that card in the NAS machine and access the tuner card via a VM, so I can get rid of the old Windows 98 machine. When I went to set up the PCI passthrough, the card showed up in the drop-down box. However, when I try to start the VM with the passthrough device on the list I get the following Error:

Error: Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py”, line 182, in start
if self.domain.create() < 0:
^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/libvirt.py”, line 1373, in create
raise libvirtError(‘virDomainCreate() failed’)
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2024-11-19T18:52:24.204940Z qemu-system-x86_64: -device {“driver”:“vfio-pci”,“host”:“0000:06:04.0”,“id”:“hostdev0”,“bus”:“pci.0”,“addr”:“0x6”}: vfio 0000:06:04.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 198, in call_method
result = await self.middleware.call_with_audit(message[‘method’], serviceobj, methodobj, params, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1466, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1417, in _call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 187, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 47, in nf
res = await f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py”, line 58, in start
await self.middleware.run_in_thread(self._start, vm[‘name’])
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1324, in run_in_thread
return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1321, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py”, line 68, in _start
self.vms[vm_name].start(vm_data=self._vm_from_name(vm_name))
File “/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py”, line 191, in start
raise CallError(‘\n’.join(errors))
middlewared.service_exception.CallError: [EFAULT] internal error: qemu unexpectedly closed the monitor: 2024-11-19T18:52:24.204940Z qemu-system-x86_64: -device {“driver”:“vfio-pci”,“host”:“0000:06:04.0”,“id”:“hostdev0”,“bus”:“pci.0”,“addr”:“0x6”}: vfio 0000:06:04.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy

Another error comes up on the NAS terminal:
"genirq: Flags mismatch IRQ 16 00000000 (vfio-intx:0000:06:04.0)) vs. 00000080

In addition, now the device shows up as “Not Available: Not Available” where it previously listed the brand and function of the device in the PCI passthrough dropdown. If I remove the passthrough device, the VM starts, but obviously with access to the device.

I’m running TrueNAS Scale on a Dell T3610 Workstation with a Xeon 6-core processor and 32 Megs of memory.

I understand that I may need to make sure Linux doesn’t load any drivers for the device, but I’ve never been able to get the device to work on Linux, despite much trying in the past so I don’t think it can be loading any drivers for it, certainly none that work. I believe that virtualization is enabled in the BIOS as the processor is successfully being passed through to one of my other VMs.

Any ideas?

That should be obviously without access to the device.

Mind providing output of

dmesg | grep -e DMAR -e IOMMU

If iommu is enabled might need to isolate the device before it gets passed through.

The command below would list all the pcie devices & if they are in use by TrueNAS. Since it also looks like that device crashed after you tried to pass it through, you might need to reboot. for valid results.

lspci -k

Here’s the dmesg output. I’ll send the lspci output when I get the chance to reboot. Looks like IOMMU is enabled. I’ve seen some information on how to isolate the device, but not from the web gui, so I’m worried changes won’t stick.

[ 0.009791] ACPI: DMAR 0x0000000088FFFCE8 0000B4 (v01 A M I OEMDMAR 00000001 INTL 00000001)
[ 0.009805] ACPI: Reserving DMAR table memory at [mem 0x88fffce8-0x88fffd9b]
[ 0.019610] DMAR: IOMMU enabled
[ 14.384774] DMAR: Host address width 46
[ 14.431019] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[ 14.495098] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[ 14.590646] DMAR: RMRR base: 0x0000008c6df000 end: 0x0000008c705fff
[ 14.666251] DMAR: ATSR flags: 0x0
[ 14.706196] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[ 14.781816] DMAR-IR: IOAPIC id 0 under DRHD base 0xfbffc000 IOMMU 0
[ 14.858489] DMAR-IR: IOAPIC id 2 under DRHD base 0xfbffc000 IOMMU 0
[ 14.935158] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000
[ 15.000277] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 15.104556] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 24.536734] DMAR: No SATC found
[ 24.574545] DMAR: dmar0: Using Queued invalidation
[ 28.396329] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 30.151172] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.

If it doesn’t require dev mode, then at worst you can make it into a post-init script that runs after you reboot - best way to make something stick.

Won’t that get removed if I ever upgrade?

Post-init scripts specifically wouldn’t - you set them to run during boot. Upgrading may have some level of unforeseen impact, but considering you’d run the script as the system boots, it’d bring you back to expect results.

That being said, if you need dev mode to get it done, then yes - it would not survive after an upgrade & you’d need to re-enable dev mode to get it working again.

So, I’ve been using the root console that comes up on the head to make changes to settings I can’t find in the interface. Is this what you mean by dev mode?

Okay, took the time to restart the machine. So many things were broken I don’t even know where to start or that I even know what they all are. The web interface wouldn’t come up, middlewared didn’t start, the menu on the head didn’t come up, the ethernet interface wasn’t up, didn’t even have an IP address, got dumped into root shell, when I ran lspci, dozens of errors. So I turned everything off, removed the ATSC tuner card from the machine and started it up again. Now everything is working again. This doesn’t look promising, does it? If I get in the mood for being raked over coals and decide to try putting it in again, anything I should try or look out for?

No, I meant the following: warning caution danger; here there be dragons

Maybe the NAS just really doesn’t like the tuner card - maybe you really are stuck with a separate system for it…

Yeah, kinda looks that way. The only thing I’ve used the root console for was changing some settings for one of my VMs that just wouldn’t run properly without settings that qemu supports but that aren’t options in the web gui. I’m hoping that’s pretty harmless compared with what I could screw up in dev mode.