Cant separate IOMMU groups

Apps and Virtualization
,
1

Hello,

I am trying to separate the IOMMU in a new TrueNas Scale bare metal build.
I have the following specs:

OS Version:TrueNAS-SCALE-24.04.2.3
Product:Z490 AORUS ULTRA
Model:Intel(R) Core(TM) i9-10900K CPU @ 3.70GHz
Memory:47 GiB
System Serial:Default string

I have tried the the pcie_acs_override=downstream and i can see it in the kernel parameters after reboot by typing cat /proc/cmdline.

BOOT_IMAGE=/ROOT/24.04.2.3@/boot/vmlinuz-6.6.32-production+truenas root=ZFS=boot-pool/ROOT/24.04.2.3 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N pcie_acs_override=downstream

The IOMMU groups doesnt seems to be separated and more specific the PCIE (16x) device. Thus i cannot pass it through for a gaming VM.

IOMMU Group 2:
00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 05)
02:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]

Any insights would be very helpfull. I have searched hours for a solution but i found nothing. Maybe is a motherboard issue ?

Thanks in advance.

2

The ACS Override Patch does not work on Truenas SCALE.

Therefore your options are to try to plug your GPU into another PCI slot, sometimes that separates it into its own IOMMU group, getting server grade hardware where this usually is not a problem, or switching to Proxmox for VMs.

1 Like
3

Hello ,

I’m facing the same issue.
Any updates on this would be much appreciated!

Thanks in advance!

4

IOMMU grouping is a matter of implementation by the manufacturer.
Server/workstation-class hardware usually gets it right. Consumer-grade hardware might not.

5

Thank you for your input. Do you have any Server/workstation-class hardware compatible with my CPU. I dont have the availability to buy a fresh setup. It is a homemade nas server.

Thanks

6

Question: Why are you trying to pass a PCIe root port through rather than the GPU device itself?

Show the IOMMU group that contains the GPU, please.

7

W480D4U

This nice little board should support your CPU (according to ASRock Rack).
However you can never be sure about the IOMMU Groups until you try it.
They do nicely seperate on my ASRock Rack board EPC621D8A, but thats no guarantee.

1 Like
8

Also Supermicro X12SCA-F, X12SAE-F.

1 Like
9

They seem to be easier available than the AsRock board :+1:t2:

10

sorry for the delay:

IOMMU Group 2:
00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 05)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] [10de:1c82] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
02:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]

I succesfully isolated the GPU. But know i get the following error when passing it to VM

[EINVAL] gpu_settings.isolated_gpu_pci_ids: pci_0000_00_01_0, pci_0000_01_00_1, pci_0000_02_00_0, pci_0000_01_00_0, pci_0000_00_01_1 GPU pci slot(s) are not available or a GPU is not configured.
11

Definitely a hardware-based inability to isolate the full group. You’ll need to use a more server/workstation-oriented board as suggested by some of the other users. The Supermicro X12SCA seems to be the cheapest of the suggestions here, but I’ve never used one personally and can’t attest to a guarantee as to its ability to split IOMMU groups down.

12

Careful - your Ethernet controller seems to be in the same group. Booting that VM may result in your TrueNAS system going offline. Disable Autostart on the VM first before trying.

2 Likes
13

You could also ask Gigabyte technical support for an updated BIOS with better IOMMU grouping.
Gigabyte did it recently for the MC12-LE0 board, after users inquired to be able to pass through the iGPU of their ryzen APUs. But that was the server arm of Gigabyte; don’t put your hopes too high on the consumer arm.

14

Thank you all for your help. I believe i will try also True nas over Proxmox i think its the best bet.