VFIO Passthrough on a 9070XT

Apps and Virtualization
1

I’m trying to GPU passthrough the new 9070 XT, but an not having any luck. From what I’m seeing someone else was successful with getting passthrough to work on the latest and greatest firmware, kernel, and drivers:

Seeing as 25.04 is coming out supporting kernel 6.12, which should be the minimum kernel supported by the 9070 XT, I wanted to bring up this issue in hopes the new stable release will have current enough drivers and firmware to be able to actually use the 9070 XT.

If not, would I be able to update the relevant firmware manually myself? I could use some guidance on that.

Here’s my dmesg output for reference:

[  237.061315] vfio-pci 0000:03:00.1: enabling device (0000 -> 0002)
[  237.064358] vfio-pci 0000:03:00.0: enabling device (0002 -> 0003)
[  238.153324] pcieport 0000:02:00.0: broken device, retraining non-functional downstream link at 2.5GT/s
[  239.157226] pcieport 0000:02:00.0: retraining failed
[  240.409305] pcieport 0000:02:00.0: broken device, retraining non-functional downstream link at 2.5GT/s
[  241.417307] pcieport 0000:02:00.0: retraining failed
[  241.417317] vfio-pci 0000:03:00.0: not ready 1023ms after bus reset; waiting
[  242.457341] vfio-pci 0000:03:00.0: not ready 2047ms after bus reset; waiting
[  244.761357] vfio-pci 0000:03:00.0: not ready 4095ms after bus reset; waiting
[  249.113304] vfio-pci 0000:03:00.0: not ready 8191ms after bus reset; waiting
2

You basically need bleeding edge firmware to get these to run properly.
The kind of OS tinkering required makes TrueNAS ill suited for the job.

AFAICT, even 6.12 is not good enough.

Check out this topic over on Level1Techs, where wendell summarises it pretty well:

3

Another post saying the cause was due to early binding of the vfio-driver:
https://www.reddit.com/r/VFIO/comments/1j59c94/amd_radeon_rx_9070_xt_reset_bug/

He didn’t have to use the latest linux-firmware-git nor patch the vbios. So unless kernel 6.13 is required, which from what I’ve been reading, 6.12 is the minimum, I don’t see much difficulty implementing a fix.

4

I’ll fully admit that there’s things I’m learning about TrueNAS as I learn and use it that have made me reconsider installing it on bare-metal. Though my usage of VMs is limited albeit a bit advanced. I just really don’t want to have to deal with the headache of migrating to something like TrueNAS in Proxmox if my VM needs can be solved by TrueNAS’ current implementation.

5

You can switch to dev mode if you want to make modifications. You can probably get it working with this, now, learning from the issues others have run into.

TrueNAS aims for stability over bleeding-edge hardware support. Intel Arc 3 support wasn’t immediately available and wasn’t supported in the official image until early 2024 iirc, over a year after launch. I was able to get it working by copying the needed files after switching TrueNAS to dev mode. The 9070 hasn’t even been out for a week :slight_smile:

I ended up moving to Scale from Core because Core didn’t properly support the 9400 HBAs I bought. I don’t regret the move, and TrueNAS will probably be more responsive to hardware support now that the codebase is shrinking, but it’s always good to check support before buying.

1 Like