Upgrade from Electric Eel 24.10.2.1 to Fangtooth 25.04.0 breaks PCI devices

TrueNAS General
1

I have working installation of 24.10.2.1 with 6 SATA drives in a single pool direct connect to the motherboard, another 6 SATA drives making up various other pools connected to a LSI SAS3008 external PCI card, and 4 m.2 PCI-E cards making up another single pool connected to the motherboard. System is a ZimaCube Pro for reference. Also, TrueNAS is running as a VM in Proxmox.

When I upgrade from the current install to Fangtooth 25.04.0, all the PCI expansion (SAS3008 and all m.2 drives) stops working. There is an error during boot, but it goes by too fast to read. All pools still show in the dashboard with the error “Offline Vdevs”. Going to System > Boot and selecting my 24.10.2.1 environment and rebooting gets me back to a working system.

Any idea what is causing my issues?

2

It sounds like the VM does not have the PCie devices passed through.

I would try hard to capture the boot error…

3

It definitely has the PCI devices passed through, checked in proxmox to be sure. Especially sure since reverting to Electric Eel everything worked immediately.

I figured i was going to have to try and capture the error when i try to upgrade again. Probably use Microsoft Snipping tool since it auto saves a screenshots of whatever you snip. Then i can make auccessive snippets as i aee erroes scroll by. When i get time to try again i will come back and post the errors.

4

My TrueNAS is running virtuelised in Proxmox with the SATA controller passthrough to the TrueNAS VM.
Running with TrueNAS-24.10.2.2, there is no issues. All the drives shows up and all is working.

When I upgrade to TrueNAS-25.04.1, all the drives on the SATA controller stops working.

This is I can see in the log from the boot:
[ 1.332448] ata7: SATA link down (SStatus 4 SControl 300)
[ 1.332532] ata9.00: failed to IDENTIFY (I/O error, err_mask=0x100)
[ 6.426358] ata10: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 6.426468] ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 6.426589] ata10.00: failed to IDENTIFY (I/O error, err_mask=0x100)
[ 11.546161] ata12: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 11.546426] ata11: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 11.546471] ata12.00: failed to IDENTIFY (I/O error, err_mask=0x100)

This goes for all the drives.

lspci shows that the SATA controller is still visable in TrueNAS but for some reason it unable to identify the drives.

Any have a suggestion as to why and what can fix the issue?

5

This is unlikely to directly help you fix your problem, but I’ve definitely observed a change in PCI and passthrough behaviour moving from 24 to 25.

My problem came to down to Intel iommu and VT-d settings. Hopefully this will help in some way.

1 Like
6

Thanks for the link WiteWulf, changing intel_iommu to off fixed the issue and all my drives are now working in 25.04.1.

Now I just need to find out how to make this change permanently.

1 Like
7

This thread explains how to permanently add kernel boot parameters:

Nb. if you add ‘intel_iommu=off’ your grub command line will have both intel_iommu=on and intel_iommu=off, but the ‘off’ switch cancels the ‘on’ one.

I also went into the BIOS settings for my server and disabled Intel VT-d (which is the hardware feature intel_iommu uses), just in case my grub command line gets reset at some point in the future.

Be aware that this may have profound effects on your system, as you’re running a hypervisor with PCI pass through, iirc. As I suggested in my thread, If really appreciate someone who knows more about the subject (ideally one of the ix folk) weighing in on this.

Anyways, I’m really glad you got this sorted :+1: