Eureka!!!
The secret to making everything work:
Go into the BIOS->Chipset->North Bridge and change “VT-d” to Disabled, then save BIOS settings and reboot.
With VT-d disabled all of the issues using drives behind the ASM2806 go away - they all just work on a vanilla TrueNAS Scale 24.04 install.
I was able to create a raidz3 with four drives behind the ASM2806 (NVMe_1, NVMe_2, NVMe_3, NVMe_4) and one more directly connected (NVMe_5) and it worked just fine
With a clean install and VT-d disabled, I do still get a pcie bus error severity:corrected message every minute or two - it’s not annoyingly spamming, but might still add back “pcie_aspm=off” to kernel parameters to get rid of that error.
Disabling VT-d should more or less not impact anything unless you are running VMs and trying to do something funky with virtual device passthrough. “Apps” (Kubernetes/Docker) should not be affected by VT-d
Note that I’m about 30min into testing here so who knows what else works or doesn’t work - but it certainly looks a lot better!
