NVMe drives not being shown in UI

TrueNAS General
1

Hey folks,

I have a TrueNAS system running on an Asrock X399 Fatality + Threadripper 1950x as a Proxmox VM with passtrough for a NVMe SSD and a SAS HBA. It’s been working totally fine and I have no issues with it, knock on wood.

However, a couple of days ago I purchased an Asus M.2 Hyper card to create another pool of 4 flash drives in RaidZ1 (drives are Samsung 990 Evo Plus, 2TBs).

I configured everything in BIOS, PCIE Biforcation for that X16 lane to 4X4, NVMe RAID to Off…and I think that’s it.

Now this is the point where I started having issues. I passed trough all 4 drives, they get recognized in Proxmox, and first time i plugged everything in and booted TrueNAS, only 3 out of the 4 drives were shown as available in the UI.
Thinking this was a sporadic issue, I tried rebooting the entire host and to my surprise, now only 1 of the drives did shows up. :\

I started tinkering a bit with it and it seems everything should connect correctly, but sadly it does not. I am at the point where none of the drives are recognized by the UI, nor by the OS itself. It seems the nvme driver is just not picking up the drives.

lspci -knn     # in truenas
..
03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a [144d:a80d]
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a [144d:a801]
        Kernel modules: nvme
04:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a [144d:a80d]
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a [144d:a801]
        Kernel modules: nvme

and on the host,

45:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a80d]
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a801]
        Kernel driver in use: vfio-pci
        Kernel modules: nvme
46:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a80d]
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a801]
        Kernel driver in use: vfio-pci
        Kernel modules: nvme

I only posted the output of 2 drives, but all 4 are there.

Any help with this would be appreciated, I am really clueless on where to go next.

2

How many PCIe slots are occupied?

3

All 4 of them. In the MoBo specs is says it should be able to do x16 / x8 / x16 / x8. I’ve the card on the first slot, the same one that is configured to do 4x4 bifurcation.
According to the docs, the mobo should be able to do all of them, at

4 x PCI Express 3.0 x16 Slots (PCIE1/PCIE2/PCIE4/PCIE5: single at x16 (PCIE1); dual at x16 (PCIE1) / x16 (PCIE4); triple at x16 (PCIE1) / x8 (PCIE2) / x16 (PCIE4); quad at x16 (PCIE1) / x8 (PCIE2) / x16 (PCIE4) / x8 (PCIE5))*

Maybe it would be worth mentioning that on the Proxmox host I was able to access all 4 drives and write to them after removing the driver blacklist while testing.

4

It’s odd that it which disks are visible to the guest isn’t consistent. I use similar nvme cards, but I don’t use TN virtualized.

Is it possible that however Proxmox is enumerating the disks, that they aren’t being passed to the guest properly? I assume passthrough has to be set for each disk? Something with IOMMU grouping?

5

Maybe it’s possible, probable even?

It is however not related to IOMMU, the groups are separated already :\


root@lab-02:~# pvesh get /nodes/lab-02/hardware/pci --pci-class-blacklist ""
┌──────────┬────────┬──────────────┬────────────┬────────┬──────────────────────────────────────────────────────────────────────────┬──────┬──────────────────┬───────────────────────────────────────────┬──────────────────┬────────────────────────────────────┬────────────────────────────────────┐
│ class    │ device │ id           │ iommugroup │ vendor │ device_name                                                              │ mdev │ subsystem_device │ subsystem_device_name                     │ subsystem_vendor │ subsystem_vendor_name              │ vendor_name                        │
╞══════════╪════════╪══════════════╪════════════╪════════╪══════════════════════════════════════════════════════════════════════════╪══════╪══════════════════╪═══════════════════════════════════════════╪══════════════════╪════════════════════════════════════╪════════════════════════════════════╡
│ 0x010802 │ 0x5017 │ 0000:42:00.0 │         39 │ 0x2646 │                                                                          │      │ 0x5017           │                                           │ 0x2646           │ Kingston Technology Company, Inc.  │ Kingston Technology Company, Inc.  │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼────────────────────────────────────┼────────────────────────────────────┤
│ 0x010802 │ 0xa80d │ 0000:45:00.0 │         42 │ 0x144d │                                                                          │      │ 0xa801           │                                           │ 0x144d           │ Samsung Electronics Co Ltd         │ Samsung Electronics Co Ltd         │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼────────────────────────────────────┼────────────────────────────────────┤
│ 0x010802 │ 0xa80d │ 0000:46:00.0 │         43 │ 0x144d │                                                                          │      │ 0xa801           │                                           │ 0x144d           │ Samsung Electronics Co Ltd         │ Samsung Electronics Co Ltd         │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼────────────────────────────────────┼────────────────────────────────────┤
│ 0x010802 │ 0xa80d │ 0000:47:00.0 │         44 │ 0x144d │                                                                          │      │ 0xa801           │                                           │ 0x144d           │ Samsung Electronics Co Ltd         │ Samsung Electronics Co Ltd         │
├──────────┼────────┼──────────────┼────────────┼────────┼──────────────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼────────────────────────────────────┼────────────────────────────────────┤
│ 0x010802 │ 0xa80d │ 0000:48:00.0 │         45 │ 0x144d │                                                                          │      │ 0xa801           │                                           │ 0x144d           │ Samsung Electronics Co Ltd         │ Samsung Electronics Co Ltd         │```

> I assume passthrough has to be set for each disk?
What do you mean by this, though? I have passed them trough to TN trough the UI, bu that is it. The Kingston NV2 drive in the above snippet is also passed trough and that works ok in TN, so I assume it's something related to how drives are enumerated using this adapter. :\
6

The recommendation with SAS HBA is to passthrough the whole controller, not just the individual drives, and to blacklist drives in proxmox. Promox can read zfs drives and sometimes tries to mount them before truenas, taking down the whole pool. There is a document somewhere on the this forum with hints for setting up a virtualized truenas server

Edit:

7

I could definitely be wrong, but since this card requires bifurcation I would expect it to be closer to an adaptor than an HBA. I’d expect to need to pass each disk to TN as if they were separate devices.

Not sure what to tell you from here. I don’t virtualize TN.

8

Yeah, that’s exactly the case @Jorsher.

Since the PCI-E slot was bifurcated, there’s no adapter like my LSI HBA, but rather Proxmox gets the 4 drives as NVMe PCI-E 4x ones.

However, they are blacklisted and using vfio-pcie driver which is the case for others who are actually correctly passed trough and working in TN.

root@lab-02:~# cat /etc/pve/qemu-server/100.conf 
bios: ovmf
boot: order=scsi0;ide2
cores: 4
cpu: host
efidisk0: local-lvm:vm-100-disk-1,efitype=4m,size=4M
hostpci0: 0000:08:00,pcie=1
hostpci1: 0000:42:00,pcie=1
hostpci2: 0000:45:00,pcie=1
hostpci3: 0000:46:00,pcie=1
hostpci4: 0000:47:00,pcie=1
hostpci5: 0000:48:00,pcie=1
machine: q35
memory: 36864
meta: creation-qemu=8.1.2,ctime=1702216538
name: warehouse
net1: virtio=BC:24:11:64:C9:DD,bridge=vmbr1,firewall=1,tag=3
net2: virtio=BC:24:11:BB:F6:26,bridge=vmbr1,firewall=1,tag=4
numa: 1
onboot: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,cache=writeback,discard=on,iothread=1,size=62G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=5c09209d-08cf-4cf8-8221-31c0118ff99c
sockets: 1
vga: virtio

First 2 PCI devices are an SAS/SATA HBA with 8 drives attached and working already, second is a NVMe directly attached to the mobo which is also working, but the last 4 are the drives from the adapter which are not actually picked up by the TN kernel driver.

01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a [144d:a80d]
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a [144d:a801]
        Kernel modules: nvme
02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a [144d:a80d]
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a [144d:a801]
        Kernel modules: nvme
03:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
        Subsystem: Dell 6Gbps SAS HBA Adapter [1028:1f1c]
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas
04:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. NV2 NVMe SSD SM2267XT (DRAM-less) [2646:5017] (rev 03)
        Subsystem: Kingston Technology Company, Inc. NV2 NVMe SSD SM2267XT (DRAM-less) [2646:5017]
        Kernel driver in use: nvme
        Kernel modules: nvme
05:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a [144d:a80d]
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a [144d:a801]
        Kernel modules: nvme
06:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a [144d:a80d]
        Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a [144d:a801]
        Kernel modules: nvme
9

Just to give back to the community with the hope someone who encounters the same issue finds this useful later, I managed to find and fix the root cause of this. Skimming journalctl on the proxmox host, I saw 4 lines similar to this:

kernel: vfio-pci 0000:09:00.0: Unable to change power state from D3cold to D0, device inaccessible

Altering modprobe to disable the d3cold state finally fixed the issue for me and all 4 drives are now instantly picked up by TN.

root@lab-02:~# cat /etc/modprobe.d/vfio.conf 
options vfio-pci ids=2646:5017,144d:a80d disable_idle_d3=1

^ The last option, disable_idle_d3=1 is the important one.

I’m also not sure why this did it, but I would expect the drives not supporting that low-power state.
For future reference, drives are Samsung 990 EVO Plus, 2TB.

Thanks everyone and have a nice weekend!

2 Likes