EDIT: Changing physical drives didn’t “move” the problem but showed 2 other drives being offline. The backplane for the NVMe is broken.
Summary
(Good) morning,
Today I woke up with lots of alerts about 2 pools in a degraded state. Two NVMe drives were not available anymore. Thank god they were not pool critical.
My setup:
Pool 1
Z1 - 3 wide HDDs + Mirrored Log VDEV + Cache VDEV
Pool 2
Mirrored SSD
Drives are:
3 x Ultrastar HDDs
2 x Intel Optane NVMe
3 x Samsung 990 Pro NVMe
In Pool 1 - the Cache VDEV suddenly was unavailable. In order to troubleshoot the recommendation was to remove it from the pool and replace it with itself. I tried that but the drive is not shown.
Before I did this in the second pool, I made sure I got some information with:
sudo zpool status -v
I dont have the exact output anymore but I saved the crucial part:
4026472286734630819 UNAVAIL 0 0 0 was /dev/disk/by-partuuid/e45ce975-7c45-40a0-909a-84182c73a117
The error message was about this:
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
lsblk output doesnt show all the NVMe drives:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 21.8T 0 disk
└─sda1 8:1 0 21.8T 0 part
sdb 8:16 0 21.8T 0 disk
└─sdb1 8:17 0 21.8T 0 part
sdc 8:32 0 32G 0 disk
├─sdc1 8:33 0 1M 0 part
├─sdc2 8:34 0 512M 0 part
└─sdc3 8:35 0 31.5G 0 part
sdd 8:48 0 21.8T 0 disk
└─sdd1 8:49 0 21.8T 0 part
sr0 11:0 1 1.9G 0 rom
nvme2n1 259:0 0 3.6T 0 disk
└─nvme2n1p1 259:1 0 3.6T 0 part
nvme1n1 259:2 0 110.3G 0 disk
└─nvme1n1p1 259:3 0 109.2G 0 part
nvme0n1 259:4 0 110.3G 0 disk
└─nvme0n1p1 259:5 0 109.2G 0 part
lspci -vvv does with the following identifiers and I was able to single out the working one:
00:1c.0 = nvme-Samsung_SSD_990_PRO_4TB_S7DPNJ0WB04705A
03:0d.0 = does not show up with ls /dev/disk/by-path
03:0e.0 = does not show up with ls /dev/disk/by-path
The drives have been purchased in the same time period but from different vendors. They are also passed through to TrueNAS using Proxmox as hypervisor. The drives are installed on the same daughter board as the working NVMe drives which makes it so annoying to isolate the problem.
Yesterday(before this issue) I reinstalled TrueNAS and restored my configuration. I did this because the drives initially were not passed through and decided to give TrueNAS full hardware access. The restore went without ANY issue.
I would like to wipe the drives but I cant because they are not recognized. Currently I do not know which physical drive is affected(dont know how to cross-reference pci id to physical drive) so I am assuming the next step is to use another socket and if that doesnt work, I have to use them in another system to check if I can wipe them.
Anyone else have suggestions or ideas I can try?