Your question, while fascinating, is does not give enough data to formulate an answer. Please read Joes Rules (linked below in my signature) to post the data we need. I suspect the hardware is the issue off the top of my head.
When you list the hardware, also list “exactly” how all drives (including the 7.3TB formatted drives) are physically connected to the system. Example: drive sda is connected to SATA port 1, nvm0 (serial number) is connected to motherboard M.2 slot 1. I really do think it is hardware however let’s see what you have going on and see if we can fix it.
That is a lot of NVMe drives, I like it. If you can include a few photos of the system (insides), that would be nice for me to see, however it is not required.
As requested pics of the insides, so the x21 which has the m2 drives is a single pci card, that hosts 21 nvme drives
The x21 is connected to the top PCIe 4.0 x16 slots - all drives simply plug into this card
The SAS2308 is connected to Middle 4.0 x16 slots - sas cables them connect to the 5 x 8tb drives
I will remove all the drives and test them off the board to see which drives are not appearing and see if they appear within windows or if they damaged.
I would not do that. There is no reason for these drives to fail, especially so many of them.
What I would do is to examine the M.2 card, and record which slot each NVMe is plugged into, by serial number. Why do I ask that? To see if there is some commonality.
Let me digest this data, I have something else in mind.
You have 14 drives on the PCIe card and the 2 boot-pool drives are on the motherboard, correct?
The nic is plugged into the very bottom of the motherboard it’s a pci 1x slot, working fine, I don’t get full 10g throughput, the hba is old gear that the x21 is replacing and will be removed once I get the x21 full working along with its attached drives.
so getting back to this, I reinstalled scale this morning, after removing the onbaord nvme drives and the old lsi card plus the old hdds as well.
During the install of scale, I could see a further four nvmes I could select to install scale on, I installed scale on two sata ssds I had installed, however once in scale, no new drives were present.
Looking at dmesg, I can see that four drives simply just don’t bother appearing.
