I went to add a new nic to my TrueNAS server, and after doing so noticed my main pool showed half the vdevs unassociated. I moved the card back to the original PCI slot and now the pool is back. Has anyone seen this before ?
TrueNAS-SCALE-23.10.1
Could be motherboard related - maybe when you put things into the second pcie slot the first gets less lanes or something.
the slot may look x16 but depending on what is plugged where, the motherboard will make them electrically x8, x4 or x1 !.
To know for sure you’d have to read the motherboard’s caveats ( I mean manual )
So yes, nothing new 
This info was in my signature, ill add it back guess havent done that since forums switched…
Motherboard is Supermicro X11SSM-F
Card was originally in CPU SLOT7 PCI-E x8, moved it to Slot 4 which looks like x4
So does the addressing of the disk get lost when changing slots? I thought zfs wrote metadata on the drives so if they move around it doesnt matter?
Thanks for the info, appears this is what happened. Why do the lanes matter, i thought metadata was written to each drive for zfs to recognized which drives belong to which vdevs, and therefore pools. Thanks!
Two different beasts.
There has to be more to this, ill google it…
Is the HBA recognised by Truenas in the new slot ?
lspci -v
I didnt read the manual, but maybe it gets deactivated when using other slots/sata lanes.
AFAIK ZFS vDevs can be registered either with a device name e.g.:
# zpool status -v boot-pool
pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:00:41 with 0 errors on Wed Oct 2 03:45:44 2024
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
sdf3 ONLINE 0 0 0
errors: No known data errors
or a GUID e.g.
# zpool status -v hdd-pool
pool: hdd-pool
state: ONLINE
scan: scrub repaired 0B in 02:57:09 with 0 errors on Sat Sep 7 11:57:16 2024
config:
NAME STATE READ WRITE CKSUM
hdd-pool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
e120bc5d-5515-4e1b-a515-58576570fd93 ONLINE 0 0 0
f0370910-036b-49d2-be08-3fe5b84d06da ONLINE 0 0 0
9a82df37-47a6-414f-975f-4a9a25b17f11 ONLINE 0 0 0
ec1b0953-d204-4bd8-85a9-666cd5f91b0f ONLINE 0 0 0
4036ded8-f619-4f8c-aaad-6129cbaa1636 ONLINE 0 0 0
errors: No known data errors
I have seen my device-name pools be successfully imported when the device name has changed, but a GUID is supposed to be more resilient to such changes.
ZFS pools created through the TrueNAS UI should have GUIDs. Pools created through earlier versions of e.g. FreeNAS may have device names.
1 Like
Somewhere I think I missed is the make/model of the HBA you are using. It is important.
If your HBA requires 8 PCIe lanes and you moved it into a 4 PCIe lane slot, that of course makes a difference. If this is the case, you can use slots 6 or 7 only unless the HBA can be configured to use 4 PCIe lanes only.
Also, you should provide a detailed description on where each one of your drives is physically connected to the computer. As I count you have 8 drives for your pools total, and I have no idea about your boot drive as you do not mention it. That is the same amount of SATA ports you have built into the motherboard. I do not promote this as a general rule however for the time being, you could boot from a USB Flash Drive and run your system that way for now. Or move the HBA into slot 6 or 7 which is your best option right now based on the limited information you provided.
With today’s hardware, a person needs to educate themselves on what is physically/electrically required to make it work. It is not as simple and, if it fits it must work. This has nothing to do with TrueNAS, it is basic computer crap these days.
The card is an LSI 9211-8i.
Here is the pool in its current working state appears to guid. I might move this card again and see if its just not populating or what the deal is.
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 22:34:54 with 0 errors on Tue Sep 17 07:34:56 2024
config:
NAME STATE READ WRITE CKSUM
DeepEnd ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
21793e9e-0f99-447f-ba20-db9304e0d343 ONLINE 0 0 0
2850ec98-09ef-449c-b56d-6ea30be94f31 ONLINE 0 0 0
23118ac0-6fdd-40ed-a5d8-af65e86151a3 ONLINE 0 0 0
2f7bf464-265d-4c8d-ae7c-5c229ff692d6 ONLINE 0 0 0
e3d5770a-6c60-46c6-b3cc-cf5e1377da96 ONLINE 0 0 0
Weird - it should work in just a slower way with only 4 lanes assigned to it.
I noticed in the image you provided is that the top PCIE slot connects directly to the CPU, while the bottom slot instead goes through the chipset… might cause some oddities?
Lastly I guess, while it shouldn’t make a difference, you can check that the card is on latest firmware & flashed to IT mode.
sas2flash -list
Should return the following:
Firmware Product ID : 0x2213 (IT)
Firmware Version : 20.00.07.00
2 Likes
I agree with that assessment, however who made the HBA? Some hack shop?
Hopefully it is a firmware issue and a simple update will make it all work again.
Hey majerus i would check the bios and see it bifrocation is turned on on that pci port it may disable one of the ports on the card worth a look. because i have taken my hba from one server to the other and the drives all showed up even with my disk shelf
1 Like
It works just fine on that 1st slot (x8) even if it does not in that last slot (x4).
When software has a design issue is soft. When is hardware, it may very well be architectural, and no way to flash a fix
But I believe that the OP is ok if it has to keep it in the x8 slot, that after all for a NAS, is better to have full speed than half in a x4, that for what it has shown didn’t work anyway.
Ill take a look, good suggestion.