I have just upgraded to ElectricEel-24.10.1 my NAS. I am running TrueNas on a Dell Poweredge R730 Server, 256GB ECC, Dual Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz, connected to an MD1400 storage unit with 12x4TB Toshiba drives.
After the upgrade I noticed that on the Storage Dashboard I have 12 unused disks. But I can also see the the Volume which contains the 12 drives.
I can access the volume, create SMB shares and access the files, copy new files.
If I select “Add To Pool” I can see * 3.64 TiB HDD x 12 drives unused. If I select add to existing pool, I get this error:
Warning: There are 12 disks available that have non-unique serial numbers. Non-unique serial numbers can be caused by a cabling issue and adding such disks to a pool can result in lost data.
Why are the drives showing unused? Is there a way to clear this?
Hi Arwen,
Thanks. The drives are connected to the server via 6Gbs PERC card. The interesting piece is this server is showing this issue, which was not there in previous BlueFin installation. I initially upgraded, but since I have re-installed fresh. Still the same issue.
If that PERC card is not running IT firmware, (which it probably is not because you have the non-unique serial number issue), you can have other problems.
Hardware RAID controllers, even in JBOD mode, are not recommended for use with ZFS. This is because some don’t guarantee in-order writes and write flushing. That can break one of ZFS’ most important features, always consistent on disk.
Normally ZFS uses COW, Copy On Write, to keep the pool always consistent. Even on power loss or server crash. Either a write completed all the way, or it is thrown out on such an event. Part of the way this is handled is that these are written in order;
New data is written to free space
Metadata / directory entry for that new data is updated and written to free space
Critical Metadata that points to #2 is updated and write is complete.
Out of order writes can corrupt a ZFS pool beyond repair. Having Metadata, (aka directory entries), point to garbage or having Critical Metadata point to garbage due to out of order writes is really bad. There are few repair tools for that problem.
All that said, you could go years without any problems. Maybe the entire life of your TrueNAS server. But, this forum, (and the old forum), is littered with sad tales of pool corruption due to hardware RAID controllers not working right with ZFS.
Hi Arwen good to know. But the issue started at current release update, not on any other release before. Also the repeated serials os because the drives are seen as free and taken.
I will look through the card details. By the way my other server uses 2x LSI 300 cards. Hope this setup is not at risk too
I seem to be getting this warning as well. (New to the forum as well!)
The new version of TrueNAS Scale, ElectricEel-24.10.1 give the same error you describe when trying to create a pool and lists a number of unused disks that are already in pools. These disks are listed twice in the disk section of the pool and as unused disks.
Dragonfish-24.04.2.5 works fine however no errors and no unused disks. So I have rolled back the update for the time being. I can always boot into the new version if I need to test stuff for you guys if anyone has any ideas.
My System:
Dell Poweredge R730XD; 2 Intel Xeon E5-2667 CPU’s; 256gb RAM; and Boot Disk 256gb running Proxmox and Virtualizing TrueNAS. Internal raid card (PERC H730P Mini) set to IT Mode and PCIe LSI Card (LSI SAS 9207-8e) for the disk enclosures The VM has the following specs: 8 CPUs, 64gb RAM, bootdisk 32gb. Attached to the LSI Card are 1xMD1220 and 2xMD1200 with many disks I can get into specifics here if you like but I have a number of 1Tb, 1.2 Tb, 2 Tb, and 22 Tb disks in the.
I also get the error:
Warning: There are 30 disks available that have non-unique serial numbers. Non-unique serial numbers can be caused by a cabling issue and adding such disks to a pool can result in lost data.
when I try to make a pool. I was able to import my existing pools but I worry about my Data… Which I do have backed up. BUT this is my main TrueNAS environment.
It is possible that the lack of warning before was a bug. Then Electric Eel fixed that bug.
Next, multi-pathing of disks is not supported in TrueNAS SCALE. Too many people accidentally re-used active disks and trashed a pool. Or came close to it.
@Seserie - Do you have more than 1 SAS controller card wired to the same disk enclosure?
That would be considered multi-pathing. Using multiple cables from the same controller card is NOT considered disk multi-pathing, and simply allows more data bandwidth to the attached disks.
@Inertia - It does not appear your instance is disk multi-pathing. But, it is possible if you selected the “wrong” ports on the disk storage array.
Some higher end storage arrays support 2 SAS expander domains. One would be to SATA disks and port 0 of SAS disks. One would be to port 1 of SAS disks, and potentially a SATA interposer card which has a simulated SAS port 1.
Some of these higher end storage arrays would then have 2 ports in SAS domain 1 and 2 ports in SAS domain 2. This would allow daisy chaining the disk array to another disk array. But, selecting 1 port in each domain might cause the duplicate serial number problem.
I just don’t know. Fibre Channel disks used a different method on disk identification for multi-pathing.
However, you both bring up a valid point. We need to find the cause of your issue.
This is actually “legal” and “legitimate” SAS configuration. The caveats, (which you may know), are:
A single SAS controller chip accessing a single SAS expander of the disk array, regardless of using 4, 8, 12 or 16 SAS ports, (1 - 4 cables).
A disk enclosure with dual SAS expanders can only have 1 SAS expander wired. Otherwise it would be multi-pathed.
In years gone by, I supported multiple medium end disk arrays with Fibre Channel connections. All the disk trays had dual path and the arrays had dual controllers. Then controller A was wired to path 1, and controller B was path 2. Plus, each disk controller had at least 2 paths on the host side, sometimes even 4 paths.
Those were an annoying mess to make sure you had the proper redundancy designed in. All paths had to be taken into consideration: server to SAN switch; SAN switch to disk array controllers; disk array controllers to disk trays.
My Cabling is as follows:
2 Cables from the LSI Card to the MD1220 In Ports.
2 Cables from the MD1220 Out Ports to the first MD1200 In Ports.
2 Cables from the first MD1200 Out Ports to the second MD1200 In Ports.
I do have a mix of SAS drives and SATA Drives (a Number of SATA Drives in the second MD1200). The SAS Drives in the MD1220 and first MD1200 seem to be detected twice. But the SATA Drives in the Second MD1200 are only detected once. @Arwen If I am reading your post correctly this should not result in multipathing, or am I reading things wrong?
A SATA drive, (which are all single port), in a dual SAS expander disk chassis will show up as a single pathed device.
Their are special disk mounting brackets for some disk arrays, that cause a single pathed SATA device to appear as 2 paths. That does not seem to be the case because your SATA devices show up once.
As for the SAS drives showing up twice, it is likely the case you are using too many cables. Meaning 1 input connector on the disk arrays is probably path A, and the 2nd input connector is path B. Same for the output connectors.
Basically you have to learn you disk array. But, you can probably direct wire 2 disk arrays, and daisy chain just one. I’m a bit burnt out, so I can’t walk you through fixing that dual path issue. Especially in a way to do so without data loss. Perhaps someone else can help you.
