Setup recommendation

TrueNAS General
, ,
1

Hi,

I have experience with hardware RAID and have played with truenas as bit but looking for zfs setup suggestions.

I have the choose of a server with 6x 3.5” bays or a server with 26x 2.5” bays. To one of these I will connect a 60 bay unit, to populate that I have 40x 2TB drives and 10x 4TB drives. I also have a PCIe to m.2 card which has 2x 512GB and 2x 246GB drives as well as two Dell BOSS cards which each have two 256GB SSDs.

If I go with the 26 bay server I’ll use two 240GB SSDs in a mirrored setup for the boot pool, if I use the 6 bay server I’ll either use one of the BOSS cards or the PCIe card (could also use that in the 26 bay server).

From what I’ve managed to work out RAM is key, the 26 bay has 128GB of RAM and the 6 bay has 172GB of RAM (either could be increased if needed).

I’m looking to use iSCSI ultimately to a windows server, I will have a 10G dedicated connection to the windows server.

The use case will be mainly media to Plex which is on a separate PC due to used an antenna (don’t have an antenna connection where the servers are housed so won’t be installing Plex on truenas) and document storage.

How would people suggest I setup the drives for a combo of best storage and redundancy? I’m thinking raidz3 but unsure about vdev setups.

If I want going to use hardware raid it would raid6 with at least one maybe two spares for the 40x 2TB drives and raid6 with one spare for the 10x 4TB drives.

Advice, suggestions, guidance and pointing to reading material welcome.

EDIT
iSCSI use isn’t a must but as I don’t have a 10G switch at the moment and only have the ability to direct connect not sure if it would be more beneficial to group 1G connections and use SMB or not.

All suggestions welcome

2

ZFS uses multiple RAID groups, each defined by the smallest disk in used in that group. Those groups are called vDevs, Virtual Devices, in ZFS. A ZFS pool is made up of at least 1 data vDev, but can contain more vDevs. Including special purpose ones that many home users don’t need.

RAID-Zx, (with 1, 2 or 3 columns worth of parity), has a suggested maximum width. Some people think 12 disks, others use more. But, as a RAID-Zx vDev becomes fragmented over time, very wide RAID-Zx vDevs seem to get slower. The wider the vDev, to more chance of experiencing that problem.

Now with 40 x 2TB disks, and the use case of Plex, perhaps 3 RAID-Z2 vDevs of 12 x 2TB disks will be fine. Or you can go slightly asymmetrical and have 2 RAID-Z2 of 13 x 2TB disks and 1 RAID-Z2 of 12 disks. This gives you 2 more data columns, and only 2 spares instead of 4.

Note I assume your reference to hardware RAID is for what you would have used, had you chosen to use hardware RAID.

One of the problems selecting a pool layout, is use case. For pure media storage, RAID-Z2 offers decent speed, good data to redundancy overhead and the ability to survive 2 disks lost, PER VDEV.

But, if you plan on using iSCSI, the general recommendation is to use Mirrors and smaller block sizes because that is more responsive to small block sizes used by the client OS than RAID-Zx.

3

Thank your your suggestion Arwen, you are correct my mention of hardware raid is what I would do if not using truenas.

I like your asymmetrical suggestion, I still have 10 free slots so could get some more 2TB drives or use the 4TB ones and slowly work on replacing the 2TB ones.

On the basis of preferably using mirrors with iSCSI I take it using SMB would be better? I could use the 10G link for faster transfer speeds from the windows server to the truenas and group the 1G connections for direct access to the truenas shares.

Would there be any benefit in using drive for Log, Metadata or Dedup? I’m guessing from what I’ve read there would be little to no benefit in using SSDs for cache drives? I have a few 1TB and 240GB SSDs.

If I setup the 4TB drives I’m guessing 1 RAID-Z2 vdev for now and I can combine more later?

4

Yes, that works. ZFS vDevs use the smallest disk as the size for all disks. So, over time replacing 2TB disks with 4TB disks will work. But, you won’t see the new space until the last one is replaced.

In someways, when someone has smaller disks and has to replace one due to failure, a general recommendation is to use the current cost effective disk size per 1GB. Even if they don’t replace all soon, it is waiting for that day.

It is all up to your application. SMB does pretty well with RAID-Zx for larger files like Media.

But, some people want the TrueNAS server to run Plex AND perform trans-coding to the client’s desired resolution. That can be tricky to set up.

Special vDevs can be complicated.

Metadata vDevs add complications. Whence added to a RAID-Zx pool, they can not be removed. Loss of a Metadata vDev, means loss of the ENTIRE pool. So they should have the same redundancy as the RAID-Zx vDevs, 3 way Mirrors if using RAID-Z2. They can be set to allow small files, which tend to be less efficient on RAID-Zx. But, that could cost space for metadata. To be clear, whence a Metadata vDev is “full”, further writes go back to the normal data vDevs.

Only synchronous writes benefit from a LOG / SLOG device, and in general the device should have power loss protects. In general, iSCSI LUNs should use sync writes because they foreign OS could be completely corrupted on a TrueNAS crash / unexpected reboot.

De-Dup is really tricky. An improved version of De-Dup, Fast De-Dup is becoming available but, like all new things, perhaps waiting for others to find all the bugs is prudent. I wrote this Resource on the old forum about the old De-Dup:

Yes.

However, ZFS has limitations. One is that a RAID-Zx vDev can not be removed from the pool. Nor can you shrink a RAID-Zx vDev, either by removing a column or use smaller disks. That was just not built into ZFS.

5

Thank you Arwen, I’m used to it being about the smallest disk in the set with hardware RAID so I normally (with hardware RAID ) try to keep to one disk size in each setup / pool.

I’m not looking for TrueNAS to host Plex or do any transcoding as I have a PC setup for that which is connected to and antenna for TV. I don’t have an antenna connection where the server lives (at least not at present) so it will just be for file storage and running of some apps. I won’t be doing any VMs on it either at least not at the moment as I’m very comfortable with Hyper-V but it is something that I’m considering playing with in the future.

Per your recomendation I will stay away from special vDevs and dedup. I think the ideal setup would be 2 RAID-Z2 @13 1 RAID-Z2 @ 12 leaving 3 spare. The 4TB drives I will keep whilst I get more to then replace the 2TB ones.

I’ll use a PCIe 4x m.2 card to give me 2 drives for mirrored boot pool and 2 drives for mirrored app pool.

I’ll do a direct 10G link to the windows server for data transfer between them and aggrigate the 1G links for general access by PCs on the network.

Thank you for your help and suggestions