It’s my first TrueNAS CE install and it’s running on a Intel i5-4690, Asus H97-Pro board with 16Gb RAM. I also have another NAS, an Asustor Drivestor 4 for backups.
I’m having trouble on how to organize my drives with RAID/ZRAID pools on both NAS, cause I have two 6Tb drives, two 8Tb drives and two 16Gb drives. Currenly the 6Tb drives live as a 12Tb strip pool on my TrueNAS + one 16Tb drive. The other two 8Tb live as a 16Tb RAID0 pool on my Asustor + the other 16Tb.
Before this was making sense, as the 12Tb pool would backup into the 16Tb pool, and the 16Tb drive would backup into the other 16Tb. But there’s no disk fault protection or bit rot, etc. It’s just that files are dupped on both NAS and so errors and corrupt files will also be dupped, not protected.
Aside from these drives, I have the option of returning one of the 16Tb drives in exchange for two 8Tb drives. This could make sense, to have four drives of the same size in the same pool, in Raid5 or something similar.
I don’t need to increase available storage and everything combined, probably my total used space is somewhere between 16 and 20Tb.
Mirror them in pairs.
Use the 2x 6TB (backup and important data) and the 2x 16TB (media files) in the PC for storage and the 2x8TB in the Asustor for backing up your important data.
I dont recommend using HDDs in striped configuration, If you need speed and low latency, buy a single 4TB one (also buy a 4TB HDD to provide a backup for the system (snapshoting every hour and backup every night). On that 4th gen MoBo, you need a PCIe card to use those NVMe drives.
(the H97 will most likely enough PCIe lanes for this. No, it does not… It only has 1x PCIe16x Gen 3 and a PCIe4x Gen2, also, surprisingly an M.2 port too.)
Booting from them might not be possible, but for storage they will be nice.
If I suspect correctly, this device Asustor has 4 HDD slots, so you can later buy another 2x(1x first and then another one as you save up)16TB for back up your media files too.
I dont recommend using HDDs in striped configuration, If you need speed and low latency, buy a single 4TB one (also buy a 4TB HDD to provide a backup for the system (snapshoting every hour and backup every night). On that 4th gen MoBo, you need a PCIe card to use those NVMe drives.
(the H97 will most likely enough PCIe lanes for this. No, it does not… It only has 1x PCIe16x Gen 3 and a PCIe4x Gen2, also, surprisingly an M.2 port too.)
Booting from them might not be possible, but for storage they will be nice.
Also recommend to buy a cheap, small SSD (sATA with 16+GB) for the OS.
That will be sufficient for most of the expected tasks.
(I recently retired an i3-4130 with 8GB DDR3 desktop RAM)
So, conclusion:
PC:
- 2x6TB in mirrored
- 2x16TB mirrored
- 1xx GB SSD for OS
Later expansion for speed:
- 2xM.2 NVMe (2-4TB depending on your budget)
- 1xHDD the same size as the NVMe
Asustor
- 2x8TB mirrored
Later, as finances allow:
- 2x16TB mirrored (1x first, and a second one later)
1 Like
Thank you so much for the input. I left it out because one was the obvious thing, the OS drive, and the other I didn’t consider what to do with it or have much concerns with speed.
I have an 128GB SATA Samsung EVO SSD for the OS. And I have a 1TB WD Black SN750 NVME drive on the PCIe 3.0 16x slot, which is basically just sitting there to quickly dump data (allows to reach maximum speed on the 2.5Gbe network).
About slots, the H97-Pro has some limitations on PCI-e, but I think it can hold more PCI-e NVME disks if they are on PCI-e 2.0 or don’t push more than 1x/4x. Not sure about this.
But there’s only one m.2 NVME slot but if I use it, it disables two SATA ports, leaving 4 remaining (with one taken by the OS SSD).
The Asustor has 4 SATA slots, plus it allows expansion with external USB3.0 drives (at least two ports, not sure if it would allow for a hub).
About media files, in my case the critical data are media files and quite space consuming: RAW photos (87Mb a shot), video files, music, etc. The smaller drives have some TV shows, movies and documentaries, phone backups, PDFs, etc.
While I was waiting for replies, I asked GPT for advice, it suggested for me to consider the setup below.
PC:
- 2x16Tb (16Tb usable, redundancy)
Asustor:
- 2x8Tb Mirrored (8Tb usable, redundancy)
- 2x6Tb Mirrored (6Tb usable, redundancy)
Total capacity 16Tb for the PC vs 14Tb for the Asustor (backups), becomes more balanced between both NAS.
Or, if in need of more space and are able to get a third 16Tb drive:
PC:
- 3x16Tb RAIDZ1 ( 32Tb usable, redundancy)
Asustor:
- 2x8Tb Mirrored (8Tb usable, redundancy)
- 2x6Tb JBOD (12Tb usable, no redundancy, for less critical backups)
This would leave the PC with 32Tb vs 20Tb for the Asustor. JBOD because this is already a backup of a dataset that is redundant, so it’s less important to have redundancy.
Or go full on without redundancy on the Asustor:
- 2x8Tb JBOD (16Tb usable, no redundancy)
- 2x6Tb JBOD (12Tb usable, no redundancy)
This leaves the PC with redundancy and almost 90% can be backed up into the Asustor with 28Gb.
OK, so it is a nicet hing, you have all those additional drives for all those specialised tasks and you use them too.
According to your description, you have a lot of unique data, that most likely need more data safety. ( I guess your photos are important for you.)
Some additional questions (however, your original post was about setting up your current drives):
- Is the i5 based PC going to run TrueNAS on bare metal?
- Will that be the only OS for that PC (Do you plan to run VMs or containers)?
- Do you have any backup plan in place besides this two setups?
- I guess, you also have a working computer too that runs some kind of desktop OS.
Then my modified recommendation is:
- I still recommend to use those drives in Mirrored instead of striped.
- YOU should buy a HBA card for your i5 based PC, (an LSi card with at least 8 sATA channels (in total of 12 now), FLASHed into IT mode. (decent ones cost you about 50 USD second hand, and will provide you with 8 additional sATA channels for future expansion.
- If you buy a bigger case, that can hold 12 drives, you will have a beefy setup available
- Add your Backup and Media pools (2x6TB +2x16TB) to the HBA
- Use your built in NVMe M.2 slot for yout drive (It is way faster then any sATA SSD can give you
- Add your other drives, like theSamsung EVO and the WD Black to the 4x sATA ports of the PC
- This will leave you with 1x4x CPIe slot + 1x1x PCIe slot for any future expansion (However, with this setup, you already have 4+2 sATA ports for future expansion)
- If you buy additional drives, you can rearrange your setup, more suitable for your needs.
- If you have some cash to spend right now, the ideal way would be to purchase 4x8TB HDDs, add them to your main PC in a RAIDZ2 setup, that will provide you 32 TB of future data with the option of be able to used ANY 2 HDDs in that pool
- To back it up, you need to buy another 16 TB drive to your Asustor device in a RaidZ1 config (2x16TB data + 16TB redundancy, enabling you to lose any one of these drives) to back up your 32TB data
- You sacrifice your 8TB data, but you can create an additional Vdev on the 32TB pool for that data
- The 2x6TB you could simply retire, if you want, or move them to your PC on the remaining 2x sATAports on the motherboard in a mirroroed configuration, enabling you to lose any ot those two drives (and additional 6TB storage)
- Backing it up, you can future buy another 6TB drive into your Asustor for backup, but without any redundancy.
- DONT use any drives on USB interface (USB is not good enugh connection method for ZFS. they dont play along well)
- DONT buy SMR drives (they are practically incompatible with ZFS, and your scrub and resilvering times will be waay too long)
- I dont know, how much data you currently have, but it is quite complicated to move yround nuge amount of data with ZFS, since, you cannot really resize the pools, you already created. (So be VERY wise, what you do!)
- You can create and destroy then , losing ALL data on them.
- There are attempts to increase flexibility of ZFS pool changes (But it mostly works int toe direction of increasing the size and, if I understand correctly, by being able to upgrade RAIDZx levels more flexible, as today. I am really not experienced in this topic, so I guess anybod,y with more knowledge can jump in to help out.)
- I prefer to keep my original configurations as long as I am finished my migration and 101% sure, that no data is lost.
- Then I think of any new prupose for the current system
Yes, you’re right, photos, videos and music, can be both personal and work related, which means that either way they have a lot of value to me.
-
Yes, TrueNAS is on bare metal
-
Yes, I don’t plan to install other OS or to run VMs
-
No, I used to have iDrive and Backblaze but had bad experiences recovering data, that’s why I decided to invest on a cheap second NAS (the Asustor) for backups.
-
Yes, I have three work computers. An M4 Mac Mini, a 13th Gen i7 64Gb + 4070 12Gb for heavy workloads and an HP Ryzen 7 laptop for mobile work. All of these will backup into the TrueNAS system.
-
Total amount of data I currently have should be close to 16Tb, maybe a bit more, but this includes non-critical data.
-
I think my case can hold 10 3.5 drives, not sure if that’s including a conversion of the 5.25" bays to 3.5" or not. So it might fit more than 10. It’s an Fractal Design Define 7 or XL or something like that.
-
About SMR/CMR, one 6Tb (WD Red) is SMR and all the 8Tb (Barracuda) are SMR too. Only on 6Tb (Ironwolf) and the 16Tb Exos are CMR HDD’s.
-
About investments, I currently can get a nice deal on a 16Tb Exos CMR drive for 250€. The cheapest 8Tb drives around here are Barracuda SMR and would cost 125€ per drive, so total cost per Tb is similiar, although the drives are inferior models and SMR. I can’t currently afford 4x 8Tb CMR drives.
So, to stick to CMR drives and within my current budget, I guess the only option is to get the HBA card and another 16Tb Exos and put 3x16Tb in RAIDZ1 or even RAIDZ2 into the PC. Everything else goes to the Asustor (don’t think it supports RAIDZ, only RAID0, 1 or 5).
Well, it seems, you have a plan at hand!
First, I recommend to expand your main storage. If the 16TB is almost full, you will have problems with TrueNAS most likely.
Then you can expand your backup storage.
I hope so. 
Thanks again for the help.
Is there any common brand/model for the HBA cards to look for? I’ve only heard about these cards for SAS drives, so I know 0 about them.
Also, what use would you give to the m.2 1TB NVME drive? I was wondering if it can act as a “buffer” to transfer files to the NAS at maximum speed and then write do the HDD pool.
Well, I have an IBM 1015. That is a bit older one, but Still works for me.
YOu neet to bus LSi 9300 chipset card, if OI am correct, but I did not follow this for along time, so you need to search more alone.
SAS cars are backwards compatible with sATA drives, but not the other way around.
If your MoBo can boot from an M-2, then it is much faster.
a sATA SDD maxes out at 550-600 MB/sec.
An M.2 is like 2000MB/sec for Gen2 and about 4000MB/sec for Gen3 (Those are theoretical values, in practice they are 1600-1800 and 3500+MB/sec)
A HDD can sustain about a 100MB/sec serial write speeds easily. SO, if you have 1 or 2,5GB/sec network speed, your drives will be able to keep up with that. Using an M.2 is usually not helping on speed, but wears out the NVMe.
TrueNAS uses system RAM for that kind of cache.
ANd it writes data from RAM to the drives. So the maximal filesize, it can buffer is the difference between your network speed and HDD write speed divided by the free RAM you have. (assuming, that your CPU can keep up with the task)
So, if you have a Gbit network spees and 3 HDD in RaidZ1, then your approx HDD write speed will be about 200MB/sec, so it will use minimal amount of RAM)
If you have a 2,5GBit Network, your network throughput will be about 230MB/sec. In that case, you have a HDD speed deficit of 30 MB/sec. If you have 16GB system RAM, and your TrueNAS uses about 3,5 of it, and will leave like 2 free for reserve, then about 11 GB RAM left for caching. Considering the 30 MB, the time it takes to fill your RAM cache is 11000MB/30MB=360 sec (6 minutes). During that time theoretically you can copy 360sec*230MB%sec=82800MB= about 80 GB. It is highly unlikely, you have such big files.So your system is safe without any M.2 cache device.
