I’m considering creating a TrueNAS SCALE for professional use by about 5 users.
You mainly need to save technical drawings and convert them into images and PDFs. It is estimated that around 6TB of space is now needed.
I was evaluating different server configurations with two mirrored disks for boot and 6/8 disks for workspace.
The server will have one or two network cards, but I don’t know yet whether 1Gb or 2.5Gb.
The CPU could be Xeon or Intel. in my opinion 4 or 6 cores are enough.
Cost is an important factor because we are talking about a small group of users.
In my opinion I don’t think there are major performance needs and therefore I prefer RAIDZ2/3 for greater protection.
With 8x2TB disks I could use about 10TB with RAIDZ3 and 12TB with RAIDZ2.
Instead, I am skeptical about how to configure the two SSD/NVME disks because using the entire 400GB capacity seems like a waste to me.
TrueNAS operates as a firmware and as such, requures the whole drive. There are ways to utilize the boot drive, but such condiguratjons are unsupported and would almost certainly break during upgrades. Also, a small capacity SSD is ultra cheat these days. I got 2x512 GB during black friday for $20 (BOGO deal).
Why bother to mirror it anyway? It is so easy to reinstall/re-upload config (~5-10 mins top). Backup your config instead unless you actually do need 100% uptime (90% of home users don’t).
Raidz2, 6-8 drives. Fine configuration. But 2 TB drives??? This is ridiculously small, and probably overly expensive per TB. Just go for larger drives.
As for the boot drive(s), get anything cheap (and small), and do not worry about “wasted” space.
To get you a better answer that is more fine tuned, how is the data going to be used specifically? Are you having people manipulate the data directly on the NAS? Or hopefully you are having the people copy the needed technical drawings to a local workstation/computer to do the manipulating and then saving the new PDF files, saving the originals of course?
The above scenario will likely over double the amount of technical drawing space depending on the format. At my work we (the company) has been migrating files of various formats into PDF. It is time consuming because you need to verify the final product is usable. The PDF files tend to be a little larger than the originals. Your situation may be different however it is worth looking at. And I agree with @etorix that a bunch of 2TB drives, unless they are free and new, is not a good plan.
I would triple the estimated capacity and purchase probably 6TB drives and place in a RAIDZ2 or RAIDZ3 configuration. Six 6TB drives in RAIDZ3 will net you about 16TB of storage and you can lose up to 3 drives due to failure. RAIDZ2 of five drives, same outcome but only 2 drive failure. And 6TB drives take a lot less time to resilver over 12TB drives.
As for boot drives, one single SSD/NVMe of any capacity will be fine, do not think about wasted space, you are purchasing a boot drive, that is all it needs to do. And no cache drives or anything else should be added. If you find a bottleneck, only then figure out what is causing it and then take action.
Have lots of RAM, 16GB minimum however I would personally recommend 32 to 64GB. Make sure it is ECC RAM.
As for cost, think about how much it would cost you if the data disappeared. But I do understand that a startup business needs to keep the costs reasonable. For a business don’t be too cheap. Buy quality components or you will wish you did later when you have problems.
An Intel Xeon is a good CPU however there are many versions. There are other CPUs out there as well. When buying a CPU, you must factor in the motherboard you will be using. Make sure the motherboard fits your needs.
I answer “on the fly”.
Are you having people manipulate the data directly on the NAS?
Good question! I don’t know and I hadn’t thought about it. I’ll look into it straight away.
Now I can only hypothesize that he draw on the PC and then every hour a tools copies the modified files to the NAS.
64GB RAM ECC?
I calculated 5 people: a secretary who works only with Office and ERP (which is on a separate Windows server); a person who does technical drawing; a person who consults the drawings, but does not modify them; two other people who I hypothesize as “future”, but who aren’t there yet.
Even assuming that they will draw too, I don’t think they will create a lot of traffic on the NAS.
So, isn’t 64GB too much?
There’s really no such thing as “too much RAM” with ZFS, but 64 GB are probably more than necessary; if all you’re doing is file storage (which it sounds like it is), 16 GB is likely fine and 32 GB would be plenty. ECC remains a good idea. If you’re planning on apps, VMs, or other services, memory needs increase accordingly.
I’m going to continue the trend of recommending fewer and larger drives and suggest 4x 8-12 TB drives in RAIDZ2. That gives you ample capacity, plenty of redundancy, and with only four drives lets you use a smaller server. I’m quite fond of the HPE MicroServers, though they’ve gotten quite expensive in recent years. The only downside is that with only four SATA ports and no NVMe, you’d be using a USB SSD for a boot device–not ideal, but certainly not the end of the world.
Add my vote - fewer larger drives
ECC Ram absolutely also, you are working with important files, you want every possible form of error checking in place instead of finding out in a year you cant open a bunch of files because they corrupted at some point.
Also, what is your backup plan once the files are on the TrueNAS? (3-2-1 rule…)
When you say 6TB is that the total size of the files you want to hold?
You can use Snapshots to take backups of changed files - you need to work out what your new/changed/deleted files per month will be and multiply by the number of months you may want to go back to older versions and add this to the total size you need.
There is an overhead for storing metadata (say 5%-10%) and you really mustn’t let your disks go over 90% utilised otherwise writes start to get very slow (because free space allocation gets slow). So you probably want to add 25% to gross the requirement up.
You may want to add some more capacity to account for inaccuracies in your estimation and for unexpected growth or new storage requirements.
Don’t forget that you are probably measuring TB as 2^40 but disk space is actually measured in 10^12 - so you need to scale up again by another 10% to account for this.
Summary: 8x2TB in a RAIDZ3 or even RAIDZ2 is probably insufficient.
A few other factors to consider:
CPU power - you would be surprised how little CPU power is needed by TrueNAS - I suspect that 4-6 cores is way more than you need unless you are doing some significant batch processing on the server using scripts or an app.
Network - try to have a network which is at least 1Gb end-to-end. It is the end-to-end bandwidth that is important - so your switch and wifi access points need to match the bandwidth. Unless you need the bandwidth and know how to consolidate 2x ethernet ports into one connection on your single LAN, you probably don’t want more than 1x ethernet port - so go for a faster port rather than 2x slower ones.
If you have 2x SSD/NVMe then my advice is to use one for boot and (if you need apps) use another for apps (and replicate the apps pool to HDD as a backup).
Buy a UPS for it.
Don’t use a USB SSD as a boot drive unless you absolutely have to in order to keep costs down. Whatever you do DON’T USE A USB FLASH DRIVE - it must be a SSD.
Think about how you are going to backup this data in case of catastrophic and unrecoverable failure of your ZFS pool.
I have a feeling that resilvering takes longer the wider your RAID array is. So think about using fewer bigger disks.
Snapshoot (on my home server I have hourly snapshots kept for 24 hours and a daily one kept for 7 days).
Online backup to Dropbox.
Backup to a QNAP on the LAN.
If I have space left over, I could put an extra disk to use for internal backup.
Capacity: 6TB is the current need. In fact I assumed to install 10/12TB. I keep the your advice on disk capacity.
Motherboard: I’m looking for; I found some with 4 SATA ports; what do you think if I connect the disks to a PCIe card? I could use the SATA for booting and apps.
I had thought about using NVME on a PCIe card, but then we need to see if it is possible to boot from here. I had tried on an old Proliant that I used for testing; I saw it as a disk, but I was never able to use it to boot.
“A PCIe card to connect SATA drives” would be a SAS HBA. But the basic recommendation if you’re moving to relatively few (4-6) large hard drives is to find a motherboard with enough SATA ports. Keep it simple.
Keep in mind that RAIDZ is not easily expandable. I would see 12 TB as the absolute minimum you should aim for, keeping in mind that snapshots also add space and that you probably want to keep usage of the pool under 80 % for the best performance.
I’m evaluating a Supermicro-based server with these disks but I don’t know how to evaluate the technical difference.
3.84TB 2.5" PM893 SATA 6Gb/s Solid State Drive (1 x DWPD)
3.84TB 2.5" 5400 PRO SATA 6Gb/s Solid State Drive (1.5 x DWPD)
3.84TB 2.5" D3-S4620 SATA 6Gb/s Solid State Drive (<3DWPD)
7.68TB 2.5" PM893 SATA 6Gb/s Solid State Drive (1 x DWPD)
7.68TB 2.5" 5400 PRO SATA 6Gb/s Solid State Drive (0.6 x DWPD)
What operating life can these SSDs have?
In the case of HHD (WD Red) some people advise me against using RAID-Z because the files are distributed fragmented across all the disks and therefore reading is very slow.
In this case they advise me to use Mirroring instead.
From your first post, you’ll be storing (presumably somewhat large) documents, not block storage, so raidz# should be fine.
The lowest endurance setting of “0.6 Drive Writes Per Day” litterally means that the manufacturer endorses writing 60% of the drive’s capacity each and every day during the warranty period as “normal use” and commits to exchanging the drive if that would result in premature failure. Do you expect your users to generate over 4.6 TB of new drawings everyday (Sundays included, multipled by the number of data drives)?
