Regarding performance: it comes down to what your expectations are. A NAS is not a local disk and never will behave like one because of several factors, protocol overhead has proven to have the highest impact by far on performance difference for us between our truenas server and local disk space. As I said we have to use SMB shares. We have just updated our server to a new machine, so I will tell about experiences with our old setup first, which we have used since late 2020. Small post-production company with about 3-5 daily users.
Most stuff we have to deliver as FullHD video, Broadcast quality. But our footage we receive mostly as 4K Video. So handling is a mixture of 4K Video streams and tons of small to medium exr-Sequences, mostly in HD1080.
Our old main setup is a supermicro x10 srl-f with not even the fastest available CPU, 256 GB Ram, 2 TB L2Arc NVME, a (6+2) RaidZ2 pool of 12 TB exos drives and second pool of some 4TB sata ssd which we use as a fast accessible library for stuff that’s re-used very often in different projects.
Clients are windows workstations and some dedicated 3D-Renderservers, all connected through a separate 10G network, that’s exclusively used for data transfer and thus can safely utilize jumbo frames without mixing things up.
The server has been running freenas/truenas core.
Based on our daily work, we experienced this, which may, in parts, help you evaluating your needs:
-
even 256 GB ram seemed to be able to contain a good amount of our frequently used data. Once re-read from disk space, the data is in Arc and can be accessed very fast. Sure, 256 GB is just a fraction of a pool’s size, and still just a fraction of the size of a typical project, but on the other hand, the data of a project which has to be accessed frequently is also just a fraction of the whole project’s size. So yes, the more Ram, the better, but with 512 Gig you should have fun. if I remember correctly the dual xeon board you intend to use is capable of utilizing up to two Terabyte of (Reg)Dimm. In my opinion that’s the only reason to use the dual xeon boards instead of a single xeon board, so maybe you could plan your ram acquisitions (i.e. get 64 GB RegEcc-Dimms instead of smaller ones) in a way that gives you options for later upgrades.
-
while we were able to saturate our ARC, we seemed not to get much profit from our L2Arc. Hit rate was very, very low. Very few accesses over a whole day. So if I interpret that correctly, the stuff that was flushed from Arc, was not to be used again anytime soon. Which adds up to my assumption, that our data was mostly cramped up actively in arc, or was sitting on our drives to be sporadically read.
-
that leads to storage performance. Which we have to decide in two parts:
-
write speed: this was good for our needs, all the time. We have no use for synced write, so our stuff goes to the server’s ram through 10GbE and is subsequently, in the background, written to the rust pool. Not a big deal. A lot faster than writing to local HDD, feels like writing to local SSD. Local NVME is another thing, but again we are talking about NAS storage via SMB.
-
read speed #1: even our 4K footage could be read smoothly from that RaidZ2-Pool into black magic resolve. Some hiccups here and there, but overall smooth, and the video editing software does it’s local caching quite effectively. With 6K/8K footage things would probably be different. If that’s what you use, you should indeed try to build your storage in a way that makes out your 10G network even for stuff thats accessed directly from disk. If using HDD, that’s more vdevs (add another mirror vdev and your fine to go.)
-
read speed#2: as I have written, things look completely different with lots of small files. In our case the CGI-generated EXR-Sequences. Here the nasty effects of low HDD access time, horrible, single threaded SMB overhead and probably Windows defender Antivirus check add up and make things very slow, compared to local storage. Even stuff that’s 100% stored in cache is significantly slower accessible than on local NVME, maybe even sata SSD storage. Maybe things are better if you are able to use other protocols, but for SMB, things are what they are right now. Compression by the way is not a big deal here, so there is no point in disabling compression. Encryption we do not use,would probably slow things down a bit.
As things are, we were not able to play a stream of EXR-images smoothly, when it had to be read directly from the RaidZ2-Pool. However, once in Arc things get better, and most times those EXR-Sequences have to be processed in our local compositing software, where processing time of each frame is magnitudes slower than the data transfer time. So we were not bothered too much, yet sure, faster would be better.
Accessing the stuff from our sata-SSD-Pool was smoother, due to the higher access time. SMB overhead still applies, obviously.
-
snapshots and backup:
Yes, with windows zFS snapshots give you access to shadow copies. Which is a huge thing. It’s just so good. I would recommend to do some staggered snapshots, each few minutes for about an hour, each 30 minutes for about a day, and each day for about two weeks, and go from there depending on your available disk space and/or replication capacity (once you decide to get a separate truenas machine as a replication server,which you really should consider, in the long run).
We decided to do replication to a separate machine. Just a low powered build, not much ram needed, with several HDDs just as big as we could afford, to replicate a copy of our most relevant datasets. Initially we kept that pool just at RaidZ1, to maximize the capacity. Meanwhile, we have added some disks and rebuilt as RaidZ2, but if on a budget I would still consider that raidZ1 sufficient, as it’s just intended as an additional layer of security of already existing, properly redundant data on the main server.
That machine is kept completely off-line, just connected to the main server over a 10GB connection on a different subnet, and gets it’s replication using pull, once a day. So the data on the replication machine is practically unaccessible from outside, to keep off those pesky ransomware attackers. Adding up with off-site backup of the absolute crucial stuff.
Final projects, once finished, end up on another cold-storage solution, to free up space on the main server.
Our updated server:
As we have made good experience with that setup for our needs, we decided to do some updates once it was time to replace the HDDs.
We definitely wanted to get bigger ARC and opted for 512 GB RegEcc-Dimms, each 64 GB. That maxes out our Motherboard at an really affordable price. Other option would have been to switch over to a board capable of even more Ram, but that would have meant either going Dual Xeon or using newer hardware. Maybe we will do that in the next years, if necessary.
We also updated CPU to one with a high clock rate, as that should improve SMB processing speed.
We kept the concept of just a single RaidZ2-HDD- Pool for large video footage files and additional bulk storage of seldom used stuff, and some local replication.
But we have also acquired a bunch of used 3,84 TB Sm883 enterprise sata SSDs at a good price, which allows us to build a 60TB flash Pool, 3x(6+2) RaidZ2. That saturates our network quite nicely and gives us smooth playback of our EXR file sequences, independent from Arc Cache.
I have good hope the SSDs will last some time, they have write endurance we will never exceed even in the next two decades. So if the controllers age well, we should be good to go with that setup over the next few years. Will see how this system performs compared to the old one under heavy load.
Hopefully that information is of some help for your setup. Cheers!
Do not consider iSCSI unless it is absolutely necessary.