My hardware setup currently is a 10core/20thread older Xeon CPU, 128GB ECC memory, and 5x24TB (Seagate Enterprise 7200 drives in RAIDZ2).
Question 1:
I currently run Plex and have set it up to with 5 cpus and I also run Immich with 4 cpus. That comes to 9 CPUs used by apps alone. What is my restriction on running apps. Is it the 10 cores or the 20 threads, or can I exceed 20 at the cost of context switching and slower performance? I’d like to run a few other apps, but I’m concerned I’d be hitting a limit or tanking my performance.
Question 2:
When looking at the memory in the dashboard, I’ve often seen ZFS cache taking most of the available memory. Do I need more memory for optimal performance or is ZFS just grabbing what it can. Again, I’d like to be using some additional memory for more apps.
Question 3:
I’m considering purchasing an additional 5x24 additional drives. My current drives are in RAIDZ2 since I was assuming at build time I’d just expand the single ZDEV to be a 10x24 eventually. I’ve read that adding a second ZDEV would increase my write IOPS but that would cause a smaller pool size since I’d have 2 drives wasted on each ZDEV. I’ve also read that adding drives to an existing ZDEV wastes a small amount. This is all for my homelab and usage is fairly light (backups of systems at night, plex occasionally). I’m interested to hear suggestions as to what approach to take with the new drives.
Cores are not dedicated to apps, and most apps will not use 100% of all the cores they are allowed to use. What you want to do is to only assign as many cores as an app needs to give you the performance you need in order to keep other cores available for other apps and (of course) for TrueNAS itself.
ZFS cache uses as much memory that isn’t used by other apps as it needs. ZFS cache follows the law of diminishing returns - the first 4GB of cache gives you the most performance gains, the next 4GB gives you less of a gain. If you look at the ZFS reports, you can see what your cache hit rate is, and if it stays above 99% I would say you are going to be happy - and my own small NAS (with 1/6th the disk capacity) achieves 99.8% cache hits with only 3GB-4GB of ARC.
You are right on all the facts here. a 2nd vDev will give you more IOPS, but you will get more throughput with a single vDev (because you will have more data drives and less parity drives). IOPS is only important when either A) you are doing a high quantity of very small random I/Os e.g. for virtual disks - and you need mirrors for that to avoid amplification or B) you have a lot of users (doing paralel I/Os). So you should be fine with a 10x 24TB RAIDZ2.
Adding a drive to an existing RAIDZ vDev moves some of the existing blocks to the new drive in order to make the free space on each drive approximately the same, but the parity is unchanged so you will start off with 3 data blocks and 2 parity blocks, and this won’t change during expansion. But if you expanded 1-by-1 to a 10-wide RAIDZ2, then rewriting your data would change 8 records (each of 3 data + 2 parity = 40 blocks) into 3 records (of 8 data + 2 parity = 30 blocks) - so you would get approximately 25% of your existing used space back.
Thanks for the reply. I admittedly struggled a bit with the last paragraph and read it a few times, but if I’m understanding you correctly, you are saying, I’d lose 25% of the potential space on the data occupied blocks, but if I rewrite data, I can recapture the 25%. If I’ve got that right, what would be the best approach to rewrite the data that is safe.
Right now there are some unofficial scripts to rebalance a pool, but iX have announced they’re working on a build in solution. Sadly there’s, as far as I can remember, no eta for the buildin solution
You are averaging a 99.9% hit rate on cache. That is pretty good - it means that only 1 in 1000 reads actually need to be fetched from disk, with the remaining 999 being served immediately from memory (the ARC) without any wait.
