I have two TrueNAS machines running that I’ve had up for several years. One is a backup machine (raidz1, 5 disks + cold spare), and the other is my main NAS (striped mirrors, 4 vdev + hot spare).
I have started toying around with special VDEVs to improve performance of the pool on my main NAS . This is because I now have several servers set up for 25gb networking and I have an interest in running some VMs off of the pool. Particularly, I am interested in using ISCSI shares to use for storing games for several physical and virtual gaming machines. My hope is to eventually create a separate high performance pool specifically for gaming and VM storage on another machine, and then rebuild my main NAS pool as two raidz1 VDEVs since the NAS is mostly media files.
My questions:
What is the best way to store small blocks in a ZVOL on a Special VDEV? I am aware of the small blocks size parameter, and currently have it set to 64K. My ZVOL is set to 128K block size. I’m specifically wondering about this for game storage. Also, I don’t want all of the data on my special device, but it would be nice if everything under 128K was on the 375GB Optane drives I will be using.
Would it be worth using an L2ARC drive in addition to the Special VDEV?
If so, how does the L2ARC work in combination with the Special VDEV?
Special vdev does not work (yet) for zvols.
See: github
s vdev does not work retroactively. You have to move the data on and off.
Older games? Even a 25GB NIC is way, way slower than a cheap local NVME drive. And it does not support features like direct storage.
That might be fine for games, but for VMs 128k will lead to extreme read and write amplification, poor performance and fragmentation. The default 16k is pretty good for VMs.
Not really. L2ARC caches stuff that evicted ARC. When you have metadata (and potentially small files, depending on your setting) on s vdev, why “cache” it twice? Also the s vdev can also “cache” writes. On the other hand L2ARC does support zvols, so there is that.
Not sure how to quote everything that you replied to, so apologies if this is messy.
I know nothing is moved retroactively and will be creating a new pool for gaming storage only. I may also create a pool for the storage of the guest OS systems, but also may not.
New and old games. I know a 25Gb NIC is a lot slower than a local NVMe. What is better is the 4k random read/writes with a pool, particularly with Optane drives as the SVDEV or L2ARC drives. This is what I want to configure so I have a pool of many TBs that store several machine’s game storage.
ZVOL 128Kib would be only for game storage. If I also store my VM’s OS on the pool, those datasets will use a smaller block size. Mostly, I want the gaming stuff set up and working first.
I’ve noticed games having textures “pop in” when using the ISCSI shares. It isn’t the final config, but it is currently happening on a 4-wide striped mirror of 12TB Exos drives, with a 3-way mirror of SAS flash drives used as the SVDEV. Leaving an area in-game and returning basically loads up super fast because it is in RAM. This is why I’m wondering if it is worth having an L2ARC. The cached info for my gaming ZVOLS would seemingly boost performance even when using a SVDEV.
Even if that would be the case (which I doubt), 4k random rw is not what is relevant for a gaming drive. Sequential read is probably the most important metric. Stuff like Direct Storage might be.
ARC is read cache in memory.
Something can evict ARC and go into your L2ARC.
L2ARC is (hopfully) an SSD, and faster than your pool.
Svdev for zvols does only metadata, no matter your settings.
So to answer your question, a L2ARC might help, since it might store small files evicted from ARC.
Considering that you can get a 1TB SSD Samsung 990 EVO for like 65$, that will severely outperform any NAS, no matter if sequential read (56GBit/s) or IO, iSCSI gaming has become a Youtuber tech influencer meme (LTT and Lawerence Systems).
Don’t get me wrong, if you do this for fun or to see how much storage you can save by using deduplication or creating an inital snapshot you then clone, more power to you.
If you expect a good solution on the other hand, this isn’t the way to go IMHO.
I am curious about ISCSI and shared storage in general, so regardless, this is something I want to try to get working as well as possible.
That said, there is definitely no way to have small blocks from a ZVOL stored on the SVDEV? Info that I’ve found online has not really clarified that for me.
Same question really for L2ARC as well. I know you can store metadata on L2ARC. Can you adjust the small block size too?
Thanks for the info. I missed where you provided the github link.
I ended up creating a few NFS shares for a couple machines and it seems to perform better in games. 4k r/w is just over half that of ISCSI, but I think that you are right regarding game storage being more sequential. I looked through several game folders and a ton of it is more than 128KiB.
If ever ZVOLS had the ability to use the small block parameter, it would be really kick-ass. Being that it’s block level storage, I wonder if it will ever be a thing.
Back when SSDs were more expensive and at the same time, games did not need more than something like 80MB/s sequential read performance, you could in theory have a 10GBit network that has multiple of 256GB SSDs gaming stations that all run of a single TrueNAS that hosts a huge steam library for them. And thanks to Snapshots, you could even “kinda deduplicate” the storage.
It never really made sense, since you could just have installed a second HDD. Even if you did not have a fast internet connection, you could have moved games from the HDD to SSD if one game really needed fast storage. Steam even integrated that feature later on.
The same is also true for the PS5 IMHO. I never understood why you should spend money on a bigger or second SSD to have a huge library, when you just can attach a HDD and move games from there.
Ahh slow internet connections so you could not just download games. Good old times! Reminds me of the good old days when I registred as a fake steam cafè so we could get the steam cache server up and running in your local LAN for a LAN party
Necro-ing my own post, but I don’t think it’s been nonsensical at all, particularly with the RAM and storage bullshit that is going on now.
I’ve had my shared storage working really damn well as NFS shares to 4 machines. I currently have a 3 way striped mirror of 8TB HDDs with Intel Optane drives as mirrored special VDEVs for anything under 64Kb. I also have a 1TB NVMe disk as L2ARC, and while one or two games (Oblivion and STALKER 2) have some slow-ish loading “pop-in”, L2ARC prevents it after the first time, even after a reboot. The pool holds basically only games so L2ARC for that pool always has recent enough data to make shit run crazy fast. This was originally over 25gb with a SSD pool, but the HDD pool runs very well with 10gb as long as there is a special VDEV and L2ARC.
