I have a pair of 750 GB Optane 4800X that I’ll be using for iSCSI VMs, but I also have a pair of 1 TB Intel’s P4510 SSDs that I am looking to put to good use. What I am considering is mirroring both of them in a single pool to serve a bunch of zvols.
One way is to mirror this naively and put Optanes in one vDEV and Intel NAND in another vDEV, make a pool that way but then I am thinking I’d be running into unpredictably different latencies since I couldn’t control which zvol would be placed on which vDEV.
So what if I used the NAND drives as mirrors of Optanes only, i.e. two vDEVs each containing an Optane and a P4510? I’d gladly trade 250 GB of “lost” NAND space to gain Optane latencies on a 1.5 TB pool (and I am thinking the extra 250 GB on each NAND drive in such a vDEV will add considerable endurance to the drives).
Reliability-wise, I think I should be good - P4510 is PLP-backed, Optane 3DXpoint doesn’t need PLP (but I think Intel did add some extra super-capacitors on P4800X?).
The big question is - will ZFS prioritise read/write ops on the lower latency drives (i.e. Optanes)? If yes, then I’d expect the NAND to be used to store extra copy of data. If any Optane fails (as unlikely as it may be), I’d be ok with running degraded until I can source another Optane.
The answer to the big question is “no”. Either way, you’d throttle the Optane drives to the level of the P4510.
I’d rather keep these DC[1] P4800X as special vdev to a big HDD pool, or as SLOG and/or[2] L2ARC.
Since we have no information about your configuration (like memory size or numbers and size of HDDs) it is impossible to know whether a special (metadata) vDev or L2ARC will be a good idea or not.
If you have VMs writing to HDD (i.e. synchronous writes), then an SLOG might be very beneficial.
Thank you - looks like it will be two separate pools then. I am already carving out a pair of Optane P1600Xs for SLOGs and have some remaining space for L2ARC but my limited testing thus far isn’t showing concerning ARC miss rates. I’ll keep monitoring in the future and decide whether to use the L2ARC or simply add more RAM.
I’ve updated my signature with my current storage configuration - please feel free to suggest improvements.
I am trying to keep the pool occupancies low for best performance, hence I plan on dispersing the VMs according to how much I value IOPS for those particular VMs.
