Been doing some FIO testing on a large NAS for a business, this machine has 16 8TB Micron 5400 Pro SATA SSDs in it and has been an absolute monster; but they have a need to get more specific random 4k read IOP performance numbers.
8 vdevs, so mirrors, all in a single pool. System has 256GB of RAM and an EPYC 7281.
I’ve been doing a lot of testing with FIO but the numbers aren’t where I would expect them, I’m thinking there’s something I’m just not understanding and maybe this is totally fine.
According to the spec sheets these drives should be capable of nearly 90k IOPS for 4k random reads on their own, reading from 16 simultaneously in theory should be at least that high.
I’m running FIO with a test file of 1TB (to avoid using ARC for the majority of it), queue depth of 32, 4k block size, random reads, 8 threads (100GB of reads per thread), and letting this run for half an hour. Results are roughly 20k IOPS. I believe this is enough for the specific needs on this machine anyway, but it feels low to me considering what the single performance of a drive should do.
Is this possibly ZFS related or something? It just seems odd since I can get about half a million from the ARC, so the system itself should be capable of pretty high numbers.