I currently have a bunch of cameras recording to SSD drives due to higher iops. The software we use kept complaining the drives were to slow. I record about a day worth and have the software transfer the recordings to spinning disk. I have about 6 vdevs of spinning disk. It seems the way the software moves recordings archives is it plays it back at a really fast rate and records it on the spinning drives. It seems to slow down after some time. Sometimes it doesn’t transfer over fast enough. We have 1TB of Ram will adding a NvMe Fusion Drive help? Or will a Fusion Drive allow me to record directly to the spinning disk pool. I am trying to find a way to increase iops on the spinning disk pool
We need to know the details. How much data, at what rate, sustained. What is your pool and vdev setup. Hardware details, etc.
We have about 300 cameras recording at 20fps 6mbps bit rate 5MP.
6 vdevs of WD Ultrastar drives
Intel Gold CPU 1TB DDR4 25gb x2 nic.
Would be nice to record directly to the spinning drives but the software was complaining about iops. So now data is moving from
Nvme drives to spinning via the software archive back up.
What type of vdevs? Mirrors or raidzx ? Whats the exact model of the xeon gold ? What cpu usage do u see during that archiving operation ?
Edit: if these are sync writes, a good slog might help.
Edit 2: so 300 cameras = 300 files ?
2 x Intel Xeon Scalable Gold 6346, 3.1GHz (16-Core, HT, 3200 MT/s)
It’s about 20-40%
RaidZ 2
Would adding a fusion pool help?
300 cameras it records 1 min files so lots of files.
I would try to record directly to pairs of mirrors. Then move that data for archival purposes to a raidz2 storage.
Mirrors are alot better for that kind of operation.
Are there any threads at 100% on the cpu ?
1 Like
Nothing at 100% right now I am writing a day worth to Nvme drives and it sends it to spinning disk every 8 hours but it’s really slow maybe 400mbps
How wide are your z2 vdevs ? How full is the pool ? How many drives ? Are they in a jbod enclosure or how are they connected ?
Is that recording software running directly on Truenas ? If its doing sync writes you might benefit from a SLOG.
But as i said. For high IOPS you need lots of pairs of mirrored drives - or go SSD.
No. The special vdev takes up metadata and small files, it will not help with large video files and is absolutely NOT a cache where incoming files would be taken up before being moved to slower storage during a less active period (the way of Apple’s Fusion Drive).
More IOPS = more vdevs.
Striped mirrors are recommended for block storage because it requires high IOPS for many small transactions. For your big video files, I suppose that raidz is more appropriate but you’ll need a lot of drives to have many vdevs.
Please elaborate about your drives (both solid state and spinning rust) layouts.
There are 1 minute with 6mbit/s so 45 MB per file. Is this already considered large ? I have no clue…honest question.
Yes. Usually the threshold is 1MB, but it depends on the performance and use case… and as such on the choosen block size.
2 Likes
It’s 11 wide Raidz 2 x6
Yes an Apple Fusion Drive would be nice
They are 11 wide Raid Z 2 and like 50% full. They are connected via SAS3-9500-16E
Recording software is on another server tired NFS, SMB, Iscsi all the same performance.
Which would mean a grand total of 66 HDDs, correct?
Assuming 300 100 iops per drive, that would mean 1.800 600 iops… hardly possibly low enough to complain. What about your SSDs?
That’s why I can’t figure out why it’s slow or struggling. NVME is like 800,000 iops micron 9400
Really ? Isnt it more like 100 for a 7200rpm drive ? Then iops would be 600.
1 Like
iX’s value in their ZFS Storage Pool Layout - Resources - TrueNAS Community Forums was 250. Going from memory it was a conservative number, hence I added a little bit… clearly I remembered wrong.
Here is how to calculate them.
However, being these streamed writes, and assuming these VDEVs are doing nothing else, there should not be an iops issue.
With block storage (iscsi) mirrors are recomended . Nfs -, you could try sync writes + and an intel optane as a SLOG.
Your hardware is beefy, i just dont see another reason than “wrong” vdev layout.