Test write latency to disk

TrueNAS General
1

I am using TrueNAS Scale 25.04.2.4
Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
768GiB RAM
25gb nic

A software I am using is saying there is a bottleneck somewhere when writing to disk.

Its saying at times I am at 100ms write times I need to be at about 33ms or less.

I am connecting a share via SMB to a window file share writing to spinning disk.

Is there a way to see or monitor write latency or how busy the disk are if anything is holding up writing? Im wondering how many more vdevs I would need to add to improve performance or if its something else.

2

You should be specific exactly what software you are using to test with. I recommend iperf3 as it is reliable.

As for write latency, if you really have 768GiB of RAM, then you would be writing to the RAM and it caches the data and writes the data to disk as it can. This would be extremely fast, not part of a slowdown for a typically configured NAS.

I would think your latency is your network but you would need to test and verify, hence the recommendation to use iperf3.

You also say “at times”, well you need to define what is happening during these times to identify why you are experiencing the slowdowns, and the frequency of them.

Let us know when you know more.

3

My 2p’s worth

  1. If the OP is using sync writes that would add extra latency as the data has to actually be written to disk before being acknowledged. But given SMB from windows, that would be something deliberately set on the NAS
  2. If the app in question is expecting local disks and not designed for network shares then it might complain about latency

As @joeschmuck says, using async writes the data gets written to a file in memory and the write is acknowleged back to the client. The data later being flushed to disk - but windows wouldn’t know or care about that.

More details required please

1 Like
4

Thank you,
The network latency is fine. I can push 25G it all comes off the same switch.
We are using a video recording software so it’s always writing. It show us using all the RAM either.

5

I am not sure if its sync or async writes but we are using windows file share to record cameras.

6

How much data are you writing per second total? If this were continuous, then you could be filling up your ARC as well. Look to see what your RAM is doing, is there no free RAM and it is all in the ARC (ZFS Cache) ? If your cache is full, you need to use faster storage media such as NVMe, or as you suspected, possibly more vdevs, or at least a better configuration. But there is probably some tuning you could try first. You need to know how much continuous data you are pushing to the NAS.

You have not provided any details about your configuration, the pool/vdevs, the CPU usage, etc.

I did a little math and taking some liberties with the values, however you may be recording at ~8Gbps when you have the 100ms delays happening, however at 33ms it would be ~24Gbps data stream, which is basically the limit of the network in an ideal situation. I could be wrong about this as well.

I see two things you may need to do:

  1. Redesign your pool to allow more bandwidth.
  2. Possibly invest in 50GbE.

I am not the expert on this topic, not even close. I can’t really offer any more advice.