Hi there, i am new to truenas and have be using truenas scale for about a month but I have noticed that I am getting slow transfer speeds when using the SMB share when lots of files (total size 10Gb). it runs fine but about halfway through it slows down to a write speed of 6.2MB/s. i am using 4 SSD each 2tb in size and so far have only used 3% of the space.
SSD Brand: Silicon Power A55 2TB
Is this normal behaviour? does anyone know why this could be happening?
Start by posting a list of your hardware. There are a few things it could be, off the top of my head, SMR drives, Low RAM, Realtek NIC. What version of TrueNAS are you running? Remember one thing, we can provide better and faster answers if we don’t have to guess.
This new forum was to inform new users of these forum rules, hopefully that will be fixed this week.
Brand of SSD will be relevant, maybe their cache gets overloaded after a few TB.
Else yes, a lot of small files are usually slower than fewer larger files of same total size. The reasons are various (from metadata writing on target side to tcp window scaling on source side…)
You should be able to see some of the hardware in the picture above. Im using a 10Gbe sfp nic, just trying to find name of it now. On the windows side its transfering from a NVME drive a firecudda 530 500gb drive using a tplink 10Gbe rj45 nic. Ssd in truenas are 4 silicon power 2tb drives.
Need to know the exact model of the SSDs.
Should be able to see it in the disks view
Some SSDs’ performance crashes on large sustained writes, for example the Intel 660p in this graph
They are the siicon power a55 2tb version. Im using a LSI 9300-16i SAS HBA - P16.12 IT Mode - Unraid ZFS TrueNAS FreeNAS Proxmox JBOD with the ssd
The drive uses SLC caching. I found reviews showing the 1TB model will drop to 90MB/s sequential writes, but that’s far above.
I also found reviews showing just 30 IOPs. Which is slow.
It could be choking on the sync write every 5 seconds once the receive buffer flushes as it waits for the saturated SLC to do its job
Is it after 5 or 10s when the issue occurs?
A silly question, but how are you cooling the 9300-16i ?
Does it have enough airflow over the heatsink ?
Are you using a server case ?
Edit: the TP link NIC could also be the culprit. Make sure its cooled enough.
I had to aim a fan at my LSI HBAs.
That does indeed seem to be a speed reduction caused by a full cache, but others have pointed out plausible alternatives: heat might tackle your performance significantly, although in this case I am inclined to blame the drives’ inability to keep up with the 10G data flux.
Comes off as a bit annoying, but after trying and providing the feedback to a few of the previous comments. Try the same transfer a few time and you should see the files moved into arc. The transfer should slowly speed up as cached files are pulled from arc as opposed from disk.
I believe the default in Scale is to utilize 50% of ram for cache, so you should have plenty of space to occupy.
This could help to rule out disk and controller issues.
This can also exaggerate other limitations given the data can be so quickly available for transfer.
Ill give that a go and see if it makes any difference. When trying the same transfer again, i assume you would leave the orignal in its place on the truenas system?
Its a HP ML350 gen9 system so is enterprise fans and has a fan baffle. Pc side im using 3 140mm fans to cool my pc
Hmm ok, but mine is dropping considerably lower than the 90MB/s down to 6.22MB/s. Do u have a link to some of thoes reviews of the drive?
Are you transferring the files to or from the server?
Does the behavior exist in both or only one scenario?
I was thinking you were pulling files from the server. Which case I’d clear the local copy on your desktop and copy it back off the server.
But you now have me curious if the issue exists with transfers both ways.
I was orignally copying files to the truenas from my pc as use it to backup pictures. Ill try copoying the file from my server to my pc to see how it behave- it should be quick as wouldnt it be stored in the cache if im coppying from the server to my desktop?
just tried copying the files from the server to my PC and no issues there. But i suppose that may be because they are in the cache
I tried copying the files again from my Windows PC to the truenas server and no issues- constant 600MB/s. I then tried it for a second time to see if I got the same result and like before it dropped to 6/7MB/s after about 10/15 seconds of transferring images so it does seem to be a cache issue with the drive as people have mentioned.
Thank you everyone for the help
The way ZFS works is that it buffers incoming data for 5 seconds before blasting it out to the disks.
Then while that is being blasted it buffers incoming data.
If it gets to the point that it needs to write its buffer and it’s still writing the previous buffer, it pauses everything.
This causes massive issues with protocols like iSCSI (disconnections) and would probably show up us slow downs like you see in SMB.
Thus you need drives which can write, in aggregate, as fast as the network can ingest
At potentially 1GB/s, you’d need closer to 12 of these drives.
Ok thank you.
So if i put a nvme cache drive would that resolve this issue or would i still face the same issue of speed slowdown after a big data transfer? Because at the moment i only expierence the issue if i am transfering 2 bunches of files eg transfer 1 is 7gb and then once that transfer is done transfer 2 is 7gb and its only on this second transfer after about 10seconds that i see the speed decrease.
