Slow transfer speeds

[Davvo]

Increasing the ARC is the likely cheaper way.

[charlie20231]

Does a nvme drive as a cache drive work slightly different to using memory as a cache?

[Davvo]

It does. Basically ARC is RAM, much faster; L2ARC is PCIe. Additionally, L2ARC is subordinate to ARC and works in a different way.
Then there is SLOG, not a write cache.

[charlie20231]

Ok. So if its writing to the RAM cache before it is writing to the SSD why would i be getting the slowdown on the SSD on the second transfer of files if i have 41gb free in RAM as surley for the second transfer it should be writing to the RAM.?

I may be getting confused here. Is the L2ARC (NVME ssd) used in file transfers eg when transfering files to the system they are writted in the L2arc wheras with ARC (RAM) it only stores files that are access regularly and wont store the files from a file transfer in the ARC or is this only the case when L2ARC is used in conjunction with arc and when no L2ARC is available, when transfering files to the system they will be written into RAM which writes it to the storage?

[charlie20231]

This video says that if you write a lot of files then L2arc wont bring any benefits and that your better off adding more memory to the system. Does this mean that the speed slowdown when transfering a second batch of files is due to there not being enough RAM as someone else also mentioned that truenas only uses 50% of ram?

Explaining ZFS LOG and L2ARC Cache: Do You Need One and How Do They Work?

[Davvo]

You are a bit confused, but the TLDR here is that there is no write cache in ZFS.

When you transfer data TO your NAS it gets handled by the RAM for a few (5 iirc) seconds first and then it is flushed to the drives, which have an internal cache each that can be relatively big (some SSDs) or small (up to 500MB for large disks) and it’s used not to swamp the transfer as the drive writes the data (writing speeds are universally lower than reading speeds).
The larger your RAM, the larger these transaction groups (txg) are; however, even if you have TBs of RAM if your drives cannot keep up with the data flux your transfer will hit a wall.

A L2ARC is a READ CACHE that helps with, as the name suggests, reads; moreover, its referenced into the ARC so if you don’t have enough RAM (at least 64GB) or have too much L2ARC (suggest ratio is 1:4 to 1:6 of ARC:L2ARC) your system’s performance will actually decrease.

A SLOG is a write LOG, not a write CACHE, and is useful only if you enforce synchronous writes; it concerns almost exclusively block storage.

Also, TrueNAS SCALE being based on Linux was able to only use up to 50% of its memory for ARC, but the recent dragonfish/dragonfin (fact-check me here, totally going by memory) update should have introduced a fix that finally delivered an usable ARC experience. The Linux kernel writers had to wait for FreeBSD to come teach them… but let them continue believe their BeTteRFS is actually better

[ericloewe]

@Stux

also found reviews showing just 30 IOPs. Which is slow.

“slow” doesn’t even begin to describe such a result, if true. Spinning rust will do 100+ IOPS!

[ericloewe]

@Davvo

TLDR here is that there is no write cache in ZFS.

There is a write cache, it’s just in RAM. So there’s nothing to gain from adding a “write cache” SSD, hence why the option does not exist.

[winnielinnie]

Someone needs to write up a resource that demystifies the concepts of “cache”, “write cache”, ARC, L2ARC, ZIL and SLOG in the context of ZFS.

Ideally, the resource should be brief, so that new users don’t find it intimidating or bewildering.

[ericloewe]

You mean like jgreco’s? We can probably port it over.

[Davvo]

Will do.

[charlie20231]

Thank you all for the help. Does anyone have any ssd drive reccomendations that are better performing than my current drives that would be able to handle transfering files that amount to 10Gb in size without slowing down? Idealy drives that have a good SSD Endurance (TBW) spec.

[DjP-iX]

We have some of that available on the docs hub:

ZFS Primer

ZFS ZIL and SLOG

[charlie20231]

Thank you ill take a look at that

[HoneyBadger]

I dragged over an old post of mine that explains how the “write throttle” works, but it doesn’t pull apart the issue with different NAND types and SLC caching tricks.

Some Insights (and generalizations) on the OpenZFS Write Throttle

If you’re wondering “why do the writes to my TrueNAS system start off really fast, but slow down after a few seconds?” then this post may help elaborate a little bit on the nature of the OpenZFS “Write Throttle” - the mechanism by which TrueNAS adjusts the incoming rate so that your system is still able to respond to remote requests in a timely manner while not overwhelming your pool disks. The below comments should be considered for users who have made no changes to their tunables. I’ll talk a…

[Stux]

@ericloewe

“slow” doesn’t even begin to describe such a result, if true. Spinning rust will do 100+ IOPS!

I rechecked. The review (for 0.5TB model) was testing 16MB IOs when it found 30 Iops. This is expected on 600MB/s SATA.

Silicon Power A55 512GB Review - Is It Any Good?

The drives might not be that crap.

[Davvo]

Dig the IOPS

[charlie20231]

Update

I have purchased Samsung 870 EVO 2 SSD’s TB to replace the Silicon Power A55 2TB and wow the performance is so much better. i did a test by copying a folder that is 9.16 GB and the drives were a constant 723MB/s