system:
9100f, 2x32GB RAM, GT1030, 3x4TB HDD, 2.5G+1G NICs
transfer speeds over SMB are very low for moderate sized files and unstable for big sized filed.
record size 1MB
sync disabled
iperf give 2.2Gbps as expected
for smaller files ~100MB it falls to 10MB/s
What’s up with the nondescript Discord link?
282MB/s sounds like what you would expect given a 2.5 Gb/s link and especially your 2.2Gb/s measured iperf. Note the capitalised B.
Small files not achieving that kind of speed is also fully expected, especially with HDDs.
It’s actually a link to our Discord server, but the nature of Discord means that preview links won’t generate correctly. (Probably ever.)
I’ll see if I can copy the TL;DR from the thread there to here.
The peak of 282MB/s is fine for a 2.5Gbps network, but it’s the “sawtooth” or “sine wave” pattern in the transfers that tells me that it might be ZFS write throttling kicking in, which means “network too fast for disks.”
@Arjun_Gupta do a transfer, and while it’s in the middle of “doing the wave” can you do the following from a command shell:
cp /proc/spl/kstat/zfs/YOURPOOLNAME/txgs ~/txgs.txt
From there, you can copy that txgs.txt file from your TrueNAS machine somewhere else, and attach it here as a text file. We’ll see if you’re having big surges of dirty data.
txgs.txt (13.9 KB)
this is while writing to the pool
hey…
any solution ?
Sorry, it’s been busy. Taking apart the txgs file there, it seems like it got trimmed a bit at the end so we don’t see multiple “cycles” so to speak, but from txg 9871178 you can see that your stime (time required to sync to disk) starts to shoot up rapidly as you’re trying to write. It starts slower but ramps up quickly, and with 3x4TB you really have “two disks writing data” and you’re asking them to keep up with a 2.5Gbps network. They don’t seem capable of doing it.
damn…
so hardware change is the only solution then?
More an addition than a change - a greater number of disks is needed.
will a DAS(USB 5gbps T-T) work or through HBA only?
Internal is most preferred, then through external shelves over SAS.
USB isn’t recommended; some community users have leveraged it for backup or replication purposes but I wouldn’t be confident enough to state that you should use it for a routine, daily-purpose pool.
will a SSD cache help?
internal is pretty much not possible
and external shelves are way too expensive here (to the point I might build a complete new NAS or make this one all flash)
Afraid not - it’s the main vdevs that are limiting you, not anything in the transactional/write logs.
