Slow RaidZ1 Read/Write Speeds (20 mbps)

zamnzim · October 10, 2024, 3:19pm

I’ve hit a wall with this issue and cannot seem to get any read/write speeds above 25mbps.
I have tried removing compression and disabling sync. I’ve been attempted striping with raid 0 and mirrored raid 1. No matter the raid configuration the speed never changes. Currently I’m running raidz1.

Specs:
Processor: Intel(R) Xeon(R) CPU E3-1230 v6 @ 3.50GHz
Memory: 32 GB 2400 mhz (2x 16GB)
NIC: Intel X520-DA2 10GB sfp+
Hard Drives: 4x HGST 12gbps SAS HDD - HUS726040ALS211 - raidz1
HBA: Dell HBA330+

Tests:
SMB: (Transfer speeds via SCP and SFTP are about the same)

Fio:

testfile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.33
Starting 1 process
testfile: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=513MiB/s][w=513 IOPS][eta 00m:00s]
testfile: (groupid=0, jobs=1): err= 0: pid=9925: Wed Oct 9 12:12:33 2024
write: IOPS=473, BW=474MiB/s (497MB/s)(27.8GiB/60055msec); 0 zone resets
bw ( KiB/s): min=333824, max=550912, per=100.00%, avg=485324.01, stdev=47828.97, samples=120
iops : min= 326, max= 538, avg=473.86, stdev=46.75, samples=120
cpu : usr=1.41%, sys=10.00%, ctx=29630, majf=0, minf=37
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,28458,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):

DD Write:

4194304000 bytes (4.2 GB, 3.9 GiB) copied, 0.645907 s, 6.5 GB/s

DD Read:

4194304000 bytes (4.2 GB, 3.9 GiB) copied, 0.389345 s, 10.8 GB/s

Iperf3 running 4 parallel streams:

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec 129 sender
[ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec receiver
[ 7] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec 135 sender
[ 7] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec receiver
[ 9] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec 124 sender
[ 9] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec receiver
[ 11] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec 142 sender
[ 11] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec receiver
[SUM] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 530 sender
[SUM] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec receiver

joeschmuck · October 10, 2024, 3:29pm

Use iperf for network speed tests. That is the issue you are facing as I read it. Do not trust the Crystal Disk Benchmark blindly. ATTO is another good test and these two do not always match for results.

So, using iperf (included in TrueNAS) is the best method to go initially.

Good luck.

zamnzim · October 10, 2024, 3:35pm

Iperf3 Single Stream:

zim@NAS-1 ~ % iperf3 -c 10.0.50.1
Connecting to host 10.0.50.1, port 5201
[ 5] local 10.0.50.15 port 36028 connected to 10.0.50.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 473 MBytes 3.97 Gbits/sec 232 1.57 MBytes
[ 5] 1.00-2.00 sec 470 MBytes 3.94 Gbits/sec 0 1.77 MBytes
[ 5] 2.00-3.00 sec 468 MBytes 3.92 Gbits/sec 0 1.96 MBytes
[ 5] 3.00-4.00 sec 470 MBytes 3.94 Gbits/sec 244 1.57 MBytes
[ 5] 4.00-5.00 sec 562 MBytes 4.72 Gbits/sec 92 1.39 MBytes
[ 5] 5.00-6.00 sec 630 MBytes 5.28 Gbits/sec 7 1.32 MBytes
[ 5] 6.00-7.00 sec 498 MBytes 4.17 Gbits/sec 5 1.10 MBytes
[ 5] 7.00-8.00 sec 599 MBytes 5.02 Gbits/sec 0 1.44 MBytes
[ 5] 8.00-9.00 sec 650 MBytes 5.45 Gbits/sec 11 1.39 MBytes
[ 5] 9.00-10.00 sec 651 MBytes 5.46 Gbits/sec 13 1.33 MBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 5.34 GBytes 4.59 Gbits/sec 604 sender
[ 5] 0.00-10.00 sec 5.34 GBytes 4.59 Gbits/sec receiver

iperf Done.

Iperf3 4 parallel streams:

zim@NAS-1 ~ % iperf3 -c 10.0.50.1 -P 4
Connecting to host 10.0.50.1, port 5201
[ 5] local 10.0.50.15 port 51450 connected to 10.0.50.1 port 5201
[ 7] local 10.0.50.15 port 51452 connected to 10.0.50.1 port 5201
[ 9] local 10.0.50.15 port 51456 connected to 10.0.50.1 port 5201
[ 11] local 10.0.50.15 port 51468 connected to 10.0.50.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 180 MBytes 1.51 Gbits/sec 0 1.57 MBytes
[ 7] 0.00-1.00 sec 180 MBytes 1.51 Gbits/sec 101 1008 KBytes
[ 9] 0.00-1.00 sec 181 MBytes 1.52 Gbits/sec 66 1015 KBytes
[ 11] 0.00-1.00 sec 180 MBytes 1.51 Gbits/sec 54 168 KBytes
[SUM] 0.00-1.00 sec 721 MBytes 6.05 Gbits/sec 221

[ 5] 1.00-2.00 sec 212 MBytes 1.78 Gbits/sec 348 631 KBytes
[ 7] 1.00-2.00 sec 214 MBytes 1.79 Gbits/sec 336 452 KBytes
[ 9] 1.00-2.00 sec 211 MBytes 1.77 Gbits/sec 587 451 KBytes
[ 11] 1.00-2.00 sec 212 MBytes 1.77 Gbits/sec 0 240 KBytes
[SUM] 1.00-2.00 sec 849 MBytes 7.12 Gbits/sec 1271

[ 5] 2.00-3.00 sec 205 MBytes 1.72 Gbits/sec 51 551 KBytes
[ 7] 2.00-3.00 sec 205 MBytes 1.72 Gbits/sec 5 607 KBytes
[ 9] 2.00-3.00 sec 206 MBytes 1.73 Gbits/sec 6 568 KBytes
[ 11] 2.00-3.00 sec 206 MBytes 1.73 Gbits/sec 0 256 KBytes
[SUM] 2.00-3.00 sec 822 MBytes 6.90 Gbits/sec 62

[ 5] 3.00-4.00 sec 210 MBytes 1.76 Gbits/sec 101 539 KBytes
[ 7] 3.00-4.00 sec 208 MBytes 1.74 Gbits/sec 70 471 KBytes
[ 9] 3.00-4.00 sec 209 MBytes 1.75 Gbits/sec 47 452 KBytes
[ 11] 3.00-4.00 sec 209 MBytes 1.75 Gbits/sec 0 281 KBytes
[SUM] 3.00-4.00 sec 835 MBytes 7.01 Gbits/sec 218

[ 5] 4.00-5.00 sec 214 MBytes 1.79 Gbits/sec 4 570 KBytes
[ 7] 4.00-5.00 sec 214 MBytes 1.79 Gbits/sec 20 390 KBytes
[ 9] 4.00-5.00 sec 214 MBytes 1.79 Gbits/sec 1 546 KBytes
[ 11] 4.00-5.00 sec 213 MBytes 1.79 Gbits/sec 0 281 KBytes
[SUM] 4.00-5.00 sec 854 MBytes 7.17 Gbits/sec 25

[ 5] 5.00-6.00 sec 212 MBytes 1.78 Gbits/sec 35 584 KBytes
[ 7] 5.00-6.00 sec 214 MBytes 1.79 Gbits/sec 14 358 KBytes
[ 9] 5.00-6.00 sec 214 MBytes 1.79 Gbits/sec 0 660 KBytes
[ 11] 5.00-6.00 sec 213 MBytes 1.79 Gbits/sec 0 281 KBytes
[SUM] 5.00-6.00 sec 853 MBytes 7.16 Gbits/sec 49

[ 5] 6.00-7.00 sec 214 MBytes 1.79 Gbits/sec 20 526 KBytes
[ 7] 6.00-7.00 sec 215 MBytes 1.80 Gbits/sec 0 634 KBytes
[ 9] 6.00-7.00 sec 211 MBytes 1.77 Gbits/sec 23 335 KBytes
[ 11] 6.00-7.00 sec 213 MBytes 1.79 Gbits/sec 0 286 KBytes
[SUM] 6.00-7.00 sec 853 MBytes 7.16 Gbits/sec 43

[ 5] 7.00-8.00 sec 212 MBytes 1.78 Gbits/sec 24 443 KBytes
[ 7] 7.00-8.00 sec 214 MBytes 1.79 Gbits/sec 12 522 KBytes
[ 9] 7.00-8.00 sec 216 MBytes 1.81 Gbits/sec 20 522 KBytes
[ 11] 7.00-8.00 sec 216 MBytes 1.81 Gbits/sec 0 297 KBytes
[SUM] 7.00-8.00 sec 858 MBytes 7.20 Gbits/sec 56

[ 5] 8.00-9.00 sec 195 MBytes 1.64 Gbits/sec 70 518 KBytes
[ 7] 8.00-9.00 sec 194 MBytes 1.63 Gbits/sec 0 694 KBytes
[ 9] 8.00-9.00 sec 192 MBytes 1.61 Gbits/sec 103 478 KBytes
[ 11] 8.00-9.00 sec 193 MBytes 1.62 Gbits/sec 0 303 KBytes
[SUM] 8.00-9.00 sec 774 MBytes 6.49 Gbits/sec 173

[ 5] 9.00-10.00 sec 211 MBytes 1.77 Gbits/sec 36 400 KBytes
[ 7] 9.00-10.00 sec 212 MBytes 1.78 Gbits/sec 1 543 KBytes
[ 9] 9.00-10.00 sec 214 MBytes 1.79 Gbits/sec 33 540 KBytes
[ 11] 9.00-10.00 sec 212 MBytes 1.78 Gbits/sec 0 307 KBytes
[SUM] 9.00-10.00 sec 850 MBytes 7.13 Gbits/sec 70

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.02 GBytes 1.73 Gbits/sec 689 sender
[ 5] 0.00-10.00 sec 2.01 GBytes 1.73 Gbits/sec receiver
[ 7] 0.00-10.00 sec 2.02 GBytes 1.74 Gbits/sec 559 sender
[ 7] 0.00-10.00 sec 2.02 GBytes 1.73 Gbits/sec receiver
[ 9] 0.00-10.00 sec 2.02 GBytes 1.73 Gbits/sec 886 sender
[ 9] 0.00-10.00 sec 2.02 GBytes 1.73 Gbits/sec receiver
[ 11] 0.00-10.00 sec 2.02 GBytes 1.73 Gbits/sec 54 sender
[ 11] 0.00-10.00 sec 2.02 GBytes 1.73 Gbits/sec receiver
[SUM] 0.00-10.00 sec 8.08 GBytes 6.94 Gbits/sec 2188 sender
[SUM] 0.00-10.00 sec 8.06 GBytes 6.92 Gbits/sec receiver

iperf Done.

Protopia · October 10, 2024, 3:50pm

There are Lies, Darned lies, and Statistics - and then there are performance tests.

If you want to understand ANY performance test you need to know what you are testing and how you are testing it and that means understanding ZFS i.e. asynchronous vs. synchronous writes, caching, pre-fetch, how RAIDZ works, block size, record size, compression, etc. and also understanding exactly how each performance testing tool is working. And even if you understand literally everything about the artificial tests, you still need to relate these to your real-life workload.

For example, is your dd write test doing asynchronous or synchronous writes? Is your dd read test using pre-fetch or not? Did you clear your ARC cache before any read tests? If your dd write test is using asynchronous writes, does it complete as soon as sends its last write to ZFS, or does it wait until all the writes have been committed to disk?

And this is only scratching the surface of ZFS’ complex optimisation code.

That said, I am not sure I am seeing a problem in any of your tests:

fio write - 474MB/s is pretty respectable for a 4x HDD RAIDZ1.
dd - far too many unanswered questions to know whether these are accurate or not. But the figures are IMO pretty darn respectable IMO.
iperf - 9.38Gb/s on a 10Gb/s network is pretty darn respectable IMO.

So I am not sure I see any performance bottlenecks here.

zamnzim · October 10, 2024, 4:04pm

This cloud server supports 50GB/s and will write to my PC’s SSD’s at 5GB/s but cant write above 13.4MB/s to the Truenas server.

Below is an image of me transferring a movie from the cloud server to my local Truenas server.

My main issue is that the test all reflect good performance but in practice the transfer speeds are terrible.

Farout · October 10, 2024, 4:25pm

So you say your PC has a 40Gbit/s NIC and you have a 40Gbit/s Internet connection ?

Protopia · October 10, 2024, 5:11pm

No - that is a redacted image where all the information that might give context has been hidden i.e. useless.

I say again. There are Lies, Darned lies, and Statistics - and then there are performance tests. And you need to understand literally every link in the technology chain and how it works, and how your test works in order to interpret the results in a meaningful way.

Q: My car is capable of 70MPH, and yet my 70M trip on a motorway speed limit of 70MPH took me 5 hours? Why?

A: Wrong fuel in the tank or traffic conditions or the motorway was only 35 of the 70 mile journey or I was towing a trailer at the time or my car is only capable of 70MPH downhill with a following wind or there was heavy rain and I had to slow down or I had to stop for a rest break or …

Details matter.

Exactly what I have been saying. Lies, Darned Lies, Statistics and Artificial Performance Tests.

What you are effectively saying is that real-life isn’t like theory and “shit happens”.

Congratulations and welcome to the real world.

A few complete guesses

Your own internet connection is not 50GB/s i.e. 400Gb/s.
You are using sftp to transfer the file which is known to be extremely slow[1].
You are not writing to an SSD on TrueNAS so not a reasonable comparison.
You were never achieving writes to your PC SSDs at 5GB/s because a single SATA channel is only capable of 600MB/s and I doubt that you were writing to 9 SSDs in parallel on your PC.
There is literally NOTHING to suggest that the reason your sftp transfer was only going at 13.4MB/s was a bottleneck in the HDD performance.

But aside from all the above, it is of course a problem with TrueNAS.

[1] See this explanation: SFTP will almost always be significantly slower than FTP or FTPS (usually by several orders of magnitude). The reason for the difference is that there is a lot of additional packet, encryption and handshaking overhead inherent in the SSH2 protocol that FTP doesn’t have to worry about. FTP is a very lean and comparatively simple protocol with almost no data transfer overhead, and the protocol was specifically designed for transferring files quickly. Encryption will slow FTP down, but not nearly to the level of SFTP. SFTP runs over SSH2 and is much more susceptible to network latency and client and server machine resource constraints. This increased susceptibility is due to the extra data handshaking involved with every packet sent between the client and server, and with the additional complexity inherent in decoding an SSH2 packet. SSH2 was designed as a replacement for Telnet and other insecure remote shells, not for very high speed communications. The flexibility that SSH2 provides for securely packaging and transferring almost any type of data also contributes a great deal of additional complexity and overhead to the protocol.

zamnzim · October 10, 2024, 6:35pm

I have a 10gbit network drop to the house. I rent a cloud server that has 50gbit network as service. I expect the the bottleneck from 50 to 10. File transfers via SMB/SFTP/SCP between the sites from my PC are achieving 1.5 gbps MAX. (i realized the typo i sent earlier. 1.5gbps is on the best day ever.) I have 2 NVMe SSD’s running in a Windows software Raid 0 on my PC which i use as a gaming drive which has been my comparison.

I’m just trying to find the bottleneck. I have never used Truenas before. I’ve never used ZFS before either. I work in a data center but i work primarily on switches and routers. I know a lot about networking but this stuff is shooting right over my head.

I’m just hoping for some helpful pointers.

Protopia · October 10, 2024, 7:41pm

As I said in an earlier post, 470+MB/s write speed from a single thread is pretty good. That is pretty much what I achieve on my own 5x RAIDZ1. So your disk write speeds look good.

With iperf you were achieving almost 10Gb/s on your 10Gb/s LAN - so that also looks good.

So looking at the local technology components, you have all the right basic building blocks.

But whilst your LAN is completely under your control, what happens on t’internet is different - you have contention with other traffic, you may have traffic shaping that prioritises real-time video and audio over web traffic over file transfer, you have much much higher latency when the sending node is waiting for acknowledgements / confirmations from the receiving node etc.

And then you are layering SMB or SFTP protocols on top of that - and these both have encryption overheads. According to the web site I linked to SFTP is particularly bad cf. normal FTP. (And that is why there are alternatives to SFTP that are designed to have parallel streams and so boost the speed.)

Protopia · October 10, 2024, 7:45pm

As a suggestion, you can run SFTP on your TrueNAS box sending the output to /dev/null and this should hopefully allow you to confirm that writing to disk is not the constraining factor.

zamnzim · October 13, 2024, 6:52pm

After messing with it for a while I found that spanning-tree was flapping on my switch. The bridge interface on turenas was triggering bpduguard to flap the switch interface in and out of err-disable state.

My network setup:

br0 = bond0 & Static IP
bond0 = 10gig SFP, 10gig SFP

I set the interface on the switch as a trunk connecting to TrueNAS and the switch stopped its spanning-tree shenanigans.

I wasn’t aware that the truenas servers bridge interface was going to send bpdus to the switch. But it makes sense as all bpdu’s are just switches negotiating “bridge” id’s and bridge is kind of in the name of the interface. lol.

whooops.

Thanks for the help!!!