How to diagnose poor read/write performance/identify bottleneck

TrueNAS General
, ,
1

Hello,

I am experiencing unexpectedly low read/write performance on my TrueNAS Scale server running version Dragonfish-24.04.2.

The pool in question consists of x4 Western Digital Red Pro 16TB 7200 RPM drives in RAIDZ-1. Each of these drives have been benchmarked to read and write in the ballpark of ~210 MB/s. I found a RAIDZ calculator which suggested that I should get read speeds of ~3x that of an individual drive, which would be ~630 MB/s.

The connection between my laptop and the TrueNAS server is limited to 1 gbps - this is my laptop’s NIC. The cable between my laptop, the switch between them, and the NIC on my TrueNAS server all support 2.5G. I did test multiple cables and got the same results. Therefore, I would expect my bottleneck to be 1 gbps / 125 MB/s.

However, the best I can achieve is 80 MB/s writes and 70 MB/s reads over SMB, which is significantly less than my network connection. Useful notes:

  1. No SMART tests, scrubs, or other tasks were running to the best of my knowledge
  2. No apps should have been putting any significant load on the pool
  3. The dataset had no compression and no encryption
  4. Not very scientific, but I was copying large files (~1-2 GB) to get ~an ideal scenario because this is faster than many small files (RIP git users)
  5. I have DDR5 RAM and a Core i9, so I don’t believe these are the issue

Questions

  1. Any ideas on what’s causing this poor performance?
  2. I acknowledge the calculator may have been wrong, or I interpreted the results incorrectly, so does anyone have any experience which indicates whether these are reasonable values for my setup?
  3. I’m not familiar with SMB overhead - could this be the issue?
2

Copying includes the speed of the laptop drive. Is laptop a SSD?
Try testing with a larger, compressed file like a zip. You might also use a large video like mp4.

3

Yes - the laptop drive is a Samsung 970 EVO Plus, so this should not be the bottleneck.

4

Sorry I missed this - this is exactly what I was copying :upside_down_face:

5

Hardware details please, especially for the TrueNAS side.

Like what Ethernet controller manufacturer and model. That can make a serious difference since RealTek brand is known to transfer at lower than wire speed.

6

Perhaps try iperf between client and server to confirm your potential network speed.

7

Sorry for the lack of detail, but what I know right now is the TrueNAS sever has an Intel 2.5 Gb NIC. I have heard of there being issues with RealTek, but not Intel (LMK if that’s not the case). I would assume it should at least support gigabit speeds, so I would hope this wouldn’t be the bottleneck.

1 Like
8

Quick update: I noticed that when I do backups via rsync, read speeds can get to 100-110 MB/s, which isn’t too far off from the theoretical gigabit bottleneck. This indicates to me that SMB has pretty substantial overhead.