Subpar Performance, Link State (not) Down, Intermittent iPerf3 Connection

TrueNAS General
, ,
1

Hello all,

This is my first time attempting to build and configure a NAS. I’m running into some issues surrounding performance, iPerf3, and link state that I can’t seem to squash. I do not know if all or some of these issues are related, any advice and help is appreciated. I am new, but functional, in Linux, but networking is proving difficult for me to grasp. I am trying to learn!

Hardware (Server [made from leftovers]):
CPU: 2X Xeon E5520 | RAM: 12GB (6x2GB) ECC | NIC: 2X Intel 82574 1GbE (onboard) | 5x WD10EZEX 1TB HDD, 120 GB SSD boot drive

TrueNAS Setup:
SCALE Ver. 25.04.0 | one pool [RAIDz2 Data, no other VDEV] | one SMB dataset+share [Std SYNC, LZ4 compression, no Atime or dedup]

Networking:
known good Cat6 cable from onboard 1GbE NIC to 2.5Gb port on Arris router | Server IPv4 address set to static in router settings | Router DNS set to local PiHole address | TrueNAS nameserver set to PiHole/DNS address | Interface set with DHCP deselected, MTU set to 1500 (have tried 9000), and server static IP set as Alias

Client:
Win10 Ver 10.0.19045 Build 19045 (WiFi 6 connected)
Samsung S24+, Android 14 (WiFi 6 connected)

Issues: I’m getting 1-11 MB/s on data transfer between my PC and the SMB share folder while transferring one large video file. Given the drive performances in this pool I’d expect close to 300 MB/s or near the 1GbE NIC limit. Transfer speeds fluctuate significantly. The Interface shows LINK STATE DOWN on the dashboard despite being able to connect to the share. Yesterday, iPerf3 was unable to run, error iperf3: error - failed to read /dev/urandom: Bad address after accepting connection from the client. Unable to diagnose. Today, iPerf3 works fine, showing 40Mb/s (why so low?), then reverts back to Bad Address error after an autologout from the web GUI. iperf3 results:

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  1.50 MBytes  12.5 Mbits/sec
[  5]   1.01-2.00   sec  3.50 MBytes  29.6 Mbits/sec
[  5]   2.00-3.01   sec  3.88 MBytes  32.1 Mbits/sec
[  5]   3.01-4.00   sec  9.62 MBytes  81.6 Mbits/sec
[  5]   4.00-5.01   sec  13.8 MBytes   115 Mbits/sec
[  5]   5.01-6.01   sec  9.00 MBytes  75.3 Mbits/sec
[  5]   6.01-7.01   sec  2.50 MBytes  21.0 Mbits/sec
[  5]   7.01-8.01   sec  1.25 MBytes  10.4 Mbits/sec
[  5]   8.01-9.01   sec  2.62 MBytes  22.0 Mbits/sec
[  5]   9.01-10.01  sec  2.25 MBytes  18.8 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  49.9 MBytes  41.8 Mbits/sec                  sender
[  5]   0.00-10.02  sec  49.5 MBytes  41.4 Mbits/sec                  receiver

iperf Done.

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec   640 KBytes  5.20 Mbits/sec
[  5]   1.01-2.01   sec  3.25 MBytes  27.2 Mbits/sec
[  5]   2.01-3.01   sec  1.62 MBytes  13.6 Mbits/sec
[  5]   3.01-4.01   sec  2.75 MBytes  23.1 Mbits/sec
[  5]   4.01-5.00   sec  3.50 MBytes  29.6 Mbits/sec
[  5]   5.00-6.00   sec  1.00 MBytes  8.38 Mbits/sec
[  5]   6.00-7.01   sec  2.12 MBytes  17.7 Mbits/sec
[  5]   7.01-8.01   sec  2.75 MBytes  23.3 Mbits/sec
[  5]   8.01-9.01   sec  1.62 MBytes  13.6 Mbits/sec
[  5]   9.01-10.01  sec  5.00 MBytes  41.8 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  24.2 MBytes  20.3 Mbits/sec                  sender
[  5]   0.00-10.02  sec  24.1 MBytes  20.2 Mbits/sec                  receiver

iperf Done.

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  3.50 MBytes  29.3 Mbits/sec
[  5]   1.00-2.01   sec  2.88 MBytes  23.9 Mbits/sec
[  5]   2.01-3.01   sec  6.88 MBytes  57.3 Mbits/sec
[  5]   3.01-4.00   sec  1.50 MBytes  12.7 Mbits/sec
[  5]   4.00-5.00   sec  4.00 MBytes  33.5 Mbits/sec
[  5]   5.00-6.01   sec  5.75 MBytes  48.0 Mbits/sec
[  5]   6.01-7.01   sec  12.4 MBytes   103 Mbits/sec
[  5]   7.01-8.00   sec  11.5 MBytes  97.8 Mbits/sec
[  5]   8.00-9.00   sec  4.12 MBytes  34.5 Mbits/sec
[  5]   9.00-10.01  sec  2.25 MBytes  18.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  54.8 MBytes  45.9 Mbits/sec                  sender
[  5]   0.00-10.02  sec  54.3 MBytes  45.4 Mbits/sec                  receiver

Note the wild swings in bitrate. This seemingly correlates with my file transfer speeds and variance.

Attempted troubleshooting with commands pulled from this forum and across other searches:

  • All SMART tests came back clean, despite the drives being high hours.
  • All drives were reading the appropriate individual read/write speeds (~160MB/s) using hdparm -Tt.
  • dd checks as below:
dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync 
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.920226 s, 1.2 GB/s

dd if=/dev/random of=/tmp/test1.img bs=1G count=1 oflag=dsync 
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.28256 s, 203 MB/s

Why such a disparity? I’m assuming the /zero is falsely inflating due to cache/RAM read writing?

  • I am able to successfully ping the server from the Win10 client with both the hostname and ip address, although with some fluctuation in ping between 1 and 5ms.
  • I can read and write files on the SMB share from both phone and Win10 clients.
  • ip route shows the link as up

Any insight as to why I’m seemingly an order of magnitude off in transfer/read/write speeds, why the dashboard claims the link is down when connection is holding, or why iPerf3 will not execute, would be greatly appreciated. Thanks!

2

Additionally, the login page to the web GUI will go through “reconnection” cycles… is this just an onboard NIC failure?

3

LINK STATE DOWN on the dashboard was resolved by editing the dashboard display to show the active port (out of the two) on the server… the link is in fact not down, it was just displaying the inactive port. I feel very dumb, but hopefully this helps someone else with multiple ports and are getting this “error”.

I was also able to bypass the /dev/urandom/ failures in iPerf3 by transmitting the random file generated by the dd command:

sudo dd if=/dev/random of=/tmp/test1.img bs=1G count=1 oflag=dsyncdd if=/dev/random of=/tmp/test1.img bs=1G count=1 oflag=dsync

sudo iperf3 -c [server ip] -F /tmp/test1.img

Across multiple tests I am averaging about 80Mb/s, with a max of 111 Mb/s… which seems really slow, given a 1GbE port and three 150MB/s drives. Any ideas on what could be constraining me here?

4

Additional dd tests, now that I think I’ve figured out how to actually test the pool… I think I was effectively testing the boot drive with /tmp/test1.img. Can anyone confirm? Now these numbers seems exceedingly high for my configuration. What am I missing?

edit2: I THINK this is actually normal behavior given the second batch of no-cache dd tests. So if the drive pool is performing as expected, there seems to be no other hardware issues. Is it just a poor performing NIC?

truenas_admin@truenas[~]$ sudo dd if=/dev/zero of=/mnt/pool0/ddfile bs=10G count=1 oflag=dsync
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 1.67164 s, 1.3 GB/s
truenas_admin@truenas[~]$ sudo dd if=/dev/zero of=/mnt/pool0/ddfile bs=2M count=10000       
10000+0 records in
10000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 10.2866 s, 2.0 GB/s
truenas_admin@truenas[~]$ sudo dd of=/dev/null if=/mnt/pool0/ddfile bs=2M count=10000          
10000+0 records in
10000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 4.31352 s, 4.9 GB/s
truenas_admin@truenas[~]$ sudo dd if=/dev/zero of=/mnt/pool0/ddfile bs=20M count=10000
10000+0 records in
10000+0 records out
209715200000 bytes (210 GB, 195 GiB) copied, 120.66 s, 1.7 GB/s

truenas_admin@truenas[~]$ sudo dd if=/dev/zero of=/mnt/pool0/ddfile bs=2M count=10000 oflag=dsync
10000+0 records in
10000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 149.747 s, 140 MB/s
truenas_admin@truenas[~]$ sudo dd if=/dev/zero of=/mnt/pool0/ddfile bs=20M count=1000 oflag=dsync
1000+0 records in
1000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 46.4986 s, 451 MB/s
truenas_admin@truenas[~]$ sudo dd if=/dev/zero of=/mnt/pool0/ddfile bs=200M count=100 oflag=dsync
100+0 records in
100+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 24.7269 s, 848 MB/s
truenas_admin@truenas[~]$ sudo dd if=/dev/zero of=/mnt/pool0/ddfile bs=2000M count=10 oflag=dsync
10+0 records in
10+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 16.1689 s, 1.3 GB/s
5

ZFS optimises blocks containing all zeros. use /dev/random.

/tmp tends to be an in memory file system.

6

Understood, thank you. Results are shown below with different bs and count magnitudes, all averaging about 200 MB/s. Given a RAID5/Z2 of 150 MB/s drives, I’d still expect more performance from the drives… am I misunderstanding something about the capabilities?

iPerf3 is still averaging about 110 Mbps between the Win10 client and the pool. ifconfig and ethtool confirm the 1GbE port is at 1000 Mbps with full duplex and auto-negotiation. I feel like this is networking setup issue at this point, because I should be able to see about 100 MB/s transfer with my current NIC.

Do I just need a new (to me) NIC?

dd results:

truenas_admin@truenas[~]$ sudo dd if=/dev/random of=/mnt/pool0/ddfile bs=2M count=10000 conv=fdatasync
10000+0 records in
10000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 101.545 s, 207 MB/s
truenas_admin@truenas[~]$ sudo dd if=/dev/random of=/mnt/pool0/ddfile bs=20M count=1000 conv=fdatasync
1000+0 records in
1000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 102.478 s, 205 MB/s
truenas_admin@truenas[~]$ sudo dd if=/dev/random of=/mnt/pool0/ddfile bs=200M count=100 conv=fdatasync
100+0 records in
100+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 103.567 s, 202 MB/s
truenas_admin@truenas[~]$ sudo dd if=/dev/random of=/mnt/pool0/ddfile bs=2000M count=10 conv=fdatasync
10+0 records in
10+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 153.561 s, 137 MB/s
truenas_admin@truenas[~]$ ```
7

As someone who used to do performance testing professionally, I will repeat that cautions previously given in other threads that you need to understand ZFS at a detailed level to be able to decide exactly what you want to measure (in this case write throughput), which tests to run, how to setup TrueNAS so that you are actually measuring the right thing, and how to analyse the results you get. So for example you need to know the detail that ZFS optimises zeros in order to know not to use zeros for your tests.

In this case, your tests still have the following issues:

  1. You have not provided enough detail to know whether you are doing synchronous writes or not. I suspect not because if you were doing synchronous writes, your speed would be orders of magnitude slower.

  2. According to the WD10EZEX specs, the internal bus is 150MB/s. According to benchmarks you should expect sequential writes (no seeks) to be c. 138MB/s per drive. With a 5x RAIDZ2, you should expect the throughput to be 3x this i.e. c. 400MB/s. From this you then need to allow for seeks which will reduce this number.

  3. The measurements given by dd is the data written which includes the overheads of updating the metadata blocks with each TXG. You really need to be using Linux or ZFS iostats to be measuring the write throughput because these will include these overheads.

  4. dd is single threaded - you really need to use fio with multiple threads to measure throughput. If you use fio then you need to choose the correct type of writes in order to avoid synchronous writes.

  5. I am guessing that the block size for writes over SMB will be c. the ethernet block size of 1500Bytes - lets say 1400bytes when you deduct overhead. A block size of 2GB is crazily unrealistic!!!

  6. Your nearest measurement at a block size of 2MB achieved 207MB/s which seems a little on the low side cf. 400MB/s, but when you take into account overheads and seeks this is in the right ball-park.

P.S. In TrueNAS/ZFS we don’t use hardware RAID terminology like RAID5 - but rather ZFS terminology like RAIDZ2. In any case, the hardware RAID equivalent to RAIDZ2 is RAID6.

1 Like
8

Turning now to network performance:

For jumbo frames to work you have to have these supported and configured on every single box in the chain of network boxes i.e. TrueNAS, your Arris Router, your workstation etc.

The variance in iperf speed is not necessarily related to your file transfer variances (because you would expect to see file transfers start fast whilst it has enough memory to cache the writes, and then slow down when it runs out of memory and has to write to disk before it can accept any more data over the network).

You should NOT (IMO) run iperf using a file - the entire point of iperf is that it is a pure network-subsystem test that doesn’t involve the disk-subsystem.

Whilst you have described how your server is connected to the switch (router) you haven’t described how your workstation is connected. (I assume that it is using wired Ethernet and not wifi because wifi will introduce a whole extra level of complication on network speed.)

First you need to check that each and every link in your network has a negotiated speed of 1Gb full duplex.

Then you need to run an extended iperf run and check your network stats to see a) what speeds are reported and b) how many errors / retries are happening.

1 Like
9

@sudo_apt-get_brain you have too many different things going on at once. My advise would be to separate your issues and address them one at a time. Start with your network. Your iPerf3 results are really bad but this is over wifi, correct? Before thinking about replacing your NIC in the server or anything else, take everything down to first principles. If possible remove both wifi and your wifi router from the equation and connect an ethernet client straight into your server. Remove any settings you’ve tinkered with for MTU etc. Re-run iPerf3. Hopefully you’ll see closer to 1Gbit and more consistent results.

After that, test and optimise your wifi, if this is where your clients will be.

1 Like
10

Finally, your system is somewhat unbalanced. Dual Xeons is quite a lot of CPU, and 12GB memory and only 5 HDDs is insufficient to be able to utilise that amount of CPU.

That said, if you are not running any apps, 12GB should be perfectly adequate to max out your HDD drives. Of course, more memory will help.

11

Thank you Protopia and Runge. As you can see, I still don’t know the right questions to be asking as I don’t know what I don’t know. I appreciate the time to re-explain the entry level stuff yet again. This is my first time benchmarking drives in this config, in this OS. I’ll take some time here to respond to the discussion topics, with some quick tests, and then I will come back later tonight or tomorrow with full test results. This ended up being pretty long, my apologies.

Testing Discussion:

I should have started here. Apparently I’ve forgotten the basics of testing.

The client (my main pc) is on WiFi 6 (AX200, 802.11ax). Haven’t yet run the ethernet cable across the house. I will switch to a hardwire connection to my laptop (Win10, RealTek 1GbE onboard) for further testing.

After some research, I do not intend to be forcing synchronous writes. The pool has Sync=STANDARD, and there is no SLOG. In my last round of dd testing I was using conv=fdatasync as an attempt to force non-cached writes. Admittedly, I don’t know if this is doing what I think it is, or if what I was trying to do is necessary. Going forward I will assume no sync, as I have a feeling these drives are not very effective in a synchronous write scenario without SSD SLOG backup.

Glad my original theoretical 300MB/s number is somewhat reasonable.

I will constrain testing to reasonable values. I need to better understand block size and how SMB works in general. A quick dd shown below (recognizing limitations to dd as mentioned), is indicating… inadequate performance:

truenas_admin@truenas[~]$ sudo dd if=/dev/random of=/mnt/pool0/ddfile bs=1000 count=1000000 conv=fdatasync
1000000+0 records in
1000000+0 records out
1000000000 bytes (1.0 GB, 954 MiB) copied, 17.6872 s, 56.5 MB/s
truenas_admin@truenas[~]$

The smaller the block, the seemingly worse my pool performs.

Networking Discussion

I would agree, however this is my workaround as I can’t figure out why iperf3 seems to ~randomly~ (lol) be unable to find /dev/urandom. It works right now with a bitrate of 129 Mbps. --bidir results are 45-110 Mbps, need to run longer tests.

I will be starting here in case this is self inflicted.

Hardware Discussion:

This is true. I am aware of the goofy hardware config, but this is a reuse/recycle build. If I was doing this from the ground up, I’d choose a high clock, low core CPU on modern architechture. However, this was $Free.99, and I haven’t had to spend anything on hardware to start learning. I definitely see that I am way over on compute, and way under on storage and RAM. I intend to be running apps eventually (photo/media server, music, VMs?, relocate PiHole to server, etc.), so I’ll probably be burning that bridge when I get to it.

12

As mentioned, fix your network first. Absolutely no point worrying about raw write performance on your ZFS array while your clients are connected via a 50-100Mbit link. Your server hardware should have no issues saturating a link 10-20x that.

1 Like
13

After many times around, I figured out that there was some issue with being connected to the 2.5Gbps port on the router. Additionally, iperf3 does not test the same as file speeds, showing near 1Gbps from server to client, and about 200 Mbps from client to server. Changing TSO had no effect. After reading other posts, this appears to be a discrepancy in window sizing, or other iperf3 test configuration settings.

With actual files, I’m averaging about 800 Mbps server to client, and about 700 Mbps from client to server. I am now seeing 100% utilization of RAM, so I think that is next upgrade to this machine.

See fio test below. I have not done enough research to know what exactly this is testing, so this is just for reference:

fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=rw --size=50g --io_size=1500g --blocksize=1500 --iodepth=16 --direct=1 --numjobs=16 --runtime=120 --group_reporting --output=/mnt/pool0/pool_SMB/fiotest.txt

Run status group 0 (all jobs):
   READ: bw=326MiB/s (341MB/s), 326MiB/s-326MiB/s (341MB/s-341MB/s), io=38.2GiB (41.0GB), run=120002-120002msec
  WRITE: bw=325MiB/s (341MB/s), 325MiB/s-325MiB/s (341MB/s-341MB/s), io=38.1GiB (41.0GB), run=120002-120002msec

Thanks for the help and direction. I believe I’m at a functional performance level with this machine.

14

Summary of solutions:

  1. Transfer speeds: Connection to 2.5Gb port on router was causing issues. Speeds stabilized when moved to 1Gb port.
  2. iperf3: It Just Worked, eventually. (I really didn’t change anything that seemed to have any effect)
  3. Drive Performance: Run more applicable tests and know what/how/why you’re testing.