Slow write speed over 10G link

TrueNAS General
1

Hi,

I have set up a TrueNAS server using an old Ryzen with a 1920X, 48GB RAM, 4 x 1TB NVMe disks configured as a VDEV 4 x 1 disk wide. I have created a NVMe-oF share and mounted this onto another identical Ryzen which runs as a Jenkins node. I have tested the network between the two machines and it tops out at 9.7GB/s. When i do a write i am getting around 103MB/s.

I feel like i`m missing something as I was hoping to get much faster write speeds.

Looking for some pointers on where to look. Thanks

2

Hey @jumpingkangaroo

You’ve validated the link between the systems (assuming you used iperf or similar) but you’ll also want to validate the local disk performance from the TrueNAS machine itself using something like dd or fio

sudo fio --filename=/mnt/yourpoolname/test.file --size=5GB --rw=write --bs=1M --ioengine=libaio --iodepth=128 --runtime=30 --numjobs=12 --time_based --group_reporting --name=throughput-test-job --eta-newline=1

This will create a new file at /mnt/yourpoolname/test.file of 5GB and blast 1MB writes into at full speed (or roughly that much, one thread per core at 64 depth should do) for 30s - that will tell you what your pools “peak sequential ingest” is.

Let it run and the output will look like below:

throughput-test-job: (groupid=0, jobs=8): err= 0: pid=2552219: Tue Oct  6 10:17:14 2026
  write: IOPS=2101, BW=2101MiB/s (2203MB/s)(61.6GiB/30012msec); 0 zone resets
    slat (usec): min=525, max=1055.9k, avg=3695.15, stdev=19706.76
    clat (usec): min=10, max=3009.5k, avg=239533.22, stdev=279858.46
     lat (msec): min=7, max=3010, avg=243.23, stdev=282.63
    clat percentiles (msec):
     |  1.00th=[   55],  5.00th=[   69], 10.00th=[   71], 20.00th=[   72],
     | 30.00th=[   75], 40.00th=[   94], 50.00th=[  121], 60.00th=[  180],
     | 70.00th=[  257], 80.00th=[  338], 90.00th=[  527], 95.00th=[  793],
     | 99.00th=[ 1519], 99.50th=[ 1569], 99.90th=[ 2165], 99.95th=[ 2433],
     | 99.99th=[ 2903]
   bw (  MiB/s): min=   23, max= 6981, per=100.00%, avg=2125.18, stdev=230.87, samples=465
   iops        : min=   18, max= 6980, avg=2122.85, stdev=230.91, samples=465
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%
  lat (msec)   : 10=0.01%, 20=0.01%, 50=0.41%, 100=42.33%, 250=26.01%
  lat (msec)   : 500=20.14%, 750=5.67%, 1000=1.84%, 2000=3.46%, >=2000=0.11%
  cpu          : usr=7.10%, sys=17.62%, ctx=298439, majf=0, minf=95
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,63066,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=2101MiB/s (2203MB/s), 2101MiB/s-2101MiB/s (2203MB/s-2203MB/s), io=61.6GiB (66.1GB), run=30012-30012msec

The line at the end is the total aggregate speed. Latencies are helpful too. But let’s make sure you’re getting solid results from the drives first before we go deeper down the rabbit hole :slight_smile:

3

You mean 9.7Gbps not GB/s. Multiplicity of 8x is a pretty huge difference. What OS? Are you using Linux NVMe-oF/TCP or are you using Starwind adapter for Windows workstation or Windows Server? Are you copying NVMe to NVMe (source/destination)? A DAC, direct cat5e cable, fiber, virtual, etc…Is the NIC on a pci-e 3.0 4x or faster? How are the NVMe’s mounted on the systems? Is the cable greater than 100ft? Are jumbo packets set to 9,000? What are you using to test the throughput (Crystal Mark, native disk tool, copy/paste)? What version of TruNAS is running? Does the interface negotiate at 10Gb?

Generally, if you run a test like Crystal Disk Mark, the test will repeat and you’ll be reading/writing directly to ARC, so it should transfer as fast as the interface allows.

Linux native test will vary a bit and you’ll see more direct read/writes to disk and not ARC.

4

Hi, Thank you for the reply.

10G network connection so these are the latest results. (Thanks for pointing out the multiplicity error)

Network speed is fine measured with iperf3

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.86 MBytes
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.86 MBytes
[  5]   2.00-3.00   sec  1.15 GBytes  9.91 Gbits/sec    0   1.96 MBytes
[  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.96 MBytes
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.96 MBytes
[  5]   5.00-6.00   sec  1.15 GBytes  9.91 Gbits/sec    0   1.96 MBytes
[  5]   6.00-7.00   sec  1.15 GBytes  9.89 Gbits/sec    0   2.54 MBytes
[  5]   7.00-8.00   sec  1.15 GBytes  9.91 Gbits/sec    0   2.54 MBytes
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec    0   2.54 MBytes
[  5]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec    0   2.54 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec                  receiver

fio speed test

fio-3.36
Starting 12 processes
Jobs: 12 (f=12): [W(12)][9.7%][eta 00m:28s]
Jobs: 12 (f=12): [W(12)][13.3%][w=935MiB/s][w=935 IOPS][eta 00m:26s]
Jobs: 12 (f=12): [W(12)][20.0%][w=923MiB/s][w=923 IOPS][eta 00m:24s]
Jobs: 12 (f=12): [W(12)][23.3%][w=922MiB/s][w=922 IOPS][eta 00m:23s]
Jobs: 12 (f=12): [W(12)][30.0%][w=898MiB/s][w=898 IOPS][eta 00m:21s]
Jobs: 12 (f=12): [W(12)][36.7%][w=908MiB/s][w=908 IOPS][eta 00m:19s]
Jobs: 12 (f=12): [W(12)][40.0%][w=891MiB/s][w=891 IOPS][eta 00m:18s]
Jobs: 12 (f=12): [W(12)][43.3%][w=894MiB/s][w=894 IOPS][eta 00m:17s]
Jobs: 12 (f=12): [W(12)][46.7%][w=891MiB/s][w=891 IOPS][eta 00m:16s]
Jobs: 12 (f=12): [W(12)][50.0%][w=928MiB/s][w=928 IOPS][eta 00m:15s]
Jobs: 12 (f=12): [W(12)][56.7%][w=934MiB/s][w=934 IOPS][eta 00m:13s]
Jobs: 12 (f=12): [W(12)][63.3%][w=897MiB/s][w=897 IOPS][eta 00m:11s]
Jobs: 12 (f=12): [W(12)][70.0%][w=875MiB/s][w=875 IOPS][eta 00m:09s]
Jobs: 12 (f=12): [W(12)][76.7%][w=907MiB/s][w=907 IOPS][eta 00m:07s]
Jobs: 12 (f=12): [W(12)][83.3%][w=933MiB/s][w=932 IOPS][eta 00m:05s]
Jobs: 12 (f=12): [W(12)][90.0%][w=891MiB/s][w=890 IOPS][eta 00m:03s]
Jobs: 12 (f=12): [W(12)][96.7%][w=774MiB/s][w=773 IOPS][eta 00m:01s]
Jobs: 12 (f=12): [W(12)][100.0%][w=798MiB/s][w=798 IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=12): err= 0: pid=161333: Thu Oct  8 09:38:50 2026
  write: IOPS=876, BW=876MiB/s (919MB/s)(25.7GiB/30010msec); 0 zone resets
    slat (usec): min=1066, max=43241, avg=13365.86, stdev=2613.46
    clat (usec): min=3, max=2031.0k, avg=1650389.84, stdev=229213.41
     lat (msec): min=5, max=2047, avg=1663.76, stdev=228.97
    clat percentiles (msec):
     |  1.00th=[  368],  5.00th=[ 1536], 10.00th=[ 1586], 20.00th=[ 1620],
     | 30.00th=[ 1636], 40.00th=[ 1653], 50.00th=[ 1670], 60.00th=[ 1687],
     | 70.00th=[ 1703], 80.00th=[ 1737], 90.00th=[ 1770], 95.00th=[ 1921],
     | 99.00th=[ 2005], 99.50th=[ 2005], 99.90th=[ 2022], 99.95th=[ 2022],
     | 99.99th=[ 2022]
   bw (  KiB/s): min=315392, max=1069232, per=100.00%, avg=909476.55, stdev=8642.12, samples=660
   iops        : min=  308, max= 1044, avg=888.07, stdev= 8.44, samples=660
  lat (usec)   : 4=0.01%, 10=0.02%, 20=0.02%, 50=0.01%
  lat (msec)   : 10=0.01%, 20=0.04%, 50=0.06%, 100=0.13%, 250=0.41%
  lat (msec)   : 500=0.71%, 750=0.77%, 1000=0.65%, 2000=96.36%, >=2000=0.81%
  cpu          : usr=0.21%, sys=95.97%, ctx=3139, majf=0, minf=328192
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.7%, 32=1.5%, >=64=97.1%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,26297,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
  WRITE: bw=876MiB/s (919MB/s), 876MiB/s-876MiB/s (919MB/s-919MB/s), io=25.7GiB (27.6GB), run=30010-30010msec

Disk stats (read/write):
  nvme1n1: ios=15/7812, sectors=30/3942535, merge=0/0, ticks=228/630702, in_queue=630930, util=43.54%

Slightly worried about the average write latency value of 1650ms.

Some more information on the system.

  • 4 disks SAMSUNG MZVLB1T0HALR-000L7
  • Connected via a ASUS HYPER M.2 X16 CARD V2 GEN4
  • Share is set up as a NVMe-oF Subsystems
  • TrueNAS 25.10.4 - Goldeye Community edition.
  • 5m Cat6e cables
  • mtu set to 9000
  • Ryzen 1920X
  • 32GB RAM

Hope this helps.

5

Your write speed looks fine for the mounted drive over 10G. 919 MB/s seems about right.

I think HoneyBadger was inquiring about the local disk writes on the NAS itself & not the mounted NVMe-oF drive, but your test shows the mounted drive is writing as expected over 10G.

What are you doing when you tested the drive write speeds for 103 MB/s?

6

You CPU usage for FIO seems really high for a 1920X. Are you doing encryption/dedupe/compression on the dataset? If so, what are the settings?

7

Seems like he’s running fio from the client machine that’s mounting NVMe-oF, but interesting if encryption is enabled on both the zvol and the client side.

8

Yes, running fio from the client to a mounted NVMe-oF share without encryption. I did some work over the weekend and have found a few interesting pieces to the puzzle, one area i failed to mention in the original post which would have been useful.

Running a test locally on the TrueNAS hit the write and read speed of the drives so i know the hardware is okay. Network speed is fine as that hits the limits with the iperf3 test. The client is running Oracle Cluster File System 2 as we wanted to machines to access the share but not the same folder at the same time. This was using a local heartbeat. If i removed the o2cb then i was getting decent write speeds.

We are now extracting a tar.gz file which contains a lot of files as the client is running an AOSP build but not getting speeds close to the local test. This maybe due to the number of files or something else, i`m not sure yet.

Thanks for your help.

9

Noticed you stated 48GB or RAM initially to 32GB of RAM. The volume of RAM might be a limiting issue for you as well.
For dealing with larger number of files, this is heavily reliant on iops, and you may not see the full utilization throughput of your network as this is highly reliant on parallel and operational actions.