Slow SMB sequential copies

TrueNAS General
1

I have two TrueNAS boxes and sequential copies (a single multi gigabyte file) to/from either of them hover around single gigabit speeds (120-130 MB/s at most), even though:

  • Both of them are connected via 10Gbps links
  • Clients use at least 2.5Gbit links
  • One uses an NVME SSD, the other one 8x non SMR HDDs (Toshiba MG08ACA16TE) in RaidZ2
  • One is bare metal, the other is virtualized in Proxmox but with PCIe passthrough for the NVME SSD and the CPU set to host
  • I can iperf from one box to the other and get stable ~9Gbps transfers

I tried turning SMB encryption off in the SMB service settings in TrueNAS but it makes no difference.

Not expecting to saturate the full 10 gig, but I would expect this kind of workload from either box to be at least twice as fast as this and saturate 2.5g links.

What should I attempt in order to further troubleshoot?

2

Is the 10Gbps card the only card in those boxes or do they also have a 1Gbps card?
If they also have a 1Gbps card did you make sure that the smb service uses the 10Gbps card?

3

The virtualized one has a single link at the very least. The other one has multiple physical NICs but only one is connected (there is a 1Gbps one but that is for the IPMI).

4

The first thing i would do is an iperf test to determine the link speed between the client and the hosts

5

Just tried your suggestion: Iperf shows a healthy 2.3Gbps, as expected of the link speed

6

Next step then should be a pool speedtest with either fio or something like

7

It doesn’t matter how many drives you have if you only have 1 VDEV, for the most part. ZFS scales with the number of VDEV’s not the number of drives. 1 VDEV basically restricts you to the performance of just one drive.

8

Ok, but a modern day HDD like the one I’m using should be able to do about 200MB/s sequential.

To say nothing of the SSD.

Also, I thought RaidZ pools had reads scale with the number of drives.

9

You are right, your speed looks like you’re getting bottlenecked to 1G link speed.

I will just quote Matthew Ahrens (core developer of ZFS):

For best performance on random IOPS, use a small number of disks in each RAID-Z group. E.g, 3-wide RAIDZ1, 6-wide RAIDZ2, or 9-wide RAIDZ3 (all of which use ⅓ of total storage for parity, in the ideal case of using large blocks). This is because RAID-Z spreads each logical block across all the devices (similar to RAID-3, in contrast with RAID-4/5/6). For even better performance, consider using mirroring.“

10

Just ran the benchmark you linked. Here’s the output:

############################################################
#                 tn-bench v2.2 (Modular)                  #
############################################################

tn-bench is an OpenSource Software Script that uses standard tools to
Benchmark your System and collect various statistical information via
the TrueNAS API.

* tn-bench will create a Dataset in each of your pools for testing purposes
* that will consume 20 GiB of space for every thread in your system.

! WARNING: This test will make your system EXTREMELY slow during its run.
! WARNING: It is recommended to run this test when no other workloads are running.

* ZFS ARC will impact your results. You can set zfs_arc_max to 1 to prevent ARC caching.
* Setting it back to 0 restores default behavior but requires a system restart.

============================================================
 Confirmation
============================================================

Would you like to continue? (yes/no): yes

------------------------------------------------------------
|                    System Information                    |
------------------------------------------------------------

Field                 | Value
----------------------+---------------------------------------
Version               | 25.10.3.1
Load Average (1m)     | 0.0546875
Load Average (5m)     | 0.0380859375
Load Average (15m)    | 0.00341796875
Model                 | Intel(R) Atom(TM) CPU C3758R @ 2.40GHz
Cores                 | 4
Physical Cores        | 4
System Product        | Standard PC (Q35 + ICH9, 2009)
Physical Memory (GiB) | 7.74

------------------------------------------------------------
|                     Pool Information                     |
------------------------------------------------------------

Field      | Value
-----------+-----------
Name       | tank0
Path       | /mnt/tank0
Status     | ONLINE
VDEV Count | 1
Disk Count | 0

VDEV Name  | Type           | Disk Count
-----------+----------------+---------------
9a221409-b080-4021-804a-d801498fa329 | DISK           | 0

------------------------------------------------------------
|                     Disk Information                     |
------------------------------------------------------------

* The TrueNAS API returns N/A for the Pool for boot devices and disks not in a pool.
Field      | Value
-----------+-----------------------
Name       | nvme0n1
Model      | KINGSTON SA2000M81000G
Serial     | 50026B7282816A99
ZFS GUID   | 2985107751441254243
Pool       | N/A
Size (GiB) | 931.51
-----------+-----------------------
Name       | sda
Model      | QEMU_HARDDISK
Serial     | drive-scsi0
ZFS GUID   | None
Pool       | N/A
Size (GiB) | 16.00
-----------+-----------------------

############################################################
#                      Pool Selection                      #
############################################################

* Available pools:
• 1. tank0
* Options:
• 1. Enter specific pool numbers (comma separated)
• 2. Type 'all' to test all pools
• 3. Type 'none' to skip pool testing

Enter your choice [all]: 1

############################################################
#              ZFS Pool Benchmark Iterations               #
############################################################

* How many times should we run each test?
• • Enter any positive integer (1-100, default: 2)
• • Enter 0 to skip this benchmark

Enter iteration count [2]:

############################################################
#                Pool Block Size Selection                 #
############################################################

* Select the block size for pool testing:
* This sets both the dd block size and dataset record size.

•  1. 16k  - Small blocks (metadata-heavy workloads)
•  2. 32k  - Small blocks
•  3. 64k  - Medium-small blocks
•  4. 128k - Medium blocks (general purpose)
•  5. 256k - Medium blocks
•  6. 512k - Medium-large blocks
•  7. 1M   - Large blocks (default, sequential throughput)
•  8. 2M   - Large blocks
•  9. 4M   - Very large blocks
• 10. 8M   - Very large blocks
• 11. 16M  - Maximum block size

* Smaller blocks test metadata-heavy workloads and IOPS.
* Larger blocks test sequential throughput.
* 1M is the default and matches prior tn-bench behavior.

Enter block size (1-11) [7]:

############################################################
#           Individual Disk Benchmark Iterations           #
############################################################

* How many times should we run each test?
• • Enter any positive integer (1-100, default: 2)
• • Enter 0 to skip this benchmark

Enter iteration count [2]:

############################################################
#                 Disk Test Mode Selection                 #
############################################################

* Select the disk benchmark test mode(s):

• 1. SERIAL - Test disks one at a time (baseline performance)
• 2. PARALLEL - Test all disks simultaneously (controller stress)
• 3. SEEK_STRESS - Multiple threads per disk (seek mechanism stress)

* You can select multiple modes by entering comma-separated numbers.
* Examples: '1' (serial only), '1,2' (serial then parallel), 'all' (all modes)

* SERIAL is recommended for baseline measurements.
* PARALLEL tests controller/chassis backplane throughput.
* SEEK_STRESS tests disk seek performance under heavy load.

! WARNING: ⚠️  RESOURCE WARNING:
! WARNING: PARALLEL mode will heavily load your storage controllers.
! WARNING: SEEK_STRESS uses multiple threads per disk and may cause:
! WARNING:   - High CPU usage (can saturate all cores)
! WARNING:   - System instability on heavily loaded systems
! WARNING:   - Significantly longer test durations
* For production systems, use SERIAL mode only.

Enter test mode(s) (1/2/3, comma-separated, or 'all') [1]:

############################################################
#                Disk Block Size Selection                 #
############################################################

* Select the block size for disk testing:

• 1. 4K (small random I/O)
• 2. 32K (medium I/O)
• 3. 128K (large sequential)
• 4. 1M (very large sequential)

* 4K tests small random I/O performance.
* 1M tests large sequential throughput.

Enter block size (1/2/3/4) [4]:

############################################################
#                  DD Benchmark Starting                   #
############################################################

* Using 4 threads for the benchmark.
* ZFS tests will run 2 time(s) per configuration
* Disk tests will run 2 time(s) per disk

############################################################
#                   Testing Pool: tank0                    #
############################################################

* Creating test dataset for pool: tank0 (recordsize=1M)
✓ Created temporary dataset: tank0/tn-bench
✓ Dataset tank0/tn-bench created successfully.

============================================================
 Space Verification
============================================================

* Available space: 887.64 GiB
* Space required:  80.00 GiB (20 GiB/thread × 4 threads)
* Test iterations: 2 (space freed between iterations)
✓ Sufficient space available - proceeding with benchmarks
* Starting zpool iostat collection for pool 'tank0' (interval: 1s)
* Warming up zpool iostat collector (3 samples)...
✓ Zpool iostat collector warmup complete
* No L2ARC on pool 'tank0' — L2ARC metrics will be omitted
* Starting arcstat collection (interval: 1s, fields: 18)
* Warming up arcstat collector (3 samples)...
✓ Arcstat collector warmup complete

============================================================
 Testing Pool: tank0 - Threads: 1
============================================================

* --- Iteration 1 of 2 ---
* Zpool iostat collector: benchmark phase started
* Zpool iostat collector: segment → 1T-write
* Arcstat collector: segment → 1T-write
* Iteration 1: Writing...
* Iteration 1 write: 149.35 MB/s
* Zpool iostat collector: segment → 1T-read
* Arcstat collector: segment → 1T-read
* Iteration 1: Reading...
* Iteration 1 read: 225.97 MB/s
* Space freed after iteration 1
* --- Iteration 2 of 2 ---
* Zpool iostat collector: segment → 1T-write
* Arcstat collector: segment → 1T-write
* Iteration 2: Writing...
* Iteration 2 write: 152.61 MB/s
* Zpool iostat collector: segment → 1T-read
* Arcstat collector: segment → 1T-read
* Iteration 2: Reading...
* Iteration 2 read: 226.92 MB/s
* Space freed after iteration 2

============================================================
 Testing Pool: tank0 - Threads: 1
============================================================

* --- Iteration 1 of 2 ---
* Zpool iostat collector: benchmark phase started
* Zpool iostat collector: segment → 1T-write
* Arcstat collector: segment → 1T-write
* Iteration 1: Writing...
* Iteration 1 write: 136.62 MB/s
* Zpool iostat collector: segment → 1T-read
* Arcstat collector: segment → 1T-read
* Iteration 1: Reading...
* Iteration 1 read: 225.53 MB/s
* Space freed after iteration 1
* --- Iteration 2 of 2 ---
* Zpool iostat collector: segment → 1T-write
* Arcstat collector: segment → 1T-write
* Iteration 2: Writing...
* Iteration 2 write: 151.01 MB/s
* Zpool iostat collector: segment → 1T-read
* Arcstat collector: segment → 1T-read
* Iteration 2: Reading...
* Iteration 2 read: 226.54 MB/s
* Space freed after iteration 2

============================================================
 Testing Pool: tank0 - Threads: 2
============================================================

* --- Iteration 1 of 2 ---
* Zpool iostat collector: segment → 2T-write
* Arcstat collector: segment → 2T-write
* Iteration 1: Writing...
* Iteration 1 write: 148.09 MB/s
* Zpool iostat collector: segment → 2T-read
* Arcstat collector: segment → 2T-read
* Iteration 1: Reading...
* Iteration 1 read: 217.16 MB/s
* Space freed after iteration 1
* --- Iteration 2 of 2 ---
* Zpool iostat collector: segment → 2T-write
* Arcstat collector: segment → 2T-write
* Iteration 2: Writing...
* Iteration 2 write: 154.49 MB/s
* Zpool iostat collector: segment → 2T-read
* Arcstat collector: segment → 2T-read
* Iteration 2: Reading...
* Iteration 2 read: 227.80 MB/s
* Space freed after iteration 2

============================================================
 Testing Pool: tank0 - Threads: 4
============================================================

* --- Iteration 1 of 2 ---
* Zpool iostat collector: segment → 4T-write
* Arcstat collector: segment → 4T-write
* Iteration 1: Writing...
* Iteration 1 write: 145.94 MB/s
* Zpool iostat collector: segment → 4T-read
* Arcstat collector: segment → 4T-read
* Iteration 1: Reading...
* Iteration 1 read: 227.66 MB/s
* Space freed after iteration 1
* --- Iteration 2 of 2 ---
* Zpool iostat collector: segment → 4T-write
* Arcstat collector: segment → 4T-write
* Iteration 2: Writing...
* Iteration 2 write: 146.72 MB/s
* Zpool iostat collector: segment → 4T-read
* Arcstat collector: segment → 4T-read
* Iteration 2: Reading...
* Iteration 2 read: 224.36 MB/s
* Space freed after iteration 2
* Zpool iostat collector: benchmark phase ended
* Cooling down zpool iostat collector (3 samples)...
✓ Zpool iostat collector cooldown complete
✓ Zpool iostat collection complete: 3629 samples
* Cooling down arcstat collector (3 samples)...
✓ Arcstat collector cooldown complete
✓ Arcstat collection complete: 3624 samples

============================================================
 Zpool Iostat Telemetry Summary for Pool: tank0
============================================================


╔══════════════════════════════════════════════════════════╗
║       Zpool Iostat Telemetry Summary for Pool: tank0     ║
╚══════════════════════════════════════════════════════════╝
  • Total samples: 3629  |  Steady-state samples: 0
  • Duration: 3681.95 seconds

────────── Per-Thread-Count Steady-State Analysis ──────────
  WRITE telemetry only (READ excluded due to ZFS ARC cache interference)

  1 Threads (550 samples):
  ┌─ IOPS ────────────────────────────────────────────────────
  │ Mean: 203.3  │ Median: 221.0  │
  ├──────────────────────────────────────────────────────────┤
  │ P99: 344.1 [High] │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 66.5 [Excellent] │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 32.7% High Variance │
  └──────────────────────────────────────────────────────────┘
  ┌─ Bandwidth (MB/s) ────────────────────────────────────────
  │ Mean: 146.3  │ Median: 153.0  │
  ├──────────────────────────────────────────────────────────┤
  │ P99: 278.2 [High] │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 47.8 [Excellent] │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 32.7% High Variance │
  └──────────────────────────────────────────────────────────┘
  ┌─ Latency (ms) ────────────────────────────────────────────
  │ Mean: 56.9  │ Median: 3.0  │
  ├──────────────────────────────────────────────────────────┤
  │ P99: 772.1 [High] │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 183.6 [High Variance] │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 322.9% High Variance │
  └──────────────────────────────────────────────────────────┘
  2 Threads (528 samples):
  ┌─ IOPS ────────────────────────────────────────────────────
  │ Mean: 198.8  │ Median: 241.0  │
  ├──────────────────────────────────────────────────────────┤
  │ P99: 415.4 [High] │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 104.1 [Good] │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 52.4% High Variance │
  └──────────────────────────────────────────────────────────┘
  ┌─ Bandwidth (MB/s) ────────────────────────────────────────
  │ Mean: 149.1  │ Median: 179.5  │
  ├──────────────────────────────────────────────────────────┤
  │ P99: 381.1 [High] │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 73.5 [Excellent] │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 49.3% High Variance │
  └──────────────────────────────────────────────────────────┘
  ┌─ Latency (ms) ────────────────────────────────────────────
  │ Mean: 123.4  │ Median: 3.0  │
  ├──────────────────────────────────────────────────────────┤
  │ P99: 738.0 [High] │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 217.4 [High Variance] │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 176.2% High Variance │
  └──────────────────────────────────────────────────────────┘
  4 Threads (1095 samples):
  ┌─ IOPS ────────────────────────────────────────────────────
  │ Mean: 194.0  │ Median: 214.0  │
  ├──────────────────────────────────────────────────────────┤
  │ P99: 433.0 [High] │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 115.0 [Good] │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 59.3% High Variance │
  └──────────────────────────────────────────────────────────┘
  ┌─ Bandwidth (MB/s) ────────────────────────────────────────
  │ Mean: 144.8  │ Median: 185.0  │
  ├──────────────────────────────────────────────────────────┤
  │ P99: 353.1 [High] │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 78.8 [Excellent] │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 54.4% High Variance │
  └──────────────────────────────────────────────────────────┘
  ┌─ Latency (ms) ────────────────────────────────────────────
  │ Mean: 160.5  │ Median: 3.0  │
  ├──────────────────────────────────────────────────────────┤
  │ P99: 2,000.0 [High] │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 292.0 [High Variance] │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 182.0% High Variance │
  └──────────────────────────────────────────────────────────┘

────────────────────────── Legend ──────────────────────────
  Statistical Measures:
    • Mean:    Average of all samples
    • Median:  Middle value (50th percentile), less affected by outliers
    • P99:     99th percentile - 99% of samples fall below this value
    • Std Dev: Standard deviation - measures spread/consistency
    • CV%:     Coefficient of Variation (std dev / mean × 100)

  CV% Rating (Consistency):
    • Excellent:    < 10%  (highly consistent)
    • Good:         10-20% (good consistency)
    • Variable:     20-30% (some variability)
    • High Variance:  > 30%  (significant inconsistency)

  P99 Latency Rating (Lower is better):
    • Excellent:    < 10ms   (very fast)
    • Good:         < 50ms   (acceptable)
    • Acceptable:  < 100ms  (may impact workload)
    • High:          > 100ms  (significant latency)

  Std Dev Rating (Consistency - Lower is better):
    • Excellent:    Low spread    (very consistent)
    • Good:         Moderate      (acceptable spread)
    • Variable:     Noticeable    (some spread)
    • High Variance:  Wide spread   (inconsistent)

============================================================
 ARC Statistics Summary (READ Phase) for Pool: tank0
============================================================


╔══════════════════════════════════════════════════════════╗
║    ARC Statistics Summary (READ Phase) for Pool: tank0   ║
╚══════════════════════════════════════════════════════════╝
  • Total samples: 3624  |  Read-phase samples: 1456
  • Duration: 3682.30 seconds
  • L2ARC: not present (L2ARC metrics omitted)

──────────── Per-Thread-Count READ ARC Analysis ────────────
  ARC cache performance during READ benchmark phases

  1 Threads (361 samples):
  ┌─ ARC Hit % ───────────────────────────────────────────────
  │ Mean: 60.5% [Poor] │
  ├──────────────────────────────────────────────────────────┤
  │ Median: 58.6%  │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 8.0 [Variable] │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 13.3%  Good │
  └──────────────────────────────────────────────────────────┘
  ┌─ ARC Size (GiB) ──────────────────────────────────────────
  │ Mean: 3.14  │ Median: 3.44  │
  ├──────────────────────────────────────────────────────────┤
  │ P99: 4.24               │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 0.89               │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 28.4% Variable │
  └──────────────────────────────────────────────────────────┘
  2 Threads (367 samples):
  ┌─ ARC Hit % ───────────────────────────────────────────────
  │ Mean: 58.0% [Poor] │
  ├──────────────────────────────────────────────────────────┤
  │ Median: 56.8%  │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 6.1 [Variable] │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 10.5%  Good │
  └──────────────────────────────────────────────────────────┘
  ┌─ ARC Size (GiB) ──────────────────────────────────────────
  │ Mean: 3.49  │ Median: 3.65  │
  ├──────────────────────────────────────────────────────────┤
  │ P99: 5.33               │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 0.70               │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 20.1% Variable │
  └──────────────────────────────────────────────────────────┘
  4 Threads (728 samples):
  ┌─ ARC Hit % ───────────────────────────────────────────────
  │ Mean: 56.7% [Poor] │
  ├──────────────────────────────────────────────────────────┤
  │ Median: 55.3%  │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 8.0 [Variable] │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 14.0%  Good │
  └──────────────────────────────────────────────────────────┘
  ┌─ ARC Size (GiB) ──────────────────────────────────────────
  │ Mean: 3.31  │ Median: 3.41  │
  ├──────────────────────────────────────────────────────────┤
  │ P99: 3.70               │
  ├──────────────────────────────────────────────────────────┤
  │ Std Dev: 0.50               │
  ├──────────────────────────────────────────────────────────┤
  │ CV%: 15.2%  Good │
  └──────────────────────────────────────────────────────────┘

──────────────────────── ARC Legend ────────────────────────
  ARC Metrics:
    • ARC Hit %:       Percentage of reads served from ARC (higher = better)
    • ARC Size (GiB):  Total ARC memory usage

  ARC Hit % Rating:
    • Excellent:    ≥ 95%  (nearly all reads from cache)
    • Good:         85-95% (majority cached)
    • Variable:     70-85% (moderate caching)
    • Poor:          < 70%  (frequent cache misses)


============================================================
 Pool Write Summary
============================================================

* Total data written: 320.00 GiB
* Pool capacity: 928.00 GiB
* Benchmark duration: 3689.67 seconds
* Drive Writes Per Day (DWPD): 8.07
* Cleaning up any remaining test files...

############################################################
#               Disk Benchmark Configuration               #
############################################################

* Test mode: serial
* Block size: 1M (very large sequential)

############################################################
#                  Serial Disk Benchmark                   #
############################################################

* Testing disks one at a time for individual performance baseline
* Block size: 1M (very large sequential)

============================================================
 Testing Disk: nvme0n1
============================================================

* Read size: 7.74 GiB
* Model: KINGSTON SA2000M81000G
* Run 1 of 2...
* Run 1: 829.62 MiB/s
* Run 2 of 2...
* Run 2: 982.27 MiB/s
✓ Average: 905.95 MiB/s

============================================================
 Testing Disk: sda
============================================================

* Read size: 7.74 GiB
* Model: QEMU_HARDDISK
* Run 1 of 2...
* Run 1: 531.08 MiB/s
* Run 2 of 2...
* Run 2: 538.60 MiB/s
✓ Average: 534.84 MiB/s

############################################################
#           Disk Benchmark Results - SERIAL Mode           #
############################################################


------------------------------------------------------------
|                       Mode: SERIAL                       |
------------------------------------------------------------

• Disk: nvme0n1
• Model: KINGSTON SA2000M81000G
•   Run 1: 829.62 MiB/s
•   Run 2: 982.27 MiB/s
•   Average: 905.95 MiB/s

• Disk: sda
• Model: QEMU_HARDDISK
•   Run 1: 531.08 MiB/s
•   Run 2: 538.60 MiB/s
•   Average: 534.84 MiB/s


------------------------------------------------------------
|                    Overall Statistics                    |
------------------------------------------------------------

* Disks tested: 2
* Average speed: 720.39 MiB/s
* Speed range: 534.84 - 905.95 MiB/s

############################################################
#                    Benchmark Complete                    #
############################################################

✓ Total benchmark time: 63.57 minutes

Delete testing dataset tank0/tn-bench? (yes/no): yes
* Deleting dataset: tank0/tn-bench
! WARNING: Dataset not fully deleted. Performing diagnostics...
* No processes found using lsof
Force delete dataset? (yes/no) [no]: yes
* Attempting force deletion...
! ERROR: Force deletion failed. Dataset still exists.
✓ Dataset tank0/tn-bench deleted.
✓ Benchmark results saved to: /root/tn-bench/tn_bench_results.json

############################################################
#                        Analytics                         #
############################################################

* Running post-benchmark analytics...
✓ Analytics data saved to: /root/tn-bench/tn_bench_results_analytics.json
✓ Analytics report saved to: /root/tn-bench/tn_bench_results_report.md

============================================================
 Analytics Summary
============================================================

* Pool: tank0
• Write peak: 151.3 MB/s @ 2 threads
• Read peak: 226.4 MB/s @ 1 threads
11

Don’t know what to tell you except that you’re running a pretty low end system with an intel atom and only 8gb of ram, so i would say that your results are expected for the hardware you’re using…

12

The number of drives does matter as does the raid configuration. For example, my 6 wide raidz2 will achieve 540 MB/s sustained transfers while my 10 wide raidz2 will achieve 920 MB/s sustained transfer rates with identical hdds. Number of vdevs limits I/O and not sustained transfer, so if you are trying to achieve higher I/O’s, you’ll want more vdevs.

13

You are on something here as TCP network overhead (SMB) is dependent on speed of single thread, but his iperf speeds should be close to his sequential transfer rates and any repeat transfers will be cached in arc which will transfer as fast as the network interface will allow. What virtualization is being used? Proxmox, XCP-ng, VMware, etc? The hold up could be at the vitualization layer. Network packet inspections will also have a massive impact on network throughput.

14

I would try a direct connect to each NAS to test your speeds. Make sure you use a good cable. Short distances of 10ft or so won’t matter for 10Gb using CAT5e but anything beyond you’ll want to use CAT6a or better.

15

Here’s a quick breakdown of the setup:

NAS 1 (1x NVME SSD stripe):

  • VM In Proxmox with NVME drive passed through and VIRTIO NIC
  • Host is a Qotom Q20331G9-S10
  • Connected to switch via DAC cable form one of the SFP ports
  • Using Proxmox’s SDN network stack (VM is attached to a VLAN with Proxmox doing the 802.1q tagging). Port is set as trunk with multiple VLANs on switch.

NAS 2 (8x 16TB HDD RaidZ2):

Both NASes are in the same VLAN and connected to the same SKS8300-8X switch (flashed to run OpenWRT)