Long time TrueNAS user, it’s been working great for years on other machines. I’m upgrading our primary video storage server and based on a bunch of reading here in and in other forms decided to try out a Dell R740XD in 24xNVME configuration which was a pretty good deal. The specs and configuration are below.
I’m well aware of the PCIe lane limitations on the R740XD and similar 14th gen platforms which dedicate 32 lanes to the 24xNVME drives resulting in 1.33 PCIe Gen 3 lanes per drive which should be about 1.33GB/s, significantly lower than what the drives themselves are capable of (~3GB/s). Even worst case on a single drive, this should still be faster than a traditional HDD array…. Right? Not so far. Happy to be the lab monkey and document the definitive guide to making this type of project work for the next guy so bring your ideas, tweaks and tests please.
Project expectations & success criteria:
Need ~40TB of raw storage so can use 2 drives for parity (2xRAIDz1 or 1xRAIDz2, data will be nightly backed up).
Data will mostly consist of large compressed video files (1GB-80GB, avg is probably in the 15GB range).
Would expect ~2-2.5GB/s (2x PCIe limitation) sustained write speeds over SMB or from another server via 100Gbps (replication/ssh/netcat).
Would expect significantly higher read speeds given the number of drives, probably capable of saturating the 100Gbps link.
Secondary use case, the machine will have 2-3 VMs/apps hosted focused on services, not much data activity.
Initial configurations:
BIOS configured per Dell recommendations for performance & low latency.
Configure TrueNAS Scale Kernel parameters to optimize for 100Gbps as recommended by Level1Techs.
Configure 24xNVMe drives as 2xRAIDz1 vdevs, RS=128k, compression off, sync=standard (have seen suggestions on larger record size, tested below)
Configure zfs primarycache=metadata per forum recommendations on pools using NVMe
Initial test results:
SMB test (128k RS):
Create new dataset (RS=128k, compression off, zfs primarycache=metadata)
WRITE: Copy 75GB .mkv video file to Darkstar from Workstation (Windows rock solid 1.7GB/s—this is ok)
READ: Copy 75GB .mkv video file from Darkstar to Workstation (Windows rock solid at 1.3GB/s—this is sad)
SMB test (1m RS, a little better read performance):
Create new dataset (RS=1m, compression off, zfs primarycache=metadata)
WRITE: Copy 75GB .mkv video file to Darkstar from Workstation – Windows rock solid 1.6GB/s—no change with record size
READ: Copy 75GB .mkv video file from Darkstar to Workstation – Windows rock solid at 1.84GB/s—better……, probably at the limit of 2x10Gbps)
READ: Use dd to copy 19GB .mkv video file from test dataset to ramdisk (dd status=progress if=/mnt/dspool/1mtest/test.mkv of=/mnt/ramfs/test.mkv bs=1m) – shows 1.3GB/s… that seems very slow……
WRITE: Use dd to copy 19GB .mkv video file from ramdisk to test dataset (dd status=progress of=/mnt/dspool/1mtest/test.mkv if=/mnt/ramfs/test.mkv bs=1m) – shows 2.4GB/s… faster writes than reads? WTF?
If someone knows how to properly use fio, happy to test your suggested commands and report back. I’ve tried a bunch of different tests from various threads which result in wildly varying numbers that I’m not confident are representative at all.
So…… any ideas? The write speeds seem ok, not spectacular but ok given the PCIe limitations. The read speeds are terrible. For the level of performance this is currently providing, we might as well go back to spinning disk which I find shocking. I know there are similar systems in production out there, I’d be really surprised if this is the top performance it can provide.
Appreciate the ideas and support!
Thanks,
For reference, the current server is an R730XD “Apollo” with 12x8TB drives that manages real world SMB (multi-channel on) performance of ~1.2GB/s read and write speeds which seems great given the spinning disks.
SMB test (128k RS):
Create new dataset (RS=128k, compression off, zfs primarycache=metadata)
Copy 75GB .mkv video file to Apollo from Workstation (Windows ranges around 900MBs-1.3GB/s)
Copy 75GB .mkv video file from Apollo to Workstation (Windows rock solid at 1.12GB/s)
Use dd to copy 19GB .mkv video file from test dataset to ramdisk (dd status=progress if=/mnt/STORAGE96TB/test/test.mkv of=/mnt/ramfs/test.mkv bs=128k) – shows 820MB/s… that seems slow……
Use dd to copy 19GB .mkv video file from ramdisk to test dataset (dd status=progress of=/mnt/STORAGE96TB/test/test.mkv if=/mnt/ramfs/test.mkv bs=128k) – shows 1.2GB/s which syncs with SMB performance
Hardware:
R740XD “Darkstar”
Built using the official Dell parts for 24xNVMe support
2x Xeon Gold 6140s 18C/36T @ 2.3Ghz
256GB Ram
100Gbps ConnectX-4 NIC
24x Intel DC P4510 2TB drives
2x 120GB SATA SSDs in BOSS card for boot
Everything on the latest firmware
Running TrueNAS Scale 24.10.2.3
R730XD “Apollo”
2x Xeon E5-2660 v4 14C/28T @ 2.0Ghz
128GB Ram
100Gbps ConnectX-4 NIC
8x Seagate Enterprise 8TB SATA 6Gb/s via Dell H200 HBA in IT mode
2x Samsung 980 Pro 500GB NVMe SSDs (VM storage)
2x 120GB SATA SSDs for boot
Running TrueNAS Scale 24.10.2.3
Workstation
Intel Core i9-9900KF 8C/16T @ 4.97Ghz (OC’d)
64GB Ram
2x 10GB Intel X520 NIC
1x Samsung 960 Pro 512GB (boot)
1x Corsair Force MP600 1TB (storage)
Nvidia 2080 Ti
Windows 10 Enterprise
Networking
Dell S4148FE-ON 48x 10GB + 2x 25GB + 4x 100GB
All machines directly connected via DACs
Verified network is good with iperf3 (multiple instances needed for 100Gbps but get ~98Gbps between servers, 9.8Gbps between Workstation & servers)
Not sure what the difference between ok and sad is, TBH.
What speed do you expect from a 10Gbps card? Note the lowercase b. FWIW, SMB protocol can be bottlenecked by the CPU, because samba is single-core (per connection).
Also, what is your CPU utilisation during the copying? Do you use encryption?
It can be the case that the CPU isn’t fast enough to calculate such amount of parity data and checksums. You can try to test mirrored VDEVs (eliminating the need for parity).
Kinda normal for sequential writes on SSDs. You can write to any cell, but you must read from particular cells.
You didn’t mention multichannel. Even though (from my experience) multichannel would work ok when not properly configured, I would not use it for tests.
Smoking guns.
Regarding the fio. I’m not very familiar with it and used it only once. Take all the considerations below with a pinch of salt.
You can create a test dataset fio with params you wanna test. Then run something like:
This will create test files and run sequential read test. You can read more about options here. The important ones (IMO) are rw, numjobs, and iodepth.
rw=read is for sequential reads. Numjobs can represent the number of simultaneous users – if you plan to use it solo, then it should be 1. The correct iodepth value depends on the application. I suspect that it is also 1 for the SMB sequential R/W. Change the values respectively. Change both to 32 to see the values from the manufacturer’s ad.
I’m not sure about sequential QD1 performance, but random QD1 performance of SSDs can be very different from QD64. Also, the sequential performance of SSDs is not that good (considering the price). Buying SSDs for mostly sequential workloads is not the best idea.
Do not forget to delete fio dataset after the tests.
I think this might be more related to dd being a single threaded. If the parity or array was CPU limited, I would expect to see pegged single cores in the SMB tests as well.
Did the same write test with rsync from ramdisk to the nvme pool and while rsync is slower (800MB/s), the CPU load is 4% with 2 cores at about 70%.
This particular test results in a 11.5GB/s read. If I increase the number of jobs to 8, it hits 16.2GB/s. Those are more like the numbers that I’d expect to see, I just can’t get anywhere close to that in real world.
Same test but sequential write (4 jobs) is 9GB/s with 2 cores in the 80% range, briefly spiking to 100% for about 2 seconds before dropping again. Overall load 43%. This seems suspiciously high given the PCIe lane limitation and in theory only 2x write advantage on this pool…
Sure, fair enough. If I re-run with only 1 job, read and write are both 3GB/s. That single thread is running a CPU load of 89%. I’d be stoked with 3GB/s real world write and something north of that in read.
1.3GB/s is very close to 10Gbps. It’s even so close that I would suspect your multichannel doesn’t work. Just to rule out this possible cause, you can borrow the 100Gbps card from your second server, install it into the workstation, and perform the test again. Better yet, make a direct connection between windows and truenas. Thus ruling out other possible network-related causes.
Wowwwww, wish I knew about TN-Bench earlier, 10/10 work my friend. Results below. Individual disk tests show the PCIe 1GB/s limitation. Read & write results also look about as expected. Do you see any issues? Currently only running 4x channel memory per CPU, could go the full 6x for 384GB total.
############################################################
# TN-Bench v1.10 #
############################################################
TN-Bench is an OpenSource Software Script that uses standard tools to
Benchmark your System and collect various statistical information via
the TrueNAS API.
* TN-Bench will create a Dataset in each of your pools for testing purposes
* that will consume 20 GiB of space for every thread in your system.
! WARNING: This test will make your system EXTREMELY slow during its run.
! WARNING: It is recommended to run this test when no other workloads are running.
* ZFS ARC will impact your results. You can set zfs_arc_max to 1 to prevent ARC caching.
* Setting it back to 0 restores default behavior but requires a system restart.
============================================================
Confirmation
============================================================
Would you like to continue? (yes/no): yes
------------------------------------------------------------
| System Information |
------------------------------------------------------------
Field | Value
----------------------+-----------------------------------------
Version | TrueNAS-SCALE-24.10.2.3
Load Average (1m) | 0.25439453125
Load Average (5m) | 0.3037109375
Load Average (15m) | 0.2958984375
Model | Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
Cores | 72
Physical Cores | 36
System Product | PowerEdge R740xd
Physical Memory (GiB) | 251.51
------------------------------------------------------------
| Pool Information |
------------------------------------------------------------
Field | Value
-----------+-----------------
Name | samsungtest
Path | /mnt/samsungtest
Status | OFFLINE
VDEV Count | 0
Disk Count | 0
VDEV Name | Type | Disk Count
-----------+----------------+---------------
------------------------------------------------------------
| Pool Information |
------------------------------------------------------------
Field | Value
-----------+------------
Name | dspool
Path | /mnt/dspool
Status | ONLINE
VDEV Count | 2
Disk Count | 24
VDEV Name | Type | Disk Count
-----------+----------------+---------------
raidz1-0 | RAIDZ1 | 12
raidz1-1 | RAIDZ1 | 12
------------------------------------------------------------
| Disk Information |
------------------------------------------------------------
* The TrueNAS API returns N/A for the Pool for boot devices and disks not in a pool.
Field | Value
-----------+--------------------
Name | sda
Model | DELLBOSS_VD
Serial | ba8d9f70e1100010
ZFS GUID | None
Pool | N/A
Size (GiB) | 119.18
-----------+--------------------
Name | nvme5n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ916401982P0BGN
ZFS GUID | 5628909457530630758
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme7n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ9183001B2P0BGN
ZFS GUID | 3814524054512588667
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme1n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ918000LP2P0BGN
ZFS GUID | 2194625387536678897
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme3n1
Model | INTEL SSDPE2KX020T8
Serial | BTLJ914503AD2P0BGN
ZFS GUID | 3466362619159729249
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme9n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ917600K12P0BGN
ZFS GUID | 15220527152320368478
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme11n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ918000DS2P0BGN
ZFS GUID | 1050510391378178878
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme13n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ918000HQ2P0BGN
ZFS GUID | 7534853165912083445
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme15n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ918100VX2P0BGN
ZFS GUID | 1525451702046694273
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme17n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ918600HH2P0BGN
ZFS GUID | 10029036033193565748
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme19n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ914400UN2P0BGN
ZFS GUID | 13956023050041575681
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme21n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ917100Y82P0BGN
ZFS GUID | 15958474702000390886
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme23n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ914500052P0BGN
ZFS GUID | 10066802367696162294
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme0n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ917100Y52P0BGN
ZFS GUID | 17754276955237960209
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme2n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ916500JF2P0BGN
ZFS GUID | 10891426602178177056
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme4n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ918300M12P0BGN
ZFS GUID | 15866509507358187499
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme6n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ918600HP2P0BGN
ZFS GUID | 4638906162398781126
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme8n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ916500FS2P0BGN
ZFS GUID | 13339094842106156350
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme10n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ917000AP2P0BGN
ZFS GUID | 424422250124590131
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme12n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ917000MX2P0BGN
ZFS GUID | 8601114335767534927
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme14n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ9145001V2P0BGN
ZFS GUID | 6658900461204691257
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme16n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ914300Z82P0BGN
ZFS GUID | 10174431092960666542
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme18n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ919200CF2P0BGN
ZFS GUID | 8969679944657584878
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme20n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ9182000X2P0BGN
ZFS GUID | 8266252572902876577
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
Name | nvme22n1
Model | INTEL SSDPE2KX020T8
Serial | PHLJ917500S22P0BGN
ZFS GUID | 1381494103913799328
Pool | dspool
Size (GiB) | 1863.02
-----------+--------------------
############################################################
# Pool Selection #
############################################################
* Available pools:
• 1. samsungtest
• 2. dspool
* Options:
• 1. Enter specific pool numbers (comma separated)
• 2. Type 'all' to test all pools
• 3. Type 'none' to skip pool testing
Enter your choice [all]: 2
############################################################
# ZFS Pool Benchmark Iterations #
############################################################
* How many times should we run each test?
• 1. Run each test once (faster)
• 2. Run each test twice (default, more accurate)
Enter iteration count (1 or 2) [2]: 1
############################################################
# Individual Disk Benchmark #
############################################################
Run individual disk read benchmark? (yes/no) [yes]: y
############################################################
# Individual Disk Benchmark Iterations #
############################################################
* How many times should we run each test?
• 1. Run each test once (faster)
• 2. Run each test twice (default, more accurate)
Enter iteration count (1 or 2) [2]: 1
############################################################
# DD Benchmark Starting #
############################################################
* Using 72 threads for the benchmark.
* ZFS tests will run 1 time(s) per configuration
* Disk tests will run 1 time(s) per disk
############################################################
# Testing Pool: dspool #
############################################################
* Creating test dataset for pool: dspool
✓ Created temporary dataset: dspool/tn-bench
✓ Dataset dspool/tn-bench created successfully.
============================================================
Space Verification
============================================================
* Available space: 39343.45 GiB
* Space required: 1440.00 GiB (20 GiB/thread × 72 threads)
✓ Sufficient space available - proceeding with benchmarks
============================================================
Testing Pool: dspool - Threads: 1
============================================================
* Running DD write benchmark with 1 threads...
* Run 1 write speed: 268.03 MB/s
✓ Average write speed: 268.03 MB/s
* Running DD read benchmark with 1 threads...
* Run 1 read speed: 6671.78 MB/s
✓ Average read speed: 6671.78 MB/s
============================================================
Testing Pool: dspool - Threads: 18
============================================================
* Running DD write benchmark with 18 threads...
* Run 1 write speed: 4182.46 MB/s
✓ Average write speed: 4182.46 MB/s
* Running DD read benchmark with 18 threads...
* Run 1 read speed: 24989.48 MB/s
✓ Average read speed: 24989.48 MB/s
============================================================
Testing Pool: dspool - Threads: 36
============================================================
* Running DD write benchmark with 36 threads...
* Run 1 write speed: 6563.14 MB/s
✓ Average write speed: 6563.14 MB/s
* Running DD read benchmark with 36 threads...
* Run 1 read speed: 20132.82 MB/s
✓ Average read speed: 20132.82 MB/s
============================================================
Testing Pool: dspool - Threads: 72
============================================================
* Running DD write benchmark with 72 threads...
* Run 1 write speed: 9369.59 MB/s
✓ Average write speed: 9369.59 MB/s
* Running DD read benchmark with 72 threads...
* Run 1 read speed: 22761.50 MB/s
✓ Average read speed: 22761.50 MB/s
############################################################
# DD Benchmark Results for Pool: dspool #
############################################################
------------------------------------------------------------
| Threads: 1 |
------------------------------------------------------------
• 1M Seq Write Run 1: 268.03 MB/s
• 1M Seq Write Avg: 268.03 MB/s
• 1M Seq Read Run 1: 6671.78 MB/s
• 1M Seq Read Avg: 6671.78 MB/s
------------------------------------------------------------
| Threads: 18 |
------------------------------------------------------------
• 1M Seq Write Run 1: 4182.46 MB/s
• 1M Seq Write Avg: 4182.46 MB/s
• 1M Seq Read Run 1: 24989.48 MB/s
• 1M Seq Read Avg: 24989.48 MB/s
------------------------------------------------------------
| Threads: 36 |
------------------------------------------------------------
• 1M Seq Write Run 1: 6563.14 MB/s
• 1M Seq Write Avg: 6563.14 MB/s
• 1M Seq Read Run 1: 20132.82 MB/s
• 1M Seq Read Avg: 20132.82 MB/s
------------------------------------------------------------
| Threads: 72 |
------------------------------------------------------------
• 1M Seq Write Run 1: 9369.59 MB/s
• 1M Seq Write Avg: 9369.59 MB/s
• 1M Seq Read Run 1: 22761.50 MB/s
• 1M Seq Read Avg: 22761.50 MB/s
* Cleaning up test files...
############################################################
# Disk Read Benchmark #
############################################################
* This benchmark tests the 4K sequential read performance of each disk
* To work around ARC caching, reads data = min(system RAM, disk size)
============================================================
Testing Disk: sda
============================================================
* Testing disk: sda
* Run 1: 472.20 MB/s
✓ Average: 472.20 MB/s
============================================================
Testing Disk: nvme5n1
============================================================
* Testing disk: nvme5n1
* Run 1: 871.13 MB/s
✓ Average: 871.13 MB/s
============================================================
Testing Disk: nvme7n1
============================================================
* Testing disk: nvme7n1
* Run 1: 1062.68 MB/s
✓ Average: 1062.68 MB/s
============================================================
Testing Disk: nvme1n1
============================================================
* Testing disk: nvme1n1
* Run 1: 1072.47 MB/s
✓ Average: 1072.47 MB/s
============================================================
Testing Disk: nvme3n1
============================================================
* Testing disk: nvme3n1
* Run 1: 1070.45 MB/s
✓ Average: 1070.45 MB/s
============================================================
Testing Disk: nvme9n1
============================================================
* Testing disk: nvme9n1
* Run 1: 1080.92 MB/s
✓ Average: 1080.92 MB/s
============================================================
Testing Disk: nvme11n1
============================================================
* Testing disk: nvme11n1
* Run 1: 1095.67 MB/s
✓ Average: 1095.67 MB/s
============================================================
Testing Disk: nvme13n1
============================================================
* Testing disk: nvme13n1
* Run 1: 1106.80 MB/s
✓ Average: 1106.80 MB/s
============================================================
Testing Disk: nvme15n1
============================================================
* Testing disk: nvme15n1
* Run 1: 1105.67 MB/s
✓ Average: 1105.67 MB/s
============================================================
Testing Disk: nvme17n1
============================================================
* Testing disk: nvme17n1
* Run 1: 1054.59 MB/s
✓ Average: 1054.59 MB/s
============================================================
Testing Disk: nvme19n1
============================================================
* Testing disk: nvme19n1
* Run 1: 1044.52 MB/s
✓ Average: 1044.52 MB/s
============================================================
Testing Disk: nvme21n1
============================================================
* Testing disk: nvme21n1
* Run 1: 1035.04 MB/s
✓ Average: 1035.04 MB/s
============================================================
Testing Disk: nvme23n1
============================================================
* Testing disk: nvme23n1
* Run 1: 1081.54 MB/s
✓ Average: 1081.54 MB/s
============================================================
Testing Disk: nvme0n1
============================================================
* Testing disk: nvme0n1
* Run 1: 1115.62 MB/s
✓ Average: 1115.62 MB/s
============================================================
Testing Disk: nvme2n1
============================================================
* Testing disk: nvme2n1
* Run 1: 1094.79 MB/s
✓ Average: 1094.79 MB/s
============================================================
Testing Disk: nvme4n1
============================================================
* Testing disk: nvme4n1
* Run 1: 1069.38 MB/s
✓ Average: 1069.38 MB/s
============================================================
Testing Disk: nvme6n1
============================================================
* Testing disk: nvme6n1
* Run 1: 1074.32 MB/s
✓ Average: 1074.32 MB/s
============================================================
Testing Disk: nvme8n1
============================================================
* Testing disk: nvme8n1
* Run 1: 1088.86 MB/s
✓ Average: 1088.86 MB/s
============================================================
Testing Disk: nvme10n1
============================================================
* Testing disk: nvme10n1
* Run 1: 1099.50 MB/s
✓ Average: 1099.50 MB/s
============================================================
Testing Disk: nvme12n1
============================================================
* Testing disk: nvme12n1
* Run 1: 1084.92 MB/s
✓ Average: 1084.92 MB/s
============================================================
Testing Disk: nvme14n1
============================================================
* Testing disk: nvme14n1
* Run 1: 1092.99 MB/s
✓ Average: 1092.99 MB/s
============================================================
Testing Disk: nvme16n1
============================================================
* Testing disk: nvme16n1
* Run 1: 1100.55 MB/s
✓ Average: 1100.55 MB/s
============================================================
Testing Disk: nvme18n1
============================================================
* Testing disk: nvme18n1
* Run 1: 1087.87 MB/s
✓ Average: 1087.87 MB/s
============================================================
Testing Disk: nvme20n1
============================================================
* Testing disk: nvme20n1
* Run 1: 1096.02 MB/s
✓ Average: 1096.02 MB/s
============================================================
Testing Disk: nvme22n1
============================================================
* Testing disk: nvme22n1
* Run 1: 1054.69 MB/s
✓ Average: 1054.69 MB/s
############################################################
# Disk Read Benchmark Results #
############################################################
------------------------------------------------------------
| Disk: sda |
------------------------------------------------------------
• Run 1: 472.20 MB/s
• Average: 472.20 MB/s
------------------------------------------------------------
| Disk: nvme5n1 |
------------------------------------------------------------
• Run 1: 871.13 MB/s
• Average: 871.13 MB/s
------------------------------------------------------------
| Disk: nvme7n1 |
------------------------------------------------------------
• Run 1: 1062.68 MB/s
• Average: 1062.68 MB/s
------------------------------------------------------------
| Disk: nvme1n1 |
------------------------------------------------------------
• Run 1: 1072.47 MB/s
• Average: 1072.47 MB/s
------------------------------------------------------------
| Disk: nvme3n1 |
------------------------------------------------------------
• Run 1: 1070.45 MB/s
• Average: 1070.45 MB/s
------------------------------------------------------------
| Disk: nvme9n1 |
------------------------------------------------------------
• Run 1: 1080.92 MB/s
• Average: 1080.92 MB/s
------------------------------------------------------------
| Disk: nvme11n1 |
------------------------------------------------------------
• Run 1: 1095.67 MB/s
• Average: 1095.67 MB/s
------------------------------------------------------------
| Disk: nvme13n1 |
------------------------------------------------------------
• Run 1: 1106.80 MB/s
• Average: 1106.80 MB/s
------------------------------------------------------------
| Disk: nvme15n1 |
------------------------------------------------------------
• Run 1: 1105.67 MB/s
• Average: 1105.67 MB/s
------------------------------------------------------------
| Disk: nvme17n1 |
------------------------------------------------------------
• Run 1: 1054.59 MB/s
• Average: 1054.59 MB/s
------------------------------------------------------------
| Disk: nvme19n1 |
------------------------------------------------------------
• Run 1: 1044.52 MB/s
• Average: 1044.52 MB/s
------------------------------------------------------------
| Disk: nvme21n1 |
------------------------------------------------------------
• Run 1: 1035.04 MB/s
• Average: 1035.04 MB/s
------------------------------------------------------------
| Disk: nvme23n1 |
------------------------------------------------------------
• Run 1: 1081.54 MB/s
• Average: 1081.54 MB/s
------------------------------------------------------------
| Disk: nvme0n1 |
------------------------------------------------------------
• Run 1: 1115.62 MB/s
• Average: 1115.62 MB/s
------------------------------------------------------------
| Disk: nvme2n1 |
------------------------------------------------------------
• Run 1: 1094.79 MB/s
• Average: 1094.79 MB/s
------------------------------------------------------------
| Disk: nvme4n1 |
------------------------------------------------------------
• Run 1: 1069.38 MB/s
• Average: 1069.38 MB/s
------------------------------------------------------------
| Disk: nvme6n1 |
------------------------------------------------------------
• Run 1: 1074.32 MB/s
• Average: 1074.32 MB/s
------------------------------------------------------------
| Disk: nvme8n1 |
------------------------------------------------------------
• Run 1: 1088.86 MB/s
• Average: 1088.86 MB/s
------------------------------------------------------------
| Disk: nvme10n1 |
------------------------------------------------------------
• Run 1: 1099.50 MB/s
• Average: 1099.50 MB/s
------------------------------------------------------------
| Disk: nvme12n1 |
------------------------------------------------------------
• Run 1: 1084.92 MB/s
• Average: 1084.92 MB/s
------------------------------------------------------------
| Disk: nvme14n1 |
------------------------------------------------------------
• Run 1: 1092.99 MB/s
• Average: 1092.99 MB/s
------------------------------------------------------------
| Disk: nvme16n1 |
------------------------------------------------------------
• Run 1: 1100.55 MB/s
• Average: 1100.55 MB/s
------------------------------------------------------------
| Disk: nvme18n1 |
------------------------------------------------------------
• Run 1: 1087.87 MB/s
• Average: 1087.87 MB/s
------------------------------------------------------------
| Disk: nvme20n1 |
------------------------------------------------------------
• Run 1: 1096.02 MB/s
• Average: 1096.02 MB/s
------------------------------------------------------------
| Disk: nvme22n1 |
------------------------------------------------------------
• Run 1: 1054.69 MB/s
• Average: 1054.69 MB/s
############################################################
# Benchmark Complete #
############################################################
✓ Total benchmark time: 106.65 minutes```
Tested both of these scenarios, the workstation is ultimately limited to a PCIe Gen 3 x8 slot so it maxes out around 50Gbps using iperf3 multi-server and multi-client.
Ran the test again twice and this appears to be a test anomaly, same speeds as the other disks reported in both follow on runs.
Noted, we’re going to put the machine into production and see how it performs until multi-client loads. It seems to perform as expected from a TN-Bench perspective. I’ll report back in a few days.
It was better, ~2GB/s read, ~1.5GB/s write. Write speed makes sense, read still feels slow. Receiving workstation SSD can definitely take the incoming data faster than 2GB/s… not sure.
Perhaps I should do the first read test, mix up the order of disks, do the second read test and then average the results.
This may help fix this false positive, and I can allow the user to run 4 times instead of 1/2.
My goal is to eliminate any outliers like this with the exception of hardware related ones. This should make the whole benchmark idea and leaderboard comparison more useful.
According to this, Xeon 6140 has ~1.75X better single-core performance. Which leaves us with maxing out at 17.5Gbps. 2GB/s is 16Gbps. Looks like a close call to me.
I’m not saying that it’s definitely the cause. But it easily could be.
Now, what is more interesting, what is your planned workload scenario exactly? I wonder when 10GBps is not enough for a single connection.
Apologies, there must have been a misunderstanding.
I wish I was maxing out the 10GbE connection. It’s not even close. Instead, my 1.7GHz CPU keeps maxing out single cores / threads, so i know my NAS could be faster but for now the CPU is holding it back.
The most I get in writes has been about 400MB/s over the 10GbE connection or about 40% of the theoretical max. Bandwidth. That’s with a 1/4 full pool and a sVDEV. If I wanted a better throughput, I’d likely need more VDEVs and a faster CPU. Or set up a dataset that solely resides on the sVDEV.
Well, it doesn’t really change the statement that single-core performance can bottleneck “single-connection” transfers. Perhaps my “math” is incorrect, though.
