Well, a single drive is a degenerate form of mirror, so the proposed stripe is a “mirror” configuration. The point is: No raidz for iSCSI.
1 Like
Get three drives. Enjoy lower latency and faster transfers.
Even if you put 12 HDDs in mirrors, you will not be able to reach half of the speed a midrange NVME drive will offer you. And that is ignoring IOPS.
You can still offload games to a network share.
Always thought it made more sense to just have a massive steam cache / lan cache on the nas…
Well, you don’t even have to anymore.
Nowadays this is enabled for every normal Steam installation by default.
Ofc you did implicitly, by not mentioning a single available alternative to bridge. Thus creating false impression for new users.
Seems like you have no idea how tcp/ip works.
Not really.
Please enlighten me bugacha 
While you are at it, could you also explain why your upload is slower than your download?
The problem with bad performance in iperf3 is inside win 11.
Same HW but using Fedora 41 (dual boot) give me better performance.
iperf3 -c 192.168.252.10 -P 1
Connecting to host 192.168.252.10, port 5201
[ 5] local 192.168.252.20 port 35208 connected to 192.168.252.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.84 GBytes 32.9 Gbits/sec 0 1.08 MBytes
[ 5] 1.00-2.00 sec 2.03 GBytes 17.4 Gbits/sec 0 1.08 MBytes
[ 5] 2.00-3.00 sec 3.74 GBytes 32.1 Gbits/sec 0 1.08 MBytes
[ 5] 3.00-4.00 sec 3.87 GBytes 33.2 Gbits/sec 0 1.08 MBytes
[ 5] 4.00-5.00 sec 3.74 GBytes 32.1 Gbits/sec 0 1.08 MBytes
[ 5] 5.00-6.00 sec 3.82 GBytes 32.8 Gbits/sec 0 1.08 MBytes
[ 5] 6.00-7.00 sec 3.73 GBytes 32.0 Gbits/sec 0 1.08 MBytes
[ 5] 7.00-8.00 sec 3.46 GBytes 29.7 Gbits/sec 0 1.08 MBytes
[ 5] 8.00-9.00 sec 2.78 GBytes 23.9 Gbits/sec 0 1.08 MBytes
[ 5] 9.00-10.00 sec 2.86 GBytes 24.6 Gbits/sec 0 1.08 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 36.6 GBytes 31.5 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 36.6 GBytes 31.5 Gbits/sec receiver
iperf Done.
iperf3 -c 192.168.252.10 -P 1
Connecting to host 192.168.252.10, port 5201
[ 5] local 192.168.252.20 port 35208 connected to 192.168.252.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.84 GBytes 32.9 Gbits/sec 0 1.08 MBytes
[ 5] 1.00-2.00 sec 2.03 GBytes 17.4 Gbits/sec 0 1.08 MBytes
[ 5] 2.00-3.00 sec 3.74 GBytes 32.1 Gbits/sec 0 1.08 MBytes
[ 5] 3.00-4.00 sec 3.87 GBytes 33.2 Gbits/sec 0 1.08 MBytes
[ 5] 4.00-5.00 sec 3.74 GBytes 32.1 Gbits/sec 0 1.08 MBytes
[ 5] 5.00-6.00 sec 3.82 GBytes 32.8 Gbits/sec 0 1.08 MBytes
[ 5] 6.00-7.00 sec 3.73 GBytes 32.0 Gbits/sec 0 1.08 MBytes
[ 5] 7.00-8.00 sec 3.46 GBytes 29.7 Gbits/sec 0 1.08 MBytes
[ 5] 8.00-9.00 sec 2.78 GBytes 23.9 Gbits/sec 0 1.08 MBytes
[ 5] 9.00-10.00 sec 2.86 GBytes 24.6 Gbits/sec 0 1.08 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 36.6 GBytes 31.5 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 36.6 GBytes 31.5 Gbits/sec receiver
iperf Done.
With -P 4 I can reach almost 100 gbps:
iperf3 -c 192.168.252.10 -P 4
Connecting to host 192.168.252.10, port 5201
[ 5] local 192.168.252.20 port 47500 connected to 192.168.252.10 port 5201
[ 7] local 192.168.252.20 port 47504 connected to 192.168.252.10 port 5201
[ 9] local 192.168.252.20 port 47516 connected to 192.168.252.10 port 5201
[ 11] local 192.168.252.20 port 47526 connected to 192.168.252.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.81 GBytes 24.1 Gbits/sec 0 3.14 MBytes
[ 7] 0.00-1.18 sec 3.31 GBytes 24.1 Gbits/sec 0 4.21 MBytes
[ 9] 0.00-1.18 sec 3.26 GBytes 23.8 Gbits/sec 0 4.12 MBytes
[ 11] 0.00-1.19 sec 3.32 GBytes 23.9 Gbits/sec 0 4.13 MBytes
[SUM] 0.00-1.00 sec 12.7 GBytes 109 Gbits/sec 0
...
[SUM] 0.00-10.00 sec 113 GBytes 96.9 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 113 GBytes 96.9 Gbits/sec receiver
iperf Done.
Reverse direction not so good:
iperf3 -c 192.168.252.10 -P 8 -R
Connecting to host 192.168.252.10, port 5201
Reverse mode, remote host 192.168.252.10 is sending
[ 5] local 192.168.252.20 port 43250 connected to 192.168.252.10 port 5201
[ 7] local 192.168.252.20 port 43252 connected to 192.168.252.10 port 5201
[ 9] local 192.168.252.20 port 43254 connected to 192.168.252.10 port 5201
[ 11] local 192.168.252.20 port 43262 connected to 192.168.252.10 port 5201
[ 13] local 192.168.252.20 port 43274 connected to 192.168.252.10 port 5201
[ 15] local 192.168.252.20 port 43286 connected to 192.168.252.10 port 5201
[ 17] local 192.168.252.20 port 43296 connected to 192.168.252.10 port 5201
[ 19] local 192.168.252.20 port 43312 connected to 192.168.252.10 port 5201
...
[SUM] 0.00-10.00 sec 92.8 GBytes 79.7 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 92.8 GBytes 79.7 Gbits/sec receiver
iperf Done.
Best Regards,
Antonio
Did you try -P 8 in windows 11 ?
Iperf3 test is synthetic and quite useless in real-life
Once you see decent speeds in iperf3, you should be doing real tests by copying files via SMB.
My friend they did ban wikipedia for you?
1 Like
Yes, but the performance are much slower than in linux. Also in win 11 as soon as I perform a “reverse connection” (iperf3 -R), one core go to 100% usage and performance are bad: maybe is an iperf3 problem or win driver problem.
Best Regards,
Antonio
can you show it please ?
100% usage is normal - pushing 100gbps is not a walk in the park, especially if network card doesn’t properly support offloads
Here the win 11 iperf:
.\iperf3.exe -c 192.168.252.10 -P 8
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 6.15 GBytes 5.28 Gbits/sec sender
[ 5] 0.00-10.01 sec 6.15 GBytes 5.27 Gbits/sec receiver
[ 7] 0.00-10.01 sec 9.31 GBytes 7.99 Gbits/sec sender
[ 7] 0.00-10.01 sec 9.31 GBytes 7.98 Gbits/sec receiver
[ 9] 0.00-10.01 sec 8.96 GBytes 7.69 Gbits/sec sender
[ 9] 0.00-10.01 sec 8.96 GBytes 7.69 Gbits/sec receiver
[ 11] 0.00-10.01 sec 9.20 GBytes 7.90 Gbits/sec sender
[ 11] 0.00-10.01 sec 9.20 GBytes 7.89 Gbits/sec receiver
[ 13] 0.00-10.01 sec 9.09 GBytes 7.80 Gbits/sec sender
[ 13] 0.00-10.01 sec 9.08 GBytes 7.79 Gbits/sec receiver
[ 15] 0.00-10.01 sec 7.19 GBytes 6.18 Gbits/sec sender
[ 15] 0.00-10.01 sec 7.19 GBytes 6.17 Gbits/sec receiver
[ 17] 0.00-10.01 sec 6.02 GBytes 5.17 Gbits/sec sender
[ 17] 0.00-10.01 sec 6.02 GBytes 5.16 Gbits/sec receiver
[ 19] 0.00-10.01 sec 8.40 GBytes 7.21 Gbits/sec sender
[ 19] 0.00-10.01 sec 8.40 GBytes 7.20 Gbits/sec receiver
[SUM] 0.00-10.01 sec 64.3 GBytes 55.2 Gbits/sec sender
[SUM] 0.00-10.01 sec 64.3 GBytes 55.2 Gbits/sec receiver
iperf Done.
And reverse direction:
.\iperf3.exe -c 192.168.252.10 -P 8 -R
...
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 3.43 GBytes 2.94 Gbits/sec 0 sender
[ 5] 0.00-10.01 sec 3.43 GBytes 2.94 Gbits/sec receiver
[ 7] 0.00-10.01 sec 3.45 GBytes 2.96 Gbits/sec 0 sender
[ 7] 0.00-10.01 sec 3.44 GBytes 2.95 Gbits/sec receiver
[ 9] 0.00-10.01 sec 3.46 GBytes 2.97 Gbits/sec 0 sender
[ 9] 0.00-10.01 sec 3.45 GBytes 2.96 Gbits/sec receiver
[ 11] 0.00-10.01 sec 3.84 GBytes 3.29 Gbits/sec 0 sender
[ 11] 0.00-10.01 sec 3.83 GBytes 3.29 Gbits/sec receiver
[ 13] 0.00-10.01 sec 3.88 GBytes 3.33 Gbits/sec 0 sender
[ 13] 0.00-10.01 sec 3.87 GBytes 3.32 Gbits/sec receiver
[ 15] 0.00-10.01 sec 3.84 GBytes 3.30 Gbits/sec 0 sender
[ 15] 0.00-10.01 sec 3.84 GBytes 3.29 Gbits/sec receiver
[ 17] 0.00-10.01 sec 3.77 GBytes 3.24 Gbits/sec 0 sender
[ 17] 0.00-10.01 sec 3.77 GBytes 3.23 Gbits/sec receiver
[ 19] 0.00-10.01 sec 3.83 GBytes 3.29 Gbits/sec 0 sender
[ 19] 0.00-10.01 sec 3.83 GBytes 3.28 Gbits/sec receiver
[SUM] 0.00-10.01 sec 29.5 GBytes 25.3 Gbits/sec 0 sender
[SUM] 0.00-10.01 sec 29.5 GBytes 25.3 Gbits/sec receiver
iperf Done.
Best Regards,
Antonio
Got it
Whats your NIC btw ?
Do you use correct MTU size on Windows ? It’s supposed to be 9014 not 9000
Did you enable Hardware offloads in Windows NIC settings ?
What costs?
Thanks for all the suggestions. I now realize that my current setup is not going to cut it for PCIE lanes.
I’m looking at selling what I have and thinking of the following, any suggestions?
Pro WS WRX80E-SAGE SE WIFI & AMD Ryzen Threadripper Pro 5945WX
Pro WS WRX80E-SAGE SE WIFI & AMD Ryzen Threadripper Pro 3945WX
ASRock ROMED8-2T & AMD EPYC 7302P
The EPYC setup would be cheaper but lower clock speed, maybe for the gaming/work PC go with the 3945WX and the server EPYC 7302P?
Then get a pair of Mellanox ConnectX-5 EDR+100GbE for 300USD
For Gaming home PC I highly recommend : 7960X with Pro WS TRX50-SAGE mobo
For server get a package Supermicro H12SSL-i + 7402P + 256Gb 3200 Samsung RAM - they are sold as combos on ebay
You can buy dual port ConnectX-4 for $140
You can get plenty of performance with NVMe-oF. Given enough RAM to keep data hot, as well as an L2ARC on an NVMe SSD to capture spill-over from ARC, you can shovel tons of data over the wire using NVMe-oF with RDMA.
(L2ARC works fine for that exact scenario. I’m doing it myself and get 60%+ hitrate on my Steam library. Over a ConnectX3 with 40GBe.)
There’s however two issues:
- You can only configure it on command line and have to redo it every TrueNAS update.
- The only available Windows initiator is made my Starwind, and they removed the free personal licenses at some (to me) unknown point in the past. If you got one, it’ll continue working, but no new ones.
However:
- iX is working on NVMe-oF support in TrueNAS. Question is if it’s gonna be gated behind an enterprise license or not. Initial support will be over TCP only. If that’s gonna be free for everyone, question is the RDMA support (like the NFS stuff right now, which is enterprise only).
- Microsoft is working on their own NVMe-oF initiator. They made some medium fanfare about it, but then scrubbed details of its announcement off their site and apparently post-poned it.
2 Likes
I’ve been eyeing the ASRock ROMED8-2T as it has more 4.0 x16 slots, any reason to choose Supermicro H12SSL-i over this?
In my Truenas Core system I’m using this MB https://www.asrockrack.com/general/productdetail.asp?Model=ROMED6U-2L2T#Specifications
It works well!
But it has only 4 PCIe x16 and 6 dimm.
Best Regards,
Antonio
1 slot is shared, ASrock IPMI is crap, bunch of boards incorrectly report fan RPM speeds. Board is more expensive too.
Onboard RJ45 is useless.
Supermicro is much more stable.