Direct link iperf3 asymmetrical results 38Gb/s & 12Gb/s ,Services high ram usage

TrueNAS General
1

Hi,

I am new to the truenas world so i do appologise if this comes accross as a repeat, I have looked around but cannot find an answer or I have missed it due to lack of understanding on my part

I have just finished a build and when testing the iperf3 results I am getting 38Gb/s in one direction and only 12Gb/s in the other direction as below seen from the truenas side, This is a direct 40GbE fiber link between the truenas box and a windows 10 PC, trying -P for more threads has no effect the total is always near 12Gb/s

1
1673×615 13.9 KB

The two systems are as below ;

Truenas side ;
OS TrueNAS CORE 13.0-U6.1
Supermicro X9SRI
E5 1650 CPU
250GB Mirrord boot drives.
128GB 1066Mhz Ram
mellanox connectx-3, MTU - 9000

Pools ;
4, 2TB WD NVMe Blacks with bifurcation, They are configured as a 2 way mirror striped with 2 more drives, Empty as of these tests.

Windows 10 side ;
i5-10600
16GB 1600Mhz ram
mellanox connectx-3, MTU - 9000

If it is any help outside of the iperf3 test, I briefly tested an iscis setup and used crystal disk mark to perform a 64GB transfer, this avregas 1.2GB/s read and 1.8GB/s write over the full transfer with no apparent drop.

Am i correct in thinking the 12Gb/s issue is when reading from the NAS ? CPU usage never peaks past 10% on the NAS side and 30% on the Win10 machine duing the iperf3 tests.

I have no tunables set as of this moment, I had messed around with the ones mentions in the 10GbE primer post but I still seem to hit the 12Gb/s limit with iperf3

Any suggestions or points on where to look would be much appreacted, as I understand iperf3 tests the link speed so should ideally be near the same speed in both directions ?

thanks

2

Do you see similar results with IPerf 2? It’s not an older version, it’s an entirely different product, and it works better on Windows.

3

Running iperf 2.1.4 Truenas side and iperf 2.2 Win10 Side.

Results are the same as iperf3

iperf2

Testing with truenas as the client with and without the -R addition

R
-R

4

Last Fall I was getting setup with a 40Gbe connections between my workstation and TrueNAS servers and was facing similar issues. My fix ending up being far more simple than a bunch of tunables, it was all about the PCIe NIC placement. Here’s a link to the discussion on the old TrueNAS forums.

TLDR Making sure the NIC was in PCIe lanes that ran through the CPU made all the difference. Going through the chipset majorly impacted speeds.

2 Likes
5

Thanks for the reply, My NIC is in a PCIe x8 Gen 3.0 Direct to the CPU in the NAS, Win10 end is the same in a x16 slot direct to CPU

One issue is the antivirus I have found, Disabling it I am able to get 22Gb/s over the 12Gb/s i had, But the link is still stuck at 22Gb/s in iperf3 so there is still something off it seems given its half the rated speed in iperf3

Going in the other direction I get 38Gb/s full speed.

I’ll have a play with BIOS settings for PCIe bits but there is little/lack of in the way of options for the PCIe power and such, Least now with an iSCIS share I can get around 2.5GB/s reads over the 1.4GB/s before hand

1 Like