What is better 4x1Gbps LACP or 2x2,5Gbps LACP or 10Gbps

TrueNAS General
,
1

Hi,
i have maybe stupid question. What is better network connection (TN Core v13, RAID-Z3 3x7disks, Supermicro SAS2):

  1. 4x1Gbps LACP (metalic)
  2. 2x2,5Gbps LACP (metalic)
  3. 10Gbps for SATA (SFP+, optics)

Number of connected users are 10 and files that users generate are 12MB big, opened over network on SMB share. Used switch in network is 2,5Gbps and have also SFP+ with 10Gbps.

pool speed test, command:

fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=100G --runtime=60s --readwrite=write --name=testfile

pool speed test, result:

fio-3.28
Starting 1 process
testfile: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=465MiB/s][w=465 IOPS][eta 00m:00s]
testfile: (groupid=0, jobs=1): err= 0: pid=88063: Fri Jun 21 08:44:15 2024
  write: IOPS=591, BW=591MiB/s (620MB/s)(34.7GiB/60050msec); 0 zone resets
   bw (  KiB/s): min=264192, max=757226, per=100.00%, avg=606035.35, stdev=110979.29, samples=119
   iops        : min=  258, max=  739, avg=591.49, stdev=108.33, samples=119
  cpu          : usr=1.16%, sys=18.08%, ctx=328996, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,35490,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=591MiB/s (620MB/s), 591MiB/s-591MiB/s (620MB/s-620MB/s), io=34.7GiB (37.2GB), run=60050-60050msec

I think that 4x1Gbps is under pool disk, so 2x2,5Gbps is better, but are 2 links better than 4 in real world, or only one with 10Gbps?

2

No question here, 10 Gbps is the best option among those you mentioned.

3 Likes
3

thx for answer.

despite the fact that the connected stations have a much lower line speed (1 or 2,5Gbps)?

how is it if 10 stations access the NAS at the same time and they all want to store their own file in the same pool? isn’t it better to have LACP, for example, of 4 lines?

also, pool get me about 5Gbps. Is 10Gbps link (card) “overkill”?

4

SMB is a single client process that at best will saturate a single pipe. So the most you can get out of an aggregated couple of 1GbE lines is 1GbE per client. Ditto 2.5GbE, etc. Maximize speed to the switch, then have a fat trunk to your NAS.

For my little system here I have experienced 800+MB/s SMB transfers with the help of my sVDEV and large record sizes. I would never be able to do that with aggregated links.

Also, in practice, the speed at which my rsync backup process could traverse the NAS with my 10GbE connection vs. gigabit or WiFi is in no comparison.

That’s one reason I manually disable WiFi on my laptop before I attach the Thunderbolt SFP+ connection as MacOS still does not reliably switch to the fastest network connection even if the 10GbE connection is indicated as the most preferred one.

1 Like
5

No. A 10 Gbit link can serve 2.5 Gbit/sec to four clients simultaneously. A 4 x 2.5 Gbit LACP may be able to do that in the best case, but often will not. See:

No, because you can also be serving data out of cache.

1 Like
7

thx, now i understand.