10G network from TrueNAS to TrueNAS

Hi,
I have two servers, both installed with TrueNAS Scale 24.04.1.1. Currently, both are connected to the same (1 GBit) switch via gigabit ethernet, both have a static IP. I thought of adding a 10 GBit adapter to both, while they are still connected to the 1 GBit switch. I don’t want to change the network switch, so I’ll just buy suitable network cards and 1 cable.

Is there anything special that I need to setup within TrueNAS so the 10G connection is used automatically when transfering from server to server? Is it sufficient to put them both in the same subnet (only for the 10G card) that is different from my 1 GBit subnet?

Kind regards

That should do it.

4 Likes

buy 2x 10Gb SFP+ cards for each server (I prefer Chelsio, less chance of getting a fake Intel card, the T520’s are good and supported)

Then go on amazon and buy a 10gTek DAC cable.

Install cards - connect.

Now for IPs - you do not, nor can you actually, assign the 10G NICs with the same IP/subnet you already have on your 1GB nics, and you dont want this anyways so you are dead on for a new subnet needed

Since the 10Gb links will only be used for TrueNAS to talk to each to each other, just use something else example

1Gb Network is 192.168.1.1/24
10Gb make it 192.168.2.1/24

Then any jobs or things you set up to use the 10Gb network - just input the 192.168.2.* IPs as source/dest and it should be good.

3 Likes

It works! Now I have 2 Mellanox ConnectX-3 adapters and after some fiddling with rsync I just saw a ~ 3.6 GBit / second transfer from server to server.

Thanks all for your input! :smiley:

2 Likes

Nice!

It is nice when you see those speeds jump up vs standard 1Gb links.

If you want to do real through- put testing do some iperf testing between boxes to test the connection to try and max out that 10Gb link, then you know for sure you arent limited in some way and anything that could use and abuse that 10Gb link, will be able to! (normally it takes multiple transfers at once to saturate a link unless the content of course is in ARC it could do it.

I used the iperf tool and found that using the =P 6, or whatever. That really tested the total bandwidth.
I have been disappointed with the performance of my network. I loaded the “openspeedtest” on my server. Then I can web access and test the speed from that server. I found that the slowest speeds were with Mellqnox-X2 cards. Intel performed the best then Solarfare (some), then Mellanox.

openspeedtest will load in the Core version as a plugin

1 Like

Would this work to connect a TrueNAS box to a PC running Windows 11? I have datasets on the NAS that are shared with SMB so they appear on the PC as folders in file explorer. I would like to be able to drag and drop files between the shared folders on the NAS and other folders on the PC’s disks.

I can manage installing 10Gb cards and a cable. When I set this up I will have 2 paths between the PC and the NAS. The path through the switch at 1Gb and a second path through the new NICs and wire at 20Gb

If I drag and drop a folder to or from a dataset, how do I tell the system to use the 10Gb connection and not the 1Gb?

Just replace or remap your network drives or share to point to the new IP address or set up a hostname alias that points to the newer 10Gb ip address and use that.

Networking Primer link
https://www.truenas.com/community/resources/10-gig-networking-primer.42/

&
https://forums.truenas.com/t/10gbase-t-best-to-avoid-it-if-you-can/1594

2 Likes

You could spend $40 on amazon and pickup a cheap 6 port switch with 2 sfp+ and 4 2.5G ports.

1 Like