Help me plan a network - 10 Gbps probably

TrueNAS General
21

I am curious what 25G card would you recommend?

Kind regards, Archivist

22

Iā€™m using a Chelsio T6225 and a Mellanox ConnectX3 variant for 25G, Chelsio T580ā€™s for 40G cards, and Mellanox Connect-X4 single-port 100G cards for my various Proxmox hosts, and a MikroTik CRS520 as the switch (16x40/100G ports and 4x25G ports).

If you donā€™t need all those ports, and/or want more bang for your buck, the MikroTik RDS2216 is a router + switch, has a smattering of ports (6x10, 4x25, 2x100), has 20 NVME slots, 16 cores, 32GB of RAM, can export its drives towards TrueNAS (iSCSI or NVMEoF), and is ~$300 cheaper than the CRS520.

23

Dude. 2 gigabytes is 16 gigabit per second. Even 500MB is doable over generic stuff if you have a hoss of a nas to serve multiple nodes. Realistically skip 10g, go to 25g for all processing nodes and 100 or 400 switch to nas.

I had similar but slower requirements and went with a 2 port 40g with 24 10g poe cisco switch.

24

For the no-switches direct connection scenarios, breakout cables can be used. Thus, you would be able to connect one QSFP(28) NIC to four SFP(28) NICs. I would personally go for mellanox connectx-4 or higher (as 3rd gen is a bit old).

:warning: Disclaimer! Iā€™ve never used breakout cables myself.

1 Like
25

Break out cables arenā€™t routinely supported by all cards, at least not as often as with switches. There are customizations that often have to be set with the NIC driver to get that to work, so do a lot of research as to which cards support that with which cables and at which speeds.

26

QSFP stands for Quad Small Form-factor Pluggable. Sooo, I do believe any QSFP card supports breakout cables. But again ā€“ Iā€™ve never used them.

27

I know. I have a half dozen of these cables in use in my server room. And in some of my servers I have 40G and 100G NICs where the manufacturer says specifically ā€œBreak out cables are not supported by this adapter.ā€

More specifically, the 40G or 100G side is not supported in some vendorsā€™ NICs. I use the high speed end on my switch port and the low speed (10G/25G) for hosts.

1 Like
28

Interesting. So far, I donā€™t need breakout cables, but something to keep in mind. Thanks.