TrueNAS Upgrade/New Build

TrueNAS General
,
1

Hello,

I am currently looking to upgrade or possibly replace my older TrueNAS system.
The current setup is a Xeon E3-1230v6 on a Supermicro X11SSM-F motherboard with 16 GB of RAM.
The storage pool is configured as a 2×2-way mirror (4×4 TB HDDs) for a total of ~8 TB.

The TrueNAS server has two 1 Gbps NICs bonded via LACP and connected to an HP Aruba switch, which also provides four unused SFP+ (10 Gbps) ports.

The server isn’t doing anything fancy - it simply serves SMB shares to three Windows client machines that open files directly from the mounted network share.
The files are usually small, but certain programs used by the clients can occasionally write larger amounts of data (around 10–20 GB per project) to the share. When multiple clients do this simultaneously, the available network bandwidth to the TrueNAS system can become saturated.

Overall, I’d consider our workload relatively mixed but leaning more toward the read-heavy side.

Goals:
I’d like to either upgrade the existing system or replace it entirely so that it can fully utilize 10 Gbps networking.
Along with this, I plan to switch from TrueNAS CORE to TrueNAS SCALE.

Question 1 - Storage Performance

Do I necessarily need SSDs to achieve 10 Gbps (read/write) performance on a TrueNAS SCALE system, or can regular HDDs reach around 800–1000 MB/s without expanding the storage pool significantly?

For example, what kind of IOPS and read/write throughput could I expect from a 1×8-wide RAIDZ2 array versus a 4×2-way mirror array?

Question 2 - Hardware Considerations

I believe the Xeon E3-1230v6 should still be capable of handling 10 Gbps.
I could upgrade the system memory to 32–64 GB of RAM and either reconfigure the storage pool or leave it as it is (a 2×2-way mirror).
According to the ZFS layout whitepaper, this configuration should theoretically provide ~400 MB/s streaming reads and ~200 MB/s streaming writes — not quite 10 Gbps of course, but likely sufficient for our needs.

For network cards, I was considering the Intel X550-T2 or Mellanox ConnectX-3.
Are these good options for TrueNAS SCALE, or are there better alternatives I should look into?

New System Idea:
If I decide to build a new system, I’m considering the following:

CPU: AMD EPYC 4244P
Memory: 64 GB (2×32 GB) Kingston DDR5-5600 UDIMM RCC
Motherboard: Supermicro H13SAE-MF
NIC: Intel X550-T2

I’d really appreciate your thoughts and insights before making a final decision.
Thank you very much!

2

NAS itself requires not much in the way of CPU. So your existing CPU is (IMHO) more than adequate. Its the addons like containers and VMs that really rack up the CPU requirement. Your existing CPU is more than adaquate.

Just add a 10Gb card - the X550-T2 will just work where that Mellanox has multiple models, some of which are not suitable (assuming I am remembering correctly). I personally use Chelsio (since Core) but the choice has got wider now.

More memory is good, but I suspect not actually needed.

Change one thing at a time, but you don’t need the EPYC

2 Likes
3

Noting that this EPYC 4000P is a desktop Ryzen with official ECC support of course, but your Xeon E3v6 is still up to the job.
Save the money, and add more vdevs if you want more throughput.

1 Like
4

Speaking from experience, smb is very cpu heavy, especially since it’s single threaded. You don’t want 20 cores, you want a few cores with very strong single thread performance.

I chased the 10gb dragon and settled at some point. My spinning rust arrays could only push or pull about 4gb per second, single stream to another machine (potentially much faster from the arc). Made me think more smaller drives rather than a few big drives would be faster, but I finally reached the hardware limit on the wire above 2.5gb ethernet. So I’m satisfied with what I ended up with although I have a twin test Nas with nothing but nvme storage that is faster and only holds a percentage of stuff my main server does. Someday we’ll get some trickle down enterprise storage that is fast, dense and cheaper.

Long story short, going above 2.5gb is largely pointless without lots of expensive storage; you will spend and spend chasing that 10g dragon outside the enterprise. Nothing wrong with future proofing the network side though, 10g hardware is very affordable now.

One last note. I use the mellanox connectx3’s and they’re basically plug and play under anything. Keep your airflow up and they won’t get too hot.

2 Likes
5

Alright. Thank you guys for your feedback.
Upgrading my existing server was my preference from the start so the fact that you guys are recommending that too, is reassuring. :slight_smile:

To get a little bit more specific here:

  • Going from 16 (2x8) to 32GB (2x16) or 64GB (2x32)
  • 8x8TB HDDs (Seagate Ironwolf NAS probably)
  • New PC case and PSU
  • Intel X550-T2 NIC (or maybe Mellanox)
    (which of them stay cooler could maybe make a difference in the decision)

I will probably just buy some Fractal case that can provide enough space for all HDDs but maybe you have a good recommendation for another one I should consider. When it comes to the PSU I am not exactly sure how much watt it should have to provide enough power for MB,CPU and all 8 harddrives.
600W should probably be more than fine, right?

Here is the important part:
I was thinking to stay with the 2 drives per vdev configuration in a mirrored array but instead of doing 2x2, (4x4TB drives) I would instead go for 4x2 configuration aka 4 mirrored vdevs with 2 drives each. (8x8TB drives overall)
I assume more vdevs = more performance otherwise I could just go for 4x4.
Let me know if one configuration is preferable over the other.
What about 1x8 wide raidz2/raidz3 vdev?
How would that compare to the first example?

I should probably mention there is another backup server that gets all the data from the server I want to upgrade replicated. The backup server itself won’t be upgraded tho. Just to let you know that backups are taken care of. :slight_smile:

Thank you again for your help.

6

Generally, fiber should be cooler that copper (RJ45). I don’t know if that is the case when comparing an old fiber NIC, like Intel x520, to a newer, more power efficient copper NIC. Search for 10G networking primer for some discussions and why they recommend fiber over copper. Going fiber gives you the option for 25Gb and faster upgrades while copper is pretty much 10Gb

You might consider a server case to get good cooling airflow, in case you decide to add an HBA.

1 Like
7

Just my 2p worth on the PSU.

  1. Make sure it isn’t a cheap one (but it doesn’t have to be made from pure unobtanium)
  2. Make sure its get plenty of SATA / Molex power cables. Some have a very limited number which has caused some users here to overload the appropriate rail and causing wierd effects.

Also - cooling. Drives need cooling, but so do add on PCIe cards - some more than others. As @SmallBarky says if you get an HBA that WILL need additional cooling and a copper 10Gb ethernet could probably do with it as well. I cable tie a fairly powerful 120+mm fan above the PCIe slots to force air into that area and have never had (touch wood) an issue yet - despite 2 * HBA’s and a 10Gb NIC (but it is fibre)

1 Like
8

The thing to keep in mind is that this hardware that has heatsinks and no fans means it’s depending on tornadic winds generally available in server cases.

You don’t need to bolt fans onto everything but you gotta have positive airflow blowing the hot air out while pulling cold air in. There are plenty of command line tools that help keep an eye on temps. Consider your case design and choose either airflow or static pressure fans accordingly. Double up the front and rear fans; it doesn’t need to be noisy, just stay away from little zinger fans. Simple as.

9

Alright. Thank you guys once again for all your valuable input.
Strangely enough as if my existing server heard the message, my X11SSM-F stopped working today (i assume its the mb, i am not 100% sure tho)

So I will probably pull the trigger on a new replacing the old parts and buy a new motherboard and CPU too.:

Are there any specific recommendations I should be looking into?
Here are the options i was thinking about:

  • Supermicro X13SCL-F + Intel Xeon E-2434
  • Supermicro H13SAE-MF + AMD Epyc 4244P

Any better options that come to mind in your opinion?

Thank you very much. :slight_smile:

10

With these requirements it seems that your Xeon E-2400 or EPYC 4004P options are already too fancy, too new, and too expensive. Look into older generations as well.
If this were a home build, I’d suggest an AsRock Rack E3C246D4U2-2T from eBay, but I suspect this is a work machine and you need “new with invoice an vendor warranty”.

1 Like
11

Yeah it is a bit of both I would say. But if it’s a bit overkill thats totally alright. This one was running for over a decade just fine so I don’t mind paying a bit extra. :slight_smile:

1 Like