Need advice migrating a server to TrueNAS

TrueNAS General
,
1

Hello everyone,

I would like to ask for your advice regarding a possible migration of a data server to TrueNAS. I am part of a research group focused on atmospheric and climate studies, and we currently have a couple of powerful servers (EPYC, 128 cores / 1 TB RAM each) with their own local SSD storage. In addition, we have a separate data server with larger capacity, which we use to share a large folder via NFS.

At the moment, this is the only intended use for the server. We do not run virtual machines or databases on this storage—only data files, which we occasionally process directly without copying them locally.

The server specifications are as follows:

  • Chassis: Gigabyte S452-Z30-00

  • RAM: 64 GB (8 × 8 GB ECC DDR4 3200 MHz)

  • RAID controller: LSI MegaRAID SAS-3 3108

  • Disks: 18 × 14 TB drives in RAID 6

  • Network: Dual 10 GbE with LACP

  • Two sata SSD’s for OS.

Since the disks were approaching 5 years of use, we decided to replace them with 26 TB drives, and we have purchased a total of 29 disks. The issue is that I had originally included 4 × 64 GB RAM modules (for a total of 256 GB), but the order has been delayed and now the price has increased to over €1000 per module (procurement in the Spanish public sector is not exactly straightforward).

As an HBA, I have an AOC-S3008L-L8E, which allows me to avoid using adapters since the SAS expander cables use the same SFF-8643 connectors.

My current plan is to create 3 RAIDZ2 vdevs with 9 disks each (27 disks total) and leave 2 disks as spares. The server is located in a data center with limited physical access, so having spares available is important (especially since hardware procurement at the university can be slow).

The main concern I have is the amount of RAM. I am aware that 64 GB is quite low for this kind of setup, but at the moment it does not seem feasible to obtain more. I could take RAM from the other servers (it is the same type), but that would leave them in asymmetric configurations and would involve mixing modules.

Would you recommend any changes to this setup?

Thank you very much.

2

I will let others comment on the RAM and NFS.

The MegaRAID probably needs to be replaced. You need a plain HBA that won’t interfere between ZFS / TrueNAS and the hard drives. The Controller may be able to be flashed to ‘IT Mode’. You can run sudo sas3flash -list and post the results back here using Preformatted Text mode and others should be able to comment on the data. Preformatted Text (</>) or Ctrl+e.

What’s all the noise about HBA’s, and why can’t I use a RAID controller?

2 Likes
3

Thank you for your reply. Yes! I have a AOC-S3008L-L8E that worked well in another setup ready to be installed in the server.

The original one is a 9361-8i which according to the forum (sorry, I can’t include links) can be converted to JBOD, but I think it is better to don’t cut costs here and put a pure HBA adapter here (a Supermicro one it’s only 40€ in eBay). The sas3flash version that I found online does not see any adapter.

4

Truenas has this fatal flaw of working with lots of things that aren’t recommended… until it suddenly doesn’t & data recovery becomes a nightmare :frowning: Some quick google-fu does say that the card can be flashed to IT mode, so I can’t speak for certain on if it’d actually work fine or not long-term.

Agreed.

TrueNAS will work, even with that amount of data, just fine with 64gig of ram - however ARC space will be limited & could impact speeds depending on how hard you’re hitting the server. Considering you have spinning rust I’m making the assumption that this’ll be perfectly acceptable.