I’m planning to build a new NAS (the plan got changed). I plan to use 8373C or a similar 3rd Gen Xeon Scalable CPU (single) and 16*64GB DDR4 ECC RAM. This is going to be all flash NAS (NVMe) and 8xHDDs for bulk storage for the moment and scale it later. The SSD i plan to use is 8xD7-P5520 3.84TB. For networking, i plan to use 200GbE (Mellanox CX6 or Intel E830-i know its not released yet).
Regarding the client machines, I’ve the following setup
8x25GbE Intel E810/XXV710-XXVDA2. Out of these 8 client machines, some of them are Xeon Silver 1st Gen Scalable and above 64GB RAM on each machine. These machines are running Ubuntu.
Now, i’ve a couple of questions:
Just like how TrueNAS have boot pool where the OS and configuration sits and the data pool is on a different drive and the read/write speeds only matters to the data pool as in terms of share. So, is it the same thing with client machines as well? I mean can I have a small sized boot drive for the client machines and have a different SSD for the main data storage? Will that affect the speed? Is it different for other OS than TrueNAS? Or should i just use a big sized and use it for OS+data together? What would be best? I planned to use a separate boot drive for easier management and separate data disk.
Will i be able to saturate 200GbE using 8 client machines at 25GbE speeds via NFS or SMB share?
answering your questions:
1., Seeing your server spec, it seems, you can afford some additional cost to build the clients. I recommend to abandon the “small SSD” idea completely.
(if your OS SSD gets full the Ubuntu system gets really instabe, if it can be able to boot up at all. That happened to multiple linux machines, I installed and used during the years. This is how I know, that it is not good to use a separate /boot partition, since the kernel updates will fill it sooner or later and once I miscunfigured the sftp server and placed the =hoomes on the main drive. It got full and it took like 4 hours of KVM debugging to get it working again. )
Also small SSDs (like <64GB) Will wear out sooner, since the TBW will be much lower for them for a marginally lower price.
So, if you want to use small SSDs, you will have a risk of wearing them out and fill them thus making your system instable.
One possible option is to buy second hand OPTANE drives, but there is only the 16GB versions are dirt cheap, the bigger ones are almost new priced even second hand, it is not available second hand, since Intel discontinued it.
Therfore they are like Gen2 or best case Gen3 by 4x so their seepds will not be really impressive.
For like 20-50 USD you can buy up to 1TB NVMe, if you are lucky!
I guess a 120-128GB-256GB Gen4x (or Gen5x) M.2 will be more than sufficient for ANY Ubuntu installation (My VMs are usually sit on a 20GB partition, and my Crafty server had 500 GB which never crossed the 20% full mark)
I recommend to use the locam NVMes for only the OS and all user data should be stored on a NFS (or even iSCSI) Drive on the server.
Another option is to install a TFTP server to your main Server and use simply network boot for your 8 clients.
It will eliminate all the above questions anf also the possible bandwidth bottleneck of the NVMe M.2 drives.
2., Well, I am not too experienced in these speed ranges, but I am more worried about whether your CPU willbe able to serve this speed.
I guess, you are not planning to use SMB, since as of my knowledge, it is still implemeted single thread, so the practical limit of using is is around 40 Gbit. SO theoretically your clients can get the full bandthwidth on their side, IF the server can serve the 8x25Gbit. Howeve, thi is so insane network speed, that I think like editing 4k or 8k videos or some AI cluster is the only relevant application I can imagine, where you get even close to this.
In general, I would appreciate to know, what is your planned apllication for this beast…
It took me two years approx to do this much of research and then got to know bits and bytes. No wife as of now. So, I’m all good
Yes, I planned to remove all the Synology and currently each client machines have their own 10TB/20TB HDDs, so for easier management and centralization, i thought to plug all those drives here, create a single NAS server and all people can access and the benefit is easier management+space+redundancy.
Regarding the RAM, i’ve couple of spares so not an issue. I know the prices are up now. xD
These are for video editing and some AI/ML on the Xeons
These client machines already exists in the workplace. Its just the NAS i’ve to build.
For the boot disk, I plan to use normal consumer grade 512GB Gen3/Gen4 SSDs. 1TB is overkill for boot disk i guess. Although 512GB could be as well but I can manage that much as i’ve couple of spares from system upgrades.
Regarding storing the data, its just for saturating the 25GbE speeds on a single client machine to/from the NAS (200GbE).
Yes, I doubt the CPU selection as well. Maybe i need a Dual CPU. But even if i say i use SMB, then i guess for the client machine specs i have, i can easily push 25GbE with some optimizations and tuning.
The Xeon machines are for AI/ML and then the rest for video editing purposes. For editing, its the Resolve Studio and Photoshop (sometimes).
I am not too experience with this high end enterprise HW, but I am not sure, a dual CPU will help.
Also Xeons are usually low clocked CPUs with a lot of cores and cache memory.
The 64 GB RAM in the Xeons will limit you to like 50GB of LLM model size. This is currently not a hard limit, since there are about 5 models, in LM studio that will not fit into this limit.
However, I just bought another 2 sticks of 32GB DDR4 ECC to max out my MoBo for my AI server to 128GB.
(It is absloutely not necesary, but I prefer to be safe than sorry. That is why I bought a 1000W PSU too.)
I am not sure, you are better off by using multiple PCs for an AI cluster.
It might bebetter to (re)build one of your Xeon machines for specifically be an AI server (128-256GB RAM, and 1-2 beefy GPU, like the 3090 or 4090, and maybe just a brand new commercial MoBo with DDR5 RAM and GPUs))
As I understand, Local AI benefits the best from high clocked CPUs and fast RAM, if not ran from a GPU.
Some of the GPU workstations have RTX 4070Ti and some 5070Ti and then there are main GPU servers that has 4xRTX 4090. There are some RTX A5000 and A6000 as well. Those are primary GPU servers.
Yup, all this stuff is way over my head and interest level.
I need to get back to something I do need to finish for the wife, installing wood blocks behind the drywall, from the mighty warm attic, and without falling through the ceiling of the first floor. It would be a long drop.
Before making a move, are you expecting to do all your video manipulations over the network cabling? I’m pretty sure you will find that placing the video file on the workstation to be the best way forward. If you are worried about the file becoming lost due to a drive failure on the client, setup an automatic save feature that will save your files from the Client maybe once every 30 minutes? Some value that you feel good about.
I agree. I have one more block of wood to install and can call it done, for the hot part of it.
It is almost like building a NAS, you start and you think you are completed, but something comes up and you change things around, costs more money, sometimes a pain, but gratifying once done. See how I linked this into @Fastline changing his mind. Now the project has dwarfed the original specs and cost.
I know the feeling so well. Every project usually begets another 5 because something something snowballs in order to get the original task done. Houses are never ending projects. Thankfully, I am able to turn off my NAS occasionally.
