I need some recommendations for my setup - Looking to improve

TrueNAS General
1

Hi

Background

I have been running my TrueNAS setup for a while but I am not an admin expert. So in my recent research I discovered that recordsize is very important and this led me to re-look at my VDEVs, Pools and datasets so I could get better performance and be ready to fully get into Apps and Containers, however my knowledge is a little limited here.

MySetup

  • 1 x RAIDZ2 | 4 wide | 931.51 GiB (Rust Drives)
  • 1 x RAIDZ2 | 4 wide | 931.51 GiB (NVME Drives)

But to try and save power, I removed the Rust drives but still have them.

On the NVMe I am running a couple of virtual machines, in particular pfSense and Virtualmin.

Going forward

  • I will be using a lot of Apps
  • I probably will use containers in the future
  • I want a server that runs well but has good data safety.
  • I don’t have a huge amount of images, films and all that stuff, but I do have some.

Questions

What is the best setup using my current kit?

Are these my 2 best options

  • 2 NVMe drives in a VDEV mirror, 4 Rust drives in a RAIDZ
    • Have the VMs and Apps regular backed up Rust Drives
    • The rust drives would also hold all of the large media files
    • User the 2 spare NVMe for something else
  • Carry on using my VMs on the RAIDZ2 as is
    • Put all of my media on the NVMe as well because I do not have that much

If there is any other recommend layouts I would welcome those suggestions or clarification which option above is the best option for me.

thanks for looking

2

If you’re running raidz2 with only 4 drives, data resilience must be your priority. If you’re going to backup your nvme pool to rust pool, you could cut back to raidz1 on both to increase storage capacity or at least on the nvme pool because you will have a backup copy on the rust pool. It’s all based on need.

Your current setup is for minimal down time & high resiliency. The mention of you getting rid of the rust drives says you want to save power, so would you really want to re-integrate them? Default record size of 128 pertty much works for most if not everything unless you are dealing with lots of small files and want to imrove optimization for small files over general use. Performance wise, you nvme are blinding fast, so it wouldn’t even be noticeable.

3

I want to challenge that premise. The 128k is already pretty good. Sure, 1MB would be better, but 128k is also fine.

Depends on where at what your are planning to run and if it is blockstorage or datasets. IMHO TrueNAS is a great NAS, not so great Hypervisor.

4

I agree with this very much.

5

I want to challenge that premise. The 128k is already pretty good. Sure, 1MB would be better, but 128k is also fine.

For file storage with limited or no changes this is probably true but for:

  • Apps with database need small 16K writes and they maybe have complex logging whic also requires small writes.
  • Going forwards containers might need their recordsize tweaking
  • Although not affected by recordsize my VM running my webserver is on 16K volblocksize and I think that is the best for it.

I am reseaching recordsize and I have found a lot of information on it, but my main issue at the moment is how I should configure my VDEV/Pool

6

@Sara and @Glitch01

The Hypervisor has been ignored for a long time and I would agree with you both here. The funny thing it could be a lot better and a massive asset to TrueNAS. I have made the team aware of this.

7

“Need” is a strong word. Are you really optimizing your App’s databases for thousands of hits per second & want to reduce latency to the lowest point possible as it could impact critical transactions that absolutely can suffer no delay?

Or are we talking about a database that’ll update occasionally when you add a movie or a few dozen photos a day & the recordsize would have almost no reasonable impact? (arguably taking up maybe dozens of additional MiBs in storage space over an eventuality)

8

If you’re so worried about setting a low recordsize for your apps’ databases you absolutely SHOULD run your apps and VMs on striped mirrors in the first place. Raidz2 with 4k blocks is a terrible idea…

9

I never said setting my recordsize to the lowest value was the aim. :blush:

I understand the smaller the block the larger metadata and overhead.

8k use to be recommended for post free but it is now 32k with lz4 compression.

Tailoring the recordsize to the workload is a consideration, but primarily this thread is about setting up my vdev to match my needs. Recordsize might of been a consideration that why I mentioned it.

The main question, should I put my pool and a striped mirror and if I did should I change the recordsize.

I will look at the link you sent me.

10

A very useful article and probably answer my question.

Don’t put your VM disks on RAIDZ.

Thanks