I’m new to TrueNAS and could use some guidance for my first big project I’m embarking on. I’ve recently purchased a Dell T640 and plan to install VMware ESXi. My goal is to run TrueNAS Core as a VM to manage all storage.
Here’s my setup plan so far:
One storage pool with large HDDs for main storage
Configured in RAID 5
One storage pool with SSDs for the OS and VMs
Configured in RAID 1
Considering adding a PCIe card for an M.2 SSD cache in the future as i have an unused m.2
Currently, i have only one SSD and one HDD in posession. Is it feasible to start with this configuration and expand later when the other arrive?
Final (for now) setup 2x 1TB SSD, 3x 12TB HDD
After discovering ZFS, I’m considering how best to leverage its features.
My ultimate question boils down to ask for your guidance i recommendations or advice on optimizing this setup and all other toughts regarding this.
For new TrueNAS users I would always suggest bare metal, especially if you are planning on storing important data. TrueNAS can be virtualized, but is it far more difficult to troubleshoot and is far more likely to have issues.
Also, a few points in your early plan. The drive the OS is installed on cannot be used for anything else. You will need a separate boot pool. Which leads to the desire to have share the drive with VMs, if you are planning to install on top of ESXi, host your VMs all on ESXi. Nested virtualization will only further complicate things.
It is possible to start with single drive pools, again not recommended for important data. Be aware that this would leave you commited to mirror vdev layouts, which will result in 50% storage utilization. RAIDZ is more efficient with drive utilization. Intended use case will help drive vdev layout decisions.
Perhaps the most important question at this point is what HBA are you using and is it properly in IT mode?
Thank you for your detailed response and the warm welcome!
To address your points and provide some additional context:
Virtualizing TrueNAS: I understand that running TrueNAS on bare metal is generally preferred. However, I need ESXi for my development work and to separate different environments. My intention is to run TrueNAS Core as a VM within ESXi. I’m aware of the complexities this might introduce, but I am willing to tackle these challenges. Could you provide any best practices or specific configurations that could minimize potential issues with this setup?
Boot Pool and Storage Configuration: As TrueNAS needs a dedicatet pool, is it possible to:
Use the SSDs to create a RAIDZ pool to boot ESXi and one pool dedecated to the TrueNAS VM? (Or does ESXi not need its dedicated boot pool?)
Configurate the HDDs in RAIDZ and seperate them into different pools as needed?
Have two RAIDZ, one for the SSD and one for the HDD? or should i have all the disks as one RAIDZ?
Starting with Single Drive Pools: The intention behind starting with single drive pools is to get familiar with the TrueNAS UI and functionality without risking data. As you mentioned, this limits me to mirror vdevs initially, but once I have all the drives I plan to reconfigure them into a more efficient RAIDZ layout. Can you confirm that this transition is straightforward with TrueNAS, or would you recommend a different initial approach to avoid complexities later?
HBA Mode for the RAID Controller: My system includes a “DELL PERC H730P RAID-Controller with 2GB Cache.” I can configure the H730P in HBA mode to allow TrueNAS to directly control the disks. If there are any additional steps or settings required to ensure proper functionality in this mode, please let me know.
To properly virtualize TrueNAS you will want to pass the entire HBA to the TrueNAS VM. This means you will need 2 controllers - 1 for ESXi and 1 for the TrueNAS VM. Your system likely has some an on board controller in addition to any HBAs, so you will need to have the on board controller connected to a drive(s) for ESXi boot/storage, and then the HBA with any drives you wish to expose to TrueNAS. Suggest at minimum 2 SSDs, one for ESXi boot and one for TrueNAS boot. TrueNAS requires very little capacity for the boot pool, I use a 16GB SSD.
In general you do not want to mix SSD/HDD as data drives in a vdev. A vdev is only as fast as its slowest data drive and will not use the full capacity of all drives if sizes are mixed. So you end up losing the speed benefits of SSDs and the capacity benefits of HDDs.
etorix covered this well. If you want to start with a small number of drives and grow mirrors are the best option, with the 50% capacity caveat. If you start with a RAIDz vdev it is better to have a larger number of drives on day 1. Currently RAIDz pools can only be grown by adding additional vdevs or replacing all drives in a vdev with larger capacity drives. RAIDz expansion is planned for the SCALE release in October, so if you want to take advantage of that it is better to use SCALE over CORE.
Definitely get a proper HBA over what is in the system currently.
As an aside you mentioned in your first post that you could potentially add a drive as cache in the future. Be aware that cache drives in ZFS probably do not work like you expect them to. You will benefit far more by adding additional RAM before adding an L2ARC drive, and SLOG drives only benefit synchronous writes and will wear out consumer devices very quickly. Intel Optanes are often suggested for SLOG devices, but I cannot recommend a SLOG drive without knowing the work load in greater detail.
Thank you so much for the great answers and readings!
I have an update regarding the single drive pools: this will not be an issue now since I have all the drives (3x12TB HDD, 2x1TB SSD, 1x500GB SSD from the old system). I prefer using CORE because it is more robust and I don’t need the extra features of SCALE as I will have dedicated VMs for the applications.
Storage Controller
I understand that I should replace the H730P with an HBA330. Even though the H740P has an HBA passthrough mode, it might not be true passthrough, so a dedicated HBA like the HBA330 would be better to minimize risks related to the controller.
Should I invest in the HBA330 rather than using the HBA passthrough mode from the H730P?
I believe the system has an integrated controller, “DELL PERC S140.”
Drive Configuration
The system has 8 available bays. My plan is to use 3 bays for the HDDs and 2 for the SSDs. Should I connect all these drives to the TrueNAS VM and additionally use one SSD as the boot drive?
Is it possible to create one vdisk from the 2 SSDs and use them as the boot drive? If the boot drive for ESXi and the TrueNAS VM is on the 500GB spare SSD, it wouldn’t be redundant.
Can I create the HDDs in one RAIDZ1 and separate them into vdisks as needed? And another RAIDZ1 for the 1TB SSDs?
I currently have no plans to expand this storage.
Additional Note
Given my setup:
Hardware / Software
3x12TB HDD, 2x1TB SSD, 1x500GB SSD
Tower server with 8 bays (hoping for future expansion)
Including H740P and S140
ESXi Hypervisor
TrueNAS
Use
Plex / Jellyfin
One VM dedicated to media streaming.
Development
Prod and Dev VM for homemade services and applications.
How would you set up and configure this? Would you buy more hardware?
Suggest replacing current RAID card. Often those passthrough mode on cards like that doesn’t really pass the raw device, but passes each device as a standalone RAID0 device.
You will need at least 1 drive to install ESXi on. Suggest using an SSD that was passed through with the HBA for TrueNAS boot, but it can work with a virtual drive for boot. Redundancy isn’t the most important for boot drives, just make sure to backup your TrueNAS config from time to time or when making major changes/updating. The disks connected to the HBA and passed through to the TrueNAS VM will not be available for ESXi.
For your dev/media VMs, suggest hosting those on ESXi. I am not sure what your plans for storage for the ESXi VMs is. We have seen issues when someone tries to host TrueNAS virtualized and then share the storage back to the hypervisor that is hosting TrueNAS, creates a circular redundancy. We strongly suggest not using the virtualized TrueNAS instance to provide storage to ESXi.
This is unfortunately a complicated setup, and would still suggest using TrueNAS bare metal. We are usually unable to investigate bug reports from virtualized TrueNAS systems due to the number of variables, and community support will also be more limited. TrueNAS SCALE uses KVM for virtualization, this setup could easily be designed with TrueNAS SCALE bare metal and use KVM to host your VMs. CORE virtualization is no longer being tested actively, and is provided without support.
