I’m planning a small, quiet build with Odroid H4 Ultra with 48GB RAM and considering using NVMe drives instead of SATA SSDs due to their lower price (atm) and compact size. Specifically, I’m looking at using an Odroids NVMe adapter to connect four NVMe drives… I do actually really want quiet and efficient drives for long term , and SSDs in any kind would be best…
Would it be a good idea to set up my first RAID 5 (RAIDZ1) array with this configuration while keeping my SATA SSDs for future expansion (possibly another 4 drive RAID 5 setup)? I’ve read that combining different RAID arrays into one pool isn’t recommended—can anyone confirm?
Additionally, I plan to use SSD that i own already for Proxmox instalation + VM data via USB3. And also would like to install host on the removable eMMC. Does this setup make sense, or should I consider a different approach?
Sorry , those question might look silly , i am just very new and would like to do the things “right” from the beginning. Thinking long term
That if you use their own 4x NVMe expansion card, it should deal OK with sharing the 4x PCIe lanes with the 4 cards.
Obviously the 4x NVMe themselves will be faster than the 4x SATA SSDs, however the 4x NVMes will share 4x PCIE Gen 3 lanes, whereas the 4x SATA SSDs will share a single lane - so the NVMe solution should be significantly faster.
It doesn’t come with an eMMC disk - they are an optional extra, and since you still have 4x SATA slots left, using a small SATA SSD as a boot drive is a viable alternative - so whilst your boot drive doesn’t need to be ultra performant, you might want to consider performance, or alternatively to save the SATA slot for a future bulk SATA HDD RAIDZ1 pool.
I think you should consider running TrueNAS native rather than under Proxmox. Running TrueNAS under Proxmox requires both dedicated storage and dedicated storage controllers, and that may need more hardware than you have. Plus this is NOT the most powerful of boxes, and to a significant extent I would expect the combination of TrueNAS Docker / LXCs / VMs to offer a much broader choice of virtualisation solutions which will allow significantly more efficient virtualisation than you could achieve with pure Proxmox VMs.
Given that you have SATA ports free, my advice (as someone who does use USB SSDs) is NOT to use USB SSDs.
If you are going to do VMs i.e. virtual disks/Zvolumes then you will really want mirrors for that rather than RAIDZ - so although it will have lower storage efficiency, my advice would be to use the 4x NVME as a 2x 2x mirror pool. I would then plan for a 2nd 4x SATA SSD RAIDZ1 pool at some point, and use a 32GB or 64GB eMMC as a boot drive. So buy the case suitable for a 4x 2.5" SSD stack and the largest PSU listed.
All in, this looks like a truly excellent technical solution for an SSD/NVMe NAS.
Hello ,
thank you for your answer . I have few questions first and then i will be able to start the purchasing spree (sales now lol)…
Could you elaborate what is the reason for that?
As in future i would like the option to populate my 4 sata ports , the SSD on SATA would make this hard.
Is it because speed concern or another reason? As the default readability looks good for those nvmes.
Thanks ! I am considering this as well , seems like TN is developing many new thingsd for VMs .
VMs do small random reads and writes to virtual disks, so high IOPS and mirrors are what you need to support high IOPS and to avoid read and write amplification (where the size of reads and writes to the virtual disk are smaller than the RAIDZ records and so to read a virtual block you need to read several real blocks and to write a virtual block you need to read several additional blocks and then write several blocks.
If I was building a new NAS, I would definitely be thinking about using an H4 Ultra with 4x SATA SSDs and 4x NVMe (using the Odroid card) and 1x Odroid eMMC for boot drive, plus a Type 3 case and SATA power cables and a PSU. Buying these from the same supplier looks to me to avoid a lot of risks of incompatibility.
I would need to buy the 48GB of memory and the SATA/NVMe drives elsewhere.
You can build the hardware in about an hour by the looks of it.
Thanks a lot for the support! Now just waiting for the shipment to arrive.
Quick questions…
If I were to use this machine solely for TrueNAS without virtualization, would you still recommend using only mirror vdevs?
Also, when it comes time to expand with additional SATA drives in the future, what’s the best approach?
Should I add them to the same pool with the same RAID configuration?
^ Can i expand the original one somehow?
Or would it be better to create a second pool?
I’m a bit concerned about how expansion might affect the folder structure and overall setup.
Also , buying Samsung nvme / ssds, how am i expected to update the firmware ? I know samsung magician on Windows, but then have no idea about Truenas and how can i update them .
Still learning, so any advice is greatly appreciated!
Mirror vDevs for virtual disks/zVols/iSCSI and database files because they have high IOPS and read/write amplification - along with synchronous writes (sync=always) for these and a SATA/NVMe SSD SLOG if the data isn’t on SSD already.
If you don’t need 1. or if you are having separate pools, then for sequential access to files then use RAIDZ and sync=standard.
Again generalised rules of thumb…
For mirrors, either replace each mirror drive in an existing vDev one-by-one with larger drives, or add another mirror vDev of any size.
For RAIDZ, either replace drives in an existing vDev one-by-one, or add another RAIDZ vDev ideally of the same width, or use RAIDZ expansion up to max 5-wide if RAIDZ1 or up to 12-wide if RAIDZ2/3.
