Recommended disk setup

TrueNAS General
1

Hi All

Building a nas to replace my qnap setup.

I want to use proxmox and virtualise truenas scale.

I have a LSI 9305 16port hba with an Core Ultra 7 cpu/64gb ram.

My disk caddies accept sas and sata

  • I have 8 x 2TB WD reds for long term storage
  • I have 2 x mSATA 500Gb samsung drives

I want to run a few heavy IO VMs including Truenas, Plex , Blueiris etc

What disks/setup are best for the VMs and also for proxmox ?

I was thinking a mirror for proxmox using SSD (any recommendations ?)

And some enterprise class SAS SSD for the VMs OS

My motherboard has plenty of nvme pcie slots also.

Thanks!

2

Virtualization of TrueNAS on Proxmox needs to be done like a production server. You need to pass through the entire controller to TrueNAS and, possibly, blacklist it from Proxmox, so Proxmox doesn’t try to mount the ZFS systems of TrueNAS.

It’s unclear about where you were going to run the other VMs, on Proxmox or TrueNAS. Can you be detailed about your current hardware and software plan?

3

Thanks for the fast reply.

I intend to do all VM’s within Proxmox.

Hardware:

Asus Z790 Prime MB
64GB DDR5
Intel Core Ultra 7 285K CPU
Silverstone RM41-506 Case & Corsair 850watt psu
2 x Silverstone SST-305E Drive Bays (accepts 5 x SAS/SATA)
LSI 9305 16 HBA Adapter

I have 8 x WD RED 2TB SATA disks and 2 x mSATA 500GB Samsung SSDs from the old QNAP setup.

I want very high performance and have 10 drive bays available plus many in the case.

I need to work out how best to host Proxmox, Truenas etc in terms of what discs.

I planned to buy a bunch of things, but its not clear what I need for max performance.

It seems like having enterprise SSDs are the go for the VMs and some disks for SLOG

In terms of whtat its going to be used for:

  • Storage of videos, photos and various personal things
  • Hosting BlueIris a CCTV server - which needs Quick sync and has about 12 cameras on it. Will be writing 24/7 so not sure what disks this is best on
  • Hosting OpenHab, very low weight automation system - needs a USB stick passed through for ZWave
  • Hosting Plex for transcoding 4K videos
  • Hosting a Windows VM as a workstation

Hope that helps!

4

My Network card is a Mellanox Connect 3 Dual SFP based 10Gbit NIC - sorry forgot that one

5

I think you need to have separate servers. One for Proxmox and one for TrueNAS.
Your motherboard gives you 3 M.2 slots and 4 SATA. That would be usable for Proxmox.

The HBA and it’s attached drives would all be only for TrueNAS since you have to pass through the entire controller and blacklist from Proxmox.

You are also out of usable PCIe slots with the HBA and NIC occupying two out of three available. You need to match the card requirements to the slots. If you wanted more drives attached to the Proxmox, you need another HBA.

I think you need to plan one server for Proxmox and one server just for TrueNAS.

I was guessing TrueNAS was going to store the videos, photos, and various personal things.

Try reading through the following on ZFS primer, pool layout whitepaper and the special devices (sVDEV). SLOG is for ‘sync’ writes and has special device requirements or suggestions.

BASICS

iX Systems pool layout whitepaper

Special VDEV (sVDEV) Planning, Sizing, and Considerations

SLOG is not a write cache. You only need it for sync writes. So databases, Block storage (iSCSI, zvols for VMs), and NFS.

1 Like
6

I second having one box for ProxMox, the other as a file server. Not saying it cannot be done, just that there are enough mentions here of folk blowing up their data that I’d avoid it. But if you’re an expert, by all means go down that path! :slight_smile:

1 Like
7

Thanks using two machines is off the table, I definitely only want a single machine.

This Motherboard has 4 x PCIE slots and yes, lll consume 2 of those for the the HBA/NIC which are PCIE 3.0 x8

That still leaves me with 2 slots left.

The HBA can accomodate up to 16 drives, Ive only got 10. And thats without using any of the onboard SATA/NVME ports

8

Is your plan to have almost all storage on the TrueNAS VM and have the other VMs in Proxmox access it for all their storage? Were you planning on a second HBA?

9

Yes, all via TrueNas. So i guess how do you install a TrueNas VM on storage that doesnt exist without TrueNas

No plans for an additional HBA

10

Proxmox would get the boot device that the TrueNAS VM would use. (a motherboard M.2) TrueNAS would get everything attached to the HBA. I don’t know how well it will work to have all the storage on TrueNAS as you would have to have TrueNAS running before your bring up all the Proxmox VMs that require it, first.

I tried looking for Proxmox / TrueNAS posts but I didn’t see any good examples. You can try searching but YouTube has some bad examples, also.

11

Confirm if this is a better high end approach:

Buy 4 x Intel S3710 SATA SSD (smallest/cheapest I can find)

2 setup by Proxmox as a RAID1/ZFS managed by Proxmox for boot
2 setup by TrueNas as RAID1/ZDS managed by TrueNas for boot

Pass through my LSI 9305 with a mix of SAS and SATA disks to TrueNas and any NVME PCIE drives

Would this work OK? I want OS/Data separated and you cant install Truenas on the same bootdrive as Proxmox