I’m quite happy with my first TrueNAS Scale deployment. It is running on a Dell Precision T1650 with 16GB of DDR3 and a 4c/8t Xeon. I modded the case to accommodate up to 6 LFF HDDs. Currently using 4x3TB Seagate Constellation ES.3 drives in RAIDZ1, on the integrated SATA controller.
Got my Apps sorted and even a Win7 VM for some legacy software. Apps are on a separate SATA SSD, as well as the boot-pool on yet another SATA SSD. Both SSDs are on a noname cheapo controller card, that just works. The machine boots directly from the SSD. The HDDs spin down after a period of inactivity, cutting power consumption from 90-ish to 47-49W.
So why change? Well for one I like tinkering. Practical reasons are better idle power economy (I think I’m gonna fail this badly) and more memory for VMs that can actually run serious software. Research put me on track for a DDR4 server platform. I like DELL and HP bc hardware, so I was on the lookout for PowerEdge T130-330 and the HP equivalents. What came up though was a T440 LFF with 64GB DDR4 and a Xeon Silver 4114.
The plan:
10x6TB SAS3 drives are in the mail. 8 are going into a raidz2. Asus Hyper M.2 16x is in the mail. That will hold my system and apps on probably raidz1.
The question: Do I go for bare metal TrueNAS and use the built in hypervisor or go for TrueNAS VM with Proxmox hypervisor. Option Proxmox will expand my knowledge and options. It will, however limit the RAM available to ARK, because I will give a limited (probably 16GB) RAM to the VM. I will be using passthrough for the PERC H730 (itself in IT mode).
Use case for this deployment will be homelab/mediaserver/hypervisor for private and company projects/bulk storage for personal and some critical stuff.lll
So, thoughts, suggestions, comments are welcome. Worst case, my server arrives FUBAR and I use it as spares for the current T440 I run as a company Hyper-V machine.
Personally, would never ever consider Scale as a VM. Additionally, we have seen a good number of proxmox Truenas installers losing pools lately, possibly due to an upgrade in Proxmox. Anything running as a VM will not necessarily behave identical to something run on bare metal, and that’s not a good thing in my book. When I see someone is on proxmox, I also tend to ignore any issues they have as it’s not worth the time expenditure. Ymmv, just speaking for me.
Am in the same pickle.
Proxmox looks good for VMs and the VM backups.
TrueNAS looks good for a multiple disks compared to storage spaces ( M$ )
Hyper-V looks good for memory management.
Am testing SCALE bear metal with just one Win VM. In that VM am running Hyper-V. And it feels slow running VMs inside that VM
Now the question is, what is more important to you ? Fast VMs ?, reliable storage ?, save memory ?, or just save electricity ?
I don’t mean to steal the thread but, what do you use to backup that Win10 VM ?.
I ask because am not sure if the snapshoots are all that is needed.
Then again the OP wants to run VMs too so am not sealing anything, I hope
I treat my VMs the same way I would treat them if they were not a VM. So, for Windows, I use Windows tools. Backup and restore as well as that, don’t recall that name, tool functions like Mac Time Machine. I don’t want or need the host to backup the VM, the VM backs up itself using whatever OS tools to do so.
That being said, not entirely true. I do have snapshots for the Windows VM. though I might stop them as the use is slowly winding down.
So for my homeassistant VM, I use homeassistant backup. Etc.
@sfatula@argumentum, would love more detail about the failings of TN-S as a VM. It is officially supported and with PCIe passthrough, it should (yeah, in theory) have full control over the storage on hardware level.
I would prio things in that order: reliable storage=good VMs(fast, easy to manage, good support for older M$ stuff)>resource management>power draw.
I started looking for a more recent machine to cut on power / have more memory, but with 2x1100W PSUs and 10 DIMM slots on 1 CPU and 16 if I put 2 CPUs in the T440, I guess the first point is causa perduta, the second is a given.
My issue is not if it’ll install and work. The problem is always down the line when something goes wrong. Therefore I mistreated the box faking failures like “now you see it, now you don’t” and CORE behaved as expected but SCALE did not.
Since the new and shiny features of SCALE can done in Proxmox, I opted for CORE for as VM in Proxmox.
All in all, CORE is king, SCALE … is what is in mode by the current fashion. If FreeBSD had proper attention from the community, SCALE would not exist.
…then again. OSs are made to run in bare metal. If the NAS aspect is important and also asking in a TrueNAS forum. Then the answer will be run it on bare metal.
And unless you’re into the YouTube hype, the VMs in TrueNAS SCALE run just fine.
My favorite VM hypervisor is Hyper-V but, all I know is windows ( hence the bias )
I have been running SCALE as a VM in ESXi for the time SCALE existed and ESXi was still a viable option for a home lab. PCIe passthrough of e.g. 2 NVMe SSDs is of course mandatory in any virtualised scenario.
With ESXi out of the game I tried a couple of open source hypervisors and was seriously underwhelmed specifically by what Proxmox offers. I still have to try XCP-ng.
Apart from that my conclusion is that TrueNAS is solid enough as is - and comes with local ZFS to boot! - we have been running all our virtualised workloads on CORE for years at my company. You should have a perfectly smooth experience with SCALE.
So for me - if you are on SCALE anyway, just use it for virtualisation, too. I don’t see anything Proxmox would add. We are still on CORE for the company, absolutely nothing to complain about. Our two domain controllers (Windows Server 2016) run on CORE/bhyve.
If XCP-ng is more of a vSphere like experience I might consider setting up an XCP-ng host using TrueNAS provided iSCSI storage like I did with ESXi.
Thanks for all the replies. For now it seems there is no benefit in running Proxmox, except the excitement factor from the possibility of torching my data.
Unless someone steps up with a really good reason to do Proxmox + TN-S VM, I’m going bare metal and running my VMs in Scale.
Is there a point of trying to migrate my current setup to the new box, or it makes more sense to just do a clean install and battle the config once more?
That’s what I’d do. But that’s me.
I believe that all these migrations carry …the old dirt with it. So to keep it as bug free as possible, I install anew.