Evaluating TrueNAS SCALE Performance and ZFS Topology on KVM VPS Infrastructure

TrueNAS General
1

I am testing TrueNAS SCALE (version 24.04.0) deployed inside a KVM-based Virtual Private Server (VPS) host to analyze ZFS pool performance and disk passthrough behaviors.

Environment & Topology

  • OS Edition: TrueNAS SCALE 24.04.0
  • Deployment: KVM Hypervisor on VPSTTT infrastructure
  • Storage Topology: 4x 50GB virtual disks in RAIDZ1 pool (tank)
  • RAM: 16 GB allocated

Diagnostics & Context

During sequential I/O benchmarking using fio, the pool experienced I/O delay timeouts resulting in faulted virtual vdevs. Running zpool status indicates checksum mismatches under heavy load across virtual block devices (vda, vdb, vdc, vdd).

Question

What hypervisor I/O caching modes (e.g., none, writeback) and disk controller settings are optimal for maintaining ZFS pool integrity when hosting TrueNAS SCALE on KVM-based VPS backends?

#TrueNAS ZFS Virtualization #KVM #Storage

2

Truenas has a whole blogpost on the topic of virtualizing truenas:

And you should propably use a more up-to-date version of truenas for your testing. Your version is more then 2 years old

3

This document is from 2023. I am not sure outside of testing it is still recommended, but I am not an expert on this.

At the bear minimum you need to have a HBA passed through to give direct access to the disks.

4

Nothing changed since 2023 so it’s still valid…

5

Pass through the entire controller to the VM, and blacklist whatever kernel modules that controller would use on the host.

6

Why are you testing on an old version of TrueNAS?

7

What is this, Twitter? :grinning_face_with_smiling_eyes:

8

Why would you expect any fundamental change in three years?
If the purpose is to play with the interface to test how it works, any hypervisor will do.
If the purpose is to host any actual data, or to get real numbers as here, the rules are simple: Type 1 hypervisor only and pass-through of the PCIe controller(s) of all data drives. Plus blacklisting said controllers if the hypervisor happens to be able to mount ZFS pools (Proxmox), which may not apply here.

9

For testing on hypervisor has always been fine, but there have been some stories on here when TrueNAS on a VM goes wrong. Putting TrueNAS on bare metal is always the preferred option.

This is my understanding, but saying that people have told me I am bonkers for virtualizing pfsense on my TrueNAS box, which by the way has been running fine for years except when the NIC failed.

10

“Always” statements are always wrong.

Self-contradicting statement aside, it’s still very close to the truth–there are many variables that play into any decision, the effect of which is that any universal statement is almost certain to be wrong in some cases. When it comes to virtualizing TrueNAS, there are pitfalls that can eat your data if you do it wrong–but they’re well-known, and well-documented (JGreco wrote them up something like ten years ago; they’re nothing new), so they can readily be avoided with a modicum of care (i.e., RTFM).

So is TrueNAS on bare metal “always the preferred option”? I don’t think so. It’s simpler, and it’s safer (because the data-eating pitfalls aren’t there), but it makes your hardware less versatile. And sure, TrueNAS can run VMs, but it pales in comparison to something like Proxmox or xcp-ng in that regard (and I expect it always will, though sadly my crystal ball is in the shop).