It’s hard to advise on exact pool specifications without more insight into the Veeam configuration since that will impact the workload, file counts, file sizes, etc. It’s also useful to consider whether you have VBR workloads in addition to VBO, how often your backups are running, whether there are concurrent backups, etc.
In general terms for someone new to ZFS, there’s a lot to learn.
Lesson one is “best practice” and the theoretical underpinnings of ZFS are inextricably linked. With that in mind, I offer some notes:
Resilience against drive failures is at the logical vdev level. If you lose more than [edit: thanks @dan] one vdev, your storage pool is gone. Newcomers to ZFS are sometimes confused because drives can be properly referred to as vdevs, but that’s physical vdev. (You can also have a single drive logical vdev, but that’s beside the point here.) TrueNAS hides this distinction for your sanity but it’s important theory.
IOPS in ZFS is related to topology. If you go with RAIDz then your slowest drive can limit pool performance. Mirrors are faster, but ZFS mirrors come along with the space cost implied by normal RAID mirrors. If your Veeam estate is essentially continuously running jobs, this might be a concern. Mirrors also have neat traits RAIDz doesn’t have, like being able to remove Special vdevs, which can be used to store ZFS metadata separately from data block storage.
With RAIDzn, n can be 2 or 3. This is the number of drives that can fail before losing a vdev. RAIDz has certain risk calculations. Can you afford the risk in losing more than n of x drives at once? As the number of drives scale that risk increases. You can have hot spares running and ZFS will use them. You should research how it uses them because it’s slightly counterintuitive, but it is elegant.
Expanding a vdev with additional drives is possible, but in RAIDz there are caveats. Data doesn’t get rebalanced across the newly expanded array. You cannot remove a data vdev from a RAIDz pool. There are some good talks on YouTube that demonstrate how RAIDz expansion works. (On that topic, anything from Klara Inc. and the OpenZFS Summit channels are worth watching.)
Mirror expansion is comparatively straightforward, but again with mirrors you have the implied capacity/drive count costs. Mirrors are useful in that they can be broken in order to seed a new pool (e.g., for sneakernet transfers), but this would be a challenge to rely upon for routine usage, partly because you should bookend that kind of use with scrubs (more on that in a moment).
With the number of drives you have, ZFS dRAID is probably something you should consider. Unfortunately I don’t have experience with dRAID and I’d defer to the wisdom of others to speak to that. Suffice it to say dRAID is for large numbers of drives (which in the ZFS world is what you’re dealing with).
Do not use port multipliers, expanders, or hardware RAID cards with ZFS. To do so is to invite serious potential for performance issues if you’re lucky, and data loss if you’re unlucky. ZFS assumes and expects that it has sole control over topology and geometry. HBAs are fine in IT mode only. The community tends to recommend LSI/Avago/Broadcom or Chelsio.
You’re going to want ECC RAM. It’s not required. At this scale it wouldn’t make sense not to have it.
Scrub your pools! This is how ZFS catches corruption particularly for data that hasn’t been accessed since the last scrub (or since it was written). A scrub will scan the metadata and all referenced blocks to confirm that they checksum as expected, and uses parity (RAIDz) or intact copies in the pool to self-heal. Intact copies can come from mirrors or in RAIDz from additional copies stored which is configurable.)
If you wish to use ZFS native encryption, you may find replication more intuitive if you don’t encrypt the pool, but rather encrypt datasets. You can use ZFS inheritance to your advantage here. @winnielinnie has a post on the old forums: Difficulty replicating native ZFS encrypted pool to new pool. | TrueNAS Community. See “pseudo-root datasets” in that post. (@winnielinnie will also helpfully correct anything I’ve gotten wrong in this post, I am sure, and is worth listening to.)
With TrueNAS and encryption, you may wish to pay attention to how TrueNAS maintains cryptographic keying material. Encryption keys are stored in plaintext on the boot pool. Encryption passphrases are not stored this way but must be entered before the datasets are available. You may mix either within a pool. Unlike with BitLocker there is no recovery key functionality.
Speaking of replication, you’ll want it. It’s the fastest way ZFS can propagate data across pools (and systems, and therefore machine level failure domains). Replication relies upon snapshots, which are immutable references to all the blocks actively used by a dataset at the time the snapshot was created. These are a cornerstone of your ZFS backup and DR strategy. Snapshots can be “held” to prevent accidental deletion. Whole expositions can be written on snapshots and replication. Snapshots only start consuming space as the dataset diverges from the snapshotted state. (Technically they do take a tiny amount of space because they’re essentially a point-in-time copy of the metadata tree and that has to be stored on disk, but the space consumed is negligible.) Blocks referenced by snapshots remain in the pool until the block is no longer referenced either by the live state of the dataset or by any snapshot.
As I recall from the documentation, Veeam compresses backup archives. Regardless, there’s virtually no appreciable performance impact to enable LZ4 compression on pools. It won’t slow things down when it encounters incompressible data but it will compress slack space.
ZFS checkpoints are good to know about. Use them before making major pool changes. A pool may only have one at a time and they are not designed to exist for appreciable amounts of time. Reverting to a checkpoint reverts the pool to the time of the checkpoint. Checkpoints are often equated to snapshots but they absolutely should not be used like snapshots.
How are you planning to point Veeam to TrueNAS? That impacts how you will want to configure datasets (which can be either POSIX filesystems or ZVOL block storage for iSCSI, and NFS is also an option).
