Hi all,
My company recently retired our old Veeam Performance Tier backup repository and purchased a 45drives Q30 to replace it with TrueNAS pre-installed. The Veeam server is connected to it via iSCSI and everything works but the synthetic full backups on the weekend are significantly slower than what we were seeing with the old hardware (a 9 hour job is now taking over 22 hours). I’ve made changes based on what I could find online regarding optimization but it still has not made a big difference. I was hoping if I posted the config here someone might spot where I have made a mistake. I am trying to tune for optimal I/O performance over everything else since the backup data is also immediately uploaded to AWS at the end of each job. Please let me know if I left any important information out and thank you in advance for any advice!
CPU Model: AMD EPYC 8124P 16-Core Processor
Memory: 125.6 GiB total available (ECC)
ZFS Cache: 112 GiB
Data VDEVs: 5 x MIRROR | 2 wide | 21.83 TiB
Spare VDEVs: 2 x 21.83 TiB
No Metadata, Log, Cache, or Dedup VDEVs
DATASET DETAILS
Sync: Disabled
Compression Level: OFF
Enable Atime: Off
ZFS Deduplication: Off
Checksum: ON
Read-only: Off
Exec: On
Snapshot Directory: Invisible
Snapdev: Hidden
Copies: 1
Record Size: 128K
ACL Type: POSIX
ACL Mode: Discard
Metadata (Special) Small Block Size: 0
What blocksize does the iSCSI have? Default 16K?
What sizes does Veeam write?
I would not disable compression. It costs nothing and can compress zeros.
Special vdev you mean
Two things. Using HDD does nothing for that, since they are not faster than your pool. You could just use these two drives in your pool.
Using L2ARC and special vdev is not really smart, since they basically “cache” the same thing.
Use SSDs or get rid of svdev use them as HDD in your pool.
What speeds are you getting and what speeds did you expect?
I would first give it a try by enabling compression and setting the block size to 1MB.
If I remember correctly, that is the size Veeam uses.
What was the old hardware?
And have you run any benchmark on the new server?
Writing backups should be a nice sequential task so just running, for example a sequential CrystalDiskMark from the Veeam server with a large size should be a good representation.
Then a local sequential write test directly to the pool would be intersting (just a large dd…) could be compared to that.
Thank you both for the quick replies!
To answer your questions, I have the iSCSI filesystem formatted NTFS with block size 64KB. Unfortunately the Q30 does not support ReFS.
I had disabled compression because the Veeam software was already compressing the data and it seemed unnecessary to have it enabled.
I have the 2 spare drives setup because I do not have access to the datacenter we are using, we need to contact someone with our parent company to go there for any on-site needs so it seemed prudent to have the spares so a drive failure would be less of an emergency and I could wait for someone to be on site vs calling in favors for a special trip.
The old hardware was a very old Dell Data Domain that worked great but was so old that the Data Domain OS could no longer be updated, which meant we could not update the Veeam software, which was preventing us from updating VMware to v8. Just a chain of headaches.
As far as benchmarks, I ran some tests with Veeam support to optimize things on the software side and we’re getting about 158 MiB/s. We also ran an independent test with DiskSpd and that resulted in 149 MiB/s.
I will confirm with Veeam support about the 1MB block size, hopefully that will help, and I can enable compression too since it can’t really hurt.
If you have any other suggestions, I’m all ears, and thank you again!
I would set compression to LZ4. Even though Veeam backups are compressed by Veeam itself, setting LZ4 will allow ZFS to compress its own metadata and you’ll be able to hold more metadata in the ARC cache as a result. There are mechanisms in place now for ZFS to quickly bail on uncompressable data, so it shouldn’t slow anything down.
Maybe I misunterstood something here but if you present the zpool to the Veeam server as blockstorage via iSCSI - shouldn’t the Veeam server be able to format is at anything it likes? What’s keeping you from using ReFS there?
Someone with way more iSCSI knowledge has to chime in, since I know basically nothing about iSCSI. My wild guesses:
I think that iSCSI shares can only be set to 4k logical block size, which then on the other hand sit in the default 16k volblocksize zvol?
So my next guess would be that your NTFS 64k does not really matter or change anything. And it probably a good thing you did not use ReFS, since that would be CoW on top of CoW.
This leads me to my next question, is iSCSI even needed? Would some SMB or NFS share also be possible? I think it isn’t really surprising that
if your older hardware was some kind of bare metal, RAID, Windows Server thing. Now you have something “virtualized” instead.
If I’m not completely mistaken Veeam also does correctly recognize and handle empty blocks - what exactly do you mean?
Honest question as I never thought about it: how would that be a problem?
With SMB or NFS OP would not get block cloning. If the zvol stays formatted with NTFS I don’t think it would make much of a difference.
How is the new system (a TrueNAS server on bare metal) virtualized and why would it be surprising to go from something bare metal to something virtualized in general?
I am sorry, I am mixing it up with Windows Backup and recovery!
The problem I had with Veeam was that the backup grew way too fast on ReFS, even though it was supposed to be incremental.
And that with Windows Server 2025 there Microsoft Visual C++ error, so you could not install it.
And that it did not run in as system task and a user had to be logged in.
I know, pretty off topic, I was just shocked how bad and unstable the market leader of backup is and swichted back to Windows Backup and recovery with NTFS (which btw does not support ReFS and at the same time has a 16TB limit for NTFS. Ahh Microsoft ).
If veeam compressed something into lets say 12k, and TrueNAS offers 16k volblocks, without compression this is a 16k stripe with 12k data and 4k zeros. With compression, it is a 12k stripe and depending on the pool geometry might use less data. But then again, I am don’t enough knowledge to know if that also applies to iSCSI, since I have no idea how volblock behaves here.
I guess only performance and write amplification wise. Just like you would not use ZFS for a VM that is on a ZFS zvol.
iSCSI is an additional layer of complexity, that might introduce additional configuration pitfalls. So I would say that a local SATA disk with NTFS will almost always outperform the same SATA disk over network, with iSCSI as an additional protocol.
Was this by any chance while Windows Server 2025 was stil a very new product?
Veeam has always been very stable for me and every colleague that I ever talked about it with…
Sorry misunderstood you there. You meant, that Veeam won’t be able to compress empty parts of blocks on the ZFS level which is correct of course - I was thinking about the blockdevice as Veeam sees it.
That’s correct but I’d still not call it virtualization. Of course adding abstraction layers rarely improves raw performance but mostly adds features and or comforts…
It would probably be worth trying the performance with SMB instead of iSCSI but I’d be (pleasantly) surprised if the performance was much better.
The additional benefit to using SMB is that you can let your pool reach 80-90% capacity without too much concern however with iSCSI that reduces to 50%.
).