Hey. I’m using Veeam B&R 12 to backup few Windows VMs to various backup repositories. Today I added TrueNAS as a repo and did some tests.
What I got is backups taking almost twice as much space as Veeam says it transferred, which makes no sense to me.
I’m using a 2 disk mirror for backup pool, encrypted and uncompressed. When I turned compression on (zstd) the numbers almost halved, so in that regard they became somewhat normal. But what is not normal is that there is 2x compression being done on files that Veeam already compresses, so something is still weird.
I also tried with unencrypted pool, with different recordsizes, etc., no difference.
I have the same backups being done on Proxmox dataset, and there is no problem there, numbers look fine and completely different than on TrueNAS dataset.
So if anyone got any ideas what might be going on… do tell.
TrueNAS SCALE version: 24.10-RC.2
Also developer mode is enabled, so Veeam Agent could be installed.
No, no snapshots at all. I’m talking about the file sizes.
I can dig up exact outputs.
This is from when zstd compression was enabled (when compression is disabled the size came out as 517G)
root@truenas[/mnt/hdd/veeambackups/Test1]# ll -lsh
total 274G
25K drwxr-xr-x 2 veeam_user 5 Oct 20 10:33 ./
8.5K drwxr-xr-x 4 veeam_user 3 Oct 20 09:26 ../
4.5K -rw-r--r-- 1 root 209 Oct 20 10:33 .veeam.1.lock
274G -rw-r--r-- 1 veeam_user 517G Oct 20 10:33 EDUS-VSRV-DC.6146a784-57bd-4434-9333-1dd2e141D2024-10-20T092624_3CC1.vbk
8.5K -rw-r--r-- 1 veeam_user 17K Oct 20 10:33 EDUS-VSRV-DC_4BD39.vbm
root@truenas[/mnt/hdd/veeambackups/Test1]# zfs get all hdd/veeambackups | grep compress
hdd/veeambackups compressratio 1.88x -
hdd/veeambackups compression zstd inherited from hdd
hdd/veeambackups refcompressratio 1.88x -
And this is same VM from Proxmox (backup from yesterday):
270G -rw-r--r-- 1 veeam veeam 300G Oct 19 21:40 EDUS-VSRV-DC.6146a784-57bd-4434-9333-1dd2e141D2024-10-19T213924_A4E6.vbk
There is no way there can be 517G of data because VM itself has 400G drive, and has 300 something GB data on it. Also Veeam compresses files already. In the first example it reports ‘Transferred: 267.8 GB’. So something is off.
zstd is good, but not THAT good. Especially not on data that is already compressed. Also, as you said yourself, it doesn’t explain where 517G came from.
The point is if compression=off then a ~300 GB Windows Server gets backed up by Veeam which compresses the backup file to ~270 GB and transfers it to TrueNAS and on TrueNAS ls -l shows a 517 GB file.
If compression is on it looks like at least file size is somewhat normal, but it still doesn’t explain where 200% compression on 517 GB file came from. Or more precisely where did 517 GB itself come from.
Just a heads up, I don’t know how Veeam writes data and these are just wild guess on my part.
Last time I used it, you were able to set how big the blocks should be.
Maybe the default is some very small number that creates a lot of metadata overhead?
That is what I suspected. Because my guess would be that even with lzo you would achieve similar compression results. Why that?
Well this is just a wild guess, but maybe Veeam does some strange writing or maybe there is a lot of metadata overhead or something.
I have read that you should never ever have compression disabled. Even for none compressable data like zip files, you still gain the metadata compression and lzo costs basically nothing.
Or sometimes it is just that ZFS is funky when it comes to pool usage and stuff like that.
ZFS has to create logical blocks of certain sizes. If your data is for example 65k, ZFS can’t use a 64k block and instead has to use a 128k block with 63k of zero padding. That takes up 128k + metadata. Compression can resolve this, because a zero padding compresses down to basically nothing, so it can store the 128k logical block in something like 17x4K = 68K physical blocks.
If your actual data is also compressible that’s even less blocks used.
This is of course a toy example, but a basically perfect 2:1 compression ratio on already compressed data looks like it might have something to do with this.
Edit: Note that the configured “record size” is a maximum. If the data is under half the size, a smaller block is used automatically. If it’s larger, two, three, etc. blocks are used, with the last one being zero padded. So a filesystem with 1M record size and 1.1M data would basically have the same issue where then 2x1M logical blocks are created and again compression has to be used.
Using smaller block sizes (especially for backups) also isn’t really advisable, as it leads to both excessive fragmentation and a lot more metadata usage
Not sure what the default for your SMB share is nowadays (it is a SMB share, right?), but according to this, it could be 512k.
Question for @Momi_V:
Assuming it is 512k. Does Veeam write 512k, which is four 128k logical blocks, but than has to add metadata and since ZFS does not allow for different sized recordsizes within the same file, there is anohter 128k block needed? So to store 512k from Veeam, it would take up 640k if compression is not enabled?
It’s not SMB share, it’s direct storage, with Veeam Agent. And recordsize and Veeam block sizes are both set to 1 MB. And I tested different values, doesn’t matter.
And again: I have Proxmox with zfs 2.2.6 which behaves totally different – normal, I’d say.
Could you zfs get all <pool/dataset> on both the Proxmox and TrueNAS pool?
Then we’d have an idea what might cause those changes. There’s conflicting information about whether PvE uses compression by default.
Also: Veam Block size of 1M - compression could very well come out to something like a worst case of having to store more than 512k, but less than 600k. That would cause the exact symptoms you are experiencing: A LOT of extra space used, but compression solves that even if the data should already be compressed.
Edit: testing different values doesn’t help, because the problem is the ratio between the application block sizes and what ZFS can work with.
If your Veeam compression comes out to 0.6 and uses a 2^n block size then no matter what record size you set ZFS to it will waste that space unless it’s significantly smaller (by like 4x or more) than the Veeam block size. And remember: changes to record size only affect newly written data.
