Hi everybody,
I am currently trying to split my big “NasStorage” dataset up into smaller child datasets.
The reason for this is that it contains data of various types and from/ for various users, and I want to specify the permissions and quotas more granularly.
I first tried to move the files using the regular mv command, but this took forever with no indication of progress, so I aborted it and tried it again using the in-built Midnight Commander.
This showed me that the move (or rather a copy, because apparently inter-dataset moves copy the files from one file system to another) is just at around 6-8 MB/s.
I think this is way slower than expected, since my pool is on a RAIDZ1 array, consisting of 4 SSDs (regular SATA ones).
I think something around 150-500 MB/s should be realistic, right?
I also tried copying the files via Dolphin and a SMB share, but this is even slower, because I think it doesn’t make use of the Server-Side-Copy feature of SAMBA.
Any ideas, what’s going on here?
Edit:
I also now tried to mount the Samba shares manually using CIFS
sudo mount --mkdir -t cifs //10.26.3.10/NasStorage /mnt/NasStorage -o credentials=/etc/samba/credentials/share,workgroup=workgroup,iocharset=utf8,uid=1000,gid=1000
In the mounted share I can move files almost instantly when they are in the same filesystem/ dataset (as expected).
When they are not, the “move” operation is not as slow as with the Dolphin integration anymore (and uses no network bandwith, so probably Server-Side-Copy is used), but it’s still just 8-10 MB/s.
So basically the same result as with mc or mv.
From a Linux perspective, moving files from one dataset to another (even in the same pool) is like moving a file from one file system to another because with ZFS each dataset is separately mounted.
Unfortunately ZFS is unable to tell Linux that it can do something clever.
I’m not an expert but I see the is a quota applied to the target dataset and I remember reading that quota can lead to lower performance then expected.
Someone with more knowledge about it can correct me if I’m wrong.
What speed do you see when copying on a target dataset without quota or from a source outside the same pool? Do you see the same slow speed or is it normal?
Okay maybe I was a bit too early with my last response.
I left the process active during the night (this time using rsync) and noticed that it only transferred about 350 GB.
The transfer speed dropped again to around 10 MB/s.
Looking at the TrueNAS stats (just one drive), it looks even worse:
Unfortunately, mc does not use syscalls to copy that allow reflink copies. I found this quite disappointing, since, like you, I occasionally would like to move files around with mc. It looks like there might be a way to make mc use an external command to copy, but I didn’t feel like going down that path.
If you want to test, cp should use --reflink=auto by default, but you could specify it and try copying that way. If the two datasets are in the same pool and don’t have different encryption keys then it should be faster.
As you can see, there were just 10 files to copy and this time definitely no hard links, which was the case with the backintime folder.
Still it took around 140 seconds to copy this folder, so around 6.8 MiB/s which is incredibly slow.
Additionally, the whole system slowed down significantly.
Something must be going wrong here…
Btw: TrueNas still tells me that I have some user/ group quotas for the dataset
root@truenas:/mnt/data/NasStorage# df -Th Filme/
Filesystem Type Size Used Avail Use% Mounted on
data/NasStorage zfs 2.5T 1.8T 694G 73% /mnt/data/NasStorage
root@truenas:/mnt/data/NasStorage# df -Th Filmarchiv/
Filesystem Type Size Used Avail Use% Mounted on
data/NasStorage/Filmarchiv zfs 1.1T 357G 694G 34% /mnt/data/NasStorage/Filmarchiv
Edit I just tried to copy (!) the same directory to /mnt/data/NasStorage, so on the same file system, and it was almost instantly (not measurable with my stopwatch).
there are various things that can make ZFS do a copy instead of reflink - I’m not clear myself on all the variations, and I believe it has gotten more flexible over time. one possible thing is copying from a larger recordsize to a smaller recordsize might make it do a full copy - kind of makes sense, since it would have to repack the data to the smaller recordsize, which is incongruent with a “thin” copy. I haven’t tested that yet, but wanted to throw that out there.
I don’t quite understand what this means for my case tbh.
Since the copy process is between two file systems (the reason why I wanted to do this in the first place, was to split my datasets into smaller chunks), I totally expect TrueNAS to do a copy.
The only issue that I have is that it’s so slow.
I mean, if I copy a directory of similar size on my workstation from the HDD to the SSD (ext4 both), it takes like 10 seconds at max and the speed is around 180 MB/s (my HDD’s read speed is probably the bottleneck).
I noticed that the copy speed varies a lot, even tho the directory consist almost exclusively of files larger than 1GB.
This can also be seen in the statistics, where some transfer speed spikes go to hundreds of MiB/s but most of the time it remains much lower:
If I read correctly, raidz1 write performance is limited by the slowest disk and looking at the spec for our SDD, the max performance for sustained write for those DRAM-Less TLC drive is around 145 MB/s (after SLC cache fill up) with dip into the low digit. If the disks, each at their time, slowdown a lot, it could bring the pool speed down.
Techpowerup review (This is assuming that they did not change the NAND to QLC without changing the model number)
I don’t think this is the full reason of the slowness, but it can set some expectation. Unless raidz1 give increased write performance, the max you can expect from your pool would be 145 MB/s. Reading from the same disks will affect the write speed but I’m not sure how significantly. My memory of single SATA SSD transfer is failing and I’m more use to multiple NVMEs
If you can, I would try different copy scenario to isolate the variable. In big sequential, even a single HDD reading would give you over 100MB/s.
Edit: there is at least 2 hardware versions, both TLC SSD Databse
Interesting to know, but even with the SLC cache being filled up (which I think shouldn’t be the case, since I wasn’t copying more than 660 GB), the performance should be much higher.
However, I have now decided to copy my movies (Filme) folder with the few large files using the regular mv command, and rename the source dataset (NasStorage) so that I can use it as the new Backup directory without having to move/ copy all the files and hardlinks.
