I’m trying to move a number of files between two datasets on the same pool.
I have a 10Gbit connection with the Pool managing 1GB/s Read/Write.
No matter whether I move over SMB share or over ssh with mv it takes about 5 minutes for 100 50mb files.
What is happening?
Can you provide more system info please?
Here are some screenshots of the configuration in question. If you need anything specific I can try to add that
I meant the system info as text. Unless screenshots are asked for, it’s better to provide your system info as a system specs list, including the hardware and ZFS setup.
CPU, RAM, Motherboard, drive models, how your drives are attached.
What is this mirror SLOG? Why is it a mirror? What type of devices is it made of? Why do you need a SLOG? Are you using sync=always on your datasets?
What recordsize are you using?
Do you mean that you log in to the server with SSH and do a local mv operation? You’re not using intermediary software?
I had a few similar concerns and questions as well. Not enough proper data has been shared to make any informed opinion.
Do you mean that you log in to the server with SSH and do a local
mvoperation? You’re not using intermediary software?
Correct. I navigate to /mnt/main/shared and use mv old-Media/Folder1 Media
It moves a single small file per couple seconds.
At least that is one of the options I tried besides SMB.
The thing is I could download the files at 1GB/s over SMB and Upload them to the other Dataset at 1GB/s over SMB. I just don’t have the local space to reasonably do this. And this shouldn’t be the fastest option.
Using Rsync (rsync -rt --inplace --remove-source-files with and without --no-owner --no-owner --no-groups) I can get to 100MB/s on move. Even this is ridiculous considering the Drives manage 1GB/s Read/Write over SMB.
I meant the system info as text. Unless screenshots are asked for, it’s better to provide your system info as a system specs list, including the hardware and ZFS setup.
CPU: Intel i7-9700k
Mainboard: Asus Code XI
RAM: G.Skill DIMM 64 GB DDR4-4400 (2x 32 GB)
Version: 25.04.01
Disks of affected pool: RAIDZ1 5 wide disks of Seagate Exos X18 18 TB, default SATA
Log VDEV with two 1TB Samsung 970 EVO, NVMe.
As to why: I had them lying around and the internet consensus seemed to be that a SLOG would be the best use.
Record size: 128k, because is was default
Sync: Standard, because is was default
If you need anything else, I can add it.
For the full list of shared
Sync*
Inherit (STANDARD)
Compression Level*
Inherit (LZ4)
Enable Atime*
Inherit (OFF)
ZFS Deduplication*
Inherit (OFF)
Case Sensitivity*
Insensitive
Checksum*
Inherit (ON)
Read-only*
Inherit (OFF)
Exec*
Inherit (ON)
Snapshot Directory
Invisible
Snapdev*
Inherit (HIDDEN)
Copies
1
Record Size*
Inherit (128 KiB)
ACL Type*
SMB/NFSv4
ACL Mode*
Restricted
Metadata (Special) Small Block Size*
Inherit (0)
Gee, this seems like a good case for Block Cloning!
Now as to how to enable BC, and then make sure the client can use it, sorry Block Cloning is just too new to me.
Background, (at least what I think I know):
Block Cloning is a new feature for file systems. Both Linux and MS-Window’s SMB seem to support it. Later ZFS added the feature.
Block Cloning acts like Hard Links, that are copied on write. So a simple file, even TBs in size, could be copied in less than a second. The only space used initially is for the directory entry, just like a Hard Link. ZFS Block Cloning even works across Dataset boundaries, (but not Pool boundaries).
In the context of moving a file / directory tree from one ZFS Dataset to another ZFS Dataset, the old way would require full copy. However, with Block Cloning a new reference to the file / directory tree would take very little time. Then, when all is good, simply erase the old path to the file / directory tree.
Now in theory, MS-Windows over SMB should support Block Cloning. But, I have very limited knowledge of how to make sure that is working. Or enable if not working.
I did some more testing:
This happens to at least use the full speed of the drive potentially slightly bottlenecked if it goes through the network, not sure.
I guess there is no solution to this problem.
The workaround is good enough for me.
While doing the local copy (mv or rsync) run the following in a terminal window (System → Shell or an SSH window), then repeat while doing the fast remote copy.
/usr/sbin/iostat -sxy 10 (first results will show after 10 seconds and then repeat every 10 seconds)
This shows per disk device statistics. Look at the %util column. That is the % utilization of the given drive. If it is between 90 and 100 for the drives that are part of the zpool you are working with, then you are saturating the drives themselves (I doubt this is the case, but good to check).
Then run
/sbin/zpool iostat -yl 10 (first results will show after 10 seconds and then repeat every 10 seconds)
This shows per zpool statistics. The last 3 columns are the zpool waiting on an internal ZFS operation, they should be - unless one of those operations are ongoing).
The total_wait disk_wait syncq_wait asyncq_wait columns indicate the time spent waiting on disk, waiting on the sync queue and waiting on the async queue (as well as total time waiting).
Based on the above, and looking at them during the local copy (slow) vs. the remote copy (fast) can tell us where the problem is.
