MacOS slow SMB for large “file”

TrueNAS General
, ,
1

Hi, I’ve just got a new Mac Mini M4. My old one was over a decade old and now very very slow cough

The new Mac had a lot smaller hard drive as now I’ve got the TrueNas server (ElectricEel-24.10.1 ) I don’t need as much local storage. I thought.

My problem is trying to move some old Apple Aperture files for us to the server - they’re about 60-70GB. If I try to copy the Aperture file as is, it’s saying it’s going to take 20 days to move them, and at the speed it’s going I can believe it.

If I zip up the same files, which end up still being the same size, I can move them to the server in a couple of minutes.

I suspect it’s something to do with the fact that these Mac files are really a folder with loads of sub folders and files inside in reality, but the staggering difference in times to copy doesn’t seem right.

2

IMO likely to be caused by synchronous writes. Make sure the dataset you are writing to has sync=Disabled in advanced properties.

3

I have already tried that and it made no difference.

4

Lots of very small files will always have a very high price penalty when it comes to performance (although 20 days does seem excessive). It took me about 4 hours to download a 7.4GB folder from my corporate Nextcloud instance the other day as it contained 5004 small files however as a test I zipped it uploaded it and downloaded again and it took about 5mins.

PS: My guess is that it wouldn’t really have taken 20 days.

2 Likes
5

And that’s the thing - zipped up, it zipped along in a couple of minutes. But zipped up I can’t do anything with it! I certainly can’t access it to get to photos contained in it. I don’t really want to have to faff around with a USB drive at the side of the Mac Mini.

6

So is this a common situation where you need to move 60-70GB of Aperture files from your local computer to your TrueNAS?

Im not very familiar with Aperture but could you not have your library point to your TrueNAS and then when you want to add photos you just simply open Aperture and import directly to the NAS?

7

Perhaps having something like SyncThing or Nextcloud would work better for you where you could save your data to a local directory (or external USB) and in the background SyncThing or Nextcloud will sync this data to your TrueNAS?

That way you get all the performance benefits of having the data local but also the data is synced to the safer longterm device i.e. TrueNAS.

8

That. SMB is really inefficient for picking the metadat for all small files inside the bundle. It would however speed up when moving the bigger data files in there.

9

Looks like I’ll just have to wait it out, or just put the zip version there. They’re being archived, as Aperture is now long dead, but I want to be able to access them when necessary. I figure there’s no easy way of copying the zip version there and then unzipping it?

10

I would use rsync to sort this. Here is an example script that I use to sync my video files from a local drive on my Mac to the TrueNAS system which is running rsycnd. I am using the “rsyncd” App on the TrueNAS system. If you want to do this without installing the rsyncd services, then mount the SMB share locally on the Mac and replace the “DST=bla” with the correct path to the mount (do not include the trailing /).

You should run this the 1st time with like this “./script test” which just does a dry-run and tells you want the rsync command will try to do without doing anything.

#/bin/bash
#PUT

SRC=/Volumes/data1/Media/Videos/
DST=rsync://barrel:30026/videos
EXCLUDE_FILE=rsync-exclude

if [ "$1" == "test" ]; then
    RSYNC_START="/opt/homebrew/bin/rsync --dry-run"
else
    RSYNC_START="/opt/homebrew/bin/rsync"
fi

if [ ! -d $SRC ]; then
    echo "Local directory is not there. Please fix and try again"
else
    $RSYNC_START \
    --iconv=utf-8-mac,utf-8 \
    --force \
    --inplace \
    --size-only \
    --no-perms \
    --no-owner \
    --no-group \
    --omit-dir-times \
    --delete \
    --progress \
    --stats \
    --recursive \
    --exclude-from=$EXCLUDE_FILE \
    $SRC \
    $DST
fi
11

SMB is notortiously horrible with transferring large numbers of small files, whether on ZFS or EXT4 or APFS.

And worse, Mac OS’ implementation of SMB (which is NOT Samba mostly because Apple doesn’t like GPLv3) is notoriously even more inefficient at SMB overall. This is particularly an issue when using SMB via the Finder (that is, mounting an SMB share and working with it with the OS’ built-in GUI tools).

I’ve been researching this issue the last few days and just saw a recomendation for using Filebrowser Pro as an SMB client. I haven’t tested it yet, but it’s supposed to be much better.

Separately, stopping Mac OS from creating .DS_Store files in SMB share directories should speed up Finder noticeably.

If you do decide to zip the files, move them, and unzip them once they’re on the server, use the TrueNAS web GUI shell to run a local unzip command in the CLI. Running a zip command from the Mac on a file on a mounted SMB share will take actual hours. Ask me how I know.

Thanks for the recommendation on using rsync, @sah . I’ll have to experiment with this, as well.

Do you need to run rsyncd on the TrueNAS server to get the best performance, or does running rsync with a mounted SMB share work just as well? If it does, that’s … a bit confusing. What’s going on there? Does rsync somehow bypass the Mac’s builtin SMB client enough to avoid the sucky parts of it?

12

I run rsynd on the system. You can install it from App library. It is pretty simple to setup.