Extremely slow scrub. Is that normal?

There is no DB direcly in there. The only container is for PBS and it saved on a SSD pool.

The backup dataset is holding backups from PBS (around 600k files from the chunks) and from Restic (smaller and I am planning to delete those).

Now, the dataset holding the Immich data contains:

  • 599,173 folders
  • 472,034 files

That folder was the one currently processed during the slow phases when I looked at the /proc/spl/kstat/zfs/dbgmsg log file.

Now, I would expect TrueNAS to be able to handle folders with images. How are you guys dealing with this?
Right now the setup is a follows:

  • Sync: STANDARD
  • Compression: Inherit (1.05x (LZ4))
  • Enable Atime: OFF
  • ZFS Deduplication: OFF
  • Case Sensitivity: ON
  • Record size: 1MB

I am happy to change the settings or the way the dataset is structured if that improves things.

:smirking_face:
that is IMHO evidence for my theory.

ZFS can handle many files. But there is no secret sauce that speeds up drives IO.

But special vdev can speed up metadata read and even smaller files.

Or getting a faster pool. 4 drives in a mirror.

If you end up being right about this, I will eat my hat… and I’m all out of hats.

If you end up being right about this, I will eat my hat… and I’m hatose intolerant.

If you end up being right about this, I will eat my hat… and we can all agree no one wants to see that.

Please don’t be right about this. :pleading_face:

Rule #1: A woman is always right.
Rule #2: If she makes a mistake, refer to rule #1.

It would be surprising to see a system with a comfortable 64 GB RAM for 6 TB of data chocking on metadata. So this must be the pool going straight to the Circle of Molasses of IOPS Hell :scream:

Raidz is inefficient at storing small files. If the data is not expected to grow quickly and massively, the best advice is to rebuild as a stripe of mirrors. More drives (possibly smaller) would be even better.

I think people overestimate RAIDZ1. The good sequential read speeds makes them ignore that the IOPS are very poor (rougly that of a single disk).
What also could be a factor (but that is way above my ZFS performance paygrade) is that RAIDZ does not read parity data in normal operation. But for scrubs, it has to read it.

The other thing I do actually know is that people overestimate RAIDZ efficiency.
I don’t want to get offtopic, but I think it might be relevant for you OP.

People expect RAIDZ1 to behave like RAID 5. But it behaves fastly different!

You think that you will get 75% storage and that a mirror would give you 50%.
But you won’t get 75%. So the storage downgrade to a mirror might be smaller than you think.

You only get 75% if the pool geometry is optimal AND there is no padding needed.
That is not always the case. Here is how that looks for your pool:

File size you try to save How the pool handles 4k sectors of your drive storage efficiency
4k 1 parity, 1 data 50%
8k 1 parity, 2 data, 1 padding 50%
12k 1 parity, 3 data 75%
64k 6 stripes needed. First 5 stripes: 1 parity, 3 data. Last stripe: 1 parity, 1 data 72.72%

If you are interested in the whole efficiency thing: The problem with RAIDZ

Thanks all for your feedback.

I am in the process of switching to a RAID10.
In addition, I did some research yesterday about ARC and L2ARC and I wondering if I should also modify the `zfs_arc_meta_balance` for ARC and adding a L2ARC only for the metadata.

@winnielinnie I looked at both of your topics, including that latest one ZFS "ARC" is getting smarter with version 2.2+ :smiley_face_emoji: | TrueNAS Community

Proper ZFS terminology is “stripe of mirrors” :wink:

64 GB RAM is probably sufficient to hold your metadata… but would be regarded as “too small” to add a L2ARC by old standards.
Rebuild, put the system back into production and after a few days of real use run arc_summary on the CLI: That will inform you whether a L2ARC would be helpful.