Rule #1: A woman is always right.
Rule #2: If she makes a mistake, refer to rule #1.
It would be surprising to see a system with a comfortable 64 GB RAM for 6 TB of data chocking on metadata. So this must be the pool going straight to the Circle of Molasses of IOPS Hell
Raidz is inefficient at storing small files. If the data is not expected to grow quickly and massively, the best advice is to rebuild as a stripe of mirrors. More drives (possibly smaller) would be even better.
I think people overestimate RAIDZ1. The good sequential read speeds makes them ignore that the IOPS are very poor (rougly that of a single disk).
What also could be a factor (but that is way above my ZFS performance paygrade) is that RAIDZ does not read parity data in normal operation. But for scrubs, it has to read it.
The other thing I do actually know is that people overestimate RAIDZ efficiency.
I don’t want to get offtopic, but I think it might be relevant for you OP.
People expect RAIDZ1 to behave like RAID 5. But it behaves fastly different!
You think that you will get 75% storage and that a mirror would give you 50%.
But you won’t get 75%. So the storage downgrade to a mirror might be smaller than you think.
You only get 75% if the pool geometry is optimal AND there is no padding needed.
That is not always the case. Here is how that looks for your pool:
File size you try to save
How the pool handles 4k sectors of your drive
storage efficiency
4k
1 parity, 1 data
50%
8k
1 parity, 2 data, 1 padding
50%
12k
1 parity, 3 data
75%
64k
6 stripes needed. First 5 stripes: 1 parity, 3 data. Last stripe: 1 parity, 1 data
I am in the process of switching to a RAID10.
In addition, I did some research yesterday about ARC and L2ARC and I wondering if I should also modify the `zfs_arc_meta_balance` for ARC and adding a L2ARC only for the metadata.
64 GB RAM is probably sufficient to hold your metadata… but would be regarded as “too small” to add a L2ARC by old standards.
Rebuild, put the system back into production and after a few days of real use run arc_summary on the CLI: That will inform you whether a L2ARC would be helpful.