Resilver too slow

TrueNAS General
1

Hi, in a TrueNAS 12.0-U4.1 with 99% used space (40Gb of 3.5tb). I’m starting the procedure of replacement of each disk from a RAIDZ1 composed of 3x2tb to 3x4tb and it took 48 hours to resilver. But, and this is the main issue, the system is unusable while resilvering, it has one VM that is so slow that the users cannot do anything.

This systems has an Intel Core i5-10400F with 16gb of RAM and a 128 gb SSD drive for boot, apart of the pool mentioned on the 1nst paragraph.

The question is, what can I do to improve the performance, or at least make it slow but usable while resilver the next 2 drives?

I could add more ram and NVME drives for cache if needed, but I don’t know if that will let run faster.

3

Pools get slower as they get fuller. If it’s 99% full, I don’t even want to think about how slow it must be.

Best to just get it over with. Maybe once you’re done with a second drive you can replicate the VM zvol over to a temporary mirrored pool on two of those 2TiB drives so the resilver doesn’t affect performance (as much).

If you’re going to add SSDs/NVMes, seeing as this seems to be a pretty small system, why not run the VM on a pair of mirrored SSDs? It will have the same amount of redundancy and be considerably faster than how you’re currently running it (mirrors are recommended for block-storage in any case).

1 Like
4

You do nothing and hope the pool doesn’t die while you resilver.

99 % is cutting it really close, you do not want to fill ZFS completely.

5 Likes
5

I moved the VM to another server, now users are connected to it. The files (this system uses thousands of files from the filesystem) are now serverved via NFS and the Resilvering continues.

The system is perfectly usable atm. As @chuck32 mention, tomorrow we’ll replace the 2nd disk and when finish the 3rd.

6

Best of luck.
For future reference, try to keep your pool under 80% capacity for best performance.

7

For block storage, the recommendation is: Mirrors and under 50%. 99% full raidz1 is bound to be horrible for zvol.

These 4 TB drives aren’t SMR, I hope?

1 Like
8

No, according to this CMR and SMR Hard Drives | Seagate US those IronWolf are CMR.

1 Like
9

Good. So the 48 hours to resilver are “just” ZFS sucking hard with a 99% full raidz pool.

I’m still flabbergasted that 4 TB spinners are still being used as new storage media when SSDs are readily available in such low capacities.

10

Probably because they’re about 3x the cost if buying new

11

Yes, also because I’ve read SSDs could die and loss everything. I don’t know if this is an issue nowadays.

12

And HDDs couldn’t do that :wink: ?

2 Likes
13

Much less of an issue these days. Samsung’s 4TB SSDs are rated for something stupid like 2500TBW (though I’m sure there are many here who probably do that in a year…)