Extremely slow scrub. Is that normal?

TrueNAS General
1

Hi,

I am using the latest version of TrueNAS Community Edition (25.10.5 as the time of writing).
My setup is a RaidZ1 pool using 4 HDD Western Digital UltraStar DC HC550 16TB SATA.
The performance of each disk is around 250MB/s.
The pool contains:

  • A dataset used by Immich of around 1 TB.
  • A dataset of media/videos of around 1.6 TB.
  • A dataset of backups of 1.6 TB which contains a mix of proxmox backups, restic backups and some mix files including some photos (raw and jpg).

When the scrub is running it is doing something for around 2h before the percentage starts to climb.
It then runs fast (between 100 and 240MB/s) for 1h which covers around 45% of the total scrub (around noon). It estimates the remaining time to 3-4h.

However, it suddenly slows down to around 5MB/s. The remaining time just keeps increasing. The disk are constantly scratching, so something is definetly happening. It is now only at 52.5%, it spent 6h to do 7.5%.

The scrub was running on Sunday when I noticed this. After more than 18h it was stuck at 55%. I stopped it as I was wondering what was happening. In the meantime I have done a long smartctl test on each disk that came back clean, no errors at all. Disks are healthy. I rebooted as well in case.

I know I have a “large” dataset of pictures (not massive big files) but it feels like it is taken way longer than it should. The scrub is setup to be executed regularly, however I never paid attention how long it was lasting before (not sure if there is an history somewhere). Hearing the disks working hours scared me a bit as I do not want to shorten their lifespan too much doing this.

Is that an expected behaviour?

Let me know if there are any commands I need to run.

Thanks

2

Forgot to mention the system has a Ryzen 5 3600 (6C/12T) CPU with 64GB ECC (20Gb free at the moment)

3

Any logged errors or failed tests on the drives? Replace X with each drive ID.

smartctl -l error /dev/sdX

smartctl -l selftest /dev/sdX

Status of the pool:

zpool status -v MyPool

zpool list -o name,size,alloc,free,cap,expandsz,frag,freeing,leaked,health MyPool

Dataset info:

zfs list -r -t fs,vol -o space,recordsize,compression,encryption MyPool

If you don’t feel comfortable sharing certain names or info, you can redact them.

HBA, cables, bad RAM, and temperatures need to be ruled out. Any of these can be and have been the reason for poor performance.

1 Like
4

Sounds pretty normal to me. My two disk mirror, filled 90% at 14TB takes roughly 26h.

5

The OP’s scrub never finished. It stalled at 55% indefinitely.

6

7

Stalled. Stuck. Tomatoes. Tomatohs.

1 Like
8

Less than 5 TB of data on 7200 RPM drives shouldn’t take 18 hours just to get to 55% and stall.

9

Thanks for taking the time @winnielinnie

Here you go

root@truenas[/home/admin]# smartctl -l error /dev/sda

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.95-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

root@truenas[/home/admin]# smartctl -l error /dev/sdb

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.95-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

root@truenas[/home/admin]# smartctl -l error /dev/sdc

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.95-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

root@truenas[/home/admin]# smartctl -l error /dev/sdd
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.95-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged


root@truenas[/home/admin]# smartctl -l selftest /dev/sda
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.95-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     19269         -
# 2  Extended offline    Completed without error       00%     19264         -
# 3  Short offline       Completed without error       00%     19228         -
# 4  Short offline       Completed without error       00%     19197         -
# 5  Short offline       Completed without error       00%     19173         -

root@truenas[/home/admin]# smartctl -l selftest /dev/sdb
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.95-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     19269         -
# 2  Extended offline    Completed without error       00%     19264         -
# 3  Short offline       Completed without error       00%     19222         -
# 4  Short offline       Completed without error       00%     19197         -
# 5  Short offline       Completed without error       00%     19173         -

root@truenas[/home/admin]# smartctl -l selftest /dev/sdc
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.95-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     19269         -
# 2  Extended offline    Completed without error       00%     19264         -
# 3  Short offline       Completed without error       00%     19222         -
# 4  Short offline       Completed without error       00%     19197         -
# 5  Short offline       Completed without error       00%     19173         -

root@truenas[/home/admin]# smartctl -l selftest /dev/sdd
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.95-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     19269         -
# 2  Extended offline    Completed without error       00%     19264         -
# 3  Short offline       Completed without error       00%     19223         -
# 4  Short offline       Completed without error       00%     19197         -
# 5  Short offline       Completed without error       00%     19173         -


root@truenas[/home/admin]# zpool status -v Rust  
  pool: Rust
 state: ONLINE
  scan: scrub in progress since Tue Jul 28 08:59:38 2026
	3.10T / 5.76T scanned at 72.1M/s, 3.04T / 5.76T issued at 70.7M/s
	0B repaired, 52.74% done, 11:12:53 to go
config:

	NAME                                      STATE     READ WRITE CKSUM
	Rust                                      ONLINE       0     0     0
	  raidz1-0                                ONLINE       0     0     0
	    efc9e50e-c803-48d3-9507-255810634dba  ONLINE       0     0     0
	    ca7bf617-7ce7-40ea-b29e-1e449969b592  ONLINE       0     0     0
	    9498bb19-7fa6-49d9-ac2a-ef8a6f08ab39  ONLINE       0     0     0
	    d5185a56-18ab-4aca-8788-a88779b93868  ONLINE       0     0     0

errors: No known data errors

You can notice the time has increased to 11h now. I am at 52.74% after 9h.

root@truenas[/home/admin]# zpool list -o name,size,alloc,free,cap,expandsz,frag,freeing,leaked,health Rust  
NAME   SIZE  ALLOC   FREE    CAP  EXPANDSZ   FRAG  FREEING  LEAKED    HEALTH
Rust  58.2T  5.76T  52.4T     9%         -     7%        0       0    ONLINE


root@truenas[/home/admin]# zfs list -r -t fs,vol -o space,recordsize,compression,encryption Rust
NAME                              AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  RECSIZE  COMPRESS        ENCRYPTION
Rust                              38.0T  4.19T        0B    360K             0B      4.19T       1M  lz4             aes-256-gcm
Rust/backups                      38.0T  1.61T        0B    453K             0B      1.61T       1M  lz4             aes-256-gcm
Rust/backups/apps-data            38.0T  28.6G      600M   1011M             0B      27.1G       1M  lz4             aes-256-gcm
Rust/backups/apps-data/portainer  38.0T   575K        0B    575K             0B         0B       1M  lz4             aes-256-gcm
Rust/backups/apps-data/syncthing  38.0T  27.1G     50.9M   27.0G             0B         0B       1M  lz4             aes-256-gcm
Rust/backups/docker-pi            38.0T  13.3G        0B   13.3G             0B         0B       1M  lz4             aes-256-gcm
Rust/backups/docker-pve           38.0T  48.5G        0B   48.5G             0B         0B       1M  lz4             aes-256-gcm
Rust/backups/immich-db            38.0T  38.4G        0B   38.4G             0B         0B       1M  lz4             aes-256-gcm
Rust/backups/pbs-storage          38.0T   241G        0B    241G             0B         0B       1M  lz4             aes-256-gcm
Rust/backups/pve-router           38.0T   150G        0B    150G             0B         0B       1M  lz4             aes-256-gcm
Rust/backups/qnap-bck             38.0T  1.10T     34.9K   1.10T             0B         0B       1M  lz4             aes-256-gcm
Rust/backups/truenas              38.0T  6.61M     4.90M   1.70M             0B         0B       1M  lz4             aes-256-gcm
Rust/media                        38.0T  1.65T        0B    326K             0B      1.65T       1M  lz4             aes-256-gcm
Rust/media/books                  38.0T  1.25G        0B   1.25G             0B         0B       1M  lz4             aes-256-gcm
Rust/media/movies                 38.0T  1.02T        0B   1.02T             0B         0B       1M  lz4             aes-256-gcm
Rust/media/tvshows                38.0T   649G        0B    649G             0B         0B       1M  lz4             aes-256-gcm
Rust/photos                       38.0T   941G        0B    279K             0B       941G       1M  lz4             aes-256-gcm
Rust/photos/google                38.0T   166G        0B    166G             0B         0B       1M  lz4             aes-256-gcm
Rust/photos/immich                38.0T   774G        0B    774G             0B         0B       1M  lz4             aes-256-gcm
10

I do not have any HBA, the disks are directly connected to my AsRock MB.
I have no issues being able to transfer data at max speed, and the early stage of the scrub is quite fast.
It is warm where I live, so the disks during summer time are between 40 and 50 deg celsius..

11

Have these parameters been changed from their defaults?

cat /sys/module/zfs/parameters/zfs_scan_issue_strategy

cat /sys/module/zfs/parameters/zfs_scan_legacy
12

I do not remember playing with that

root@truenas[/home/admin]# cat /sys/module/zfs/parameters/zfs_scan_issue_strategy
0
root@truenas[/home/admin]# cat /sys/module/zfs/parameters/zfs_scan_legacy
0
13

Everything looks good.

I really don’t know why your scrub crawls and stalls at around 55%. Scrubbing 5 TB of data on 7200 RPM drives shouldn’t take that long.

Is there other I/O currently using the pool?

Other than that, you might have to check the logs for anything that might stand out.

14

I was hoping it was something obvious :frowning:
Nothing else is going on at the moment on the HDD.
Docker apps are running separately on a SSD pool.

Which specific logs should I check?

15

According to my expertise, the logs that will help troubleshoot this are found under… it’s this file that… remember when…

@HoneyBadger @Captain_Morgan Help! :sob:

:person_running:

1 Like
16

I will let it finish this time and will see how long it takes. I just hope it will not crawl like a snail like that untill 100% :smiley: It will take days otherwise…

17

Can finally add a picture

Screenshot From 2026-07-28 21-55-06-1
Screenshot From 2026-07-28 21-55-06-11276Ă—1719 161 KB

18

are you thinking of /proc/spl/kstat/zfs/dbgmsg ? :wink:

19

I might need to try that sometime.

What would @Oli look for? Not only is the scrub very slow, but his charts show long periods of idling on all drives during a scrub. It almost looks as if his drives idle for 2 hours between reads.

1 Like
20

The drive are constantly scratching and doing something. It looks very small as it is nothing compare to the first part.
Here a closer look at what is going on right now

Screenshot From 2026-07-28 22-03-25
Screenshot From 2026-07-28 22-03-254177Ă—363 167 KB