Extremely slow scrub. Is that normal?

Does the first arrow point to when the scrub started? It’s hard to tell with the difference in time zones.

Is the drop when it reached 50%?

What is the scale? You cropped out the speeds.

At 5 MiB/s…

Those 5 MiB/s could be the “floor” you’re seeing for 2 hours at a time in between spikes.

Interesting log. I can see the following, however when doing a tail it indeed very slow to appear.

1785247369   ffff8be859683300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 4177 blocks of Rust in 10843ms (1 os's, 4070 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785247369   ffff8be859683300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for Rust txg 13424013; bm=2233/352704/0/0
1785247375   ffff8be859683300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 2233 (Rust/photos/immich) with min=1 max=13419638; suspending=1
1785247380   ffff8be859683300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 2873 blocks of Rust in 10754ms (1 os's, 2800 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785247380   ffff8be859683300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for Rust txg 13424014; bm=2233/352768/0/0
1785247386   ffff8be859683300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 2233 (Rust/photos/immich) with min=1 max=13419638; suspending=1
1785247391   ffff8be859683300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 5204 blocks of Rust in 10815ms (1 os's, 5086 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785247391   ffff8be859683300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for Rust txg 13424015; bm=2233/352864/0/0
1785247398   ffff8be859683300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 2233 (Rust/photos/immich) with min=1 max=13419638; suspending=1
1785247402   ffff8be859683300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 1866 blocks of Rust in 11055ms (1 os's, 1784 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785247402   ffff8be859683300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for Rust txg 13424016; bm=2233/352960/0/0
1785247410   ffff8be859683300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 2233 (Rust/photos/immich) with min=1 max=13419638; suspending=1
1785247414   ffff8be859683300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 1851 blocks of Rust in 11970ms (1 os's, 1784 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785247414   ffff8be859683300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for Rust txg 13424017; bm=2233/353025/0/0
1785247422   ffff8be859683300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 2233 (Rust/photos/immich) with min=1 max=13419638; suspending=1
1785247426   ffff8be859683300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 3423 blocks of Rust in 12166ms (1 os's, 3308 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785247426   ffff8be859683300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for Rust txg 13424018; bm=2233/353153/0/0
1785247432   ffff8be859683300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 2233 (Rust/photos/immich) with min=1 max=13419638; suspending=1
1785247437   ffff8be859683300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 2105 blocks of Rust in 10613ms (1 os's, 2038 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785247437   ffff8be859683300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for Rust txg 13424019; bm=2233/353216/0/0
1785247444   ffff8be859683300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 2233 (Rust/photos/immich) with min=1 max=13419638; suspending=1
1785247449   ffff8be859683300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 2890 blocks of Rust in 12035ms (1 os's, 2783 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785247449   ffff8be859683300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for Rust txg 13424020; bm=2233/353312/0/0
1785247457   ffff8be859683300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 2233 (Rust/photos/immich) with min=1 max=13419638; suspending=1
1785247461   ffff8be859683300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 3686 blocks of Rust in 12051ms (1 os's, 3562 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785247461   ffff8be859683300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for Rust txg 13424021; bm=2233/353440/0/0

I was kind of expecting something related to Immich with all the pictures.

The first green arrow shows when it started doing someting, it stayed at 0% for around 1h.
The second green arrow was at around 45% when it stopped going fast.
After that the speed was more a 5MB average.

A lot of suspending in that log. Not sure if this is normal behavior for a scrub during a period of a pool that isn’t being actively used.

@HoneyBadger or others might know more of what is to be expected and what might signify an underlying problem.

Does zpool events -v Rust show anything? You’ll have to limit it to today.

The entire scrub fell off a cliff and is now crawling, just as you observed initially. I don’t believe it has do to with your media files or datasets. I wouldn’t put that on media and photos. A block is a block to ZFS. 128K or 1M, a block is a block.

Here are the events. Not so many overall. Here is an example as I cannot attach files

Jul 28 2026 20:01:44.579571721 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Rust"
        pool_guid = 0x572cea0d9a6b11d6
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "truenas"
        history_dsname = "Rust/backups/apps-data"
        history_internal_str = "readonly=1"
        history_internal_name = "set"
        history_dsid = 0x2209
        history_txg = 0xccd30c
        history_time = 0x6a689a28
        time = 0x6a689a28 0x228b9009 
        eid = 0x14b

Jul 28 2026 20:02:16.746059282 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Rust"
        pool_guid = 0x572cea0d9a6b11d6
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "truenas"
        history_dsname = "Rust/backups/apps-data/syncthing/%recv"
        history_internal_str = " "
        history_internal_name = "receive"
        history_dsid = 0x1b90
        history_txg = 0xccd310
        history_time = 0x6a689a48
        time = 0x6a689a48 0x2c77f612 
        eid = 0x14c

Jul 28 2026 20:02:33.027269142 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Rust"
        pool_guid = 0x572cea0d9a6b11d6
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "truenas"
        history_dsname = "Rust/backups/apps-data/syncthing"
        history_internal_str = "mountpoint=/"
        history_internal_name = "inherit"
        history_dsid = 0x230c
        history_txg = 0xccd312
        history_time = 0x6a689a59
        time = 0x6a689a59 0x1a01816 
        eid = 0x14d

Jul 28 2026 20:02:44.409318576 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Rust"
        pool_guid = 0x572cea0d9a6b11d6
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "truenas"
        history_dsname = "Rust/backups/apps-data/syncthing/%recv"
        history_internal_str = "snap=apps-data-2026-07-28_20-00"
        history_internal_name = "finish receiving"
        history_dsid = 0x1b90
        history_txg = 0xccd314
        history_time = 0x6a689a64
        time = 0x6a689a64 0x1865b4b0 
        eid = 0x14e

Jul 28 2026 20:02:44.409318576 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Rust"
        pool_guid = 0x572cea0d9a6b11d6
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "truenas"
        history_dsname = "Rust/backups/apps-data/syncthing/%recv"
        history_internal_str = "parent=syncthing"
        history_internal_name = "clone swap"
        history_dsid = 0x1b90
        history_txg = 0xccd314
        history_time = 0x6a689a64
        time = 0x6a689a64 0x1865b4b0 
        eid = 0x14f

Jul 28 2026 20:02:44.409318576 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Rust"
        pool_guid = 0x572cea0d9a6b11d6
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "truenas"
        history_dsname = "Rust/backups/apps-data/syncthing@apps-data-2026-07-28_20-00"
        history_internal_str = " "
        history_internal_name = "snapshot"
        history_dsid = 0x1c37
        history_txg = 0xccd314
        history_time = 0x6a689a64
        time = 0x6a689a64 0x1865b4b0 
        eid = 0x150

Jul 28 2026 20:02:44.421316520 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Rust"
        pool_guid = 0x572cea0d9a6b11d6
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "truenas"
        history_dsname = "Rust/backups/apps-data/syncthing/%recv"
        history_internal_str = "(bptree, mintxg=13421800)"
        history_internal_name = "destroy"
        history_dsid = 0x1b90
        history_txg = 0xccd314
        history_time = 0x6a689a64
        time = 0x6a689a64 0x191cc7a8 
        eid = 0x151

There is definitely a spike every 2h, however I am not sure if it related to the scrub. It lasts around 2min. The disk went suddenly quiet haha.

Are those the latest events? “receive”? Is this pool a backup target? Were any replications/receives started?

What about near the time the drop happened?

It’s getting to the point where you might want to cancel the scrub, let things settle, then start a new scrub that you can immediately assess if it repeats the same pattern.

Make sure no other tasks are running. Replications, receives, smart tests, library scans, nothing.

Here the start. Nothing in the events at the start or end of the spikes.

Jul 28 2026 08:59:38.830437392 sysevent.fs.zfs.scrub_start
        version = 0x0
        class = "sysevent.fs.zfs.scrub_start"
        pool = "Rust"
        pool_guid = 0x572cea0d9a6b11d6
        pool_state = 0x0
        pool_context = 0x0
        time = 0x6a67fefa 0x317f7810 
        eid = 0x117

Jul 28 2026 08:59:38.830437392 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Rust"
        pool_guid = 0x572cea0d9a6b11d6
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "truenas"
        history_internal_str = "func=1 mintxg=0 maxtxg=13419638"
        history_internal_name = "scan setup"
        history_txg = 0xccc476
        history_time = 0x6a67fefa
        time = 0x6a67fefa 0x317f7810 
        eid = 0x118

I already did that on Sunday and went this rabbit hole on checking a lot of things haha.
So far the pattern looks similar. I am curious if tomorrow it will still be in the same ballpark or not.

There’s your problem. Probably running your server in a humid environment.

Maybe I missed it, but how are these drives connected? Directly to motherboard? Through an HBA? Something less kosher?

Also, any chance we can get the full output of smartctl -a instead of just the ‘passed’?

Of course. I put sda, they all look the same

root@truenas[/home/admin]# sudo smartctl -a /dev/sda
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.95-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Ultrastar DC HC550
Device Model:     WDC  WUH721816ALE6L4
Serial Number:    2BKBMDJN
LU WWN Device Id: 5 000cca 295ef8f6f
Firmware Version: PCGNW680
User Capacity:    16,000,900,661,248 bytes [16.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-4 published, ANSI INCITS 529-2018
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Jul 28 22:45:28 2026 HKT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84)	Offline data collection activity
					was suspended by an interrupting command from host.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(  101) seconds.
Offline data collection
capabilities: 			 (0x5b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 (1867) minutes.
SCT capabilities: 	       (0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   001    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   149   149   054    Pre-fail  Offline      -       44
  3 Spin_Up_Time            0x0007   083   083   001    Pre-fail  Always       -       355 (Average 357)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       16
  5 Reallocated_Sector_Ct   0x0033   100   100   001    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   001    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   140   140   020    Pre-fail  Offline      -       15
  9 Power_On_Hours          0x0012   098   098   000    Old_age   Always       -       19289
 10 Spin_Retry_Count        0x0013   100   100   001    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       16
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       830
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       830
194 Temperature_Celsius     0x0002   055   055   000    Old_age   Always       -       39 (Min/Max 25/55)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   100   100   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     19269         -
# 2  Extended offline    Completed without error       00%     19264         -
# 3  Short offline       Completed without error       00%     19228         -
# 4  Short offline       Completed without error       00%     19197         -
# 5  Short offline       Completed without error       00%     19173         -
# 6  Short offline       Completed without error       00%     19149         -
# 7  Short offline       Completed without error       00%     19126         -
# 8  Short offline       Completed without error       00%     19101         -
# 9  Short offline       Completed without error       00%     19077         -
#10  Short offline       Completed without error       00%     19053         -
#11  Short offline       Completed without error       00%     19029         -
#12  Short offline       Completed without error       00%     19005         -
#13  Short offline       Completed without error       00%     18981         -
#14  Short offline       Completed without error       00%     18957         -
#15  Short offline       Completed without error       00%     18933         -
#16  Short offline       Completed without error       00%     18909         -
#17  Short offline       Completed without error       00%     18885         -
#18  Short offline       Completed without error       00%     18861         -
#19  Short offline       Completed without error       00%     18837         -
#20  Short offline       Completed without error       00%     18813         -
#21  Short offline       Completed without error       00%     18789         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The above only provides legacy SMART information - try 'smartctl -x' for more

And yes directly to the motherboard.

As long as those fields are identical for each drive… Yeah those are looking like incredibly healthy drives.

Uhh… what other stupid things can I think of? No chance you got something silly like 3 meter long sata cables? :stuck_out_tongue: Usually a scrub could give way to something else accessing the drives, but it doesn’t really look like that. No chance you have a custom spin down script or any kind of esoteric power savings enabled?

This last part will be a long shot because it’d normally cause the system to hang instead of a scrub failure, but did you do the c-state changes in bios considering you have a Ryzen 3600 & there has been past recorded wonkyness when the cpu goes into low power state?

That is good to know :wink: I tried to move all the noise on the SSD to protect the HDD.

Haha, no they are normal size.
Nothing is really running right now on them. I also disabled Syncthing and any other replications.
I do not remember playing with the power settings.
What c-state change should I do? Was not aware of wonkiness, so it should be the default on my ASRock X470D4U

Something like C6 state disabled and Global… power state something to not whatever the lowest eco mode? Its been a few years :frowning:

Once again, this is a bit of a long shot, there used to be issues where low power states would cause the system to crash. I have maybe 8% hope that it’d fix the issue at hand.

impatient. The rest does not matter. He/she manually stopped it.

While I agree, we have no idea about the number of files and fragmentation. Also RAIDZ1.

32* 4k blocks are not the same as the same as a single 128k block.

Semi offtopic: bevor further hunt down this issue and waste any time, you @Oli realize that running RAIDZ1 with such huge disk (and resilver times) is very risky?

I didn’t use enough verbiage. A block is a block. Doesn’t matter if it is (a) 128K (block) or (a) 1M (block). Wasn’t referring to files.

After 18 hours later and only a couple TB scrubbed, after the speeds suddenly dropped from a cliff and never returned. I wouldn’t call that impatient. :wink:

OP’s intuition was correct. Something is not right.