Extremely slow scrub. Is that normal?

Helps to look at the HW specs for this motherboard: https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U#Specifications

SATA/SAS Storage
PCH Built-in Storage 6 SATA3 6Gb/s (1 SATA DOM port), support RAID 0, 1, 10
Additional SATA Storage Controller
Additional SATA Controller Asmedia 1061: 2 SATA 6Gb/s

Check to see if your pool is connected to the Asmedia controller.

1785247369   ffff8be859683300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for Rust txg 13424013; bm=2233/352704/0/0
1785247375   ffff8be859683300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 2233 (Rust/photos/immich) with min=1 max=13419638; suspending=1
1785247380   ffff8be859683300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 2873 blocks of Rust in 10754ms (1 os's, 2800 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)

What’s the recordsize on the dataset Rust/photos/immich ?

It’s showing 2873 blocks (records) in 10754ms - so if it’s the default 128K, that means it checked ~368MB in that time.

suspending=1 is also expected, it means ZFS is deferring the scrubs due to the time they’re taking:

1785268430   ffff9496d8583200 dsl_scan.c:4600:dsl_scan_sync(): scan visited 847330 blocks of cobia-raidz2 in 19204ms (8 os's, 331190 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785268430   ffff9496d8583200 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for cobia-raidz2 txg 16125585; bm=4588/34/0/0
1785268435   ffff9496d8583200 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 4588 (cobia-raidz2/MainShare@auto-2026-07-05_00-00) with min=1 max=15730654; suspending=1
1785268442   ffff9496d8583200 dsl_scan.c:4600:dsl_scan_sync(): scan visited 7074232 blocks of cobia-raidz2 in 11681ms (1 os's, 5974995 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785268442   ffff9496d8583200 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for cobia-raidz2 txg 16125586; bm=4588/60648/0/28419
1785268447   ffff9496d8583200 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 4588 (cobia-raidz2/MainShare@auto-2026-07-05_00-00) with min=1 max=15730654; suspending=1
1785268461   ffff9496d8583200 dsl_scan.c:4600:dsl_scan_sync(): scan visited 2453487 blocks of cobia-raidz2 in 19090ms (1 os's, 238741 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785268461   ffff9496d8583200 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for cobia-raidz2 txg 16125587; bm=4588/63003/0/2760
1785268466   ffff9496d8583200 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 4588 (cobia-raidz2/MainShare@auto-2026-07-05_00-00) with min=1 max=15730654; suspending=1
1785268477   ffff9496d8583200 dsl_scan.c:4600:dsl_scan_sync(): scan visited 3686588 blocks of cobia-raidz2 in 16148ms (1 os's, 1668512 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785268477   ffff9496d8583200 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for cobia-raidz2 txg 16125588; bm=4588/79423/0/3700
1785268482   ffff9496d8583200 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 4588 (cobia-raidz2/MainShare@auto-2026-07-05_00-00) with min=1 max=15730654; suspending=1
1785268491   ffff9496d8583200 dsl_scan.c:4600:dsl_scan_sync(): scan visited 8182087 blocks of cobia-raidz2 in 13379ms (1 os's, 6619351 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785268491   ffff9496d8583200 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for cobia-raidz2 txg 16125589; bm=4588/108316/0/3634
1785268497   ffff9496d8583200 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 4588 (cobia-raidz2/MainShare@auto-2026-07-05_00-00) with min=1 max=15730654; suspending=1
1785268507   ffff9496d8583200 dsl_scan.c:4600:dsl_scan_sync(): scan visited 9337920 blocks of cobia-raidz2 in 16121ms (1 os's, 8349491 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785268507   ffff9496d8583200 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for cobia-raidz2 txg 16125590; bm=4588/146682/0/0
1785268512   ffff9496d8583200 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 4588 (cobia-raidz2/MainShare@auto-2026-07-05_00-00) with min=1 max=15730654; suspending=1
1785268516   ffff9496d8583200 dsl_scan.c:4600:dsl_scan_sync(): scan visited 3154809 blocks of cobia-raidz2 in 8634ms (1 os's, 1043922 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1785268516   ffff9496d8583200 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for cobia-raidz2 txg 16125591; bm=4588/151811/0/436

what does sudo iostat -x 1 look like in the %util column?

1M

Info posted here.

Sadly no, none are connected to the Asmedia.

By any chance, do you have the onboard SATA ports configured in RAID mode vs. AHCI mode in the BIOS?

Looks like 2h ago it ended processing fast again. It means the iostat is surely less relevant But here it is.

root@truenas[/home/admin]# sudo iostat -x 1
Linux 6.12.95-production+truenas (truenas) 	07/29/26 	_x86_64_	(12 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.39    0.04    0.73    3.70    0.00   95.15

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda            100.28   9165.25     0.02   0.02   10.66    91.40    4.36     98.68     0.00   0.11    4.26    22.62    0.00      0.00     0.00   0.00    0.00     0.00    1.63    8.90    1.10  37.83
sdb             99.96   9186.50     0.02   0.02   10.63    91.90    4.36     98.64     0.00   0.11    4.06    22.65    0.00      0.00     0.00   0.00    0.00     0.00    1.63    8.41    1.09  37.60
sdc             99.17   9174.51     0.02   0.02   10.84    92.51    4.36     98.69     0.00   0.11    4.33    22.63    0.00      0.00     0.00   0.00    0.00     0.00    1.63    9.10    1.11  37.99
sdd             99.99   9168.02     0.02   0.02   10.65    91.68    4.35     98.63     0.00   0.11    4.22    22.65    0.00      0.00     0.00   0.00    0.00     0.00    1.63    8.83    1.10  37.72


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.42    0.00    1.67    0.08    0.00   97.83

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda            305.00 210484.00     0.00   0.00    9.83   690.11    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.00  99.60
sdb            625.00 209928.00     0.00   0.00    0.96   335.88    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.60  36.80
sdc            301.00 209120.00     0.00   0.00    9.88   694.75    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.98  98.40
sdd            313.00 215616.00     0.00   0.00    9.58   688.87    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.00  99.60


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.09    0.00    1.92    1.09    0.00   95.91

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda            285.00 196476.00     0.00   0.00    9.89   689.39   38.00    432.00     0.00   0.00    2.45    11.37    0.00      0.00     0.00   0.00    0.00     0.00    2.00   10.00    2.93  98.80
sdb            531.00 180084.00     0.00   0.00    0.98   339.14   35.00    428.00     0.00   0.00    1.29    12.23    0.00      0.00     0.00   0.00    0.00     0.00    2.00    7.50    0.58  32.80
sdc            274.00 188180.00     0.00   0.00    9.87   686.79   34.00    428.00     0.00   0.00    1.41    12.59    0.00      0.00     0.00   0.00    0.00     0.00    2.00    8.50    2.77  92.40
sdd            275.00 190652.00     0.00   0.00    9.75   693.28   38.00    432.00     0.00   0.00    1.50    11.37    0.00      0.00     0.00   0.00    0.00     0.00    2.00   11.00    2.76  91.60


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    1.26    0.00    0.00   98.74

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda            298.00 203152.00     0.00   0.00    9.85   681.72    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.94  98.80
sdb            297.00 202128.00     0.00   0.00    9.90   680.57    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.94  99.20
sdc            309.00 210676.00     0.00   0.00    9.50   681.80    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.94  99.20
sdd            316.00 215124.00     0.00   0.00    9.30   680.77    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.94  98.80


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.08    0.00    1.59    0.08    0.00   98.25

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda            311.00 212480.00     0.00   0.00    9.65   683.22    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.00  99.20
sdb            306.00 209680.00     0.00   0.00    9.76   685.23    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.99 100.00
sdc            339.00 216864.00     0.00   0.00    8.74   639.72    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.96  97.20
sdd            580.00 213080.00     0.00   0.00    1.96   367.38    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    1.13  53.20


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.08    0.00    1.83    0.00    0.00   98.08

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda            311.00 210924.00     0.00   0.00    9.63   678.21    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.99 100.00
sdb            314.00 213032.00     0.00   0.00    9.53   678.45    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.99  98.40
sdc            629.00 213616.00     0.00   0.00    1.12   339.61    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.71  38.40
sdd            627.00 212948.00     0.00   0.00    0.95   339.63    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.60  35.60


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.08    0.08    0.84    0.00    0.00   98.99

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda            316.00 215480.00     0.00   0.00    9.46   681.90    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.99  99.60
sdb            322.00 217540.00     0.00   0.00    9.32   675.59    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.00  99.60
sdc            438.00 212752.00     0.00   0.00    5.59   485.74    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.45  84.80
sdd            638.00 217176.00     0.00   0.00    0.96   340.40    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.61  38.80


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.42    0.00    1.34    0.00    0.00   98.24

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda            318.00 218368.00     0.00   0.00    9.42   686.69    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.00  97.60
sdb            317.00 218300.00     0.00   0.00    9.45   688.64    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.00 100.00
sdc            303.00 208724.00     0.00   0.00    9.90   688.86    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.00 100.40
sdd            640.00 217580.00     0.00   0.00    0.94   339.97    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.60  40.40

Just in AHCI mode, no RAID configured at the motherboard level

The scrub finished in 23h.
A big portion of the time the speed was more around 5-6MB.

Here is an idea of the speed across the all scrub

Between 3am and 4-ish that must be Proxmox Backup Server doing its thing.
It really started to srub faster again around 6:30am.

Still not sure why a large phase is slow.
Thanks for the help so far.

It’s almost as if it whipped your drives back into shape. :laughing: “What is all this commotion? Oh right! We’re supposed to scrub the pool!”

EDIT: Wait a second. Is this a virtualized instance of TrueNAS?

No it is not virtualized :slight_smile:

More the other way around as PBS is running in a container in TrueNAS.
My Proxmox is running on another machine.

Totally normal for HDD?

Transfer of MB, yes. But what about the more interesting part; busy time?
Could be first huge images, than small files.

And here we go :grin:

Average speed is
5,5 TB * 1024 * 1024 / 3600 * 23 = 36MB/s.

36MB/s for a RAIDZ1 with HDDs is IMHO totally fine.

Again, RAIDZ1 with such huge drives is very risky. You have 24 scrubs now, imagine the rebuild times when the pool is at 24TB or something. You will have resilvertimes measured in weeks and begging that not another drives fails in the meantime. And these other drives are even the same model…

@winnielinnie was mentioning the scrub only cares about the blocks.
Why the file size would have an impact on the overall perf then?
Why 36MB is fine considering the disks can do 250MB?

I totally get your point. Believe me, my goal was not to have such big disks. At the time where I live, it was difficult to get any reliable disks with a smaller size … or they were at the same price. I did not pay today’s crazy price to put myself in a misery later.
In any case, it is where I am today. I will have to review the backup strategy and just have a copy on smaller drives. The most critical size is 1TB right now, the rest is not an issue if I loose them.

My scrub of 8 TB of data consistently takes about 12 hours. That’s over 180 MB/s. With the new scrub method in recent versions of ZFS, it minimizes random seeking.

The issue with the OP’s scrub is not merely that the average speed of scrubbing 5.5 TB of data was 36 MB/s (which is awfully slow for those enterprise HDDs), but there was a very long period where it ran at around 5 MB/s. As a matter of fact, it ran really fast and really slow. It never cruised at 36 MB/s. If you remove the burst at the start and end, you wouldn’t see an “average” of 36 MB/s, but a crawl of 5 MB/s. That’s definitely something worth noting and possibly diagnosing. If my own pool behaved like that, I would want to try to figure out why. For all we know, there could be an underlying bug that was incorrectly stalling the scrub.

Going to have to agree that something is strange. Sure scrubs take time, I’m going to guess that lots of small files can take longer for HDDs to scrub through, & the more full a pool the longer it’ll take to scrub (assuming same hardware). But I have ~42TB of used data & I finish my scrubs in half the time OP does.

Worth looking into deeper imo.

I definitely agree those long slow sections felt, and sounded, more like a defragmentation in the old days. When it was fast it was silent.

Anyway, what could be the next step?

What kind of data can I gather to help with this?

I mean you could check fragmentation simply with zpool list, who knows, maybe that is the problem.

Edit: I’ve also come to realize that if it is the problem then the fix is basically move all the data off to another place & put it back.

@winnielinnie asked me that yesterday.

It’s currently at 7%

smaller files, more blocks.

because 250MB/s is sequential read of a single file.
Scrub is not that.

Yeah, I am also at
14 TB * 1024 * 1024 / (3600 * 25) = 163MB/s with my two wide mirror enterprise HDDs. But I only have files.

IMHO another evidence that the problem is small or fragmented files.

That if free space fragmentation. Not the fragmentation of your data.

Did you put anything like a VM or DB or something else related that could cause lots of small files on your pool?

9% capacity. I find it hard to believe that there’s a fragmentation issue with that.
Scrub takes time, but 23 hours for just under 6 TB is slow.

just throwing it out there - since we are running out of ideas - APM on drive?