VDEV Expansion Paused?

TrueNAS General
,
1

Hi all,

Relatively new to Truenas. Still on the upward learning curve.

So I attempted a VDEV expansion to a Raidz1 VDEV with three 12tb HDD. I’m adding a 4th. I started the expansion a day and a half ago. Last night the status change to “Paused at 95% for resilvering or clear”.

It’s now been in This state for about 12hours or so.

Questions:

Should I be concerned or just continue to wait?

What exactly does the message mean?

If it’s an indication of an error, what would be my next step?

thank you

running 24.04

2

The normal reason for a RAID-Zx Expansion to pause, is failure of a disk. If so, that must be fixed before RAID-Zx Expansion will continue.

Please post the output of the following command, in CODE tags:

zpool status -v

This should show the real reason why it was paused.

Ideally, the RAID-Zx Expansion section in TrueNAS would have a more complete explanation on failures.

3

Thanks for the feedback. I did have one of the other drives throw a read error (you will see in the dump from the command you asked me to do). I had planned to replace it after the expansion was completed. But now I’m unsure which way to go. Can I initiate the replacement while the expansion is still underway?

================

Linux truenas 6.12.15-production+truenas #1 SMP PREEMPT_DYNAMIC Mon Sep 8 18:50:34 UTC 2025 x86_64

    TrueNAS (c) 2009-2025, iXsystems, Inc. dba TrueNAS
    All rights reserved.
    TrueNAS code is released under the LGPLv3 and GPLv3 licenses with some
    source files copyrighted by (c) iXsystems, Inc. All other components
    are released under their own respective licenses.

    For more information, documentation, help or support, go here:
    http://truenas.com

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI, CLI, and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

Welcome to TrueNAS
Last login: Tue Oct 28 12:48:29 MST 2025 on pts/0
root@truenas:~# zpool status -v
pool: DATA_CORE
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using ‘zpool clear’ or replace the device with ‘zpool replace’.
see: Message ID: ZFS-8000-9P — OpenZFS documentation
scan: scrub repaired 192K in 11:00:44 with 0 errors on Mon Oct 13 04:32:45 2025
expand: expansion of raidz1-0 in progress since Tue Oct 28 12:41:02 2025
26.0T / 26.9T copied at 153M/s, 96.55% done, paused for resilver or clear
config:

    NAME                                      STATE     READ WRITE CKSUM
    DATA_CORE                                 ONLINE       0     0     0
      raidz1-0                                ONLINE       0     0     0
        9aafc286-f9ef-49ea-89ad-ec9ad353efa9  ONLINE       0     0     0
        92f23a64-4e92-4f71-9675-70a5cab61b5a  ONLINE       0     0     0
        73dd1d51-3139-4609-a552-bafdeb944f56  ONLINE       2     0     0
        167c212a-ac4b-4e84-b87f-0176750d06bc  ONLINE       0     0     0
    cache
      f2006e02-479c-489e-8e5a-31e608a708b3    ONLINE       0     0     0

errors: No known data errors

pool: Strip_1TB
state: ONLINE
config:

    NAME                                    STATE     READ WRITE CKSUM
    Strip_1TB                               ONLINE       0     0     0
      e2c21704-fcf4-4eb0-9dd8-4e617c30a787  ONLINE       0     0     0

errors: No known data errors

pool: Strip_2TB
state: ONLINE
config:

    NAME                                    STATE     READ WRITE CKSUM
    Strip_2TB                               ONLINE       0     0     0
      1b5eb32f-8dbe-4616-9e29-4e5b63b51665  ONLINE       0     0     0

errors: No known data errors

pool: Stripped_12TB
state: ONLINE
config:

    NAME                                    STATE     READ WRITE CKSUM
    Stripped_12TB                           ONLINE       0     0     0
      e2343e53-d768-4e73-8ac4-3adfb84cd6e1  ONLINE       0     0     0

errors: No known data errors

pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:00:26 with 0 errors on Tue Oct 28 03:45:27 2025
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      sda3      ONLINE       0     0     0

errors: No known data errors
root@truenas:~#

4

I wanted to add that I read some of the TrueNAS documentation, which suggested clearing the two read errors, and it appears that the expansion has restarted. The Prior errors on the same drive also reference an ATA error. So is that the drive, drive controller, or SATA cable issue?

This was the last error that I experienced (the one I just cleared).

Critical

Device: /dev/sdi [SAT], ATA error count increased from 31 to 143.

I’ll have to see if it completes successfully.
This is now what zpool status -v shows for the problem pool.

pool: DATA_CORE
state: ONLINE
scan: scrub repaired 192K in 11:00:44 with 0 errors on Mon Oct 13 04:32:45 2025
expand: expansion of raidz1-0 in progress since Tue Oct 28 12:41:02 2025
26.0T / 26.9T copied at 152M/s, 96.55% done, 01:46:46 to go
config:

    NAME                                      STATE     READ WRITE CKSUM
    DATA_CORE                                 ONLINE       0     0     0
      raidz1-0                                ONLINE       0     0     0
        9aafc286-f9ef-49ea-89ad-ec9ad353efa9  ONLINE       0     0     0
        92f23a64-4e92-4f71-9675-70a5cab61b5a  ONLINE       0     0     0
        73dd1d51-3139-4609-a552-bafdeb944f56  ONLINE       0     0     0
        167c212a-ac4b-4e84-b87f-0176750d06bc  ONLINE       0     0     0
    cache
      f2006e02-479c-489e-8e5a-31e608a708b3    ONLINE       0     0     0

errors: No known data errors

Opinions please:

The “failed drive” has had prior errors. But they cleared, and TrueNas kept using the drive. Of the two prior errors this drive has thrown, one of them also references ATA errors.

HOWEVER, After about 10 minutes, the problem drive experiance the same erros again and paused the expansion again.

Thanks again.

5

The drive is likely due for replacement/RMA.

6

Thanks for the advice. I got the expansion going again. I’m seeing increasing ATA errors through the process. As I mentioned, I’m going to replace the suspect drive once the expansion is done.

One question: Where are these Workflows you talked about? That would be an enormous help.

Thanks

7

I don’t know about the “Workflows”…

But, in regards to the RAID-Zx Expansion pausing, it will always pause on error. Since you are close to completion, clearing the errors may work for you. Be warned however, ZFS RAID-Zx Expansion ALWAYS runs a scrub after and this MUST complete too. It’s a quirk of the RAID-Zx Expansion…

1 Like
8

Thanks for all the help.
I was able to replace the bad drive during the expansion pause and the expansion continued to completion. And yes, it has started a scrub of the VDEV as you stated. Only 10 hours left.

Again, appreciate to help.

1 Like