Unable to replace faulted drive in Pool RAIDZ1 | 4 wide | 3.64 TiB

havedatawilltravel · June 1, 2025, 2:54pm

Hello and thanks in advance.

One faulted 4TB WD NAS HDD. Replacement is 4TB WD NAS HDD.

This is a redundant backup pool. Main pool healthy. I would like to understand the process now with this redundant pool in the instance my main pool needs a drive replaced.

Error message after taking disk off line, then using GUI to replace

[EFAULT] Could not create a partition of 4000787030016 bytes on disk sdk because the disk is too small. If you are replacing a disk in a pool, please ensure that the new disk is not smaller than the disk being replaced. Could not create partition 1 from 2048 to 7814039215 Could not change partition 1’s type code to BF01! Error encountered; not saving changes.

Under the more info

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/plugins/disk_/format.py”, line 39, in format
subprocess.run([“sgdisk”, “-n”, f"1:0:+{int(size / 1024)}k", “-t”, “1:BF01”, f"/dev/{disk}"],
File “/usr/lib/python3.11/subprocess.py”, line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command ‘[‘sgdisk’, ‘-n’, ‘1:0:+3907018584k’, ‘-t’, ‘1:BF01’, ‘/dev/sdk’]’ returned non-zero exit status 4.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 515, in run
await self.future
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 560, in run_body
rv = await self.method(*args)
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 174, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 48, in nf
res = await f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool/replace_disk.py", line 92, in replace
await self.middleware.call(‘pool.format_disks’, job, {
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1000, in call
return await self.call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 715, in call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool/format_disks.py", line 29, in format_disks
await asyncio_map(unlock_and_format_disk, disks.items(), limit=16)
File "/usr/lib/python3/dist-packages/middlewared/utils/asyncio.py", line 19, in asyncio_map
return await asyncio.gather(*futures)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/utils/asyncio.py", line 16, in func
return await real_func(arg)
^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/pool_/format_disks.py”, line 24, in unlock_and_format_disk
await self.middleware.call(‘disk.format’, disk, config.get(‘size’))
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1000, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 726, in call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 619, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/disk/format.py", line 54, in format
raise CallError(error)
middlewared.service_exception.CallError: [EFAULT] Could not create a partition of 4000787030016 bytes on disk sdk because the disk is too small. If you are replacing a disk in a pool, please ensure that the new disk is not smaller than the disk being replaced.

Could not create partition 1 from 2048 to 7814039215
Could not change partition 1’s type code to BF01!
Error encountered; not saving changes.

Summary of troubleshooting

tried the force option during replace
tried for wipe new HDD via stroage-> disks both with quick and full with zeros
removed 4TB WD NAS HDD and formatted in win11
removed 4TB WD NAS HDD and formatted in linux with gparted
opened NAS and check all cables, unplugged both SATA and power, replugged
replaced cables to the faulted drive
bought new 4TB WD NAS HDD - installed it, tried all the above steps with the new HDD.

Next steps

I bought new SATA cables.
Entertaining moving the 4x 4TB faulted pool to a freeBSD box and trying it three, though this is the least desirable option.
remake the redundant pool. Last stop in the trouble shooting journey,

Have not tried yet:

adding drive via CLI given the warnings issues by ixsystems.

If you made it this far, thanks!

etorix · June 1, 2025, 3:14pm

This looks like you have created the pool in a version of SCALE which did NOT reserve 2 GB for padding, and the new drive is slightly smaller than the old one. So you’re out of luck.

Get a larger drive.
Or backup, destroy and recreate the pool including the smaller drive.

winnielinnie · June 1, 2025, 3:38pm

If only someone predicted this over 5 months ago...

havedatawilltravel · June 1, 2025, 7:12pm

Ah. Thank you. I am new to TrueNAS.

The pools were made in FreeBSD and imported. While they work, this is a significant issue.

The work around would be to replace the drive in freebsd, then reimport. Or remake the pool.

Remaking my main pool is less desirable as there is always the uncomfortable interval of hoping the backups are a true copy and there is no data loss. (aware of zfs send)

Is there a solution other than remaking the pool? CLI to reserve that space?
If I replaced the drives 1 by 1 with a 6TB drive, will it expand after the last 4TB drive replaced with 6TB to use all the space?

I appreciate this still does not address the 2GB issue.

etorix · June 1, 2025, 7:27pm

If no padding space was left when creating the pool you cannot replace the drive in FreeBSD either. ZFS can expand, but not shrink.

I leave that question to high rank wizards, but if a procedure exists it may very well be even less comfortable than backup and restore.

Normally yes; you can check with sudo zpool get autoexpand in the shell.
And if you do this with a recent version of SCALE it should hopefully create the 2 GB padding on the new drives

havedatawilltravel · June 1, 2025, 8:01pm

Thanks for your quick replies.

I will rebuild this 4TB pool and zfs send my data as the first step. I will use the bad drive, then try to replace it with one of the news ones. Will keep you updated.

I will leave my main pool for now and follow this thread for any further info. Specifically this:

Normally yes; you can check with sudo zpool get autoexpand in the shell.
And if you do this with a recent version of SCALE it should hopefully create the 2 GB padding on the new drives

Thanks again.

Addit: one point of clarification i am use the ‘community edition’ is this also known as scale? I have just started using TrueNAS.

etorix · June 1, 2025, 10:00pm

Yes, “CE” is new moniker for what was known as “SCALE” (and which will never Scale-Out…)

Next, the “Love Symbol”?

havedatawilltravel · June 3, 2025, 3:36pm

Thanks for all the help.

Backup pool, rebuilt, working well.

Main pool, new drives, will consider my options. The fall back position is if in the short term need to replace a drive, will use freeBSD. Will revisit this when I have some time to zfs send receive and rebuild.

What would be a useful name of this post - to help others new to the journey find this info. And save them some time and hardware costs while they trouble shoot this.