Visibility for Long-Running ZFS Device Removal Operations

First, I want to say that ZFS device removal is an incredible capability. Being able to remove a top-level vdev without rebuilding the entire pool has been extremely useful.

However, the observability during the process is lacking.

I recently removed a top-level vdev from a large pool, and the TrueNAS UI only reports that a device removal is in progress. To actually understand what was happening, I had to build my own monitoring script using zpool status and zpool iostat.

For example, my script continuously reports:

  • Percentage complete
  • Data copied (TiB)
  • Current throughput (MB/s)
  • Estimated time remaining
  • Remaining async reclaim (“freeing”)
  • Per-vdev read/write bandwidth
  • Live pool bandwidth

Example output:

42.7% complete
5.18 TiB copied out of 12.1 TiB
252 MB/s
ETA: 8h 1m

Per-vdev write distribution:
10TB disk 1: 46 MB/s
10TB disk 2: 46 MB/s
10TB disk 3: 48 MB/s
10TB disk 4: 47 MB/s
10TB disk 5: 47 MB/s

Removing disk:
233 MB/s reads

Async reclaim remaining:
2.74 TiB

This information already exists inside ZFS—the UI simply doesn’t expose it.

For operations that can run for many hours (or even days), I think TrueNAS would benefit from a dedicated progress page showing:

  • Overall progress (%)
  • Bytes copied / total bytes
  • Current throughput
  • ETA
  • Async reclaim progress
  • Live bandwidth graphs
  • Per-vdev activity during the evacuation
  • Any throttling or warnings if performance drops

This would make long-running maintenance much easier to monitor and would give administrators confidence that the operation is progressing normally instead of leaving them wondering whether it has stalled.

I’m curious if this is something that others would find useful, or if there are plans to expose more of ZFS’s internal progress information in future versions.

2 Likes

@Rocketplanner83 Great idea. Don’t forget to vote for it yourself, (I am out of votes…).