Thank you @winnielinnie @etorix @Johnny_Fartpants @Farout for all your help and support in this, being new to ZFS, this forum has been VERY helpful and I appreciate you VERY much!
The array is no longer degraded.
Timeline of the recovery:
- October 30: SATA communication failures on ata4, including BadCRC, link resets, and negotiation down to 3.0 Gb/s.
- October 30: Reseated SATA data and power connections.
- November 2: Removed a 170 GB ZFS checkpoint; the repeated deferred-resilver cycle subsequently stopped.
- November 2: Resilver completed with zero reported scan errors.
- November 3: Scrub completed, repairing 268 KB with zero uncorrectable scrub errors.
- November 3: ata4’s ZFS checksum count increased to 20, although SMART interface CRC remained at 4.
Potential next steps / concerns:
My remaining concern is the 20 ZFS checksum errors on SATA port 4. The scrub repaired 268 KB and completed with zero uncorrectable errors, while SMART attribute 199 remained unchanged at 4 and there have been no additional kernel SATA errors since the initial October 30 link resets.
Could these checksum errors plausibly be historical corruption from the earlier SATA instability, or does the continued concentration of errors on one drive suggest an ongoing disk, cable, or controller issue?
I also noticed this drive is currently negotiating at 3.0 Gb/s rather than 6.0 Gb/s. Would you recommend replacing the SATA cable first, testing another motherboard port, or proceeding with extended SMART tests before changing anything?
I also find it odd that the drive extended model numbers are slightly different and their sata versions are different. In spite of the fact that what I thought I was purchasing was 4 identical drives, though I did have to portal through 5 months of RMAs and returns with amazon/WD support, so that likely explains that.
I’m thinking now I should run long smart tests on all drives to verify, and maybe look into replacing sata cables (current sata cables were bought new for this purpose this year www amazon com/dp/B08K8XVS2R but maybe they aren’t so great? recs?)
Please let me know if you agree with these steps, additional items, and any other information I could provide that would be beneficial?
# zpool status -v pool-data-03
pool: pool-data-03
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: scrub repaired 268K in 04:13:51 with 0 errors on Tue Nov 3 04:23:25 2026
config:
NAME STATE READ WRITE CKSUM
pool-data-03 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
13a0551c-7684-4efd-9830-c91f846b30e7 ONLINE 0 0 0
beaf27a1-fd08-4493-b021-320df04a349c ONLINE 0 0 0
56485580-c6e2-492d-9536-f33b7a171a2b ONLINE 0 0 20
f91b33b1-6461-4cbc-b3e9-f9cb00041385 ONLINE 0 0 0
errors: No known data errors
Additional output:
# udevadm info -q property -n /dev/sdc \
| grep -E '^(ID_PATH=|ID_MODEL=)'
ID_MODEL=WDC_WD142KFGX-68CLHN0
ID_PATH=pci-0000:00:1f.2-ata-4.0
# for sn in sata4 sata3; do
disk=$(find /dev/disk/by-id -maxdepth 1 \
-name "ata-*${sn}" -print -quit)
echo "===== $sn ====="
if [ -n "$disk" ]; then
smartctl -A "$disk" | grep UDMA_CRC_Error_Count
fi
done
===== sata4 =====
199 UDMA_CRC_Error_Count 0x000a 100 100 000 Old_age Always - 4
===== sata3 =====
199 UDMA_CRC_Error_Count 0x000a 100 100 000 Old_age Always - 22
# journalctl -k -b --since "2026-10-30 16:39:00" --no-pager \
| grep -Ei 'ata[0-9]|SError|BadCRC|I/O error|failed command|resetting link'
#
# for disk in /dev/sd?; do
echo "=== $disk ==="
smartctl -i "$disk" 2>/dev/null | \
grep -E 'Device Model|Serial Number|SATA Version'
done
=== /dev/sda ===
Device Model: WDC WD142KFGX-68CLHN0
Serial Number: sata2
SATA Version is: SATA 3.5, 6.0 Gb/s (current: 6.0 Gb/s)
=== /dev/sdb ===
Device Model: WDC WD142KFGX-68AFPN0
Serial Number: sata1
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
=== /dev/sdc ===
Device Model: WDC WD142KFGX-68CLHN0
Serial Number: sata4
SATA Version is: SATA 3.5, 6.0 Gb/s (current: 3.0 Gb/s)
=== /dev/sdd ===
Device Model: MZ7LH480HBHQ0D3
Serial Number: sata5 - boot drive
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
=== /dev/sde ===
Device Model: WDC WD142KFGX-68AFPN0
Serial Number: sata3
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
# lspci -nn | grep -Ei 'sata|ahci|raid'
00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82]
# journalctl -k -b --no-pager | \
grep -Ei 'ata[0-9].*(link up|link down|hard resetting|SError|limited to)'
Oct 30 16:38:06 truenas kernel: ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct 30 16:38:06 truenas kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct 30 16:38:06 truenas kernel: ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct 30 16:38:06 truenas kernel: ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct 30 16:38:06 truenas kernel: ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct 30 16:38:07 truenas kernel: ata4: SError: { UnrecovData 10B8B BadCRC }
Oct 30 16:38:07 truenas kernel: ata4: hard resetting link
Oct 30 16:38:07 truenas kernel: ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct 30 16:38:07 truenas kernel: ata4: SError: { UnrecovData 10B8B BadCRC }
Oct 30 16:38:07 truenas kernel: ata4: hard resetting link
Oct 30 16:38:07 truenas kernel: ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct 30 16:38:41 truenas kernel: ata4: SError: { UnrecovData 10B8B BadCRC }
Oct 30 16:38:41 truenas kernel: ata4: hard resetting link
Oct 30 16:38:41 truenas kernel: ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct 30 16:38:41 truenas kernel: ata4: SError: { UnrecovData 10B8B BadCRC }
Oct 30 16:38:41 truenas kernel: ata4: hard resetting link
Oct 30 16:38:42 truenas kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
# smartctl -A /dev/sdc | grep -Ei \
'Reallocated|Pending|Uncorrectable|CRC|Reported_Uncorrect|Command_Timeout'
5 Reallocated_Sector_Ct 0x0033 100 100 001 Pre-fail Always - 0
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 100 100 000 Old_age Always - 4
# smartctl -i -H /dev/sdc
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.33-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD142KFGX-68CLHN0
Serial Number: sata4 / sdc
LU WWN Device Id: []
Firmware Version: 83.00A83
User Capacity: 14,000,519,643,136 bytes [14.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database 7.3/5528
ATA Version is: ACS-4 published, ANSI INCITS 529-2018
SATA Version is: SATA 3.5, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Tue Nov 3 12:05:21 2026 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
# zpool events -v | tail -120
scheme = "zfs"
pool = 0xc69ef9176dab9c6b
vdev = 0x8e8dfe846eac3c8c
(end detector)
pool = "pool-data-03"
pool_guid = 0xc69ef9176dab9c6b
pool_state = 0x0
pool_context = 0x0
pool_failmode = "continue"
vdev_guid = 0x8e8dfe846eac3c8c
vdev_type = "disk"
vdev_path = "/dev/disk/by-partuuid/56485580-c6e2-492d-9536-f33b7a171a2b"
vdev_ashift = 0x9
vdev_complete_ts = 0x10e43b285b739
vdev_delta_ts = 0x3661c0d2
vdev_read_errors = 0x0
vdev_write_errors = 0x0
vdev_cksum_errors = 0x14
vdev_delays = 0x0
dio_verify_errors = 0x0
parent_guid = 0x18b21e8f9a7ff09c
parent_type = "raidz"
vdev_spare_paths =
vdev_spare_guids =
zio_err = 0x0
zio_flags = 0x2000b0 [SCRUB SCAN_THREAD CANFAIL DONT_PROPAGATE]
zio_stage = 0x400000 [VDEV_IO_DONE]
zio_pipeline = 0x5e00000 [VDEV_IO_START VDEV_IO_DONE VDEV_IO_ASSESS CHECKSUM_VERIFY DONE]
zio_delay = 0x0
zio_timestamp = 0x0
zio_delta = 0x0
zio_priority = 0x4 [SCRUB]
zio_offset = 0x52017ed000
zio_size = 0x1000
zio_objset = 0x23b
zio_object = 0x182b6
zio_level = 0x1
zio_blkid = 0x0
bad_ranges = 0x0 0x970
bad_ranges_min_gap = 0x8
bad_range_sets = 0x0
bad_range_clears = 0x2385
time = 0x6ae99706 0x1bdd4ad0
eid = 0x1f4
Nov 3 2026 03:45:02.303193623 sysevent.fs.zfs.scrub_start
version = 0x0
class = "sysevent.fs.zfs.scrub_start"
pool = "boot-pool"
pool_guid = 0x6bbfb0ec4b6ce077
pool_state = 0x0
pool_context = 0x0
time = 0x6ae9ad1e 0x12125e17
eid = 0x1f5
Nov 3 2026 03:45:02.303193623 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "boot-pool"
pool_guid = 0x6bbfb0ec4b6ce077
pool_state = 0x0
pool_context = 0x0
history_hostname = "truenas"
history_internal_str = "func=1 mintxg=0 maxtxg=4262911"
history_internal_name = "scan setup"
history_txg = 0x410bff
history_time = 0x6ae9ad1e
time = 0x6ae9ad1e 0x12125e17
eid = 0x1f6
Nov 3 2026 03:45:40.363784622 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "boot-pool"
pool_guid = 0x6bbfb0ec4b6ce077
pool_state = 0x0
pool_context = 0x0
history_hostname = "truenas"
history_internal_str = "errors=0"
history_internal_name = "scan done"
history_txg = 0x410c09
history_time = 0x6ae9ad44
time = 0x6ae9ad44 0x15aee9ae
eid = 0x1f7
Nov 3 2026 03:45:40.363784622 sysevent.fs.zfs.scrub_finish
version = 0x0
class = "sysevent.fs.zfs.scrub_finish"
pool = "boot-pool"
pool_guid = 0x6bbfb0ec4b6ce077
pool_state = 0x0
pool_context = 0x0
time = 0x6ae9ad44 0x15aee9ae
eid = 0x1f8
Nov 3 2026 04:23:25.475059870 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "pool-data-03"
pool_guid = 0xc69ef9176dab9c6b
pool_state = 0x0
pool_context = 0x0
history_hostname = "truenas"
history_internal_str = "errors=0"
history_internal_name = "scan done"
history_txg = 0x40606e
history_time = 0x6ae9b61d
time = 0x6ae9b61d 0x1c50d69e
eid = 0x1f9
Nov 3 2026 04:23:25.475059870 sysevent.fs.zfs.scrub_finish
version = 0x0
class = "sysevent.fs.zfs.scrub_finish"
pool = "pool-data-03"
pool_guid = 0xc69ef9176dab9c6b
pool_state = 0x0
pool_context = 0x0
time = 0x6ae9b61d 0x1c50d69e
eid = 0x1fa
