Identical CKSUM errors across all 7 drives in a raidz1

TrueNAS General
1

Identical CKSUM errors across all 7 drives in a raidz1 — persist after replacing HBA, testing RAM + PSU. raidz-expanded pool on OpenZFS 2.4.1 (26.0.0-BETA.2)

I was trying to use Claude to troubleshoot ) please dont judge!) and this is what i have done, any help would be appreciated.

TL;DR

All 7 disks in a single raidz1 vdev show the exact same checksum error count, incrementing in lockstep, with zero READ and zero WRITE errors and nothing in dmesg. This has survived a full hardware swap of the suspected component and clean RAM/PSU tests. The pool was raidz-expanded and is running a bleeding-edge OpenZFS build. Looking for anyone who’s seen this signature and knows whether it’s salvageable in place or a known raidz-expansion / OpenZFS 2.4.x issue.

System / Hardware

  • TrueNAS version: TrueNAS 26.0.0-BETA.2

  • OpenZFS version: zfs-2.4.1-1 / zfs-kmod-2.4.1-1

  • Motherboard: gigabyte (older)

  • CPU: 2600x

  • RAM: ddr4 32g, non-ECC

  • PSU: Corsair 850

  • HBA / controller (NEW): 10Gtek 6-Port PCIe Expansion Card, PCIe x4 to 6xSATA

  • HBA / controller (OLD, since replaced): m.2-to-SATA adapter, several years old (i dont think this was the issue)

  • Boot device: single SSD (sdb3, boot-pool)

Pool layout

  • Pool: raid

  • Topology: single raidz1-0 vdev, 7 × HDD

  • Hot spare: 1 × HDD (AVAIL)

  • Drive connections: 5 drives were on the old m.2-to-SATA adapter, 4 on motherboard SATA (9 drives total: 7 vdev + 1 boot + 1 spare). Since replacing the adapter with a PCIe HBA. 5 drives on HBA, 3 on MOBO + boot on MOBO

  • Pool history: this vdev was raidz-expanded — expand: expanded raidz1-0 copied 9.38T in 19:05:11, on Mon May 26 00:00:40 2025

The problem

Previously the pool had zero checksum errors. They appeared suddenly (I was away for a weekend; my first thought was a cable knock or a bad adapter). What’s unusual is that every drive shows the identical error count, not a spread — which is not what independent disk/cable failure looks like.

Representative zpool status -v raid (counts climb over time; snapshot below):

  pool: raid
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 440K in 2 days 09:20:28 with 36 errors on Thu Jul  9 21:45:55 2026
expand: expanded raidz1-0 copied 9.38T in 19:05:11, on Mon May 26 00:00:40 2025
config:

        NAME                                      STATE     READ WRITE CKSUM
        raid                                      DEGRADED     0     0     0
          raidz1-0                                DEGRADED     0     0     0
            79040f44-422b-4f54-9528-c0264e64bcc6  ONLINE       0     0   266
            fd0ea60d-016d-48d5-bc87-c4de4174b3aa  ONLINE       0     0   266
            6476bca5-e800-4ed4-b830-8bf312f87651  ONLINE       0     0   266
            ea73c50a-5069-49ce-aca4-9d1946e1e5e7  ONLINE       0     0   266
            c4e4c9d0-7744-4d5c-8ab6-897cfa226134  ONLINE       0     0   266
            4f6e0b01-fef0-4ba6-a207-c1d64891d94e  ONLINE       0     0   266
            f2ae3af8-40dc-4feb-b573-762bd9b97413  DEGRADED     0     0   266  too many errors
        spares
          1cf534a5-4afd-4453-b0cf-65709190c024    AVAIL

Notes on the above:

  • All 7 drives track the identical CKSUM count. READ/WRITE are 0 across the board.

  • One drive (f2ae3af8) flipped to DEGRADED / too many errors, but it shows the same count as the other six and zero read/write — it just crossed the threshold first. I do not believe it’s a failing drive.

  • A zpool clear resets the counters to 0, after which they immediately begin climbing again.

Permanent errors reported

A scrub (2d 9h, repaired 440K with 36 errors) reported permanent errors in:

raid/ix-apps/app_mounts/jellyfin/config:<0x10139>
/var/db/system/netdata/dbengine/datafile-1-0000000834.ndf
/var/db/system/netdata/dbengine/journalfile-1-0000000834.njf
/var/db/system/netdata/ml.db
/var/db/system/netdata/dbengine/datafile-1-0000000833.ndf
/var/db/system/configs-.../TrueNAS-26.0.0-BETA.1/20260705.db

The Jellyfin dataset now fails to mount:

[EZFS_MOUNTFAILED]: zfs_mount_at() failed - cannot mount
'raid/ix-apps/app_mounts/jellyfin/config': Input/output error

(The <0x...> object-ID form suggests corrupted metadata rather than a plain file, which is presumably why the whole dataset won’t mount.)

Event log — fixed-interval “heartbeat”

zpool events raid | tail shows ereport.fs.zfs.data events arriving on an almost exact ~5.12 second interval while the pool is otherwise idle, plus periodic bursts of ereport.fs.zfs.checksum events sharing an identical nanosecond timestamp (a single stripe read hitting all drives at once):

Jul 11 2026 19:10:10.850621835 ereport.fs.zfs.data
Jul 11 2026 19:10:11.138619705 ereport.fs.zfs.checksum   <-- x14, all same ns timestamp
...
Jul 11 2026 19:10:16.734578328 ereport.fs.zfs.data
Jul 11 2026 19:10:21.858540455 ereport.fs.zfs.data
Jul 11 2026 19:10:26.978502602 ereport.fs.zfs.data       <-- ~5.12s apart, metronomic

My read: this looks like something reading the same already-corrupted blocks on a timer, with ZFS counting each hit across the whole stripe — rather than new/spreading corruption. But I can’t get it to stop (see below), and it’s still climbing.

Troubleshooting already done

  1. Replaced the old m.2-to-SATA adapter with a new PCIe HBA. Errors returned with the identical-count signature. (Note: some vdev drives are/were on onboard SATA on an unchanged path, yet track the same count as drives whose entire path was replaced.)

  2. Memtest86+ on the (non-ECC) RAM — reported clean. [FILL IN number of passes / duration]

  3. PSU tested — reported fine. [FILL IN how it was tested]

  4. Moved the System Dataset from raid to boot-pool (midclt call systemdataset.update '{"pool": "boot-pool"}', then rebooted). Confirmed it moved — but the ereport heartbeat and climbing CKSUM continue.

  5. Stopped netdata, then stopped docker / docker.socket to try to identify/stop whatever is re-reading the corrupted blocks. [FILL IN whether the heartbeat stopped after this]

  6. zpool clear raid — resets counters; they climb again immediately.

  7. Attempted zfs rollback of the Jellyfin config to a pre-corruption snapshot (@1.3.11, dated before the corruption event) — this suspended the whole pool; required a reboot to bring it back ONLINE. Did not retry.

  8. dmesg shows no SATA/SAS link errors, drive resets, or controller complaints.

Deliberately NOT done (to avoid making it worse): no further scrubs, no resilver, no replacing the DEGRADED drive, no swapping in the spare.

What I think is going on (open to being wrong)

  • Identical counts + zero read/write + clean dmesg, persisting through an HBA swap and clean RAM/PSU, seem to rule out per-drive hardware and point above the disk layer.

  • The metronomic event interval reads like a daemon repeatedly hitting the same permanent (unrecoverable, whole-stripe on raidz1) blocks — i.e. the climbing counter is a re-read artifact, not active corruption.

  • The pool is raidz-expanded and running OpenZFS 2.4.1 (very new). I’m suspicious of a raidz-expansion / new-OpenZFS interaction, but I have no proof.

Questions for the community

  1. Has anyone seen identical CKSUM counts across all drives in a raidz vdev that persist after full hardware replacement (controller swapped, RAM + PSU tested clean)? What was the root cause?

  2. Is there a known issue with raidz expansion + OpenZFS 2.3/2.4 producing spurious or real checksum errors like this?

  3. Is the metronomic ereport.fs.zfs.data interval a reliable indicator that this is a reader re-hitting dead blocks vs. live corruption? How would you confirm which?

  4. Given the permanent errors are all disposable (netdata metrics DB, a Jellyfin config object, a config-backup snapshot), is there any point trying to repair in place, or is evacuate → destroy → rebuild (on a stable release, without the expansion history) the correct call?

  5. Anything I should capture now (e.g. full zpool events -v, specific zdb output) that would help diagnose before I tear the pool down?

Happy to post any additional output — zpool events -v, zdb, smartctl, full dmesg, etc. Thanks in advance.

2

First: Welcome to the community.

Second: Take your time troubleshooting. AI is not very reliable, do not trust it if you are not familiar with exactly what the commands are doing. AI lies to me every day, no kidding. It makes assumptions, bad ones too.

A few questions:

  1. What did you build this pool/VDEV on? TrueNAS 26.0.0-BETA.2 ?
  2. Is all your data saved elsewhere? Hopefully.
  3. How many passes was MemTest86 run? At least 5 is reasonable with no errors.
  4. Did you run a CPU Stress Test for at least 4 hours, to verify the solder joints in the CPU and voltage regulator sections.
  5. If you have a backup of all your data, or it is not critical to save, then the “easiest” thing to do is wipe the drives, install TrueNAS 25.x, and rebuild. Do not use 26.x in Beta if you are not willing/able to troubleshoot some serious problems.

At a minimum you should delete the files that are corrupt. They are corrupt so no need to keep them on your system. Then take a look at my Drive Troubleshooting Flowcharts (see link below). It addresses some basic ZFS problems that are common and of course drive failures. It will have you run some commands. If you have a question about a command, please ask. Also, this will allow you to clear the errors once you delete the corrupt files.

Once you have this done, you might need to see what it going on with the degraded drive. It might come back on it’s own, but…

Cheers

3

No judgement, we’ll go straight to sentencing… :hammer: :wink:

Cheksum errors on all drives are probably NOT the drives. You’ve ruled out RAM (good on you!) and PSU, that leaves the cables, backplane if any, and the main suspect: HBA!

None of that is precise enough, and none of that is good. Cheap SATA controllers with posrt multipliers are known to result in exactly what you’re experiencing. Please get a LSI SAS HBA, and give it some cooling.

Do you have a backup?

4 Likes
4

Don’t leave us hanging! I’m on the edge of my seat! [FILL IN an explanation that I was being sarcastic]

2 Likes
5
  1. version 25
  2. The important stuff is, about 20% of the total data
  3. 3 passes
  4. no, its a CPU i have been using for 3-4 years.
  5. this is what Im leaning towards
6

Then it is time to backup the remaining 80% if the pool is still mounted (hopefully so).

For rebuilding, note that your raidz1 is too wide for comfort; consider raidz2.
And check these SATA controllers: No port multiplier, no JMicron… (9300-8i HBAs are cheap)

1 Like
7

Just to make sure you do not overlook this, even a well running CPU or System has worked for years, do not write off old components failing.

1 Like