27.0-RC.1: system hard-resets during raw encrypted zfs send/recv (non-raw works)

**Version:** TrueNAS 27.0.0-RC.1 (ZFS 2.4.4-1, kernel 6.18.52-production+truenas)

**Hardware:** i7-10700, 48 GB non-ECC DDR4, LSI 9300-16i (IT mode), 7× 1 TB SATA SSD pool + 1 TB SATA SSD target, WD SN530 NVMe boot

**What I was doing:** Migrating an encrypted pool (RAIDZ1 → RAIDZ2) via a temporary single-disk pool.

- Source pool `Primary`: pool-level native encryption (AES-256-GCM, key), ~117 GiB, ~45 datasets. The apps dataset (`Primary/ix-apps`) is unencrypted; everything else inherits encryption from the root.

- Target: new single-disk pool `Temp` (unencrypted).

**Command:**

`zfs snapshot -r Primary@migrate && zfs send -Rw Primary@migrate | zfs recv -u Temp/Primary`

(second attempt also used `-x mountpoint`, run via nohup in the background)

**Result:** The system hard-reset ~30 seconds in, **both times at the same point**: the unencrypted `ix-apps` tree (~33 GB) finished receiving, and the reset happened as the stream moved on to the first encrypted dataset. Nothing was written to the journal before the reset (the last entries are routine), and no panic was captured.

**Things I ruled out:**

- The first attempt was with RAM overclocked (3200 MT/s on 2400/2666 sticks). That attempt also logged 4 checksum errors on one pool disk (repaired from parity), and a permanent error appeared in `libgallium` on the boot pool (written during the RC.1 upgrade). I reverted the RAM to 2666 and ran MemTest86 for 4 passes, 0 errors.

- The second attempt, with stable RAM, failed at the **same spot**.

- Scrubs of the source and target pools were clean afterwards.

**Workaround that worked:** Per-dataset **non-raw** send/recv (`zfs send ds@snap | zfs recv -u -o Temp/…`), which completed all 45 datasets in ~7 min without issue, and the same method back into a new encrypted RAIDZ2 pool. Data verified (logicalreferenced matched on 44/44 datasets, clean scrubs). RAIDZ expansion to 8 disks afterwards also completed fine.

**Suspicion:** raw (`-w`) send/recv of encrypted datasets on 27 RC.1 / ZFS 2.4.4. Happy to try reproducing with a console attached to capture a panic trace, or to provide a debug file, if that’s useful.

Use Feedback / Report a Bug in the TrueNAS GUI or at the top right of the forums. Attach Debug dump and please post the ticket number or a link back to the forum.

I tried using that option but got an error saying:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.


Apache/2.4.69 (Debian) Server at support-proxy.ixsystems Port 443

I have a hunch that wasn’t a bug.

  1. You were using overclocked RAM.
  2. While the RAM was overclocked, you ran a migration replication. Not only did the system crash, the data pool got 4 new checksum errors and the boot-pool got a checksum error.
  3. You then restored the RAM to stock speeds and confirmed no memory issues by running multiple memtest passes.
  4. Then you ran a scrub on the data pool.
  5. Then, with your RAM still on stock speeds, you retried the replication to migrate the source pool onto a temporary pool.

It sounds like overclocking the RAM is the original culprit which manifested as a system crash and checksum errors.

You’re definitely right that the overclock caused the checksum errors. That part I’m confident about.

Where I don’t think it explains everything is the second crash:

After the first crash I dropped the RAM from 3200 to 2666 and ran MemTest86 for ~6.5 hours (4 full passes, 0 errors).

With the RAM at 2666, I retried the same raw send/recv and it crashed at exactly the same point as the first time: right as the stream moved from the unencrypted ix-apps tree to the first encrypted dataset.

Immediately afterward, on the same RAM settings, I switched to non-raw per-dataset send/recv and it moved all ~120 GB without a problem, in both directions (out to the temp pool and back into a new encrypted pool). Scrubs were clean and the data verified.

I missed that part in your OP.

It’s possible that the original corruption from the overlocked RAM might have caused the original issue, where it’s not certain if the scrub (after the failed second attempt) is what fixed the issue, but you happened to use a non-raw replication afterward anyways.