I’m using TrueNAS Community Edition 25.10.7 (Goldeye). I have a single small SSD with TrueNAS on it as well as a single 8 TB ZFS pool mirrored across 2 drives (one is 8 TB and one is 10 TB, yes I know that I should have gotten another 8 TB but the 10 TB was gotten afterward and was cheaper than another 8 TB). I am using about 2-3 TB of space in the 8 TB pool.
Friday (25-SEP-2026) morning, I noticed that my NAS powered on but unresponsive; there was no output on the monitor I attached to it and my router was reporting it offline (which obviously meant that I could not connect to the WebUI or via SSH). I have not made any major changes to the NAS recently except for successfully upgrading the Jellyfin app from 10.11 to 12.1 on 24-SEP-2026 (which included TARing and GZIPing a copy of the .ixapps folder for Jellyfin with its config to a different spot on my ZFS pool beforehand).
With no other choice, I reset the NAS. During startup, I encountered the following error while TrueNAS was importing my ZFS pool:
PANIC: zfs: rt={spa=[MyPoolName] vdev_guid=[BunchOfNumbers] ms_id=85 ms_allocatable}: adding segment (offset=1566a2b0000 size=1000) overlapping with existing one (offset=1566a2b0000 size=1000)
Kernel panic - not syncing: zfs: rt={spa=[MyPoolName] vdev_guid=[BunchOfNumbers] ms_id=85 ms_allocatable}: adding segment (offset=1566a2b0000 size=1000) overlapping with existing one (offset=1566a2b0000 size=1000)
CPU: 2 UID: 0 PID: 1229 Comm: z_metaslab Tainted: P OE 6.12.105-production+truenas #1
Hardware name: Z77 Extreme3 BIOS P1.20
Call Trace:
[I can provide this info if absolutely needed but it’s very complex]
Kernel Offset: Ox18c00000 from 0xffffffff81000000
— [ end Kernel panic - not syncing: zfs: rt={spa=[MyPoolName] vdev_guid=[BunchOfNumbers] ms_id=85 ms_allocatable}: adding segment (offset=1566a2b0000 size=1000) overlapping with existing one (offset=1566a2b0000 size=1000)
Startup does not continue after encountering the error.
I disconnected the 8 TB and 10 TB ZFS pool drives and rebooted the NAS successfully. Then, I connected the drives afterward, put ZFS in recovery mode (using the “echo 1 > /sys/module/zfs/parameters/zfs_recover” command at the Linux shell), and imported the pool in read-only mode. While in read-only mode, I was able to back up all of the data in the pool piecemeal to my primary desktop and various USB drives connected to it (barely as I don’t have a ton of space on it). Then, I disconnected the pool and reimported it in the normal (read-write) mode and started a scrub. The scrub took several hours to run but came back 100% clean with no errors. However, the Kernel panic error still occurs when I rebooted the NAS. It is currently off.
I know that I could destroy and rebuild the pool given that I have all of the data backed up, but I’d really prefer not to have to do this if I can avoid it because it will take a long time and I don’t remember exactly what I put in every setting in each of my apps when I installed them. I’m also concerned that this issue could happen again; the whole point of using ZFS and buying a second drive/turning my pool into a mirror was to hedge against failures like this.
Is there a way to simply rebuild the metaslab/meta data given that the scrub had no errors? What causes problems like this so I can try to avoid them?
Run a memory test, at least 2 passes. Maybe run it at least 24 hours, (however many passes that is)
Try booting with only 1 of the data pool disks attached. If it still panics, try the other.
It is possible that corruption is only on one Mirror disk, though I don’t understand why a scrub would come back with 100% success. But, who knows, maybe corruption occurred before the checksum was calculated and the metadata written. (Back to memory failure…)
Update here: I ran MemTest86 against all 4 sticks of my RAM and I’m getting some errors. I previously added (a month ago) 2 sticks of RAM to my existing 2 sticks and didn’t have errors back then, but I admit I only ran 1 set of tests. All 4 sticks are exactly the same type.
I’ll try to isolate which of the sticks is hitting the errors (and if they all pass see if I can’t mess with some BIOS settings to see if I can’t help resolve the issue).
once u yank the bad stick, boot and do a readonly import b4 writing anything
-echo 1 > /sys/module/zfs/parameters/zfs_recover
-zpool import -o readonly=on YourPoolName
-copy anything important off
-then import rw and scrub on the good ram only
Sorry to hear that. But, that is likely the cause.
ZFS was originally written for servers, (SPARC), which ALL have ECC RAM. We home and small business users can and do successfully use ZFS without ECC RAM. However, it is always a risk.
Whence corruption is propagated to the pool, with bad Metadata, that passes it’s checksum, it is really hard to get the pool back. The stock answer, is “Do you have a recent backup?” We are not their yet…
If you get your TrueNAS server stable, you may be able to do what @nite_route suggest.
Or perform a roll back to an earlier write transaction group. This is part of the COW, (Copy On Write), features built into ZFS. During writes, all new Data and Metadata is written to unused storage space, and a ring buffer is updated. Going back to a prior ring buffer entry, (aka Uber block), will throw out however many writes occurred between current and the prior entry used. This is hard core recovery… and let us see where you get from other work, (getting your RAM stable…).
Update here: None of the 4 RAM sticks failed when individually tested (yay!). Google AI indicated that the issue was them likely that the issue was the Memory Controller and after increasing the “VTT Voltage” by +0.028V in my BIOS settings (as this is there setting which controls how much power the Memory Controller gets), MemTest86 is now passing even with all 4 sticks of RAM.
I’m doing another scrub right now, and if it passes, I’m going to try “rolling back a txg” as nite_route suggested. If that doesn’t work, I’ll try to “roll back to an earlier write transaction group” as Arwen suggested (but I have no clue what that entails).
I must be misunderstanding your advice, as running the “zdb -ul /dev/disk/by-id” or “zdb -ul /dev/sdb”, etc. is just giving me an error:
failed to read label 0
failed to read label 1
failed to read label 2
failed to read label 3
The two mirrorred hard drives containing my pool data are sdb and sdc, and I do see those in the /dev folder. What am I misunderstanding here? Can you clarify the command you gave me slightly?
zdb -ul wants the zfs partition node, not the bare disk and not the by-id directory.
-ls -l /dev/disk/by-id/ | rg ‘sdb|sdc’
-use the *-partN that actually holds the pool (on truenas its usually part1 or part2)
-zdb -ul /dev/disk/by-id/ata-YOURDRIVE-part2
quick check also works:
-zdb -l /dev/sdb1
-zdb -l /dev/sdc1
if labels print, grab an older txg from the uberblock list then:
-zpool import -f -o readonly=on -T <older_txg> YourPoolName
Running a scrub adds hundreds of Transaction Groups (txgs), wiping out any chance of rolling back to an earlier one (since only 128 are saved). Since I took your advice to run the scrub, I couldn’t roll back to an earlier transaction group (txg).
Update: Instead, since the scrub indicated that there were no errors with the actual data and I had a mirror, I detached one of the drives (which I’ll call drive 1). Next, I wiped and created a new pool on drive 1. Then, I created a snapshot of the old pool on the remaining drive (which I’ll call drive 2) and copied all of the datasets (and data in them) from drive 2 to the new pool on drive 1 using the snapshot. I compared the data between both the old and new pools after this copy to ensure it worked correctly and both pools now had identical data (this was easy because the datasets show how big they are in the Web UI). Finally, I wiped drive 2 and attached it to drive 1 to re-establish the mirror. After the resilvering process completed, I exported the pool temporarily and re-imported the new pool with the old pool’s name in order to rename it. As such, the metadata is now fixed and the issue is gone.
Given that the scrub was clean, I don’t understand why TrueNAS could not regenerate the metadata which was corrupted. However, I’ll try to be more aware next time I add RAM that I need to test all sticks simultaneously going forward rather than just the new sticks.
Actually, for 4KB native sectors, (aka AShift 12), it is 32 TXGs saved. (If I remember correctly, the Uber block storage was fixed in size, thus, using 4KB sectors required shrinking the count of the Uber blocks…)
There is work to fix this, but it will require a pool breaking configuration. Basically existing pools can’t take advantage of this new feature. Only newly created pools. (If I understand this feature correctly…)
If a RAM buffer used for metadata is bad before ZFS checksums this metadata, (which occurs before written to disk), the Scrub will not detect the any problem. However, if the metadata corruption is bad enough, it can cause a crash.
For others reading this in the future, this exact issue can happen to any file system. Including the other check-summing file systems, BTRFS and Hammer1/2. (And probably BCacheFS, though I know little about it.)
I previously added (a month ago) 2 sticks of RAM to my existing 2 sticks and didn’t have errors back then, but I admit I only ran 1 set of tests. All 4 sticks are exactly the same type.